Despite the fact that he was shorter than me, he was mean to me.
He wasn't very nice to me even though he wasn't as big as me.
この場合、次のような言い方ができますよ。
ーDespite the fact that he was shorter than me, he was mean to me.
「彼が私より背が低いという事実にも関わらず、彼は私に意地悪だった」
despite the fact that ... で「…という事実にも関わらず」
ーHe wasn't very nice to me even though he wasn't as big as me.
「彼は私ほど大きくなかったのにもかかわらず、私にとても意地悪だった」
even though で「…にもかかわらず」
ご参考まで!