They take advantage of the fact that I am unable to speak English.
I am so sorry this is happening to you :(
「英語が話せないのをいいことに」は英語で They take advantage of the fact that I am unable to speak English. です
英語が話せないのを = unable to speak English
いいことに = take advantage of
私が英語が話せないことをいいことに影で散々私のことを悪く言う
They take advantage of me being unable to speak English and talk bad about me behind my back.
They take advantage of the fact that I can't speak English and badmouth me behind my back.