I think it doesn't really matter to the person who said it, but it matters a lot to the person he said it to
「気にする」は訳語が結構あります。「彼は気にしていない」ーit doesn't matter to him, he isn't concerned about it, he doesn't care about itなど
it doesn't matter・he isn't concernedは一番適当だと思います。he doesn't careはちょっと冷たい・カジュアルなニュアンスです。
それで、I think it doesn't really matter to the person who said it, but it matters a lot to the person he (本人は男の人だったら)said it toを使えます。
ご参考になれば幸いです。