I think it doesn't really matter to the person who said it, but it matters a lot to the person he said it to
「気にする」は訳語が結構あります。「彼は気にしていない」ーit doesn't matter to him, he isn't concerned about it, he doesn't care about itなど
it doesn't matter・he isn't concernedは一番適当だと思います。he doesn't careはちょっと冷たい・カジュアルなニュアンスです。
それで、I think it doesn't really matter to the person who said it, but it matters a lot to the person he （本人は男の人だったら）said it toを使えます。
ご参考になれば幸いです。
この場合、次のような言い方ができますよ。
ーThe person who said that wasn't worried about what he said, but the person he spoke to was affected by his words.
「言った本人は気にしてないけど、言われた方は気にする」
the person who said ... で「…と言った本人」
the person one spoke to で「言われた人」と言えます。
ご参考まで！
アメリカ合衆国 
カナダ 
