You don't have to keep up the pace if you don't want to.
You don't have to force yourself to keep up.
Don't push yourself to fit in.
あまり興味のなさそうな話題になんとか会話を合わせようとしている友人、
A friend who is trying to keep up the conversation to a topic she/he seems to be less interested in.
誘いを断れずに行きたくもない飲み会につい参加してしまうことが悩みの友人。
A friend who is troubled to participate in a drinking party which she/he does not want to go but can't refuse the invitation.
You don't have to come to the party if you don't want to.
You don't have to come drinking with us if you prefer not to.
You don't have to come to the party if you don't want to.
You don't have to come drinking with us if you prefer not to.
上記のようにパーティーや飲み会にこなくてもいいんだよ、いうこともできます。
しかし、もっと幅広く使えるフレーズは
You don't have to if you don't want to.
(したくなければしなくていいよ)
この場合は「何を」を明示していないので状況によって食べなくてもいい・払わなくてもいい・歌わなくてもいい等、気が乗らないこと全般に使えます。