It's depressing when a bill shows up just when you least expect it.
Just when you've forgotten all about it, a bill comes along. It's so irritating.
忘れた頃 = when you least expect it; just when you've forgotten about it
「鬱陶しい」は「depressing」という意味も「irritating」という意味もあります。
「やってくる」は「come along」「show up」と言います。