嬉しいけど本音のところ苦痛って英語でなんて言うの？

If you enjoy speaking to a friend or someone you know you can say 'you are happy' or 'it's nice' to speak to them
by adding 'their accent is dfficult to undertand' or 'I would have preferred time on my own' explains why it was a pain as well

Other examples:
It was great that he came to visit me in the hospital, but he should have called or texted before showing up
I was glad to be able to see her off at the airport, but in fact it would have been better to have just called to say a goodbye. I have a terrible empty feeling within me now.

Well, with so many reasons that you do not enjoy these conversations, it seems quite amazing that you do feel happy when you receive a call.
Bittersweet = arousing pleasure tinged with sadness or pain.
"Henry had many bittersweet memories of his time in London,"
The truth of the matter seems to be that you:
"...have neither the time nor inclination for conversations - but appreciate the communication.

I love it when he calls but I have a hard time understanding him.
I like when she calls but I would rather sleep and she is difficult to understand.
I like it when you call me but can we make it at a better time?
Thank you for thinking of me but can we do this tomorrow at an earlier time? I need to get some sleep.
It's good of you to call me and I am glad to hear from you but can I get a raincheck?

A raincheck means to reschedule to another time. If you go to an event and it gets rained out then the facility will often give you a free ticket or raincheck so you can come a different day.

Some friends like calling at 'odd hours' when you are tired and would like to sleep. Such friends could be feeling lonely wherever they are and want to talk to somebody regardless of the time. However, you reluctantly hear them out. Some accents can make it difficult for a listener to understand what the speaker is saying. However, if the friend also likes to crack some jokes, you might want to try and make out what he/she is saying and therefore carry on listening. Such friends talk for long periods and can bore you to hell.

However, if one of your friends is of this type, you may say:
He/she is such a great friend but he/she calls me at odd hours and speaks in an accent that makes it difficult for me to understand what he/she says.
or
I feel happy whenever he/she calls me but I soon get bored by his/her jokes which I can't understand because of his/her accent.

The day is always busy and we find we need more rest.
Catching up with friends or those we care about is great and it makes us happy, although, there are times when you feel very tired and talking over the phone is more of a strain than a joy.

*I have mixed feelings about our conversations
= This means that you have both good and bad feelings about the conversations you have. Sometimes it's fun and sometimes it's the opposite!

*It's a tricky situation with my friend because... = in this context 'tricky' means difficult and not straight forward. If you are in a tricky situation it means that you're not sure what to do.

*Our friendship has its ups and downs = ups means good and downs mean bad so its a mixed friendship that is both good and bad at times.

You could also say...
Sometimes its so difficult to understand him on the phone
It's a love/hate relationship ( we get on equally bad and good)
I love speaking to him but sometimes he's a pain in the a* ( you can say this just as a joke) :)

I hope this is helpful!
Enjoy learning English here! : -)

"I really enjoy speaking with you but sometimes it can be an inconvenience" stating that you do like to speak with this person but it can be hard work, 'Inconvenience' is the difficulty or trouble experienced ex. "leaving fro work at rush hour is such an inconvenience because it makes me later for work!".
'To struggle' is describe as having trouble with something and enduring some pain or discomfort.