・I didn't know how to answer (the question).
（その質問に）なんて答えたらいいかわからなかった。
・I didn't know what to say.
なんて言ったらいいかわからなかった。
・I couldn't come up with the right words (to say).
いい（正しい）言葉が思い浮かばなかった。
come up with... は
「（考えや言葉）が思いつく、思い浮かぶ」という表現です。
This is a way to say that you don't know how to answer but that you do have ideas and that you are paying attention. Sometimes saying "I don't know" seems like an excuse and isn't well-received. This will allow the teacher to also ask the question in another way that you may be able to answer better.
これは、なんと答えたらいいかわからないけれど、考えがあって、自分は注意を払っているという表現です。
時には、 "I don't know"（わかりません）と言うことは、 言い訳に聞こえ、よく受け取られないこともあります。
これは、先生に別の方法で質問をしてもらうと、答えられるかもしれないということです。
I know the answer but I don't know how to say it in English.
*I know the answer but I don't know how to say it in English.- This means that you know the answer but you finding it hard to find the words to express it in English. In cases like these you can try and the teacher may be able to understand what you are saying and help you.
例文
*I know the answer but I don't know how to say it in English.
答えは分かっていますが、英語でなんと言ってよいか分かりません
答えは分かっているけど、それを英語で表現する言葉を見つけるのが難しいということです。このような場合は、頑張れば先生は言おうとしていることを理解し、助けてくれるかもしれません。
The phrase "I'm not sure." can be used to state you are not sure if you know the answer to a question you were asked (or you don't know the answer at all and just don't want to say).
"I'm not sure."とは、尋ねられた質問の答えがわかるかどうか（もしくは、答えが全くわからず、言いたくない）確かではないときに使われます。
Sorry, I'm not sure how to deal with that question
Could you please rephrase that question in simpler terms?
Well, you can wave your hands in the air and scream that you don't know the answer, or perhaps ask the teacher to rephrase the question. What is difficult about the question? The vocabulary? The subject matter? It is necessary to identify exactly what the problem is.
I understand what you are asking but don't have the vocabulary to explain.
I am not sure of the words to explain my answer
How can I explain my answer
The important point you made in your sentence above is that you are not sure 'how' to answer the question - not that you don't know the answer.
It is best to let the teacher know, so that they can help you with vocabulary words. Tell your teacher "I know the answer, but don't know the words to explain it. Can you help me?" Your teacher might try and get you to explain using words you do know and then help you to use other expressions.
上記の質問で重要なポイントは、答えが分からないのではなく、質問にどう答えるかが分からないということです。
これは、単語の手助けをしてくれるので、先生に知らせるのが一番です。I know the answer, but don't know the words to explain it. Can you help me?（答えは分かってるけど、説明するための単語が分かりません。助けて頂けますか？）と先生に言ってください。
先生は、あなたの知ってる単語を使って説明してくれ、そして他の表現を使うことを手助けするでしょう。
Either of these expressions are good to use when responding to someone that you do not know the answer to something. The first expression is a more polite and professional way of answering. The second one is more laid back and is used when you have a certain degree of familiarity with your teacher.
日本 
アメリカ合衆国 
南アフリカ 
イギリス 
オーストラリア 
