A religious building can be referred to as a "house of worship" or "place of worship" and can include churches, temples, mosques, synagogues, shrines, or other holy places. The term "House of God" is commonly used for a church, however, we would normally refer to a "Buddhist temple" or "Shinto shrine" when directly referencing these places. One could also say a place or building is holy or sacred to a particular religion : "This is a sacred place to Shintoists".
Depending on the kind of place of worship and the followers, different terms will be used. In relation to Buddhism, we will have Buddhists (followers), monks and nuns (those dedicated to service), and will be led by a abbot. These terms are all borrowed from English Christian terms.
In Shintoism, the followers are called Shintoists, but there is not much information on the leadership as the practice of Shinto is rather one of a collection of beliefs than organised religion. The phrase Kami is also acceptable.
Shintoism (Japan's indigenous religion) is not the same as Buddhism, but the two religions do share some similarities. As such, it is completely acceptable to use the Japanese terms in an English conversation or discussion as long as you can explain the terms to those who are not familiar with them.
In the case of most religions, the following terms can be used universally if you do not know the specific term:
parishioner - follower
congregation - community of followers (member of the congregation is also common)
priest - leader (female - priestess)
minister - leader (male and female)
abbot - leader (male and female)
nun - female dedicated to service
monk - male dedicated to service
If in doubt, one can simply refer to Buddhist or Shintoist followers or leaders.