US businesses are being needlessly discouraged from setting up shop in Japan or shipping exports to that country, experts say. While ''the highest-opportunity areas in manufacturing have a significant presence of US leading firms, . . . there don't appear to be any factors prohibiting other US firms from establishing or advancing niche (specialty) positions,'' says a new report on US-Japanese trade, ''Japan: Obstacles and Opportunities.'' It adds that ''new or untapped opportunities appear greatest in the wholesale, retail, and service-industry sectors.''
The report was prepared by the consulting firm McKinsey & Co. and the United States-Japan Trade Study Group, which represents business and government from both countries.
While US executives see much to praise in the report, they still point out specific problems they say need to be addressed.
The US share of direct foreign investment in Japan has declined, the study acknowledges. It also recognizes the US trade deficit with Japan, which US officials claim totaled $20 billion last year. And meeting Japanese product standards is not easy, it admits.
But there are a lot of ''on the other hands'' that need considering, the report says.
On the other hand, US companies produce and sell annually about $20 billion worth of goods and services in Japan. Japanese operations in the US total about are goods produced locally.
And, on the other hand, trade figures don't include oil shipped by US companies to Japan which came from outside the US - a total of $21 billion in 1980.
As for trade barriers, quotas and tariffs in Japan are lighter than in Europe and the US, the report says. Japan restricts imports in five non-agricultural areas, while the US restricts six, West Germany 11, and France 27. Japan's average tariff, the report says, is 3.2 percent, ''below most developed nations.''
If US firms want to make it in the Japanese market, they apparently should head for the service, retail, and wholesale sectors. The study identified some broad areas that have ''high potential'' for US firms: financial and business services, medical management, computer services, leisure-related services, and educational-vocational services. Here is a look at some of these areas:
* Financial services: Consumers are changing in Japan. They are getting older and want higher interest on their savings. They could use US expertise in pension funds and money market management, the study says. Corporations are interested in more international investing, an area where the US has a good track record. US international banks and insurance companies could work with regional Japanese banks gaining consumer access in addition to corporate access.
However, says Martin Weiss, editor of Money Forecasts in West Palm Beach, Fla., ''serious obstacles'' get in the way of business. Mr. Weiss advises a major Japanese brokerage firm.
The biggest obstacle is ''the wall'' between brokerage firms and banks. ''That refers to a very solid, protective shell which surrounds the entire banking industry and prevents any other industry from making inroads on their saver market,'' he says.
Weiss says money funds had a tough time starting in Japan, and US firms trying to edge in on the securities business find it ''rough - the market isn't big enough to begin with.'' He thinks two businesses could do very well in Japan: foreign-security firms in the institutional market and financial-planning firms.
* Medical management: The report does not paint a bright picture of Japanese hospital management. Financial failures of hospitals and shortages of qualified personnel and nursing homes are some of the problems.
But there's a major disincentive to outside participation: Japanese hospitals must be owned by a certified medical doctor, and cannot be run for profit, nor offer dividends.
The McKinsey firm and hospital executives in the US, though, are not too discouraged. ''I'm convinced there are many ways it (entering the Japanese market) can be done,'' says Robert Diener, group vice-president and head of the international division of American Medical International Inc., a $4 billion company that owns and operates private hospitals. ''It's very hard to ignore that market, especially when the Japanese are very concerned about (quality) of health care.''
Mr. Diener added that ''it appears some of these long-time regulatory bars to our entry are breaking down.''
* Computer services: The Japanese are weak on software development, but US companies ''will have to move quickly'' to take advantage of this, the study says. One area, known as CAD-CAM, computer-aided design and manufacturing, is especially attractive. Over 80 percent of CAD-CAM systems used in Japan are from the US, the study says.
An official at Computervision Corporation in Burlington, Mass., says Japan is a ''high growth'' market, but that US CAD-CAM companies need to watch out. They must pay attention to specific applications needed by the Japanese, and supply support and servicing of their product. Computervision Corporation has about a third of the Japanese market.
Other kinds of software need bolstering in Japan: operating systems software - the internal nuts and bolts software that makes computers run - and applications software, programs that accomplish specific tasks, such as budget balancing or word processing.