Washington — When 21 US ambassadors, all political appointees of the Reagan administration , publicly endorsed Sen. Jesse Helms (R) of North Carolina for reelection this fall, professional diplomats were both angry and embarrassed.
''State Department people pride themselves on being unflappable, but this struck a real nerve,'' says Dennis Hays, president of the American Foreign Service Association (AFSA), the professional group that represents career diplomats. ''There was a strong gut feeling that this was a mistake that hurt not only the individuals, but us as a nation by sending out very confusing signals.''
Although technically legal, the partisan action was widely considered inappropriate for top US representatives abroad and a violation of diplomatic tradition. The resulting fuss - including an indignant joint statement of protest by 36 former ambassadors and senior officers - probably will keep it from happening again in the near future.
But the incident raises fresh concerns about the degree to which US presidents dole out ambassadorships as political rewards, the caliber of those chosen, and the sensitivity of the posts assigned.
Since World War II, most presidents have reserved about one-third of the country's ambassadorial slots for their political friends. Under President Carter, the proportion fell to 27 percent. The political share has edged up to 41 percent under President Reagan, a figure matched only by the late President John Kennedy. Even some political appointees say privately that their proportion of slots may be too high.
And it is not just the large embassies of Western Europe and Canada which serve as the political plums. Increasingly, many of the smaller embassies, some of the more sensitive spots in Central and South America, and almost all the nations in southern Africa are in the hands of noncareer ambassadors. Most other industrialized nations assign few political ambassadors to overseas embassies.
Yet the US has such an abundance of trained diplomatic officers that it amounts to what one careerist complained is ''an officer glut.'' The US has an extensive career-diplomat training program, and a corps of 3,500 officers. Each ambassadorship siphoned off on the political side makes it that much harder for the US Foreign Service to recruit and keep good professionals, says David Simcox , president of the Foreign Service Officer Association. The problem is compounded by a 1980 US law, which says that those career officers who are not promoted within a certain period of time are ''selected out'' of the service and much retire.
''It's as if you said you wanted to build a professional Navy but that the captains of the ships would be appointed from the outside,'' notes Viron (Pete) Vaky, a former career ambassador to Colombia, Venezuela, and Costa Rica. ''If you don't provide some opportunity to rise to the top, it will have a deteriorating effect on the career service. I think a key question is, 'Do you really want a top-rate, professional diplomatic service?' ''
AFSA's Mr. Hays says a number of Foreign Service officers who see promotion opportunities growing slimmer are leaving at mid-career levels after 15 or 20 years of training and experience.
''There's a real threat that we'll lose our next generation of leaders,'' he says. The level of politically appointed ambassadors ''is already unacceptable and having a major impact on us. I think there's a danger it could go even higher, which will compound the problem.''
Still, most State Department professionals insist numbers are not the issue. What bothers them most, they say, is the caliber of those chosen. They insist such exemplary noncareer ambassadors as Mike Mansfield, George Bush, and Ellsworth Bunker - all applauded by professionals - are rare.
''The majority are not really well-qualified people - even in their own line of work,'' says Malcolm Toon, a former career ambassador to the Soviet Union. ''The principal criteria are ideology, loyalty, and reliability. That's not enough. The principal qualification should be competence.''
Yet most ''political'' nominees, regardless of qualifications, sail through Senate confirmation hearings on the premise that a president has the right to appoint whomever he wants.
In Hays's view, President Reagan's ambassadorial selections fit the ''historic average'' with about 20 percent actually ''harming'' US interests.
How can a noncareer ambassador hurt US interests?
Many are businessmen and former campaign chairmen or money organizers. Often their lack of experience in diplomacy and a tendency, based on success in business, to tell others what to do and not to hold back personal opinions, serve as handicaps. ''These people are seen as the embodiment of the US,'' Hays says. ''It can take years to rebuild trust and affection.''
US Ambassaor to Canada Paul H. Robinson Jr., for instance, an outspoken Reagan political appointee and one-time Chicago insurance executive, has been widely quoted as telling Canadians that their country is spending too much on social services and is overly concerned about the acid-rain problem. He has termed Canada's metric conversion program ''rubbish.''
In other cases, the intensity of devotion to duty among political ambassadors has been questioned. Case in point: former US Ambassador to the United Kingdom John L. Louis did not return to London from a month-long Florida vacation for 10 days after the Falkland Islands crisis had begun. Though a spokesman said Ambassador Louis had been ill, career diplomats insist one of their own would have found a way to get back sooner.
Yet career diplomats have made mistakes, too. Just a few weeks ago, US Ambassador to Israel Sam Lewis nettled the administration by criticizing the timing of Reagan's Middle East peace initiative in personal comments he thought were off the record.
Some political ambassadors have fared much better than expected. And some countries specifically request them on the theory they will have more clout with the President in making the country's views and needs known. In addition to that ''in,'' says William Sharpless of the Council of American Ambassadors, an organization representing 154 past and present political appointees, noncareer ambassadors often have better ties with Capitol Hill. From their own political experience they may also have a clearer sense of what Americans are thinking than do professionals who move from one overseas post to another, he says. Mr. Sharpless argues that there is a definite role for the ''citizen diplomat'' and that he can help his career counterpart if the two work closely. ''It's not a zero-sum game,'' he insists.
Although there is widespread concern among the American diplomatic corps that the proportion of political ambassadors is too high for the good of the United States and for the profession, there is no current move to impose a legal limit. A bill sponsored two years ago by Sen. Charles McC. Mathias (R) of Maryland would have put a 15 percent cap on the number of political appointees. The bill had the AFSA's support.
But the preferred route these days is to persuade the White House to adopt more self-restraint. One hope is that the Senate will work harder as Presidential adviser to see that better-qualified people are appointed and to assure the administration that its traditional suspicions about the loyalty of the diplomatic corps are unfounded.
And Mr. Hays stresses that he is prepared, ''if the trend gets worse,'' to take a number of steps - from speaking out more pointedly on Capitol Hill against nominees who lack qualifications for the job, to urging the Secretary of State to give the issue a higher priority on his list of concerns.