Vienna — Five months ago the United States touched off a storm of controversy by walking out of the United Nations agency in Vienna that guards against the spread of nuclear weapons.
Officials in Washington and Vienna now say in interviews that Washington is about to return under terms of an agreement being hammered out behind the scenes.
''It will be worked out by the time the agency holds its next board meeting here Feb. 22,'' says one US source optimistically.
Yet the controversy continues. The US move, which stopped all US budget payments to the agency as well, has been widely condemned by critics who say it has undermined the US commitment to nuclear nonproliferation, weakened the UN agency, and landed Washington in a ''no-win'' diplomatic tangle.
''The walk-out backfired,'' says one Washington source close to the episode.
The US suspended membership in the International Atomic Energy Agency (IAEA) in Vienna Sept. 24 to protest a sudden and successful move against Israel. By a one-vote margin on the crucial vote, Arab and other developing countries voted to reject Israel's credentials to the agency's 1982 general conference in Vienna.
The obviously political vote carried little practical effect. Israel had taken part in the entire conference, which had only one hour left. The vote ceased to apply when the conference expired. Left untouched and unaffected was Israeli membership in the agency itself.
But Arabs and developing nations were determined to ''punish'' Israel for not preventing the massacre of Palestinians in Beirut refugee camps a few days before.
The Arabs also wanted to intensify action taken the year before to signal outrage over Israel's bombing of an Iraqi nuclear reactor near Baghdad June 7, 1981. Then, at the agency conference later in 1981, a majority of members voted to suspend all technical assistance to Israel.
To the US delegation, the 1982 vote was dismaying. ''It signalled the start of a campaign to expel Israel from the UN system itself,'' says one American official. ''It was followed by anti-Israeli moves at the UN's international telecommunications body, and then in the General Assembly itself.
''We had to take a stand. As it was, we beat back the other two challenges, and as far as the atomic energy agency is concerned, we've made it clear that any further action against Israel will make it very hard for the US to stay in the agency. . . .''
Officials claim they were not merely protecting Israel, but were opposing the entry of any and all politics into agency affairs.
''Suppose one member was found to be violating safeguards against diverting fissionable material for use in a bomb,'' one official said. ''Would the agency condemn it or not? The agency should not be politicized, and the vote against Israel was a political act. . . .''
Yet this view causes deep concern inside the agency, among some of its professional staff and developing country delegations alike. It is also rejected by administration critics in Washington, who see more controversy ahead.
''For an action supposed to protest at politics in the agency,'' says one agency staff member, ''it has had precisely the opposite effect. It has led to more politicking, much more.''
''What it has done,'' says a Washington source, ''is to put us into a no-win situation. If we return to the agency, as I expect we will, we will be branded as a country that puts our support of Israel, for domestic political reasons, above the principle of nonproliferation, and developing countries will use this stick to beat us with all the time.
''Besides, the UN General Assembly has passed a resolution saying that South Africa is to be suspended from agency technical committees. We will oppose that as 'political' - and we'll be seen as defending Pretoria as well. . . .
A Capitol Hill source in Washington warns: ''The US provides the key technical scientific research to improve safeguards (inspections to ensure that nuclear have-not nations are not diverting uranium or plutonium for use in bombs).
''That research goes forward in our national laboratories - Los Alamos, Lawrence Livermore, Argonne, Brookhaven, and the rest. If the US does not start providing funds for that research by April 1, lab teams will be broken up and assigned to other tasks, and nonproliferation research could be set back at least six months and perhaps much more.''
When the US walked out it said it would review its agency membership. It has now done so. According to Sen. Charles Mathias (R) of Maryland Feb. 3, the State Department has concluded that the agency is valuable to the US and that Washington should rejoin.
First, however, a procedural knot has to be untied.
Under a continuing resolution passed by Congress late last year, the US can provide $10 million to the agency if the agency board of governors ''certifies to the US government that the state of Israel is allowed to participate fully as a member nation'' in the agency.
On the face of it, that should be easy: Israel has never ceased to be a member. Washington sources say Director General Hans Blix in Vienna could easily write to the State Department on behalf of the board saying so, or the leader of the board could do so.
The US in fact owes the agency a total of $14.5 million in back payments. The State Department intends to ask for the other $4.5 million in a supplementary budget request.
Still unknown, however, is what form of certification would be enough to satisfy the author of the continuing resolution, Sen. Richard Kasten (R) of Wisconsin.
''Kasten is the key right now,'' says one Senate source. ''He has refused all contact with the State Department, leaving talks to an aide. No one is really sure why he wrote the resolution the way he did, or what will satisfy him.''
''Yes,'' said the third-world ambassador, ''Blix could write a letter for the board. But don't imagine the board would support a formal vote in favor of Israel. Half of that board is made up of developing nations - and Israel has a general conference resolution outstanding against it, denying it technical assistance.
The State Department is pressing to find out just what Senator Kasten requires so that it can tell the agency in Vienna the form that certification should take.
If Senator Kasten wants more than a Blix letter or a formal board statement, prospects for an early solution will worsen.
Agency officials are awaiting word with considerable tension. The agency has enough operating funds to last until November, but any prolonged delay in the US return would mean reduced budget goals, imminent cutbacks, and a slowdown in safeguards that few in Vienna really want.