Geneva — Plunged into the limelight by the award of the Nobel Peace Prize, the office of the UN High Commissioner for Refugees (UNHCR) finds itself publicly feted for its achievements. But privately it is beset by a growing array of problems. These are:
* Pressure from its major Western donors to curb costs.
* A lack of cooperation from the Soviet bloc, which has been the source of massive refugee exodus from Afghanistan, Ethiopia, and Indochina.
* And the nagging task of bringing the definition of a refugee into line with the realities of the modern world.
The pressure grew last week here, during the annual session of the UNHCR's policymaking executive committee. UNHCR officials appreciate that the Nobel award - announced Oct. 14 - could not have been better timed.
In immediate terms, its impact will be measured in dollars and cents.
Poul Hartling has proposed a budget of $430 million for 1982, and next month in New York he will hold a pledging conference to raise the money. Following the Nobel award, UNHCR officials hope for both a generous and prompt response.
In many ways, the Nobel award mirrors the UNHCR's dramatic expansion from a minor body to the front rank of UN agencies. It is 27 years since the UNHCR won its first Nobel Peace Prize - three years after the agency was founded in the aftermath of World War II.
Between 10 and 15 million people throughout the world have been uprooted from their homes. In contrast to the refugees who fled after World War II, however, many thousands - in countries like Angola and Somalia - are victims of drought as well as persecution.
In Southeast Asia, meanwhile, 53,000 Vietnamese ''boat people'' fled in the first six months of this year. But increasingly diplomats feel they are fleeing not just persecution but Vietnam's devastated economy, and that they are enticed by Western broadcasts, by the UNHCR's own highly visible resettlement programs, and by large Western quotas, including the 168,000 set for 1980-81 by the United States.
Similarly, in the Caribbean, over 100,000 Haitians have fled their homeland only to face repatriation by the Reagan administration after being termed ''economic migrants.''
Faced by this, many officials in the UNHCR would like Mr. Hartling to liberalize the definitition to justify the already existing practice of aiding those who may have left for reasons other than a fear of persecution. But this faces opposition from Western countries seeking to curb spending and tighten up immigration procedures.
To the concern of many Western diplomats here, the agency has often seemed greedy for publicity, and too uncritical in accepting governments' estimates both on the extent of their refugee burden and their needs.
A further problem has been the way in which the UNHCR has continued to assist refugees in what are development, rather than strictly humanitarian, relief projects. Of the $1.15 billion requested for African refugees, only $69 million was strictly for emergency assistance. A full $312 million was for projects like roads and schools - more usually the business of the World Bank or UN Development Program.
UNHCR officials reject the charge of empire-building, they point out that as long as a government gives asylum but not citizenship to a refugee, he stays in legal limbo, denied access to development aid.
They do, however, concede that the agency has allowed itself to be sucked into emergencies that do not fall clearly within the UNHCR's mandate - as when UNHCR trucks were used to carry food to the drought-stricken province of Karamoja in Uganda.
All this has combined to cause a retrenchment in the agency's operations, and Mr. Hartling recently turned down an urgent appeal by the Angolan government to assist 131,000 people displaced by the South African invasion in August.
Shedding such responsibilities will lighten the UNHCR's load. It allows Mr. Hartling to present a budget for 1982 that is significantly below that of 1981. But many are worried that this leaves a gaping hole in the UN's humanitarian aid system.
Many UN officials see this as yet another example of the way in which Western governments project their own economic policies and values on the UN system, with too little regard for the needs of less fortunate nations.
''It ignores the fact that each day brings more, not less refugees,'' said one official.
But they also see it as part of a wide-ranging debate over refugee policy .Earlier this year officials from the Immigration and Naturalization Service (INS), which screens would-be entrants to the US, suddenly stopped accepting ''economic migrants'' in the camps of Southeast Asia.The policy was subsequently relaxed, but it sent a chill through UNHCR officials, who described it as ''a shot across our bows.'' Mr. Hartling himself in an interview pointed out that the motives of the Vietnamese boat people are almost irrelevant once they have taken to the boats - simply because they cannot return.At the same time, UNHCR officials express alarm at President Reagan's recent announcement that Haitians would be repatriated - as well as the Salvadorans who have sought asylum in the US.Mr. Hartling, in a series of notes, has urged the US to give the Salvadorans blanket eligibility, as was done with Lebanese and Nicaraguans, instead of treating them on an individual basis like other immigrants.In refusing, the US government is assumed to be anxious to deter another wave from the Caribbean. A similar motive appears to lie behind a recent change in the policy of Thailand, one of the US's chief allies in Southeast Asia.On Aug. 15, Thailand announced that no more Vietnamese boat people would be accepted for resettlement. Two weeks ago, the Thai chief of the security council, General Prasong, said Thailand would suspend its patrols against the pirates who prey on the Vietnamese boat people in the Gulf of Thailand.General Prasong's announcement is thought aimed at putting pressure on the US to increase its offer of $600,000 toward joint US-Thai antipirate patrols in the gulf. But there is suspicion some Thais see the pirates as one of the few effective deterrents against the arrival of boat people.Still, more people are concluding that ''deterrence'' has little relevance once boat people have set their sights on a goal. There is also a feeling that in today's world of disintegrating economies and a highly visible gap between rich countries and poor, it may be almost impossible to disentangle economic motives from political persecution.As a result, some UNHCR officials argue that the outflow of boat people from Vietnam will not be stemmed as long as Vietnam is isolated from Western aid, burdened by food shortages, and forced into desperate economic experiments.Mr. Hartling and his staff are too burdened with the day-to-day business of running camps to pitch into this quagmire. But even if they did, they would find the UNHCR's scope for action sharply curtailed by its lack of bargaining power in vital regions.Neither Thailand nor Malaysia accepts the 1951 convention, and with it the principle of asylum. Equally serious, the UNHCR does not have a presence in El Salvador. And relations with the United States - its main contributor - have been strained since last year, when Mr. Hartling turned down American pleas to help with the exodus of Cubans.Most serious of all, the Soviet bloc does not contribute to the UNHCR's budget or participate in its decisionmaking - even though Ethiopia, Vietnam, Afghanistan, and Laos are the source of today's major refugee outpourings.Despite this, many here argue that the UNHCR can play a crucial role in behind-the-scenes talks. Such pressure has paid dividends: In 1979 the agency initiated a program for the ''orderly departure'' of Vietnamese by plane, instead of perilous, rickety boats. After a slow start, over 1,000 now leave each month. Last week, at a meeting here, Hoang Bich Son, a vice-minister for foreign affairs, agreed to ease some of the restrictions on the program. Many of his younger colleagues hope Mr. Hartling will be emboldened by the Nobel Peace award to take a tougher line with governments. Some would like him to insist that camps in Somalia are moved to sites where they can be properly served with food and water instead of isolated way out in the Ogaden Desert. Others argue that governments must be pressured to provide accurate figures of refugees - given that these are often used in political debates: In Pakistan, the UNHCR accepts a figure of 1.7 million Afghan refugees, compared to the 2.5 million claimed by the government.Still others here would like Mr. Hartling to exert more pressure on governments to create the conditions for repatriation of refugees, as happened in 1979, when victims of the revolt in Shaba Province returned to Zaire from Angola, and - better still - to prevent further outpourings, as Vietnam agreed to do here in July 1979, at the conference on Indochinese boat people.All this points to a more aggressive and political role for the UNHCR, something that the amiable Mr. Hartling shuns. ''We are strictly humanitarian'' he insisted in an interview here last week.Despite this, many feel that the whole UN system - beset by criticism from all sides - badly needs greater thrust and vigor at the top. What better tool to use, they ask, than the Nobel Peace Prize ?