President Reagan appears to be meeting growing resistance to his appointment of a United States ambassador to the Holy See. If public opposition builds, he could be in trouble on the issue.
Two developments in Congress are reopening what seemed to be a closed question:
* At a Senate confirmation hearing, Sen. Jesse Helms (R) of North Carolina voiced his opposition to formal relations with the Vatican and suggested that a vote on the nomination by the full Senate Foreign Relations Committee be postponed a month or so.
* Sens. Mark O. Hatfield (R) of Oregon and Lowell P. Weicker Jr. (R) of Connecticut have requested a meeting of the Senate Appropriations subcommittee on state and justice to air the matter of funding for the US mission at the Vatican. The two lawmakers, who oppose the Vatican appointment, expressed concern that a State Department funding request was approved by the subcommittee chairman, Sen. Paul Laxalt (R) of Nevada, without discussion.
Civil-libertarian and religious groups opposed to the President's move have welcomed the congressional developments. ''They are important because this gives people more time to express their views,'' said a spokesman of Americans United for Separation of Church and State. ''It will also give us time to explain the issues to members of Congress. There is now a glimmer of hope that the Congress will take a look at the question.''
Commented an official of the American Civil Liberties Union: ''There needs to be time for Congress and the public to be informed about the serious constitutional questions involved, to get a dialogue going, and to let the public be heard from.''
Until recently, most observers believed the Vatican action to be a fait accompli. Last November, with little media attention, Congress repealed an 1867 law forbidding the use of federal funds for a diplomatic mission at the Vatican. On Jan. 10 the President established formal diplomatic ties with the Holy See and nominated William A. Wilson, a California land developer, to be the envoy.
Only two steps remain to complete the process: The Senate Foreign Relations Committee must confirm the nomination of Mr. Wilson; and Senate and House appropriations subcommittees must approve the State Department's request to assign $351,000 of its funds for expanding the US mission at the Holy See.
Opponents of formal diplomatic relations with the Vatican say they have had little opportunity to put their case before the Congress. They point out that no public hearings were held prior to the Senate and House votes on the legislation repealing the 1867 statute and that the legislation was passed by voice vote. They also say that the Feb. 2 confirmation hearing itself was scheduled without adequate public notice.
Some lawmakers agree. ''Procedurally, it has not been handled well,'' said Sen. Charles McC. Mathias Jr. (R) of Maryland, a member of the Senate Foreign Relations Committee. ''It was an issue of more substance than was reflected in bringing it up without notice on the floor and by voice vote. So we're back at the point of debating the thing. After a month this may take on a different proportion.''
Sensitive to charges that Reagan's allies in Congress are trying to railroad through the Vatican nomination for political ends, the Senate Foreign Relations subcommittee opened up its hearing last week to testimony on the substantive question of diplomatic recognition of the Holy See as well as on the credentials of the nominee.
Sen. Richard G. Lugar (R) of Indiana, who is chairman of the subcommittee and who sponsored the legislation removing the roadblock to formal US-Vatican ties, seemed to bend over backwards to establish an atmosphere of openness and fairness. He acknowledged the widespread opposition to the Vatican appointment and agreed to an unspecified delay of committee consideration of the nomination so the public, in line with Senator Helms's suggestion, to keep the record open for a time so the public could express its views.
Some lawmakers are beginning to hear from constituents on the issue in significant numbers. Senator Helms has run into a barrage of criticism from Baptists in North Carolina. This explains why he reversed his position on the issue, say political observers. Helms was one of the original cosponsors of the bill repealing the 1867 law but recently removed his name from the bill.
Senator Mathias said he had received 100 letters protesting the move. ''This has just broken on the public consciousness,'' he remarked. ''So as an early return it's fairly heavy.''
The White House is also receiving letters and calls but declines to say how many. ''We don't have any information out on this,'' said a press aide.
In the Senate confirmation hearing last week Mr. Wilson, who has served as the President's personal envoy to the Vatican, said that upgrading the US mission to full diplomatic relations would give the US a ''permanent presence'' at the Vatican, facilitate the arranging of visits by US government officials, and enable the US to explain its foreign policy positions to the Vatican.
But spokesmen for a number of Protestant groups, both liberal and conservative, argued strongly that sending a full-time US ambassador to the Vatican violated the constitutional principle of separation of church and state.