Washington — Doubts about the establishment of formal diplomatic ties with the Holy See continue to simmer in the US Congress. In the face of opposition from civil-libertarian and religious groups to President Reagan's action, lawmakers are allowing themselves more time to consider the question of funding of an American embassy at the Vatican.
* The Senate Appropriations subcommittee on state and justice will hold a hearing March 5 at which public witnesses will be permitted to testify on the funding issue.
* US Secretary of State George P. Shultz is tentatively scheduled to testify before the subcommittee March 28 on this and other subjects.
Meanwhile, the President's appointment of William A. Wilson to be ambassador to the Holy See still has to be confirmed by the Senate. The Senate Foreign Relations Committee is expected to vote on the confirmation Feb. 21.
Apparently surprised by the growing opposition to the move, the administration has sent top officials to put its case to Congress. At a hearing in a House Appropriations subcommittee last week, Deputy Secretary of State Kenneth Dam defended the State Department's request for a ''reprogramming'' of $ 321,000 to upgrade the US mission at the Vatican. The Congress has not specifically appropriated funds for an embassy, although it has removed an 1867 statute banning such an appropriation.
Formal ties with the Holy See are important, Mr. Dam told the subcommittee, because the Vatican is a ''significant player'' on the world scene, dealing with such questions as war and peace, immigration, narcotics control, and food distribution. Such ties will enable the United States ''to make sure the Vatican understands the US government position on vital issues of the day,'' he said.
Asked what specific benefit could be gained by a formal ambassador that is not now available through the President's personal envoy, Mr. Dam stated, ''we want to be able to influence the political positions'' of the Vatican. ''We'd like to get our point of view across so that it is taken account of at the Vatican foreign office.''
Protestant church spokesmen express concern over this and other statements by the deputy secretary. ''Is this not an entanglement of state and church?'' says James Dunn, executive director of the Baptist Joint Committee on Public Affairs. ''It clearly violates the American tradition of the separation of church and state. It's incredible the State Department would say this and announce it in advance.''
Groups speaking out against the Vatican appointment say it would take a groundswell of grass-roots response to persuade the legislature to block the presidential action either by refusing funds for an embassy or turning down the Wilson nomination. But they say they are encouraged by recent developments.
''My initial appraisal was that we were stuck with a fait accompli,'' comments Forest Montgomery, an official of the National Association of Evangelicals. ''But with the concern of some in the Congress, we have bought some time. Now we have a fighting chance - though I would not predict victory.''
As various religious organizations testified at the House subcommittee hearing on Feb. 9, some subcommittee members seemed to waver in their support of funding or at least to sympathize with the charge that the legislative action leading to diplomatic recognition of the Holy See had taken place without hearings and publicity.
Rep. Bob Carr (D) of Michigan said congressional repeal of the 1867 law was ''an accomplished fact before I was aware of it.'' He suggested an effort may be mounted in Congress to reinstate the language of the 1867 law or to repeal the Nov. 22 action abrogating that law.
The National Conference of Catholic Bishops has publicly ''welcomed'' the establishment of formal ties with the Holy See. But some Roman Catholics oppose the move because of concern about potential interference of the Roman church in the affairs of the American church. ''Full diplomatic recognition and the presence of a (papal) nuncio will slow the glacial adaption of our church institutions to modern life by giving the Curia, the executive branch, added means to pressure the bishops and their conference,'' testified Joseph T. Skehan , president of the National Association of Laity.
Fred D. Schwengel, Capitol historian and former GOP congressman from Iowa, told the lawmakers the issue was ''fraught with danger to the First Amendment.'' He urged them to give the matter fuller consideration. After the hearing, subcommittee chairman Neal Smith (D) of Iowa said he expected to delay a vote on the funding question until after the Senate confirmation vote.
In that connection, there is uncertainty over the position of Sen. Jesse Helms (R) of North Carolina. At the confirmation hearing on Feb. 2, Senator Helms stated his opposition to the Vatican appointment and suggested the confirmation vote be delayed for a month. At this writing, the Senate Foreign Relations Committee has not received any formal request from the senator for a postponement, a staff aide said, and plans to vote on the issue at its Feb. 21 meeting. Mr. Wilson is expected to be confirmed, the aided added.