A legal test of the constitutionality of sending a US envoy to the Vatican could be very complicated, say proponents as well as those who oppose such a move.
Those who oppose the move on church-state grounds, including the Baptist Joint Committee on Public Affairs and Americans United for Separation of Church and State (AU), admit that their first priority is to try to block the Senate's confirmation of President Reagan's nominee, William A. Wilson.
If unsuccessful, they will take the matter to the courts. However, this would be a ''last resort,'' says John Baker, constitutional lawyer and general council for the Baptist Committee. Dr. Baker says the appointment would be opposed on three long-established constitutional grounds:
* That it constitutes preferential treatment of one church over another.
* That it promotes an ''entanglement'' of the state in religious affairs.
* That it would tend to encourage divisiveness among religious groups.
However, AU official Joseph Conn - who also strongly opposes the Vatican appointment on church and state grounds - allows that a legal resolution of the issue would also raise questions about the President's authority in foreign-policy matters. Diplomatic issues have been the province of Congress and the President, not the courts, he explains. ''If the Reagan administration had decided to set up a relationship with the National Council of Churches, we would have no problem striking it down,'' Mr. Conn says.
Harvard constitutional scholar Laurence Tribe sees even more serious legal problems for those who might want to invalidate the Vatican appointment. Professor Tribe says it may be very difficult for any group even to get ''standing'' in the courts to challenge Reagan's action.
A litigant might have to show specific injury, he says, or potential harm, as a result of diplomatic ties. They may have to show that such recognition would give the Pope unfair advantage over another religious leader. ''There is no international entity who could be considered a counterpart (to the Pope),'' Tribe points out.
The church/state legal expert predicts that the courts will likely try to ''duck the issue.'' And he says that in addition to established groups that may vie for legal standing in this case, a taxpayers' suit could be forthcoming which would base a case on objections to public funds being used to further US ties with the Vatican.
Not all American Roman Catholics are happy about the appointment, either. America magazine - a Jesuit opinion journal that caters to Catholic intellectuals - has been lukewarm about it. But it has not openly opposed the move. However, Father Joseph O'Hare, America's editor in chief, says his publication warns Catholics about potential US involvement in their internal affairs. He is concerned that the Reagan administration might use its new diplomatic sway to get US Catholic Bishops to soften their stance on the banning of nuclear weapons. Father O'Hare also suggests that it would have been better to appoint a non-Catholic envoy, to alleviate fears of the church's influence on US government.