Boston — The church-state debate, which has escalated throughout the presidential campaign, took on a new wrinkle Wednesday with a court challenge to President Reagan's appointment of a diplomatic envoy to the Vatican.
Plaintiffs - including Protestant and Jewish groups and individuals - charge that having a US ambassador in the Vatican is a violation of the First Amendment to the United States Constitution.
In a suit filed in the US District Court in Philadelphia, they say the appointment ''entangles'' the government in the internal affairs of the Roman Catholic Church and grants special status to one religion over another.
Over the years, the US Supreme Court has developed a church-state legal ''test,'' through which it has generally struck down government ''entanglement'' with religion or public ''preference'' for any church group. This year, however, the high court retreated somewhat from this strict interpretation by permitting government ''accommodation'' of religion in the case of a Christmas display erected on public grounds.
Some courts have also suggested that situations or events involving churches can have nonreligious or nonentangling purposes.
When President Reagan announced that he would appoint William Wilson as US ambassador to the Holy See late last year, there was vocal opposition from many church and constitutional groups. The administration defended its move as a diplomatic tie not to the Catholic Church but to the Holy See, the government of the Vatican City, a city-state headed by the Pope.
The US had full diplomatic relations with the Holy See for two decades during the 19th century and less formal ties for several years earlier. But official diplomatic status with the Vatican was banned by Congress in 1867. And this prohibition remained in effect until it was lifted late last year by federal lawmakers at the administration's request.
The appointment of Ambassador Wilson was confirmed in March after some balking by Congress, including initial delay by the House Appropriations Committee in approving a State Department request for $351,000 to pay the ambassador's salary and expenses. In April, Mr. Reagan accepted the credentials of Archbishop Pio Laghi as the Vatican's ''apostolic pro nuncio'' to the US.
Wednesday's legal move comes in the midst of the most controversial church-state debate that has ever surfaced during an American election campaign. President Reagan has been sharply criticized by opponents for seeming to embrace the New Right's ''Christian nation'' stance and for strongly backing such highly volatile religious issues as the classroom prayer amendment and government aid to parochial schools.
Democrats, meanwhile, are engaged in the controversy over vice-presidential candidate Geraldine Ferraro, a Catholic who favors free choice on abortion. She is battling with some elements of her church who take an absolute stand against this practice.
Many church officials have called a truce to what they call the ''religious wars of 1984,'' castigating politicians who have injected church issues into the campaign.
Robert Maddox, executive director of Americans United for Separation of Church and State, in announcing the suit against exchange of diplomats with the Vatican, said: ''We believe this formal relationship between the state and one church is absolutely improper in a nation where the principle of church-state separation has long been enshrined in constitutional law and in the hearts of the American people.''