THE Northern Ireland peace process, already wobbling because of distrust across the religious divide, has been given a severe jolt. James Molyneaux, widely seen as a powerful voice for moderation among Northern Irish Protestants, unexpectedly resigned yesterday from his post as leader of the Ulster Unionist Party. The UUP is the more dominant of Northern Ireland's two main unionist parties that favor maintaining British rule in the province. The battle to succeed the man who has led the party since 1979 will be bitterly fought and is likely to put the quest for peace on hold for some time, British government officials say. A senior figure in the London government said: ''This is worrying news. Without Molyneaux's steadying hand, the UUP could have derailed the search for a political solution in Northern Ireland months ago. Much will depend on who takes over, and the omens are not encouraging.'' A tight-lipped, unsmiling figure who made few speeches in the House of Commons and ruled his party with what colleagues said was a rod of iron, Molyneaux had a close working relationship with British Prime Minister John Major. Molyneaux often threw his party behind the government in crucial Commons votes, enabling Mr. Major's administration, which has a majority of only nine seats, to survive. This earned him enemies in the upper ranks of his party and led to a draining away of popular support. This past March, he was challenged for his party's leadership by a young ''stalking horse'' candidate. University student Lee Reynolds gained a fifth of the vote on a platform that attacked Molyneaux's approach to the peace process. Reynolds argued that Molyneaux had failed to resist British pressures to speed up contacts with Sinn Fein, the political wing of the anti-British Irish Republican Army, and had not done enough to demand concessions from the IRA. Molyneaux defends the union between Northern Ireland and Britain, but colleagues said he thought the best way to support it was by showing flexibility and political dexterity. An early sign of this approach came in 1977 when he severed the UUP's links with the Reverend Ian Paisley, who formed the more radical Democratic Unionist Party. He has been an outspoken opponent of violence on both sides of the religious divide. In October 1982, on the same day, gunmen of the Irish National Liberation Army (a rebel offshoot of the IRA) twice attempted to assassinate Molyneaux. In the 12 months since the IRA and - later - Protestant paramilitary groups declared a cease-fire, Molyneaux has been the focus of whispered criticism by members of his party. Many said his approach was failing to safeguard the Protestant position in Northern Ireland. These critics are now certain to come out into the open. Within minutes of Molyneaux announcing his resignation, a senior unionist source said: ''We can now expect not only a contest for the crown, but a battle over how the party should approach the peace process.'' Four main candidates are likely to contest the leadership election - all apparently less flexible than Molyneaux. Ken Maginnis, the party's security spokesman, is a leading contender, but there were suggestions from within Unionist ranks yesterday that David Trimble, a much younger man, would gain support. Mr. Trimble has criticized Sinn Fein leader Gerry Adams for his hard-line approach to the peace process. Earlier this year he walked out of a TV studio when asked to debate live with Mr. Adams. Molyneaux said yesterday he was stepping down so that his party could rally behind a new leader in time for a general election expected in the next 18 months.