PATTAYA, THAILAND — IN a compromise that set Cambodia's fragile peace back on track, the Khmer Rouge agreed Dec. 3 to return to the capital, Phnom Penh, after fleeing a mob attack late last month.The deal, hammered out by the four rival Cambodian factions at an emergency meeting in Thailand, was the result of pressure from the United States and the other four permanent members of the United Nations Security Council sponsoring the overall peace settlement. Under the compromise, the Khmer Rouge, who are blamed for the deaths of at least 1 million people during their brutal rule from 1975-1979, will return to the Cambodian capital and live in the headquarters of the interim reconciliation council comprised of the rival politicians. Khmer Rouge leader Khieu Samphan is expected to go back to attend a council meeting later this month. A small UN presence is expected in the building until the establishment next spring of a fuller UN administration to oversee a cease-fire, demobilization of the rival armies, and elections expected by early 1993. The crisis in Cambodia's transition to peace was triggered by the attempted lynching of Khieu Samphan hours after he ended a 13-year exile and returned to Phnom Penh. Many Asian and Western analysts believed the demonstration, staged by the Phnom Penh government of Prime Minister Hun Sen, spun out of control in a torrent of anger against the Khmer Rouge leader who was the theoretician behind the group's brutal agrarian revolution. Rejecting a Khmer Rouge demand for UN protection, Western and Asian diplomats hinted that if the interim Supreme National Council (SNC) failed to meet in Phnom Penh, newly opened diplomatic missions and international aid might be endangered. "It's true that there was a cloud over whether the missions accredited to the SNC would be able to stay in Phnom Penh," said a senior US diplomat. The Phnom Penh regime was pressured to meet its commitment to secure the presence of all factions, Western diplomats say. In turn, the Khmer Rouge, whose involvement is considered crucial to a stable peace, backed off its demands for an 800 member UN troop escort and the release of an official missing in the Phnom Penh assault.