Is there a way out of the impasse in the Middle East? Former National Security Adviser Zbigniew Brzezinski believes there is. He has made several trips to the region in the past 10 months and held intensive conversations with Arab leaders, including the Palestine Liberation Organization (PLO) chairman, Yasser Arafat. This month he returned from a visit to Tunis, where he was a guest of Chedli Klibi, secretary general of the Arab League.
Despite a sense of frustration and despair over the failure of American policy in the Middle East, says Dr. Brzezinski, Arab leaders still look to the United States to be the honest broker in a comprehensive peace settlement. And despite their own indecision and fears, they are ''desperately'' desirous of a new negotiating process.
As a result of his talks, Brzezinski is urging the Reagan administration to explore a formula which he thinks could cut through Arab reluctance and lead to serious negotiation.
The formula is for a three-stage process:
* The first stage would be the creation of some international sanction for the talks through a mandate of the United Nations Security Council. This would engage the UN secretary general and the Soviet Union, giving the Arabs some feeling that they are not ''alone'' with the United States and Israel. The UN mandate would reaffirm resolutions 338 and 242, calling for withdrawal of Israel from occupied territories to secure borders.
* The second stage would comprise two sets of substantive talks. The first would deal with the West Bank and the Gaza Strip and involve Israel, Jordan, the Palestianians, and the United States. The second set of talks would would deal with Lebanon and the Golan Heights and involve Syria, Israel, and the United States. These negotiations would be required because, without them, Syria would undermine the first.
* A final stage would be some larger forum that would ratify the agreements reached in the narrower talks.
While the plan poses obstacles, Brzezinski says he thinks it would meet with positive response in the Arab world. And he warns that if the United States does not soon resume an active search for diplomatic solutions, the consequences could be perilous.
''American policy is probably in worse condition than at any point since the beginning of the Arab-Israeli conflict,'' he said in an interview. ''The result is that American prestige and American capacity to influence events in the Middle East seems to be at a nadir. I consider that very dangerous because the longer-term consequences of that are likely to be reinforcement of the tendency in the Arab world to turn to radicalism and intensely anti-American fundamentalism.''
Further, if there is no resolution of the Arab-Israeli conflict, he warns, the US will have to consider that the first use of a nuclear weapon by a territorist group against Israel might occur, with all the grim consequences that would entail.
This week marks the fifth anniversary of the signing of the Camp David accords. A participant in that historic event, Brzezinski notes that under the timetable of the agreement, the West Bank by now should have been granted full autonomy.
Acknowledging that election politics make it difficult to come to grips with the current stalemate, Brzezinski nonetheless thinks that the administration could at least begin thoughtful exploration of the problem. And while the Camp David accords and the ''excellent'' Reagan initiative seem dead, he believes their basic elements could be embraced in a new negotiating formula.
A major dilemma for the US has been to get the Arabs to sit down and negotiate. Brzezinski says that in his talks with them, Arab leaders stressed that some way has to be found in which they can join the negotiating process ''without placing themselves at the mercy of the United States, with the United States being deferential to Israel's concerns.'' Hence the effort to find a formula based on the Reagan plan and Camp David, but which would enable the Arabs to feel that the talks have an international sanction.
Why would the Soviet Union agree to a sanction for talks to which they would not be party? ''It is difficult for the Soviets to totally oppose what seems like the beginnings of a peace process at a time when quite a few Arabs are desperately searching for such an approach,'' says Brzezinski, who is a senior adviser at the Center for Strategic International Studies at Georgetown University. ''The Soviets would be exposed too openly, too brazenly, as a party that is really interested in keeping the conflict alive for their own interest. So it would force the Soviets to eat their words . . . or to disassociate themselves from the peace process.''
Would the US accept PLO participation over Israel's objections? ''That depends on whether the PLO would be prepared to accept in the first-stage resolutions, 338 and 242. If so, at least American objection to participation would disappear. Israel might not be willing to talk (with the PLO), but we could talk indirectly.''
With respect to the PLO, Brzezinski says, Arab leaders told him that it is now free of Soviet and Syrian control and therefore more prepared to talk.
Brzezinski says he stressed to the Arabs that they could not engage the US in a negotiating process unless they became a viable partner to the negotiations. Also, that they could not expect - as they have in the past - that the US would promise a bargain in advance of the talks by pushing Israel into a compromise.
''Yet once the negotiating process was joined,'' stresses Brzezinski, ''one could reasonably expect some constructive progress, with both the Israeli and the Palestianian sides gradually coming to realize that what is now thought to be unacceptable can become at least to some degree tolerable.''