Washington — Despite new tensions, the United States grimly reopens its dialogue with the Soviet Union this week. Last week's destruction of an unarmed South Korean jetliner over Soviet territory may have made it more difficult for the Reagan administration to narrow differences with the Soviets on a variety of issues. For many Americans, the incident has merely dramatized how great US-Soviet differences are.
But going on the theory that in times of tension it is even more important to keep lines of communication open to the Soviets, the White House is moving ahead with plans to send Secretary of State George P. Shultz to meetings in Madrid this week that will include talks with Soviet Foreign Minister Andrei A. Gromyko.
President Reagan has reacted to the downing of the South Korean jet liner in what has been widely described here as a calm and restrained manner. In a televised address scheduled for last night, the President was expected to call on other nations to restrict landing rights for Soviet aircraft for a set period of time in retaliation for the downing of the South Korean plane.
Mr. Reagan was also expected to propose a strengthening of the international rules for safe air travel. He has already led in the call for a United Nations Security Council meeting regarding last week's incident.
In his encounter with Foreign Minister Gromyko on Thursday, Mr. Shultz is expected to seek a full explanation for what happened in Soviet air space on Aug. 31, when a Korean Air Lines Boeing 747 was shot down with 269 people aboard. Shultz will also be probing for signs of flexibility in the Soviet positions on arms control talks. Given the new strains in the relationship, however, discussing the possibility of a summit meeting next year between Reagan and Soviet President Yuri Andropov seems to be less likely than it was before.
''I think the reason that George Shultz wants to meet with Foreign Minister Gromyko is not just to sustain a dialogue,'' said Richard R. Burt, assistant secretary of state for European affairs, toward the end of last week. ''It is to tell him firsthand, face-to-face, what we feel about this atrocious action.''
A senior administration official says, meanwhile, that in the US the Soviet action is likely to reduce the pressure on Reagan to make concessions in arms control talks with the Soviets. It also is likely to increase congressional support for the President's proposed production of MX missiles, some observers say.
''It shows that the United States should not expect too much from arms control negotiations,'' the senior official says.
In Western Europe, he official says, the Soviet action should help unify the NATO allies and reinforce its determination to deploy new US medium-range missiles beginning in December. But this renewed allied resolve might also help produce a more realistic Soviet negotiating position in the Geneva arms control talks both on strategic- and intermediate-range weapons, the official said.
Shultz and Gromyko are to attend the final meetings Wednesday through Friday of representatives from 35 nations who have been working for almost three years in Madrid to complete a final document to be added to the Helsinki accords of 1975 on security and cooperation in Europe. This will be the first meeting between the two officials in nearly a year.
Some observers had considered the successful agreement at Madrid on human rights and other issues to be a sign of improvement in US-Soviet relations. It is the first significant East-West agreement to be reached since the Soviet invasion of Afghanistan in December 1979. But in the public eye, last week's destruction of an unarmed jetliner overshadowed the Madrid agreement.
Max M. Kampelman, the lawyer who has headed the US delegation to the Madrid meetings for the past three years, says that it is nonetheless important to push ahead with the agreement made in Madrid. Mr. Kampelman declares that the Soviet regime is ''brutal,'' but argues that the Helsinki accords give the West an opportunity to hold the Soviets accountable for violations of human rights and other internationally agreed upon norms.
''Repression is at the highest level in the Soviet Union since the Helsinki Final Act was signed in 1975,'' said Kampelman in an interview. ''There is no question about that. The repression is massive.''
''But what I would say is that in the face of that, it's all the more necessary to try to strengthen the Helsinki process and to try to do what Madrid has done,'' he said.
''What we have done for three years in Madrid is present what I think has been generally acknowledged to be the fullest and most authoritative review and detailed analysis of the nature of the Soviet Union,'' Mr. Kampelman said. ''We've gone into Soviet anti-Semitism, anti-Catholicism, the treatment of the Baltics, the nationality groups, the use of psychiatric hospitals, the use of slave-labor camps, the use of chemical and bacteriological warfare, the Afghanistan experience - in great detail, with facts and figures.''
Kampelman argues that a series of international meetings arranged as a follow-up to Madrid will provide a continuing opportunity to judge Soviet accountability on human rights, family reunification, and other issues.
''We look upon Madrid on two levels. And, in effect, I believe it's the way we should look upon East-West relations and our dealings with the Soviet Union, '' said Kampelman. ''On one level, we must seek accommodation. This is a powerful military machine.''
''We have never fought a war against Russia,'' he continued. ''And the last thing in the world that any rational, intelligent human being would want is to think in terms of war today. So this is where dialogue and discussion come in. And this is what Madrid permits us to do.
''But there is a second track, and that is a recognition that this regime is a brutal regime, and we have to use the Madrid forum to let Europe know, and remind Europe of what they know but some would like to forget - that this is a brutal regime which oppresses its people.
''And a government which declares war against its own people cannot be trusted not to declare war against its neighbors.''
During the period of detente, the West ''let down its guard,'' Kampelman contends, while the Soviets continued engaging in massive political warfare, spending billions of dollars on propaganda, espionage, and subversion.
Kampelman contends that the Soviets are currently engaged in a ''peace offensive'' aimed at gaining the sympathy of Western Europe. But he sees no evidence that this amounts to much more than a propaganda offensive.
''They're intensifying their activities, as far as I can see, in Central America and Africa,'' said Kampelman. ''The subversion has been intense - just look at the number of Soviet diplomats who've been expelled by European countries that are reluctant to expel.. . . The Afghanistan negotiations are so far only talk, a lot of talk. the Soviet troops are still there. The repression is still there.''
''It requires patience to deal with the Soviets,'' said Kampelman. ''They have patience, and we must have patience to remain a day longer than they. And this, of course, is one of the things we showed in Madrid, that we were prepared to stay a day longer than they.''
''I'm not afraid of dealing with the Soviets,'' said Kampelman. ''I think the thing to do is to deal with them, and defeat them and beat them and outsmart them and out-patience them. It's a competition. We can't avoid the competitive nature of the relationship.''