JAPANESE hopes of recovering the so-called Northern Territories from the Soviet Union have been spurred by new signals from Moscow. Both sides are proceeding with caution, knowing that, if mishandled, the demons of nationalism could be reawakened, thwarting a promising new beginning in relations between the two neighbors.The Northern Territories are four small islands (actually three islands and a group of islets), sometimes known as the southern Kuriles, that Moscow has occupied since World War II. One signal coming from Russian President Boris Yeltsin is that the issue will have to be negotiated with Russia, not the Soviet Union. In a meeting this week with Japanese officials, Ruslan Khasbulatov, acting Russian Parliament speaker, stressed the Russian Federation's determination to settle the dispute and remove it as an obstacle to an official World War II peace treaty and major economic assistance from Japan. Earlier this week, the reformist deputy head of the Soviet Union's economic management co mmittee, Grigory Yavlinsky, advocated the return of the islands to Japan based on a 1855 treaty. This is the first time a Russian of such prominence has publicly advocated returning the islands. The history of this dispute is long and tangled in competing claims of nationalism. Both Japan and Russia started out with a sense of having to catch up with the West, after having fallen behind because of feudalism in Japan's case and despotism in that of Russia. "How far we are from Europe!" sighs a liberal in one of Tolstoy's plays. A 19th century Japanese intellectual would have recognized and shared this sentiment, although he would have viewed Russia as an encroaching, imperialistic power, far ahead of his own country in the technology and means of waging modern war. Vigorously modernizing itself, Tokyo had caught up with Moscow by the end of the century and was challenging it for domination of weaker neighbors. Japan grabbed Korea, and Russia was well on its way to grabbing Manchuria from China when Japan's victory in the Russo-Japanese war of 1904-05 turned the tables. The Kuriles and the much larger island of Sakhalin were two other areas where Tokyo and Moscow competed, as Japan thrust northwards and toward the mainland while Russia expanded southwards toward the ice-free Pacific Ocean. A treaty between the two signed in 1855 - the treaty Yavlinsky cites - recognized the southern Kuriles as Japanese, with the northern Kuriles and Sakhalin open to citizens of both countries. In 1875, a new treaty assigned all the Kuriles to Japan and all of Sakhalin to Russia. In 1905 , Japan gained the southern half of Sakhalin after its victory over Russia in the 1904-05 war. Forty years later, Moscow joined the Allied coalition against Japan just one week before the emperor's surrender in World War II. Soviet troops overran Manchuria, southern Sakhalin, and all the Kuriles. Japan, its war machine destroyed, gave up all the territories it had gained by conquest in the previous hundred years. But Tokyo insisted that the southern Kuriles were inherently Japanese territory, not acquired by conquest. Successive Japanese governments have said they would sign a peace treaty with Mo scow only when the Soviets agreed to return the islands. The Soviets have argued that these islands are spoils of war. The whole post-1945 world order is based on the inviolability of the borders established as a result of the Allied victory. Why should Japan alone challenge these borders? The latest Soviet signals suggest this attitude may be changing. Yavlinsky, for instance, disclaims any direct link between Japanese economic aid and returning the islands, but recognizes that such aid would "greatly mitigate the shock" caused by the radical economic reform policies he and his group are planning. There's the rub. For all Moscow's need, a crude linkage of aid with the territorial issue could backfire. Nationalism is a dangerous, unpredictable animal, which can be harnessed not only by despots riding roughshod over their opponents, but by demagogues in a democracy as well. So the caution on both sides is understandable. But the birth-pangs of a new Russia, a new Soviet Union, offer Japanese and Russians the opportunity to build up a new relationship based on adherence to principles of freedom and democracy, not competing nationalisms. It is an opportunity not to be missed.