London — The British government has quietly begun to examine a new formula for solving the problem of the Falkland Islands. The aim, after a period of waiting to allow the political dust of battle to settle, will be to internationalize the territory. Already two men closely involved with the Falklands crisis, but now retired, have spoken favorably of the idea.
Lord Carrington, who resigned as foreign secretary after the Argentine invasion, has suggested that the islands be administered under an international agreement, possibly a modified version of the Antarctic treaty.
Sir Nicholas Henderson, Britain's ambassador in Washington during the Falklands war, supports much the same idea: ''I am sure that in some way at some stage the problem will have to be internationalized. Other countries will have to be brought in. Perhaps something will be sought along the lines of the Antarctic Treaty of 1961.''
Government officials in Whitehall say it is far too early to begin public consideration of this line of approach. In any case Prime Minister Margaret Thatcher has declared that for the time being ''fortress Falklands'' is the only option available for Britain.
But it is an option that leaves neither Mrs. Thatcher nor members of her government happy. The long-term problem and cost of defending islands 8,000 miles from Britain and only 400 miles from Argentina will be formidable.
Mrs. Thatcher, however, has a dual problem in trying to move beyond ''fortress Falklands.'' Argentina, still under military rule, is in no mood to talk and has not formally ceased hostilities following the Falklands war.
At Westminster the prime minister would come under attack from her own back benches if she proposed negotiations too soon. With a general election probable this year the Prime Minister would weaken her position politically if she suddenly appeared ready to yield.
So, things are likely to stay as they are for the time being.
But the junta in Buenos Aires is promising a return late this year to civilian rule. And if Britain's Conservatives are confirmed in power after the election, Mrs. Thatcher's room for maneuver will immediately expand.
Whitehall planners are working within these political considerations. They are virtually unanimous that the ''Antarctic option'' - as it is already being called - is a blueprint which they should start to draw up now.
Official thinking behind the scenes is still sketchy. But two ideas are current: Either the Falkland Islands are included in a newly drawn Antarctic treaty; or they are made the basis of a separate agreement.
The second approach appeals to some diplomats who conjecture that Britain and Argentina could simultaneously renounce sovereignty over the islands and then agree on a form of administration involving international guarantees.
Tactically, Britain must let its long-range thinking be affected by such ideas. At the United Nations and within the European Community, the country expects to come under pressure soon to look for ways of reopening negotiations about the islands' future. The problem will not just go away.
Sir Nicholas Henderson observes, ''British governments for years have been urging everyone to settle differences by negotiations, and we can expect to be asked to follow our own advice.''
At the same time, the mineral and food resources in the area of the Falkland Islands could be developed. But Britain accepts that this will not be possible without a concerted international effort.
For the present British south Atlantic policy appears to be in a cul-de-sac and the Argentines remain angry. In Whitehall, however, contingency planning is already under way.
''We are talking about years, certainly not months,'' says one Whitehall official, ''but sooner or later there must be talks. With other options played out, the international approach has to look appealing.''