Washington — Although Congress packed up and left town for the year, one piece of unfinished business is still nagging at its leaders. ''Unless we rein in budget deficits as soon as possible, the cost of servicing our national debt will wipe out the progress made in restraining spending,'' warns Senate Finance Committee chairman Robert Dole (R) of Kansas in what has become a standard deficit-fighting speech that he is taking from podium to podium during the legislative break.
Senate majority leader Howard H. Baker Jr. (R) of Tennessee this week put ''control of the fiscal affairs of this nation'' second only to dealing with the danger of nuclear arms.
And in the background came the drumbeat of concerns over medicare costs, as the House Ways and Means Committee sponsored a conference on the woes of the program, which offers medical aid to the elderly. Medicare is widely seen as heading for bankruptcy in the next decade, and the Congressional Budget Office reported that only dramatic reforms can avoid huge medicare deficits.
So politically explosive are the budget problems, Senator Baker told a meeting of the American Gas Association here, that only a bipartisan effort can solve them.
''Americans are good at being bipartisan for short spurts,'' he said, but the need now is to push aside partisan politics on the deficits ''for maybe 10 years.''
But as the 1984 election year nears, bipartisanship is expected to be rare even for 10 minutes. Moreover, fighting deficits means more tax revenues, which the President rejects, or cuts in federal benefit programs such as social security and medicare, which House Speaker Thomas P. O'Neill Jr. (D) of Massachusetts opposes with at least equal vehemence.
The one ''hope'' could be a change in the economy next January that would galvanize Congress.
''This place works best in a crisis,'' says an aide to House Ways and Means chairman Dan Rostenkowski (D) of Illinois. ''If there is an economic crisis, I suspect we'll move very quickly.''
If interest rates should rise a bit in January and cause concern about the economy, that might create ''a favorable atmosphere'' for action, says George A. Ramanos, an aide to Senate Budget Committee chairman Pete V. Domenici (R) of New Mexico. But short of that, prospects for action are dim, the aide says. ''Nobody ever got elected by raising taxes.''
Moreover, Mr. Ramanos adds, ''Nobody's going to touch entitlements in an election year, because entitlements include social security.''
Senator Domenici is aware of election sensitivities, since he is seeking a new term next fall. However, the budget chairman has already put his committee to work on a deficit-cutting plan for introduction early next year, just as Congress begins to study the President's 1985 budget.
Meanwhile, Senator Dole, who is not up for reelection next year, holds hearings Dec. 12-14 on the deficit at an otherwise deserted Capitol Hill.