Washington — In the aftermath of one of the worst presidential election drubbings in history, Democratic House members this week will pick their leaders and vote on rules for the new Congress in a restive mood.
House Speaker Thomas P. (Tip) O'Neill Jr. will hold his post easily. But although the white-haired and deeply partisan Massachusetts Democrat has promised that this will be his last term, there have been nippings at his heels.
Conservative Rep. Charles W. Stenholm, shocked at a loss of four House Democrats in his state of Texas, had the temerity to threaten to run for speaker.
The ''coach,'' said Representative Stenholm, is ''responsible for the losing season,'' and ''Tip is the coach.'' However, by last Friday the Texan conceded that he could count only three votes for himself, and one of those was reluctant.
If there will be no coaching change for the Democratic Party, then the liberal Representative O'Neill will probably continue to be the Reagan administration's chief antagonist. In fact, an aide predicts that few of O'Neill's fellow Democrats will want to compete for that spokesman role.
But the signs that this is a transitional year for the Democrats are clear. Some of the pushing and tugging come from members worried that the Democratic Party has lost the middle class and is losing the South.
''We've got to make a strong pitch for middle-class America,'' says Rep. Buddy Roemer, a conservative Louisiana Democrat and Stenholm ally.
He says of the Democratic decline in the South and Southwest, ''We've been saying for years now that it's coming. It's here now.''
Estimating that about 60 members are expressing concerns about the party leadership, he points out that some are younger members who say ''it's our turn'' to be leaders.
Already Speaker O'Neill has bowed to pressures from his members to expand his circle of advisers. He has established a council to meet every other week to discuss the party's legislative program.
Last week during a Stenholm-O'Neill meeting, the speaker indicated he might be willing to put a conservative, perhaps Stenholm, on that council.
The restless extend beyond conservatives, however. O'Neill is ''phasing out, '' says Rep. Vic Fazio (D) of California, a moderate just elected to his fourth term. ''There needs to be additional spokespersons'' for the party, he says.
Representative Fazio says he expects a style of more open leadership, personified in Rep. Richard Gephardt (D) of Missouri, who is expected to be elected unopposed as the new Democratic Caucus chairman this week.
Fazio also says he hopes Democrats will begin to deal tougher with interests groups, including labor. ''There can't be this holding our feet to the fire,'' he says.
The House Democratic leadership is playing down the restiveness. ''I think that what you see is a continuous, ongoing process,'' says majority leader Jim Wright (D) of Texas. ''I'm not all that alarmed by it. It's healthy.''
As the Democrats meet this week, they will also have other controversies to resolve.
TV in the House. A Democratic committee is seeking to limit the ''special orders'' time for House members to speak out on the House floor following the normal daily business to only two hours, one hour per party. The move would cut off a group of outspoken Republicans who have built national reputations by attacking Democrats during those televised periods.
The right-leaning Republicans have an unlikely ally in Rep. Don Edwards (D) of California, one of the staunchest liberals in the House. Recalling the time he helped keep the House in session most of the night with speeches against the Vietnam war, Representative Edwards says of the proposed gag rule, ''It should be defeated because it diminishes minority rights.''
Budget process. Democrats will consider a controversial and dramatic change in the way the House passes money and budget bills.
A proposal by Rep. David R. Obey (D) of Wisconsin would wrap all of the legislation into one gigantic bill. If approved, the plan would make radical changes in House operations.
Opponents say changing the process won't help. What needs to change is the will to make budgeting work, they say. But there has been such complaining about exisiting procedure that some members want an overhaul.
Arms control committee. Rep. Thomas J. Downey (D) of New York is pushing for a ''select'' House committee to review and discuss arms-control policies. Although it would have no legislative power, the panel would probably prod the Reagan White House on arms policy.