Washington — A resurgent Democratic Party is wasting no time flexing its muscles in the new House of Representatives. As the 98th Congress opened this week, Democrats in short order gained 26 seats, ushered in revised House rules to strengthen their leadership, and delivered a stunning rebuke to Rep. Phil Gramm, the Texas Democrat who led the fight for President Reagan's economic program.
Democrats, who have for decades allowed Southern conservatives to vote against the party without penalty, finally drew the line at Congressman Gramm, who was expected to announce soon whether he would switch to the Republican Party. The Texas lawmaker lost his seat on the influential House Budget Committee, where he had allied himself with Republicans and had given his name to the Reagan-backed budget bills of 1981.
''I don't believe in punishments or an enemies list,'' said Democratic House majority leader Jim Wright, the fellow Texan who two years ago recommended Gramm for the key committee. But Congressman Wright added to reporters on Tuesday, ''It's perfectly all right to have a friends list.''
And Gramm is not on that list. According to Wright, the Democratic leadership asked for support on 18 key votes during the last Congress, and Gramm voted with the Republicans on every one.
Gramm's offenses went beyond his voting record, however. He made even fellow conservative Democrats uncomfortable with his public image as a rebel.
''It's his method that got him into most of the trouble,'' says Rep. Charles Stenholm (D) of Texas, chairman of the Conservative Democratic Forum, the group known as the boll weevils. Mr. Stenholm, whose own voting record is close to Gramm's, says, ''There is a limit to how far you can go.''
Gramm reached the final limit when he fought the rules changes that had passed in the Democratic caucus. It is an unwritten rule that members can fight hard behind the closed doors of the caucus on party matters, but in public they vote together.
Even as his membership on the Budget Committee was hanging in the balance, Gramm voted on Monday against the Democratic-proposed rules on the House floor, which would strengthen the Speaker's hand on the House floor. Among other changes, the rules do away with some procedural votes and bar legislative ''riders,'' such as antiabortion amendments, to appropriation bills.
''They can tolerate conservatives,'' says a Gramm spokesman of the Democratic leadership. But Gramm did more, says the aide. ''He actually went out to win, and more than that, he did win (on budget votes).''
While Democrats point to Gramm as a ''special case,'' his removal from the Budget Committee sends a message to other conservatives.
''It got our attention,'' says Rep. G. V. (Sonny) Montgomery (D) of Mississippi, another boll weevil. Mr. Montgomery, who also went against the leadership on key votes last term, came close to losing his chairmanship of the Veterans Affairs Committee. In the Democratic Steering and Policy Committee, which nominates chairmen, the vote was 16 to 11 to retain him.
Will Montgomery change his voting this term? ''I'm still going to vote my convictions on bills I feel very strongly about,'' he says, but he adds that he will back the leadership on procedures. He also predicts that he will be taking fewer trips to the White House.
Like all boll weevils except Gramm, Montgomery voted for the new Democratic rules.
Regardless of how the boll weevils vote in the coming year, they have already lost much of the clout they enjoyed two years ago.
In fact, the group may even have a tough time recruiting new members. Some newcomers are reportedly reluctant to join. Montgomery says of freshmen House members, ''The ones I have talked to are concerned about deficit spending,'' a traditional boll weevil issue. But do they say they want to join the boll weevils? ''They don't say that,'' he answers.
Representative Stenholm says it is too early to seek new members. ''It is not our intention to solicit any members soon,'' he says, but he mentions that only one new member has asked to join so far.
Rep. Sam B. Hall Jr., another Texas boll weevil, while saying ''We'll still be active,'' adds, ''I'm not so sure we're going to pick up many new members, just looking at their biographical sketches.''
Even more important, the boll weevils can no longer provide enough votes for Republican-conservative victories, now that the Democrats have gained 26 new seats.
''We don't have the swing votes,'' says Montgomery, adding that the feelings among conservatives have also changed toward the Republican administration. ''There is not that strong feeling about the President. He was given his chance. That's what the people wanted.''
Rep. Tom Loeffler, a Republican from Texas who acted as a coalition builder with boll weevil Democrats, says, ''There is still an opportunity for a bipartisan coalition, depending on the issues. But it will take (all) 166 Republicans and 52 Democrats.''
Mr. Loeffler argues that the new Congress will not overturn the basic Reagan program that reduces federal spending and builds up defense. ''We will not go backwards,'' he says. ''We will be adjusting.'' If the Democratic leadership moves too far, he maintains, the President will use his veto and boll weevils and Republicans could join again. ''That coalition will sustain vetos,'' says Loeffler. ''Ultimately, there'll be a compromise.''