The retaking of Bani Walid comes as Libya's new leaders have struggled to unify the oil-rich North African nation three months after Qaddafi was captured and killed.
Hundreds of well-equipped and highly trained remnants of Qaddafi forces raised the green flag over buildings in the western city late Monday after hours of clashes, said Mubarak al-Fatamni, the head of Bani Walid local council.
Al-Fatamni, who fled to the nearby city of Misrata following the attack, said four revolutionary fighters were killed and 25 others were wounded. He said the Libyan Defense Ministry has not sent any forces to the area.
A top commander of a revolutionary brigade in Bani Walid, Ali al-Fatamni, who was present in Benghazi during the attack, says he has lost contact with other fighters in the town.
The bold attacks, which have led authorities to declare states of emergency in several areas, are the latest breakdown in security, three months after Qaddafi capture and killing. Protests have surged in recent weeks, with people demanding that the interim leaders deliver on promises of transparency and compensation for those injured in the fighting.
Bani Walid, 90 miles (140 kilometers) southeast of Tripoli, was one of the last Qaddafi strongholds to fall to revolutionary forces amid a months-long civil war. Gadhafi's son and longtime heir apparent, Seif al-Islam, was long believed to have been hiding in the town.
Seif al-Islam, who has been charged with crimes against humanity by the International Criminal Court, was captured in November by fighters from the town of Zintan in Libya's western mountains, who continue to hold him.