The fighting came hours after suicide bombers killed 137 people in the national capital Sanaa, in coordinated attacks claimed by Islamic State, an al Qaeda offshoot that controls swathes of Syria and Iraq.
Fighters from Al Qaeda in the Arabian Peninsula (AQAP) were forced to withdraw late on Friday night from al-Houta after holding it for several hours, the officials and residents said. Two army brigades then entered the city, capital of Lahj Province.
There were no reports of any militant casualties.
Houta is only 30 km (20 miles) from the Indian Ocean port of Aden, where President Abd-Rabbu Mansour Hadi has temporarily based the government since he escaped from weeks of house arrest in Sanaa by Houthi militia, which controls the Yemeni capital.
Yemen has been hurtling towards civil war since last year when the Houthis advanced from their northern heartland, further undermining the country's tenuous internal security and creating more space in which AQAP can operate. In the past two days, unidentified warplanes have bombed the palace in Aden that Hadi has been using.
Western countries and Yemen's Gulf neighbors see AQAP as the most dangerous al Qaeda branch after its efforts to bomb international airliners and launch cross-border raids into top oil exporter Saudi Arabia. Washington has been waging a drone air war against the militants in Yemen.