In a statement, President Barack Obama said he had accepted Summers' decision.
"Larry was a critical member of my team as we faced down the worst economic crisis since the Great Depression, and it was in no small part because of his expertise, wisdom and leadership that we wrestled the economy back to growth and made the kind of progress we are seeing today," Obama said. As director of the National Economic Council, Summers oversaw the administration's response to the economic and financial crisis early in Obama's first term.
Still, Summers faced opposition from some Democrats, including members of the Senate Banking Committee. Summers alluded to the opposition to his candidacy in a letter he sent to Obama Sunday to formally withdraw from consideration.
"I have reluctantly concluded that any possible confirmation process for me would be acrimonious and would not serve the interests of the Federal Reserve, the administration or ultimately, the interests of the nation's ongoing economic recovery," he wrote.
Summers' ascent to the top of the list to succeed Bernanke rankled some of the president's liberal supporters. He was seen as having been too cozy with Wall Street and was criticized for controversial comments he made about women and math and science.
Shaunna Thomas, co-founder of the women's rights group UltraViolet, welcomed Summers' withdrawal, saying she hopes it serves as "a reminder to all that sexism has no place anywhere in society and certainly not in the highest levels of our government." Thomas called on Obama to nominate current Fed vice chairwoman Janet Yellen, another candidate on Obama's short list, to take over Bernanke's job.
Obama vigorously defended Summers in recent weeks, both in public comments and in closed-door meetings with lawmakers.
"I will always be grateful to Larry for his tireless work and service on behalf of his country, and I look forward to continuing to seek his guidance and counsel in the future," Obama said Sunday.
The president has also been facing trouble with his liberal base on other fronts this year, including revelations about the National Security Agency's spying programs and his call for a military strike against Syria.
Summers' decision could make Yellen the leading candidate to replace Bernanke. Yellen, who became a member of the Fed's board of governors in 1994, would be the first woman to run the Fed.
Obama is expected to announce his nomination as early as this month. Bernanke's term ends Jan. 31, 2014.