Election official P.K. Agnihotri said Mukherjee had received more than twice as many votes as his rival in last Thursday's balloting by national and state legislators.
"I express deep gratitude to the people for electing me to this high office," Mukherjee said. He said he would try to justify the people's trust.
His rival, Purno Agitok Sangma, conceded defeat before the counting was complete Sunday. "I congratulate Mukherjee on his victory and I wish him success," he told reporters.
Mukherjee's elated supporters danced to the beat of drums and set off firecrackers outside his residence as Prime Minister Manmohan Singh and Sonia Gandhi, the Congress party chief, arrived to congratulate him. Several opposition groups also backed Mukherjee.
Mukherjee and Sangma, a former speaker of Parliament, were competing to succeed India's first female president, Pratibha Devisingh Patil. Mukherjee will be sworn in on Wednesday.
In India, the prime minister is the head of government and the president is mostly a figurehead.
Mukherjee has served previously as foreign minister, defense minister and finance minister. His latest term as finance minister from 2009 until earlier this year was marred by declining growth, a tumbling currency and rising inflation. India's economic growth in the last quarter was 5.3 percent, the slowest rate in years.