"This was a difficult decision and it was based on the feeling that this was the direction the franchise needed to go," Colts owner Jim Irsay said during a news conference on Tuesday.
"It's a big change for the franchise but at the same time there's players, other coaches and many people on the staff that will go forward into the new day and get on with the work of 2012."
The decision to release Caldwell, who joined the Colts in 2002 as a quarterbacks coach, marks the second key move by the Colts since the end of their season after Vice Chairman Bill Polian and General Manager Chris Polian were fired two weeks ago.
In his first year as head coach, Caldwell went 14-2 to win the AFC South division and got Indianapolis to the Super Bowl. He led the Colts to another division title in 2010 but they fell in the first round of the NFL playoffs.
Manning's neck surgery forced him to miss the entire 2011 campaign and has clouded his future with the team since their losing record awarded them the top pick in April's draft, where Stanford quarterback Andrew Luck is pegged as the top choice.
Manning, who was drafted in 1998, has won a Super Bowl with the Colts and took home most valuable player (MVP) honors in the title game. He is also a four-time league MVP and considered one of the game's most prolific passers.
Irsay said the decision to make changes at the head coach and general manager positions was similar to 1998, when the team went in a different direction by replacing those two positions and drafting Manning.
But the Colts would not say whether the latest round of changes will include a change at the key quarterback position this time around.
"We're not even there on anything involving Peyton Manning quite yet. We have to know about his medical, there's so many things. I am not even there yet," said General Manager Ryan Grigson.
"In regards to Peyton, I have not quite gone in that direction yet and that is something that's not been discusseed."