The ruling could help put to rest lingering animosities between the Balkan neighbors.
The International Court of Justice said Serb forces committed widespread crimes in Croatia early in the war, but these did not amount to genocide. The 17-judge panel then ruled that a 1995 Croat offensive to win back territory from rebel Serbs also featured serious crimes, but did not reach the level of genocide.
Fighting in Croatia from 1991-95 left around 10,000 people dead and forced millions from their homes.
Tuesday's decision was not unexpected, as the U.N.'s Yugoslav war crimes tribunal, a separate court also based in The Hague, has never charged any Serbs or Croats with genocide in one another's territory.
Croatia brought the case to the world court in 1999, asking judges to order Belgrade to pay compensation. Serbia later filed a counterclaim, alleging genocide by Croat forces during the 1995 "Operation Storm" military campaign.
Rejecting both cases, court President Peter Tomka stressed that many crimes happened during fighting between Serbia and Croatia and urged Belgrade and Zagreb to work together toward a lasting reconciliation.
"The court encourages the parties to continue their cooperation with a view to offering appropriate reparation to the victims of such violations," Tomka told a packed Great Hall of Justice at the court's Hague headquarters, the Peace Palace.
Decisions by the International Court of Justice are final and legally binding.
Tomka said crimes including killings and mass expulsions by both sides constituted elements of the crime of genocide, but the judges ruled that neither Serbia nor Croatia carried out the crimes with the "specific intent" to destroy targeted populations.
Both countries expressed disappointment that the court had rejected their claims, but said it is time to move on.
"We are not happy, but we accept the ruling in a civilized manner," Croatian Prime Minister Zoran Milanovic said. "It has been more than 20 years; Croatia is now a member of the European Union and can now build its future."
In Serbia, President Tomislav Nikolic said that although the court did not back the Serbian genocide claim it did "reverse some usual stereotypes" that Serbs were the only culprits for the war.
"Despite the injustice, an encouraging step has been made," Nikolic said in a televised statement. He expressed hope Serbia and Croatia will move on "in good faith
The case brought by Croatia was not the first time Serbia had faced allegations of genocide at the world court.
In a landmark 2007 judgment, judges cleared Belgrade of committing genocide in the 1995 massacre of 8,000 Muslim men and boys in the Bosnian enclave of Srebrenica, but said Serbia breached the genocide convention by failing to prevent the slaughter, Europe's worst mass slaying since World War II.
Associated Press Jovana Gec in Belgrade contributed.