What is a species? Ornithologists consider ways to define it.

Determining that, say, dogs and cats are different species isn't difficult. The two species of carnivorans don't just look and act very different, they also can't produce viable hybrid offspring. 

Most biologists use exactly that criteria to draw the line between species, whether or not the two groups can reproduce and yield fertile hybrids. 

The biological species concept, as it's called, seems simple enough. But it's not perfect.

To resolve such questions of speciation, biologists have dug into the genomes of organisms to see how readily two groups interbreed. But there's even ambiguity there. For example, ornithologists agree that the blue winged warbler and the golden winged warbler are distinct species. But their nuclear genomes show remarkable similarities.

Geoffrey Hill, an evolutionary ecologist at Auburn University, says he may have a fresh way to explain speciation: Nuclear DNA alone isn't enough to mark the distinction between species, at least in birds. Instead, it's the divergence of sets of coadapted mitochondrial and nuclear genes that sets species apart, he argues in a paper published Wednesday in the journal The Auk: Ornithological Advances.

Here's how it works: there are two sets of genes in eukaryotes, members of the domain of life that includes all animals and plants. One set is found in a cell's nucleus and the other is in the cell's mitochondrion, an organelle that governs respiration and energy production. As these two sets of DNA work together to ensure an organism's fitness, Dr. Hill's idea is that if the hybrid offspring of two populations have nuclear genes and mitochondrial genes that haven't evolved to work well together and instead counter each other, that could diminish the fitness of the hybrid and therefore articulate the distinction between two species.

For the blue and golden-winged warblers, Hill's hypothesis might resolve the problem.

"Without this new kind of thinking, those birds would likely go away as separate species," Hill says. Although the two species' nuclear genomes are nearly identical, their mitochondrial genomes are quite distinct, he explains in a phone interview with The Christian Science Monitor.

This mitonuclear compatibility species concept, he says, could explain more than just the speciation of these warblers. Although his focus is on birds, Hill argues in the paper that, according to his species concept, "mitochondrial genotype is the best current method for diagnosing species."

David Toews, an ornithologist and molecular ecologist at Cornell University who was not involved in Hill's work, isn't sure the relationship between the nuclear and mitochondrial genes of an organism is the whole story of avian speciation. 

"I think it's a cool idea," Dr. Toews says in a phone interview with the Monitor, and could be a piece of the puzzle, but, "it would be surprising if it was so prevalent to give rise to all the bird species that we know."

"It's always been kind of a difficult thing to put animals or plants into boxes," Toews says. "We know the process of speciation is gradual," he says, "And so trying to put those things that have evolved from a gradual process into these tightly packaged units is difficult."

This new species concept and the classic biological species concept don't have to be mutually exclusive, suggests Darren Irwin, an ornithologist and evolutionary ecologist at the University of British Columbia who was not involved in Hill's work.

"The biological species concept is based on the idea that two species have some form of reproductive isolation between them," he explains in a phone interview with the Monitor. "That can be things that happen before mating, like they don't like each other's song or appearance, or it can be things that happen after mating that make the offspring have lower fitness somehow."

"So this idea that there's mitonuclear coevolution of the mitochondrial genome and the nuclear genome that causes problems in the hybrids, that fits right into that part of the biological species concept," Dr. Irwin says. But alone, he says, the mitonuclear concept "is quite restrictive in how it defines species."

The mitonuclear compatibility species concept might not only apply to birds, Hill says, although that is what his own research focuses on. 

It could be also useful for researchers studying another particularly speciose group, fishes. "The author is likely correct that nuclear and mitochondrial coadaptation is one hallmark of the independently evolving lineages that we seek to recognize in nature," Brian Sidlauskas, an ichthyologist and evolutionary biologist at Oregon State University, writes in an email to the Monitor. "Thus, the paper calls attention to another potential diagnostic tool that can be used to help separate species."

"It is founded on a solid basic tenet that the genome of a species should exhibit co-adapted gene complexes," Dr. Sidlauskas says of Hill's hypothesis. He adds, the compatibility between the nuclear and mitochondrial genomes is "certainly an important biochemical pathway, but it is hardly the only important one."

Jerry Coyne, a biologist at the University of Chicago and co-author of the book "Speciation," agrees that Hill's hypothesis could only be one aspect of what is going on as part of the classic biological species concept. "He hasn't established that this is a better criterion for a species concept than the one that is traditionally used," Dr. Coyne says in a phone interview with the Monitor. "It always comes down to reproductive isolation."

"If you ask why nature is lumpy," he says referring to the groupings that scientists call species, "you can hardly arrive at any other conclusion other than that the things that would make these lumps a continuum instead of a lumpiness are reproductive barriers."

Trying to find a one-size-fits-all species concept might not be the best approach for biologists, Irwin says. "It may be that different concepts work better in different groups of animals or plants and it may be that different processes are sort of occurring in different cases," he says. "There may not be a perfect species concept."