The courtroom is coming under increasing scrutiny in its role as the arena where divorce cases and other sensitive family issues are settled. ''There is an undercurrent across the country to take the divorce process out from under the formal legal system,'' says James Cook of the Joint Custody Association. He senses a ''tremendous resentment'' toward the prevailing adversarial court process that gives divorcing couples the impression that ''You've got to go to war whether you want to or not. If you don't fight you're going to lose.''
In attempting to establish who is the ''better'' parent, lawyers' tactics can inflame an already emotional situation, bringing out the worst in parents rather than the best.
Dr. Stuart Berger, author of ''Divorce Without Victims'' (Houghton Mifflin, Boston), questions the ability of judges to make crucial decisions concerning children's welfare. ''A judge makes decisions according to the law, but the law has nothing to do with child development,'' he says.
Both laymen and professionals are exploring ways in which divorcing couples can have more say in settling conflicts and in making decisions regarding their children.
In mediation, divorcing couples meet on neutral ground to work out their own custody arrangement with the help of a third party.
''The move to mediation will come when parents realize it is an option,'' says family specialist Ciji Ware, adding that many divorcing couples ''settle on the courthouse steps'' after a judge's decision anyway. ''People who determine their own fate can live with the decision better than a decision that is handed down,'' she says.
Twenty-three states now use mediators in their court system. California requires couples involved in a custody fight to try to settle their dispute with a mediator before they see a judge. In Los Angeles, 60 percent of custody cases are decided before a couple enters the courtroom.
Hugh McIsaac, director of the Los Angeles Conciliation Court, says the conciliation process eliminates the worst aspect of custody fights - competing for the children.
According to Mr. McIsaac, the Los Angeles conciliation system (which does not deal with financial matters) has effectively lessened the burden on the courts. He goes as far as saying, ''California is writing the textbook on how divorce conflicts will be resolved in the 21st century.''
Some judges and lawyers are skeptical of mediation, believing it may deny people good, solid legal representation. Critics question the qualifications of individuals hanging out their shingle in a profession still in its infancy. They also question whether mediators, either in a private capacity or as part of a court system, can or should tackle the financial issues so closely intertwined in divorce cases.