Boston — These are tough times for American fiction writers. If today Ernest Hemingway were finishing his first novel, ''The Sun Also Rises,'' he might have to take it to the nearest university press to get it published.
''A Confederacy of Dunces,'' Pulitzer Prize-winning first novel by the late John Kennedy Toole, was rejected by ''eight or nine'' commercial houses, claims the author's mother, before Louisiana State University Press published it in 1980. To date, it has sold 600,000 copies in cloth and paper.
Commercial publishers, pressured by skyrocketing printing and paper costs and chain booksellers' mass-marketing needs, are reluctant to publish new writers, says William Guthrie of Little, Brown & Co., a venerable commercial house in Boston.
''Many novels aren't being published because they don't fit standard commercial genres, i.e. adventure stories, gothic romances,'' says L. E. Fillabaum, director of the LSU Press. ''Our reason for starting to publish fiction was the problem we perceived of fiction of high literary content having small market potential.''
Since the 1960s, when the short-story market began to dry up due to rising book-publishing costs, the demise of the general interest magazines, and the glut of young talent, university presses stepped in to help fill the void. While emphasizing that they continue to focus on academic works, their ''bread and butter,'' university press spokesmen also cite a responsibility to keep this American genre alive.
Even though the university press output is only a drop in the publishing bucket - 3,000 out of a total 40,000 titles a year, of which only a small fraction are new works of fiction - these presses are providing a lifeline to authors who might not find opportunities with commercial publishers.
Out of the 79 members of the Association of American University Presses, ten - Illinois, LSU, Johns Hopkins, Arkansas, Indiana, Georgia, Pittsburgh, Missouri , and State of New York - publish fiction. Some of these produce only one novel, novella, or short-story collection a year; others as many as six.
Some universities publish the winning entry of their annual fiction contests:
* Missouri's Breakthrough Program accepts manuscripts of short story collections or poetry from previously unpublished authors.
* Georgia recently introduced the Flannery O'Connor Short Fiction series (named after the late Southern author). It offers two separate awards for book-length collections.
* The Drue Heinz Literary Prize for short fiction is given by the University of Pittsburgh Press, which, besides publishing the winning entry, awards $5,000.
* The oldest contest, the Iowa School of Letters Award, presented by the University of Iowa Press, bestows a $1,000 prize.
* The Associated Writing Programs' AWP Award Series at Old Dominion University in Norfolk, Va., awards three prizes: for poetry, short fiction, and a novel, each of which is farmed out to a different university press for publication. It too pays $1,000.
Competition for these awards is fierce; contests average 300 entries and are judged by such luminaries as Kurt Vonnegut, Joyce Carol Oates, and Robert Penn Warren.
Winning an award contest may provide just the needed boost to a writer's career. ''All of our past writers have gone on to publish additional works,'' says Richard Wentworth of the University of Illinois Press. Winning authors, he says, are also in a better position to get grants from the National Endowment for the Arts and to be ''discovered'' by a commercial house.
One such success was ''In the Land of Dreamy Dreams.'' Untouched by major publishers, it was a first book (a collection of short stories) written by a woman in her 40s, ''all of which militates against commercial publishers looking at it,'' says Miller Williams, Director of the University of Arkansas Press, which did publish the book. Well-received by the critics, the collection is in its fourth printing. Mr. Williams says a commercial house recently paid author Ellen Gilchrist an unspecified five-figure advance for her second book.
The smaller scale of the university presses enables them to take the sorts of risks that commercial publishers can't afford, or aren't willing, to take.
However, these non-profit presses, generally university-subsidized, have their own financial woes. Federal and state funding cuts take their toll. The 1983 budget for the National Endowment for the Arts, which grants funds to several university presses, is expected to be slashed by 30 percent, says Florence Lowe, of the endowment.
But spokesmen for university presses claim that mere budgetary cutbacks are not reason enough for them to close the chapter on publishing valuable new writers of fiction. Charles East, of the University of Georgia Press says, ''Publishing fiction is a labor of love in a losing proposition.''