A 19th-century Hamlet -- and played by a woman; Hamlet Tragedy by William Shakespeare. Directed by Joseph Papp.

Boldness and convention jostle each other in Joseph Papp's version of ''Hamlet,'' at the Public/Anspacher Theater. The boldness consists of casting Diane Venora in the title role. Convention is observed in a straightforward handling of the text, with the action moved up to the 19th century. The overall result is not clarity but confusion.

Women Hamlets are a rarity, if not a novelty. Distaff Danes have included Sara Siddons and Kitty Clive in England, Sarah Bernhardt in France, and Charlotte Cushman, Eva Le Gallienne, and Judith Anderson in the United States. Director Papp's contribution to unorthodoxy may consist principally of entrusting so monumental a role to an inexperienced and technically limited actress.

There are other problems with this ''Hamlet.'' They begin to be evident when Horatio (James Cromwell) appears on the scene. Bespectacled, his beard tinted with gray, Mr. Cromwell's avuncular, Chekhovian Horatio suggests a middle-aged professor at Hamlet's prep school. If these friends ever attended university together, Hamlet must have been a child prodigy.

The prince's jaunty boyishness - contrasted with the appearance and behavior of his supposed contemporaries - creates an imbalance that distorts all of this Hamlet's relationships. The diminutive Miss Venora attacks the role with a powerful (though not particularly melodious) voice, a lithe athleticism, and an alienating manner. She does not so much play with her onstage colleagues as play at them. Partly as a result, the enterprise lacks coherence.

In the course of his emotional scenes with the Queen (Kathleen Widdoes) and Ophelia (Pippa Pearthree), Hamlet wrestles each woman to the ground. Pale-faced and morose, arrogant and aggressive, he is yet vulnerable enough to break down sobbing on some of those occasions that inform against him. Perhaps Mr. Papp and Miss Venora view him as a hysteric. In any case, it is difficult to visualize this Dane as ever having been ''the glass of fashion and the mould of form, the observed of all observers.''

A good deal of the performance is marked by a conversational delivery that seems to be part of the director's design to contemporize the tragedy. But like so much else, the results baffle the understanding and at times boggle the mind.

With Hamlet offstage, the self-respecting company does creditably with the four-hour version at the Public/Anspacher. Bob Gunton presents Claudius as a shallow, overweening second-rater whose fratricidal conspiracies ultimately overwhelm him. As his consort, Miss Widdoes queens it in a somewhat tentative manner while keeping a wary eye on the hapless Ophelia. George Hall doubles handily as a fussbudget Polonius and a verbally nimble Gravedigger.

George Hamlin is commandingly majestical and spectral as the Ghost of King Hamlet and makes a further contribution as the Player King. The very large cast - a number of whose members play more than one role - includes Robert Westenberg (Laertes), Rich Lieberman and Ralph Byers (Rosencrantz and Guildenstern), Rocco Sisto (Osric), and J.T. Walsh, Stephen McNaughton, Jamey Sheridan, and Raphael Sbarge.

The production, with its many movables, was designed by Robert Yodice (scenery), Theoni V. Aldredge (costumes), and Ralph K. Holmes (lighting). B. H. Barry staged the fight sequences and Allen Shawn composed the martial flourishes and other incidental music.