How Melville's family background influenced his art; Subversive Genealogy: Politics and Art in Herman Melville, by Michael Paul Rogin. New York: Alfred A. Knopf. 354 pp. $22.95.

Rogin's difficult task in ''Subversive Genealogy'' is to sweep across all the writings of Melville, showing how they reflect the writer's family relations against the backdrop of American history in which they were deeply involved.

Both Melville's grandfathers, Maj. Thomas Melvill and Col. Peter Gansevoort, were heroes of the Revolution. The next two generations of both families also figured importantly in politics and military affairs. Melville's father-in-law was a chief justice of the Commonwealth of Massachussetts.

Melville's family suffered considerable turmoil and misfortune, and the novelist's generation responded to the legacy of revolutionary heroism and subsequent family diminishment with either rebellion or attempts to emulate their patriarchs. Melville himself swayed between both reactions, and much of his writing builds on their tension.

In this book Rogin, a professor of political science at the University of California, Berkeley, attempts to follow up on the work of numerous Melville scholars, who have noted the remarkable parallels between Melville's family and problems in his writings, by gathering their findings in one study, adding to it , and presenting a comprehensive view.

Rogin is at his best in assembling evidence, but he also provides some shrewd readings of Melville's work. His chapter on the novel ''Pierre,'' is especially insightful.

Unfortunately, the attempts of scholars to draw parallels between a writer's life and work tend to lie in the misty landscape of speculation, and while the cautious investigator pursues such studies carefully and tentatively, Rogin has ridden the horse of his thesis boldly into the murk. At times it has shied at some notion and thrown him into the bushes.

He writes, for example, ''The shift of the Melvill clothing business from French fashions to fur hats anticipated the change in Melvill(e) political sympathies from European royalty to American nature.'' One wonders about the value of such observations. Rogin even finds significance in the fact that Colonel Gansevoort captured a brass drum from the British in the Revolution and that, about a century later, in writing ''Billy Budd,'' Melville notes that after Billy's hanging the crew is sent below with the ''drumbeat to quarters.''

Advanced students and specialists can sort out the truly significant from the roughage. Such readers will be ultimately enriched by the book. But the general reader will very likely find himself at sea in its welter of detail, and tenuous speculation. Any reader, though, will come from the book with an enriched sense of the novelist's toughness in dealing with the forces in himself and his times, and an admiration for his depth and sense of honor.