Pyschoanalysis: a telling critique; The Perils of the Profession: In the Freud Archives, by Janet Malcolm. Alfred A. Knopf. 165 pp. $11.95

''In the Freud Archives'' is Janet Malcolm's second book about Freudian psychoanalysis. In her previous book, ''Psychoanalysis: The Impossible Profession,'' Malcolm recalled the heyday of the discipline, the 1950s, when with unlimited optimism analysis reached out to every kind of human disorder. In the 1960s government-funded clinics made analysis accessible to almost everyone, and analysts talked about improving the health of society as a whole.

But in the '70s both the money and the optimism ran out. Today analysts acknowledge that at least as many treatments fail as succeed, and success has been more modestly defined. Most analysts won't treat the severely ill, and most prefer educated patients to uneducated ones. The average analysis takes six to eight years, 11 months a year, five days a week. Many analysts have begun to rely on antidepressant drugs to effect the ''cure'' their method promises. And other, less severe kinds of therapy have eroded the supremacy of Freudian analysis.

Having devastated the profession in her last book, Malcolm now proceeds to take on the analysts themselves. ''In the Freud Archives'' is a fascinating account of intrigue and imbroglio within the secretive psychoanalytical establishment.

The story begins in 1974 when Jeffrey Masson, a 33-year-old Sanskrit scholar who had decided in his brash way to become a psychoanalyst, met Kurt Eissler, secretary of the Freud Archives and one of the most respected psychoanalysts in the world. Masson's wit and energy captivated the diffident Eissler, who soon came to love the younger man as a son. He showered Masson with favors, the biggest of which was to persuade Anna, Freud's daughter, to show Masson some unpublished letters that her father wrote to his friend Fliess in 1897. Eissler would soon regret his generosity.

1897 was a breakthrough year for Freud. Up to that point he had believed his neurotic patients when they told him, as most did, that their parents had seduced them as children. His ''seduction theory'' simply stated that parental abuse is at the root of most forms of hysteria. But in 1897 Freud began to suspect that these tales of incest were fantasies, not real experiences. His search for an explanation led him away from the social reality of parental cruelty, toward the ''psychic reality'' of the unconscious mind. The Oedipus complex soon followed, and psychoanalysis was born.

One of the 1897 letters suggested to Masson that Freud had it right the first time, and that he knew it. His suppression of the seduction theory was in fact, said Masson, a cowardly and dishonest retreat from the miseries of the real world, and the main reason that present-day psychoanalysis is so ineffectual. Masson turned his ideas into a book, ''The Assault on Truth: Freud's Suppression of the Seduction Theory,'' published amid a blaze of controversy last winter.

True, Freud's reasons for discarding the seduction theory were less than scientific. But then the unconscious was never a scientific discovery in the sense that, say, DNA, the genetic building block, was. Freud used metaphors like ''psychic reality'' and ''dream-life'' that his followers have taken literally, and he made mistakes - they're plentiful in the Fliess letters - that his idolaters, like Eissler, have tried to cover up. If the Freudian establishment weren't so inflexible, its antagonists, like Masson, wouldn't be so hyperbolic.

But it remains to be seen whether classical Freudian analysis, in the hands of such obstinate, eccentric practitioners, will survive. All analysts are in analysis themselves (that was one of Freud's commandments), and many of them are analyzed by doctors who are their superiors within the psychoanalytic hierarchy. The fears and desires that are unleashed in analysis get confused with professional relationships outside of analysis, to the point where the analysts start acting crazier than the people they're supposed to cure. ''In the Freud Archives'' brilliantly and vividly illustrates the perils of the profession.