IS all this criticism being heaped on the head of Virginia's governor, L. Douglas Wilder, what it seems to be?Is it that Governor Wilder talks too loosely and irresponsibly and that, as some of his critics allege, he is a bit zany? Or could the source of much of this criticism be Democrats who feel he has committed two unforgivable acts: He has remained unremittingly conservative; and he has taken on a party favorite, Sen. Charles S. Robb, and in the process probably destroyed Senator Robb's presidential prospects. I asked Wilder at a Monitor breakfast the other morning whether this criticism he was getting, and his emergence as a controversial figure, was directly related to him, as I put it, "out-conservativing the Republicans" with his heavy emphasis on fiscal responsibility and cutting government spending. He said this might well be so. Could it be that the highest black elected official in the land is up against the same opposition that Judge Clarence Thomas faces as he seeks Senate confirmation to the Supreme Court: from those who believe he is too conservative? Could the intensity of the criticism and scrutiny of Wilder be connected to Wilder's presidential aspirations - together with the way he has confronted Robb? Is it part of an effort, planned or simply spontaneous, to derail a man who appears to many Democrats, particularly those in the North, to be outside the mainstream of party thinking? It is true that Wilder with barely 20 months in office is causing quite a stir over whether he is being spied upon by political foes. Stemming from this are charges that Wilder is showing unfounded anxiety and that he is even "paranoid." Yet Robb was testifying that same day before a federal grand jury about a tape recording of a Wilder cellular phone call that had landed in the Robb camp. A mysterious listening device was found in a tree outside the governor's office - which turned out to be nothing but a radio antenna, not a listening device. So while Wilder denies that he believes he is being spied upon, it might be a normal reaction for him to feel unfriendly acts may be going on. But liberals are charging that Wilder's preoccupation with being spied upon and his tendency to make damaging offhand comments are clear signs that he isn't qualified to run for president - a direction he may be heading. Actually, it would seem that Wilder's views on Judge Thomas are complex if not contradictory. In an off-the-cuff statement he suggested that Thomas' Catholicism should be an issue in his confirmation hearings. He pulled back from this comment when he found it being interpreted as an allegation that the Pope might enter into Thomas' decisions. About Thomas, Wilder said, "I think he's qualified and if I'd guess now he would be confirmed." But he stopped short of endorsement, saying he wanted to hear the Senate testimony first. Wilder would like to hear where Thomas stands on abortion. He, himself, seems to be pro-choice. How conservative is Wilder? These breakfast comments: * Asked what ideas he would bring to government if, after Labor Day, he decides to run for president, Wilder said that he would first "stress fiscal responsibility." The president should "have the line-item veto power," he said. "It is one of the best ways to deter spending." * On taxes, he said, "Raising taxes doesn't solve anything. Before I'd raise taxes I'd trim spending and eliminate layers of bureaucracy." * On where he stands on helping blacks, Wilder said: ve had some black politicians say I don't have a black agenda. Why should an agenda for blacks be different from the agenda for the American people?" He added: "There is no such thing as black English. You either speak the English language or you don't. It's a cop-out and admission of defeat when you say that. Blacks learn like everyone else and should be taught like everyone else." Sounding like a presidential candidate, Wilder did accuse President Bush of failing in his promise of cutting the budget. He insists that he himself could get that job done. Reporters I talked to as they filed out after the breakfast were saying what seemed to be fairly obvious: Wilder, with his conservative approach, would be much better positioned in the general election should he get on the ticket than in his quest for the nomination.