In the area of higher mathematics, where problems emphasize reasoning skills more than computational skills, boys are outperforming girls in the United States. When variables such as the number of math courses students have taken, verbal ability and general intelligence, and enjoyment of math are equal, test results still favor boys.

One study of talented seventh- and eighth-graders, conducted at John Hopkins University suggested ''a large sex difference in mathematical ability'' (the smallest mean difference in SAT math scores was 32 points), and argued that all students in its study had received ''essentially identical formal instruction in mathematics.''

But apart from actual curriculum choices, how ''essentially identical'' was the instruction of the students tested? And how identical are experiences in math instruction generally for males and females in this country?

A number of researchers, among them Elizabeth Fennema of the education department at the University of Wisconsin, have begun to find out. At a Washington meeting of the American Association for the Advancement of Science in early January, Ms. Fennema presented the following conclusions, based on classroom observations over a three-year period:

* Confidence in learning mathematics is highly correlated to achievement in mathematics. Boys are significantly more confident in their ability to deal with mathematics than are girls.

* From a young age, girls are encouraged to exhibit more dependent behavior than are boys, a factor that discourages the kind of independent, autonomous problem-solving required in higher mathematics.

* As early as sixth grade, girls begin to see mathematics as having little or no usefulness for them in the future. In addition, they claim that parents, teachers, and counselors are not positive toward them as learners of mathematics.

* While most overt behavior by teachers appears to be nonsexist and fair to most students, teachers interact in both positive and negative ways more often with boys than with girls.

* Where sex-related differences in mathematics are found, they are very school specific. In some schools no differences are found; in others, differences are consistently found.

These observations encouraged Ms. Fennema to develop an intervention program for schools to change affective behavior - feelings, beliefs, and attitudes -- that she believes are influencing the test scores of even very bright students.

Contrary to widespread assumptions that schools tend to reinforce sexual stereotyping, she declares, ''as loudly and as emphatically as I can,'' that schools can increase female interest in and learning of mathematics.

The program developed by Ms. Fennema and her colleagues, called ''Multiplying Options and Substracting Bias,'' is designed to change the educational environment operating upon adolescent girls, since merely asking girls to change their behavior places an unreasonable burden on their shoulders.

The program consists of four 2-hour workshops, one each for students, teachers, counselors, and parents, and four videotapes, each designed for its target audience, each making its audience aware of the behavioral and environmental factors that influence female math performance. Nine Midwestern schools, consisting of five experimental schools and four control schools, randomly assigned, beginning in the spring and fall of 1978, provided the data to test the effectiveness of the program.

The results were heartening. Both male and female students in the program increased their plans to study mathematics during and after high school, while males no longer tended to perceive mathematics as a ''male domain.''

Actual female enrollment in junior and senior year mathematics courses increased. Females tended less often to attribute their failures to a lack of ability.

A second intervention program, also described at the AAAS conference in January, developed and carried out by the San Francisco Bay Area Network for Women in Science (now called the Math/Science Network), consists of one-day conferences for junior and senior high school girls, designed to encourage girls' participation in math studies and careers in science and technology. Each conference includes opportunities for discussion with women working in math- and science-related fields.

An evaluation of pre and post conference questionnaires concluded that participants significantly increased their participation in such courses.

Just how significant programs like these might be in raising actual mathematics test scores for women is thus far unknown.

The Fennema research suggests that female performance in mathematics is influenced by subtle and not so subtle factors in the educational environment as a whole, an environment capable of ''consciousness raising'' through intervention workshop programs.

Meanwhile, the failure of females to achieve their full potential in mathematics is, according to Ms. Fennema, ''one of the most serious inequities in education that currently exists. Without mathematics knowledge and skills, women will never be able to achieve equity in society.''