Boston — Since the 1930s, the United States has been a world leader on the arcane frontier of science known as particle physics. Over the years, American scientists have recorded a number of breakthroughs, including invention of the original atom-smashing cyclotron and discovery of a number of the elementary particles.
But now, buffeted by erratic swings in research-and-development funds over the last decade, that leadership is being challenged by new programs in Japan, the Soviet Union, and, above all, Europe.
This is creating a reverse-direction ''brain drain'' as American scientific talent increasingly heads overseas to perform experiments for which US facilities are no longer adequate. The American scientific community, concerned over such trends, is scrambling to reappraise its entire approach to high-energy physics research.
A conference being held at Cornell University this week is focusing on the potential designs for ''next generation'' particle accelerators. There, scientists and government officials will discuss the possibility of constructing facilities that could help US physicists recapture their lost research momentum by the early 1990s.
And in June, a subcommittee of the Department of Energy's High Energy Physics Advisory Panel (HEPAP) will meet in Woods Hole, Mass. Its official charge is to make recommendations on ''scientific requirements and opportunities for a US high energy physics program in the next 5 to 10 years.''
Observers say that US particle physicists have already reached a consensus on proposing the construction of a single laboratory with a $1 billion accelerator 40 times as powerful as any yet constructed. The US now has several major accelerator laboratories. The willingness of normally competitive US particle physicists to pool their resources in a single national machine shows how seriously they take the overseas challenge.
Particle accelerators slam atomic fragments such as positively charged protons or negatively charged neutrons into one another at nearly the speed of light. The collision of two such particles yields a potpourri of smaller fragments which scientists study to find out what an atom is made of and what holds it together. Generally, the higher the energy of the proton or electron beam, the more interesting and significant the collisions and the deeper one can pry into the atom's mysteries.
At the 13-nation European Center for Nuclear Research at Geneva (CERN), a giant accelerator is under construction that will send protons zipping through a ring-shaped underground tunnel 16 miles in circumference. The potential of this new machine, called the large electron positron storage ring (LEP), to open a new range of physics research is a main cause of American researchers' concern.
''When the LEP comes on line in 1987, that will be the next-generation machine, and the transfer of supremacy in particle physics to Europe will be complete,'' explains Sheldon Glashow, a Harvard University Nobel Prize winner. Leon Lederman, director of the US Fermi National Accelerator Laboratory in Batavia, Ill., agrees. He adds that ''if the focus of particle physics continues to shift to Europe, the subject could end in the US.''
A US accelerator now under construction - the colliding beam accelerator (CBA) at the Brookhaven National Laboratory on Long Island, N.Y. - was to have shored up the American position. But since its authorization in 1977, the CBA has fallen four years behind its construction schedule. Also, its present projected cost of more than $600 million is double the original estimate.
Thus a number of US scientists question whether completing CBA would be the wisest use of limited funds. ''A 1984 CBA, even a 1986 CBA, would be all right, '' says Dr. Glashow. ''But a 1988 date is too long.''
For one thing, the machine would not really match the European competition. CERN has already captured one of the main prizes in the energy range of the CBA. This is the so-called ''W'' particle, whose existence suggests an underlying unity between the electromagnetic force and the ''weak'' force involved in some forms of radioactive decay. CERN announced discovery of the ''W'' particle last December. CERN researchers expect to find a companion particle, called a ''Z'' particle, within the next few months.
Although Brookhaven's CBA still could carry out much useful research, many US physicists now think its potential payoff is not worth its cost. There is strong pressure to jump ahead with a much more powerful machine.
''This is it on the CBA,'' said one Energy Department official closely connected with the HEPAP proceedings. ''Without a clear, positive recommendation to proceed, CBA will be shut down.''