By Jeffrey Mervis

U.S. high school students who regularly handle rocks or minerals in science class did much worse on a recent national science test than those who never engage in such hands-on activities. Students who never mixed chemicals or peered through microscopes in their classrooms did just as well on the test as those who often participated in those activities. 

Surprised? Those eye-catching results, from the 2015 National Assessment of Educational Progress (NAEP) in science released last week, seem to fly in the face of conventional wisdom that hands-on learning is the best way to teach science. Last week, for example, the Obama administration honored the nation's best science and math teachers by staging what it called "Active Learning Day" at the White House.

Government officials refer to the NAEP as the nation's report card. The science test, which periodically measures what a representative sample of U.S. students in grades four, eight, and 12 know about the life, physical, space, and earth sciences and the scientific process, is part of an ongoing assessment of reading, mathematics, civics, and other subjects. Results for the two younger grades are broken out by state, leading to media coverage that often focuses on why a particular state is ahead of or behind its peers.

For those keeping score, U.S. elementary and middle school students overall did a bit better in science in 2015 than the previous NAEP cohort tested in 2009, and the wide gap between the scores of white and minority students in both of those grades narrowed slightly. In contrast, the scores of the country's 12th graders didn't budge, and the racial disparity didn't shrink.

But the NAEP is more than a horse race. It also includes a 23-question survey about what actually happens in the classroom and how students engage with science. For example, one question asks students how frequently — never, rarely, sometimes, or often — they do hands-on activities in seven areas, including chemistry and the earth sciences.

In only one case, involving activities with simple machines, did students who had hands-on experience also have better NAEP scores. To be precise, those who never worked with machines scored 155 (average for all 12th graders was 150), whereas those working with them often scored 171.

Perhaps even more sobering, however, is how few students report engaging in hands-on activities at all. For example, only 29% of students "sometimes" or "often" work with simple machines. Some 58% never do, and 14% do so rarely. The share who say they never or rarely work with "living things" is almost as large, at 62%. Most students are even strangers to such standbys as microscopes and chemicals: Only 17% and 18%, respectively, report using them "often" in science class.

National Center for Education Statistics (NCES) officials say it's not their job to explain the NAEP results, including any counterintuitive findings such as the negative correlation between hands-on activities and scores. But one expert on the test who requested anonymity speculated that asking students what they did "this year" in their science class might skew the results.

For example, high-achieving students often take physics or advanced biology in their senior year to prepare them for an elite college. So they are unlikely to be working with rocks or minerals in grade 12. At the same time, a remedial student might be enrolled in a physical science course to meet distribution requirements for graduation, in which rocks are part of the curriculum. Even so, that explanation fails to address why so few students reported working with microscopes — presumably an essential tool for someone taking advanced biology.

The dismal state of 12th grade science described by the NAEP test takers might be expected to trigger immediate and widespread outrage among science educators — if they knew about them. But the results are almost impossible to ferret out on the website maintained by NCES, which manages the NAEP. The link that invites users to explore the student questionnaires, for instance, provides data on only a few, partial answers by students at each grade level. The actual data are buried in a search tool called Data Explorer that even an experienced researcher might find challenging.

"Some of the menu labels aren't intuitive," admits Grady Wilburn, an NCES statistician in Washington, D.C., who helps crunch the numbers and disseminate the results. "We've learned a lot about how to design web tools" since the current format was adopted a decade ago, Wilburn says. NCES hopes to have a better version available for the 2017 reading and math assessments that will be released in 2018, he adds.

The 2015 science results come with another twist. Every one of the items in a drop-down menu of selected results from the student questionnaire describes an activity that appears to have a positive effect on NAEP scores.

For example, the menu contains the answer to how often students "identify questions that can be addressed through experiments" in class. But that topic is only one of a seven-part question on their familiarity with scientific methodology. Half of all students say they identify testable hypotheses at least once a week, and that group scored significantly higher on the test than their peers who said never or rarely. In contrast, none of the data presented by using the drop-down menu show a negative correlation between hands-on activities and NAEP scores.

The same thing is true for a question about searching the internet for science content. Those who never go online earned an average score of 144, whereas students who surf the web two or three times a week averaged 165 on the test. But the menu ignores the finding that students who rarely read books or magazines about science score higher on the NAEP than those who are voracious readers.

NCES officials say they select which questions to feature based on what they think would most interest the public, and from topics currently in the news. But the choices seem to dovetail with the adage that politicians prefer to deliver good news.

Secretary of Education John King and presidential science adviser John Holdren were quick to tout the new results as evidence that the Obama administration's efforts to bolster science education, including training more and better science teachers and the increased emphasis on active learning, are beginning to pay off. NCES Acting Commissioner Peggy Carr in Washington, D.C., offered an even rosier assessment, telling reporters that the results show "we're kind of at the top when it comes to science."

Some might question that take, given the middle-of-the-pack ranking by U.S. students in international tests such as the Program for International Student Assessment taken by 15-year-olds in nearly 80 countries and jurisdictions, plus the fact that 40% of U.S. seniors taking the NAEP test failed to demonstrate even a "basic" knowledge of science.

The 40% figure is "appalling," says David Evans, who leads the National Science Teachers Association based in Arlington, Virginia. "I think people need to know a lot more than ‘basic' to function in today's world. I think ‘proficient' is where we need to be," he adds, noting that only 22% of high school seniors are now at that level.

It's also noteworthy that the 2015 cohort of seniors are the first NAEP science test takers to have spent their entire school careers under the 2002 education law known as No Child Left Behind (NCLB). It required annual testing of reading and math for students in grades three through eight. But many U.S. educators say NCLB gave short shrift to science and all other subjects because those topics did not factor into how the law called for assessing school performance. "We're seeing the results of a long time in which we just didn't focus on science," Evans says. A new law enacted last year retains the annual testing but puts states, not the federal government, in the driver's seat for improving student performance.

Holdren also isn't surprised that the 12th grade scores didn't go up, but for different reasons. NAEP scores reflect many factors, he told reporters in a conference call on the NAEP results, including teacher quality, family support, and a student's overall interest in the subject. "We've also known for a long time that it's important to reach kids early," he said, "in a way that makes science exciting and puts it in the context of solving important societal problems."