September 17, 2007

FOCUS ON ISSUES & DECISION MAKING: Music Education Research 101, Part 1

Why is it that research studies that seem so strong, so convincing, don’t always help us make our case for music education programs?

As advocates, we need to use every tool at our disposal to defend our programs, including the resources on this website. However, arguments that convince one group of the value of music may not affect another group. This is why our advocacy arsenal must include valid research that speaks to many different perspectives, including financial, philosophical and psychological.

This two-part series on music education research will help you correctly identify and interpret research studies about music and academic achievement. You’ll also see how to avoid common pitfalls that inadvertently weaken your case, such as overstating or inaccurately presenting research results, incorrectly using research language, or inadvertently misrepresenting research findings.

Research, Public Opinion and Policy

In our media, we are bombarded by research; polls and statistics have become an important part of how we process information. Most of us are well aware of how frequently––and often successfully––advertisers, market researchers and public relations firms use research to sell products.

But, research isn’t only a sales and marketing tool; it is often used––and sometimes misused––to sway public opinion and help create educational policy. During the 1980s, the release of the report, A Nation at Risk, led to increased attention on “the basics” of education (limited only to math, science, reading). In the 1990s, research on adolescents’ needs prompted the shift from junior high to middle schools. More recently, low achievement in some U.S. public school system sectors, revealed by the National Assessment of Educational Progress (NAEP), preceded the enactment of the federal No Child Left Behind Act of 2001 (NCLB)  and the results are felt daily in public schools nationwide.

Music education advocates are relative newcomers to evaluating the merits of research studies and then using the results to advance the case for a complete music education for every child. In the late 1970s and early 1980s, the beginnings of the music education advocacy movement (sparked by budget recessions, property tax relief, etc.) helped push interest in music education research outside the university. Advocates began to understand that research might be an effective means to sustain endangered music education programs.

A Beginner’s Guide to Research Language

Research terminology can be confusing unless you understand the meaning and usage of words within a research setting. Once you understand a few basics, you’ll more easily recognize bogus research and see the potential difficulties of using research to justify inclusion of music programs in your district’s curriculum.

Most fields of research have strict ethics and standard procedures for reviewing and publishing research results. Research journals have editorial boards that screen research before it is published, in order to ensure that it meets rigorous standards. The Journal of Research in Music Education  is an example of such a publication in our field. Please be wary of self-reported research, or research from non-refereed journals; it may not meet professional standards.

Prove––Researchers generally frown on this word and prefer to use demonstrate or support. Why? Research results are reported according to a scale of probability or confidence, which is calculated using a variety of statistical tools. It’s not really possible to calculate research results exactly (you generally can’t survey every person from a certain population of people; you have to be selective). Researchers use statistical measurements to estimate and determine the validity and reliability (see below) of their research, and tend to avoid words that indicate absolutes.

Accuracy––Accuracy means results were calculated correctly, and researchers didn’t make any mistakes in the mathematics of the report.

Correlation and Causation––The difference between these two terms is often misunderstood. If results demonstrate a high correlation between two events, it means there is some type of relationship between the two but it doesn’t necessarily mean that one item is affecting the other. Additional tests must be conducted to determine what type of relationship there is, or whether there is a cause-and-effect relationship (causation). Correlation doesn’t mean that there is causation.

Generalizability––Can the research results be applied to other people or populations? For example, do things that cause disease in mice also cause disease in people? Researchers are interested in generalizability because the results may have greater meaning for more people.

Reliability––This means results are consistent and repeatable or, if the same research were conducted again, the same results would occur. As with validity, there are statistical formulas and minimum standards for research to be considered reliable.

Significant––This term deals with statistical measurements, rather than practical importance. Reliable research findings usually demonstrate statistical significance of .01 or better, meaning the results will be true in 99 of 100 cases.

Substantial––This term refers to the size of the effect. To a layperson, there may be a substantial body of research, but this doesn’t necessarily relate to whether or not the size of the effect was substantial.

Validity––If research is valid, it actually measured what researchers intended to measure. Researchers may make incorrect claims (intentionally or unintentionally) about certain aspects of their research, but there are statistical formulas to determine if research meets the minimum standards of validity.

In Part 2, I'll address how to ask the right questions to identify the useful, valid research you need to help you defend and strengthen music education programs in your district.

Stephen Benham is an Associate Professor of Music Education at Duquesne University in Pittsburgh, Pennsylvania. He received his B.S. from the University of Minnesota, his M.M. from the University of Michigan, and his Ph.D. from the University of Rochester (Eastman School of Music). His prior experience includes thirteen years as a public school string teacher in Oregon, Michigan and New York.