Experiment: Explain an example of when the “worst case scenario” actually happens — this part is sadly typical though.

If you’d like an introduction to this topic, please check out: Distortion- Chapter 1

I was first introduced to John Hattie by an administrator who was suffering from “workshop disease”. This administrator had just come back from a conference and apparently felt like the problems of the world had been solved. This was after I had completed a significant amount of  statistical training for my Masters Degree. Perhaps if I had experienced this study before that time, I would have been less skeptical. So, let’s talk philosophically about methodology.

One of the reasons that “workshop disease” is a real thing is that, at workshops, there just isn’t enough time to deeply delve into methodology or the reasoning behind research. Presenters cater their research findings to a general audience of teachers and administrators. They are not presenting to their academic peers and this means that the audience probably isn’t very interested in the methods chosen for the study or their justification. In my experience, the researcher’s chosen methods are usually briefly described, perhaps on a single slide, and rarely questioned.

There is a common human trait to not question things that aren’t understood. John Hattie commonly presents this material himself. Anyone asking questions would surely feel less worthy than the pre-eminent scholar himself. Administrators and teachers with little statistical or research training wouldn’t want to stand up and look like fools! Instead, accepting the authority figure’s word as true and accepting the results because of the aspirations of the research is absolutely normal. Who would want to stand up in a lecture about all the clear, tangible, and effective ways proven to help students and be openly skeptical? The skeptic would seem to be advocating for children to stagnate in their academic development or seem to be making excuses for lazy teachers!

So, here’s a brief description of the methods and findings.

For his book Visible Learning for Teachers (2012), John Hattie has used a meta-analysis of smaller scale studies. His first book publishing this material “was based on more than 800 meta-analyses of 50,000 research articles, about 150,000 effect sizes, and about 240 million students . . . A further 100+ meta-analyses completed since . . . have not changed the major messages” (Hattie, 2012, p. 2).

A first criticism is that these numbers are impossible, at worst, and useless at best. The amount of time it would take even a team of researchers to read 50,000+ articles is astounding. Even if Hattie’s research team was able to read through all, it probably means that the articles were not carefully read. In Visible Learning for Teachers (2012), Hattie discusses the meta-studies that were used in his own analysis. So, the 50,000+ articles were not individually read, only the 800 meta-analyses. This means this work is not just a secondary analysis, but a tertiary analysis. The individual articles’ limitations and proof are probably simplified so much that they may lose all connection with the original authors’ intent.

The findings are summarized below in a chart from Teachers’ Toolbox 

I won’t dispute or discuss any of this here, but generally, the analysis demonstrates that feedback has a huge effect size and something like finances/money has a small effect size. Stay tuned for: Effect size!

Hattie, John. (2012). Visible Learning for Teachers. Routledge. Retrieved 6 January 2016, from <http://www.myilibrary.com?ID=363783>