This is part 4 in a series on the Heartland Institute’s supposedly rigorous study (pdf) on the state of global warming science. This flawed paper has been distributed to 10,000 Utahns by the Utah-based Sutherland Institute, a “sister” of the Heartland Institute.

Paul T. Mero, the president of the Sutherland Institute claims that, “for skeptics, we went out of our way to include a special analysis of the methodology used to create this study. This report is an honest reflection of the international scientific community…”

Let’s see how that holds up.

Flaw #4: Heartland’s study makes incorrect assumptions about levels of agreement and disagreement

Probably the most mind-numbing courses I’ve taken are in the area of measurement theory and levels of measurement, so I’ll try to keep this short and sweet. The Heartland Institute study asks respondents to record their responses on a scale of 1 to 7 – 1 being “strongly agree” with 7 being “strongly disagree.” After the data was collected, the “researchers” then interpreted scores of 1, 2 and 3 to mean “agree,” scores of 4, 5 and 6 to mean “disagree” and a score of 4 to mean “uncertain.”

The problem here is that you are making a leap of faith that this is what your respondents meant when choosing any given number. Marking a 5 in one respondents mind might mean something very different in the minds of another respondent.

This is why a well done survey is usually uses a scale of measurement that looks something like this: “Strongly Agree — Agree — Somewhat Agree – Somewhat Disagree – Disagree — Strongly Disagree — Uncertain.” A scale like this is simple and leaves almost no interpretation of what the respondent actually means.

The Heartland Institute’s use of arbitrary measurement levels leaves its claims of agreement and disagreement up to a certain amount of interpretation – something any rigorous research tries to avoid as much as possible.