Interval measures – Part 5 of "Ew, I just stepped in a Heartland study!"

Interval measures – Part 5 of  "Ew, I just stepped in a Heartland study!"
on

This is part 4 in a series on the Heartland Institute’s supposedly rigorous study (pdf) on the state of global warming science. This flawed paper has been distributed to 10,000 Utahns by the Utah-based Sutherland Institute, a “sister” of the Heartland Institute.

Paul T. Mero, the president of the Sutherland Institute claims that, “for skeptics, we went out of our way to include a special analysis of the methodology used to create this study. This report is an honest reflection of the international scientific community…”

Let’s see how that holds up.

Flaw #4: Heartland’s study makes incorrect assumptions about levels of agreement and disagreement

Probably the most mind-numbing courses I’ve taken are in the area of measurement theory and levels of measurement, so I’ll try to keep this short and sweet. The Heartland Institute study asks respondents to record their responses on a scale of 1 to 7 – 1 being “strongly agree” with 7 being “strongly disagree.” After the data was collected, the “researchers” then interpreted scores of 1, 2 and 3 to mean “agree,” scores of 4, 5 and 6 to mean “disagree” and a score of 4 to mean “uncertain.”

The problem here is that you are making a leap of faith that this is what your respondents meant when choosing any given number. Marking a 5 in one respondents mind might mean something very different in the minds of another respondent.

This is why a well done survey is usually uses a scale of measurement that looks something like this: “Strongly Agree — Agree — Somewhat Agree – Somewhat Disagree – Disagree — Strongly Disagree — Uncertain.” A scale like this is simple and leaves almost no interpretation of what the respondent actually means.

The Heartland Institute’s use of arbitrary measurement levels leaves its claims of agreement and disagreement up to a certain amount of interpretation – something any rigorous research tries to avoid as much as possible.

Interval measures – Part 5 of  "Ew, I just stepped in a Heartland study!"

Kevin is a contributor and strategic adviser to DeSmogBlog.

He runs the digital marketing agency Spake Media House. Named a “Green Hero” by Rolling Stone Magazine and one of the “Top 50 Tweeters” on climate change and environment issues, Kevin has appeared in major news media outlets around the world for his work on digital campaigning.

Kevin has been involved in the public policy arena in both the United States and Canada for more than a decade. For five years he was the managing editor of DeSmogBlog.com. In this role, Kevin’s research into the “climate denial industry” and the right-wing think tank networks was featured in news media articles around the world. He is most well known for his ground-breaking research into David and Charles Koch’s massive financial investments in the Republican and tea party networks.

Kevin is the first person to be designated a “Certified Expert” on the political and community organizing platform NationBuilder.

Prior to DeSmogBlog, Kevin worked in various political and government roles. He was Senior Advisor to the Minister of State for Multiculturalism and a Special Assistant to the Minister of State for Asia Pacific, Foreign Affairs for the Government of Canada. Kevin also worked in various roles in the British Columbia provincial government in the Office of the Premier and the Ministry of Health.

In 2008 Kevin co-founded a groundbreaking new online election tool called Vote for Environment which was later nominated for a World Summit Award in recognition of the world’s best e-Content and innovative ICT applications.

Kevin moved to Washington, DC in 2010 where he worked for two years as the Director of Online Strategy for Greenpeace USA and has since returned to his hometown of Vancouver, Canada.

Related Posts

on

The Vermont senator nevertheless supported final passage of the Inflation Reduction Act, calling it "a step forward" on climate and drug prices.

The Vermont senator nevertheless supported final passage of the Inflation Reduction Act, calling it "a step forward" on climate and drug prices.
on

Campaigners say action on climate change is in jeopardy if MPs such as climate science denier John Redwood are granted key posts.

Campaigners say action on climate change is in jeopardy if MPs such as climate science denier John Redwood are granted key posts.
on

British Airways owner IAG led the list of buyers last year, analysis shows.

British Airways owner IAG led the list of buyers last year, analysis shows.
on

The protesters attempted to draw attention to the urgent need for large-scale climate action and pressure Democrat legislators to act.

The protesters attempted to draw attention to the urgent need for large-scale climate action and pressure Democrat legislators to act.