Interval measures – Part 5 of "Ew, I just stepped in a Heartland study!"

Interval measures – Part 5 of  "Ew, I just stepped in a Heartland study!"
on

This is part 4 in a series on the Heartland Institute’s supposedly rigorous study (pdf) on the state of global warming science. This flawed paper has been distributed to 10,000 Utahns by the Utah-based Sutherland Institute, a “sister” of the Heartland Institute.

Paul T. Mero, the president of the Sutherland Institute claims that, “for skeptics, we went out of our way to include a special analysis of the methodology used to create this study. This report is an honest reflection of the international scientific community…”

Let’s see how that holds up.

Flaw #4: Heartland’s study makes incorrect assumptions about levels of agreement and disagreement

Probably the most mind-numbing courses I’ve taken are in the area of measurement theory and levels of measurement, so I’ll try to keep this short and sweet. The Heartland Institute study asks respondents to record their responses on a scale of 1 to 7 – 1 being “strongly agree” with 7 being “strongly disagree.” After the data was collected, the “researchers” then interpreted scores of 1, 2 and 3 to mean “agree,” scores of 4, 5 and 6 to mean “disagree” and a score of 4 to mean “uncertain.”

The problem here is that you are making a leap of faith that this is what your respondents meant when choosing any given number. Marking a 5 in one respondents mind might mean something very different in the minds of another respondent.

This is why a well done survey is usually uses a scale of measurement that looks something like this: “Strongly Agree — Agree — Somewhat Agree – Somewhat Disagree – Disagree — Strongly Disagree — Uncertain.” A scale like this is simple and leaves almost no interpretation of what the respondent actually means.

The Heartland Institute’s use of arbitrary measurement levels leaves its claims of agreement and disagreement up to a certain amount of interpretation – something any rigorous research tries to avoid as much as possible.

Interval measures – Part 5 of  "Ew, I just stepped in a Heartland study!"

Kevin is a contributor and strategic adviser to DeSmogBlog.

He runs the digital marketing agency Spake Media House. Named a “Green Hero” by Rolling Stone Magazine and one of the “Top 50 Tweeters” on climate change and environment issues, Kevin has appeared in major news media outlets around the world for his work on digital campaigning.

Kevin has been involved in the public policy arena in both the United States and Canada for more than a decade. For five years he was the managing editor of DeSmogBlog.com. In this role, Kevin’s research into the “climate denial industry” and the right-wing think tank networks was featured in news media articles around the world. He is most well known for his ground-breaking research into David and Charles Koch’s massive financial investments in the Republican and tea party networks.

Kevin is the first person to be designated a “Certified Expert” on the political and community organizing platform NationBuilder.

Prior to DeSmogBlog, Kevin worked in various political and government roles. He was Senior Advisor to the Minister of State for Multiculturalism and a Special Assistant to the Minister of State for Asia Pacific, Foreign Affairs for the Government of Canada. Kevin also worked in various roles in the British Columbia provincial government in the Office of the Premier and the Ministry of Health.

In 2008 Kevin co-founded a groundbreaking new online election tool called Vote for Environment which was later nominated for a World Summit Award in recognition of the world’s best e-Content and innovative ICT applications.

Kevin moved to Washington, DC in 2010 where he worked for two years as the Director of Online Strategy for Greenpeace USA and has since returned to his hometown of Vancouver, Canada.

Related Posts

on

Some residents of Dimock, Pennsylvania have not had clean water for 14 years because of Cabot Oil & Gas’ fracking operations. A “historic” plea caps off the landmark fracking case and will result in the restoration of clean drinking water.

Some residents of Dimock, Pennsylvania have not had clean water for 14 years because of Cabot Oil & Gas’ fracking operations. A “historic” plea caps off the landmark fracking case and will result in the restoration of clean drinking water.
on

"Lack of regulation makes for a Wild West of sustainable fund management", Edward Lander from Ethical Consumer said.

"Lack of regulation makes for a Wild West of sustainable fund management", Edward Lander from Ethical Consumer said.
on

Leaders in the state’s environmental justice movement see signs their pressure is bringing accountability for both government and polluters.

Leaders in the state’s environmental justice movement see signs their pressure is bringing accountability for both government and polluters.
Opinion

This bold new mechanism is what’s needed to end fossil fuel-induced violence against women, natural resources, and the climate, argue two African peace activists.

This bold new mechanism is what’s needed to end fossil fuel-induced violence against women, natural resources, and the climate, argue two African peace activists.