The challenge of reconciling our mental models with the material physical universe: Top-down and bottom-up approaches

A recurring theme that you'll find at POEM is how the practice of science is defined, in large measure, by its central value of seeking to avoid bias and by a collection of methods designed to assist scientists in avoiding bias when interpreting research results.

Even more than other methods for avoiding cognitive and logical traps, statistical measures are some of the most rigorous tools scientists have for providing clear frameworks for interpreting what the data from empirical observations and experiments actually mean.

To lay the foundation for discussing statistics in evaluating massage research, let's first talk about different approaches to the challenge of reconciling our mental models with the material physical world.

Data, information, facts, and truth

Data is a collection of factual information used as the basis for reasoning, discussion, or calculation. When a scientist talks about a fact that is rooted in research, they are referring to a piece of information that is being presented as objective reality.

Because that information is a fact, a scientist will often say "It is true that..." and then go on to state whatever that particular fact means.

It is easy for a casual listener to believe the scientist must be referring to absolute “Truth”, because of the way these words are commonly used in everyday conversation.

For example, the media may cover scientific topics in a way that implies that science points directly to “Truth” in the same way the term is used in philosophy or meaning-making and self-expression.

But this is not a faithful representation, because science—which deals only with aspects of the natural material physical universe—takes for granted that the measurement of things observed in the natural world contains a certain amount of error. By "error", we mean the Merriam-Webster dictionary meaning of "a variation in measurement, calculation, or observation due to mistakes or uncontrollable factors".

As we will discuss in Chapter 4 of the research literacy e-Book, it is impossible to observe or measure reality from a completely 100% neutral position, and there are no perfect measurement tools.

For this reason, scientists emphasize working in a way to obtain the best results possible, knowing that no observations of reality can be completely error-free.

There can be no achievement of absolute truth, just--if the process is carried out with integrity--getting closer and closer to what the facts are.

In order to work toward this goal, scientists have developed methods for managing observational errors, because those errors can be understood and controlled by making skillful choices about experimental design and statistical techniques.

The Semantic Triangle, introduced in Chapter 2 of the research literacy e-Book here at POEM and available later this month, shows how the elements of meaning can be divided among concepts (the meanings people attach to ideas), terms (the language used to describe ideas), and referents (the things in the natural world to which terms and concepts refer).

Source: http://sig.biostr.washington.edu/~raven/semantic-triangle.jpg accessed 2 May 2012

The big question is how to know—given that perceptions and experience vary so much from one person to another—that those concepts and terms in our minds really connect to the referents they claim to represent.

Sorting out how best to connect those internal aspects of meaning to the external physical world is an ongoing problem that challenges all of us.

Top-down vs. bottom-up approaches to data

One approach that has been taken throughout history is to decide in advance what the “truth” is, and then to look for empirically observed facts that will reinforce that “truth.”

This is known as the top-down approach, in which a researcher starts with a desired answer in mind and then fits the questions and the data into that answer.

Obviously, this approach implies a great deal of bias from the start.

Ptolemy, a Greek astronomer who lived in Egypt during the first and second centuries CE/AD, developed a model showing the sun and the planets in a circular orbit around the Earth. This model depicted the Earth at the center of everything, or geocentrism: a view that seemed at first to fit with what people observed when they looked up at the sky.

Source: http://upload.wikimedia.org/wikipedia/commons/7/7b/Bartolomeu_Velho_1568.jpg accessed 1 May 2012

But some careful observers noted that a planet such as Mars would sometimes be seen moving in its normal direction, but then it would come to a stop and begin to move in the opposite direction—backward across the sky—before returning to its expected path. It seemed to move in a retrograde way.

Source: http://upload.wikimedia.org/wikipedia/commons/6/6a/Retrograde_Motion.bjb.svg accessed 1 May 2012.

The left side of the drawing shows the Earth's actual motion around the sun in the blue points 1-5. Mars' actual motion around the sun is shown by the red points on the left of the diagram, and the right side of the diagram shows what Mars' motion looks like to an observer on the Earth. So there is no such thing as Mars (or Mercury, for that matter) in retrograde; it's actually an illusion produced by our motion relative to the other planet around the sun.

To reconcile this observation with the idea of the planets and sun making simple circles around the Earth, advocates for Ptolemaic astronomy used the concept of epicycles, or loops, that represented the additional movements of the planets. Epicycles were explained as looping paths that averaged out to simple circles. In the expanded Ptolemaic system, the planets and sun were continually looping around given points, which were themselves moving in simple perfect circles around the earth.

Source: http://upload.wikimedia.org/wikipedia/commons/2/29/Ptolemaic_elements.svg accessed 1 May 2012.

As in the previous image, Mars is shown in red, and Earth in blue. This is the model of epicycles introduced to account for what looked to observers on Earth to be retrograde motion.

Because of the observed referent (occasional apparent or seeming reversals in movement of the planets), it was necessary to add this new term and concept (epicycles) in order to hold onto and protect the Ptolemaic idea that something was moving in perfect circles around the earth. The advocates of Ptolemaic astronomy kept adding epicycles as necessary to force the model to fit the observations.

And for a very long time, despite the hacks and cobbled-together epicycle justifications, the Ptolemaic model continued to have a great influence on astronomy’s view of the Earth’s place in the universe, because there was not much change in the data available to observers.

But over time, new observational instruments such as telescopes were invented, and these made it possible to add new information to the accumulated body of knowledge about the sky.

Eventually, a tipping point was reached, and the weight of evidence made it clear that Ptolemy’s model of the universe no longer matched the observed facts.

A newer explanation, called the heliocentric model, was developed by Polish astronomer Nicolaus Copernicus (1473-1543) in which all the planets, including Earth, orbited around the sun.

Source: http://upload.wikimedia.org/wikipedia/commons/5/57/Heliocentric.jpg accessed 1 May 2012

Source: http://upload.wikimedia.org/wikipedia/commons/3/33/Geoz_wb_en.svg accessed 1 May 2012

A century later, Johannes Kepler introduced his laws of planetary motion, which demonstrated that the planets actually move in elliptical paths around the sun, not in perfect circles--a model which was an even better fit to the empirical data.

Those who insisted on retaining Ptolemy’s view of the universe, despite the growing evidence against it, were holding on to the top-down approach to data. They practiced apologetics, and used cherry-picking, special pleading, and other fallacious techniques, to protect their model from the challenge the material physical world confronted it with.

In contrast, the bottom-up approach of Copernicus and Kepler, who worked from the data to develop their conclusions, won out.

These new thinkers prevailed over the Ptolemaists because they were willing to let go of their previous beliefs (Kepler, in particular, was disappointed by the idea that planets moved in ellipses rather than in the perfect circular shapes he found so beautiful, but he followed his conscience in following the process where it led) and to let the data itself tell the story.

[Of course, by "prevailed", we never mean "100% accepted": there are, after all, modern-day adherents to the Flat Earth model in the incarnation of the Flat Earth society, just to name one example (Motto: "Replace the science religion...with SANITY.").

What we mean is that the majority of professionals, who have actually done the work to understand the domain, vouch for the work as having been carried out with integrity, and to be validated as showing the results it claims to demonstrate.]

Statistics is one methodology that we apply in a bottom-up approach to understand the meaning of the story that the data is telling us.

Exercise

Can you think of some real-life examples of where people try, or have tried, to protect an old model that has been discredited, despite the mounting evidence against it?

Areas where you might find examples nowadays include healthcare and politics, among others.

How far are some people prepared to go to protect old models?

What techniques do they use to do so?

What are the stakes--politically, psychologically, economically, and in other domains?