The Economist reports that computer scientists are now using Bayesian Analysis in their efforts to create software that “thinks” like a person.

Bayesian Analysis is already central to search engines and a host of other applications, so its practicality is well established. But the most interesting question is, does nature uses it in the design of organic computers, such as the human brain?

Perhaps this is how we understand language, reason and plan.

Now, some professors have figured out how to test this hypothesis. They’ve been able to conclude that we are probably all “Bayesian processors.” I’ll tell you how below…

The Bayesian approach relies upon starting with an accurate hypothesis about how something works. This hypothesis can be stated as an assumption.

In school, you probably learned about something called the “normal” or bell curve. (Draw the curve on paper, and it resembles a bell.) In such cases, outcomes in the middle are almost always the case, and extremes happen very rarely.

If a baby is hungry and screams, and the result is almost always that the baby gets fed until full, there is a normal distribution operating here. It makes sense that the baby will learn to think and act this way. Very few data points — perhaps just one — are needed to make this connection.

If, on the other hand, if after crying the baby rarely got fed to satiety or was overfed, it would draw a different conclusion.

The major alternative to Bayesian Analysis is known as Frequentism. It relies on a lot more data, but without assumptions. It’s not so good for making decisions with limited information. Yet many life circumstances require exactly that kind of decision-making.

Researchers Thomas Griffiths of Brown University and Joshua Tenenbaum of MIT studied a group of 350 people. Subjects were given a single piece of very specific, relevant information. From this, they were supposed to reach a conclusion.

The areas of inquiry were strikingly diverse…

• When told the box office receipts of a currently running film, they were asked to estimate total sales receipts over the life of its release in theaters.
• When informed that a line of poetry appeared so many lines into the poem, they were asked to estimate the total length of the poem.
• Time to bake a cake was to be estimated from time it had already been in the oven.
• The length of a Congressman’s career was to be derived from his or her years already served in Congress.
• Lifespan of a person based on current age and the hold time in a telephone wait list were also estimated.

In two cases, the test subjects failed to accurately predict from the data. However, in one of the cases the researchers concluded that the data had been poorly worded, while in the other case (how long an Egyptian pharaoh reigned) it was a topic of which most Americans are ignorant.

From that perspective, the first exception was irrelevant and the second further confirmed the research — since quality of assumptions depends on the context of knowledge.

Statisticians describe data as falling into a variety of “distributions.” The so-called normal or bell-curved distribution is the most famous, but hardly the only one we encounter.

The professors have discovered that people are able to use Bayesian data to accurately determine unusual and complex distributions.

A further topic of research will be how people develop such finely tuned assumptions in the first place. Clearly, we as a species appear exceptionally gifted at this kind of reasoning. It is, I suspect, closely attuned to what most of us call “intuition.”

This has all kinds of interesting implications. For instance, the researchers now believe that superstition — which is common among people of all levels of education and sophistication — may result from assumptions wrongly forming a self-supporting pattern with the data.

I could imagine this happening as follows: Someone meets a long-lost friend when the elevator stops on the 7th floor. This creates the assumption. Subsequent experiences of “7” are similarly pleasant, and provide supporting data. The person begins to look for evidence, and to discount contradictory data. As a result, “7” comes to be the person’s lucky number.

Causality is perceived where it’s merely coincidence. But try telling that to a gambler!

Have you heard of the “Delphi Method” of forecasting? It’s widely used in business planning. Essentially, when it’s excessively expensive to estimate something, three experts are asked to give their opinions. The average is then used as the forecast.

What the new research suggests is that, with properly formulated starting assumptions, it may not always be necessary to use experts. I predict that this, in turn, will lead to startling advances in market research and could well replace focus groups.

It will help companies make wise choices of new products and technologies, and avoid poor ones. I have a contact who is a professor of marketing at Purdue University. He co-founded a company to simulate markets for companies.

I’m going to solicit his opinion of this and how it will affect transformational technology companies. That will help guide my future recommendations.  I’ll let you know what I find out.

To your profitable future,
Jonathan Kolber
February 21, 2007

P.S.: A small, under-the-radar pharmaceuticals pioneer has brought America’s most dreaded disease to its knees… Gains of 30-60 times your money or MORE are all but certain if you’re holding shares when Big Pharma buys it out.

Investing Based on Probabilities was originally featured in the Penny Sleuth.