Which type of error do you prefer?

Mayor Bloomberg is avoiding a Type 2 error

As I write this, Hurricane Sandy is bearing down on the east coast of the United States. Mayor Bloomberg has ordered evacuations from various parts of New York City. All over the region people are stocking up on food and other essentials and waiting for Sandy to arrive. And if Sandy doesn’t turn out to be the worst storm ever, will people be relieved or disappointed? Either way there is a lot of money involved. And more importantly, risk of human injury and death. Will the forecasters be blamed for over-predicting?

Types of error

There are two ways to get this sort of decision wrong. We can do something and find out it was a waste of time, or we can do nothing and wish that we had done something. In the subject of statistics these are known as Type 1 and Type 2 errors. Teaching about Type 1 and Type 2 errors is quite tricky and students often get confused. Does it REALLY matter if they get them around the wrong way? Possibly not, but what really does matter is that students are aware of their existence. We would love to be able to make decisions under certainty, but most decisions involve uncertainty, or risk. We have to choose between the possibility of taking an opportunity and finding out that it was a mistake, and the possibility of turning down an opportunity and missing out on something.

Earthquake prediction

In another recent event, Italian scientists have been convicted of manslaughter for failing to predict a catastrophic earthquake. This has particular resonance in Christchurch as our city has recently been shaken by several large quakes and a multitude of smaller aftershocks. You can see a graph of the Christchurch shakes at this site. In most part the people of Christchurch understand that it is not possible to predict the occurrence of earthquakes. However it seems that the scientists in Italy may have overstated the lack of risk. Just because you can’t accurately predict an earthquake, it doesn’t mean it won’t happen. Here is a link to a story by Nature of the Italian earthquake.

Tornado warnings

Laura McLay wrote a very interesting post entitled. “what is the optimal false alarm rate for tornado¬†warnings?” . A high rate of false alarms is likened to the “boy who cried wolf”, to whom nobody listens any more. You would think that there is no harm in warning unnecessarily, but in the long term there is potential loss of life because people fail to heed subsequent warnings.

Operations Research and errors

Pure mathematicians tend not to like statistics much as it isn’t exact. It’s a little bit sullied by its contact with the real world. However Operations Research goes a step further into the messy world of reality and evaluates the cost of each type of error. Decisions are often converted into dollar terms within decision analysis. Like it or not, the dollar is the usual measure of worth, even for a human life, though sometimes a measure called “utility” is employed.

Costs of Errors

Sometimes there is very little cost to a type 2 error. A bank manager refusing to fund a new business is avoiding the risk of a type 1 error, which would result in a loss of money. They then become open to at type 2 error, that they missed out on funding a winner. The balance is very much on the side of avoiding a type 1 error. In terms of choosing a life partner, some people are happy to risk a type 1 error, and marry, while others, hold back, perhaps invoking a type 2 error by missing out on a “soul-mate”. Or it may be that we make this decision under the illusion of certainty and perfect information, and the possible errors do not cross our minds.

Cancer screening is a common illustration of type 1 and type 2 errors. With a type 1 error, we get a false positive and are told we have a cancer when we do not. With type 2, the test fails to detect a cancer. In this example the cost of a type 2 error seems to be much worse than type 1. Surely we would rather know if we have cancer? However in the case of prostate cancer, a type 1 error can lead to awful side-effects from unnecessary tests. Conversely a large number of men die from other causes, happily unaware that they have early stages of prostate cancer.

The point is that there is no easy answer when making such decisions.

Teaching about type 1 and type 2 errors

I have found the following helpful when teaching about type 1 and type 2 errors in statistics. Think first about the action that was taken. If the null hypothesis was rejected, we have said that there is an effect. After rejecting the null only two outcomes are possible. We have made the correct decision, or we have made a type 1 error. Conversely if we do not reject the null hypothesis, and do nothing, we have either been correct or made a type 2 error. You cannot make a type 1 error and a type 2 error in the same decision.

  • Decision:Reject the Null. Outcome is:
    • Correct or
    • Type 1 error
  • Decision:Do not reject the Null. Outcome is:
    • Correct or
    • Type 2 error.

Or another way of looking at it is:

  • Do something and get it wrong – Type 1 error
  • Do nothing and regret it – Type 2 error

Avoid error

Students may wonder why we have to have any kind of error. Can we not do something to remove error? In some cases we can – we can spend more money and take a larger sample, thus reducing the likelihood of error. However, that too has its cost. The three costs are important aspects of decision-making, and helping students to understand this will help them to make and understand decisions.

About these ads

8 thoughts on “Which type of error do you prefer?

  1. It all comes down to performing a risk/cost analysis. Does the risk of doing nothing exceed the cost of recovery should the risk become actuality?
    Considering many costs of a risk becoming actuality, it’s far preferable in many, many cases to make the type 1 error.
    In NJ right now, there are quite a few people who have committed a type 2 error and water is moving ashore toward them, along with rain and wind. Considering the risk of a hurricane that is amplified by a Nor’easter and tides, the risk is that of death or injury vs inconvenience.
    That is a classic type 2 error. With potential lethal consequences.

  2. As important (for cost/benefit analysis) is the consideration of Type 3 and Type 4 errors.

    That is, Type 3 := solving the wrong utility function(*) correctly and 4 := solving the utility function correctly but drawing the wrong conclusion. (The latter arising in medicine for example where a physician sees symptoms, identifies the right cause but then prescribes the wrong treatment). Other candidates exist for Type III and IV error; but these seem the most reasonable in my view. :)

    (*) I would consider arguing that assigning a $ value to something that does not *have* an intrinsic $ value is indicative of a possible type III error. I will refrain from giving examples lest they be too controversial.

  3. I think it is better to speak of avoiding errors of various given extents, rather than error/no error. It is the dichotomous view of tests that is the source of so very much criticism, whereas, evaluating the extent and directions of errors that are and are not ruled out well avoids them. I often discuss this issue, and related ones, on my blog
    errorstatistics.com

  4. Pingback: Types of Error and Hurricane Sandy | Notes on Learnings About Data Analysis

  5. @David, I see EVERY error in my career in information security. VERY common are 3 and 4.
    Even in the DoD, with all manner of sensors installed, people will type 4 error and add to the sensor package and forget silly things like user education, compliance checks, antivirus updates and function and patch management.
    In short, EVERY field has its share of all forms of error. The trick is as much trying to minimize errors AND recovering from them.

  6. Pingback: Which type of error do you prefer? « Guzman's Mathematics Weblog

  7. Nicely explained! Now I'm going to be thinking in a slightly new way...like adam and eve or joseph and mary... Type 1 or 2! :) on said:

    Nicely said! Adam and Eve or joseph and Mary……type1 or type 2. Wow and my own decisions! :)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s