Sampling Error Isn’t

I hope you committed to a response in the box before reading this post.

This is an important topic. Recently I read an amusing blog regarding poor sampling technique. The tweet that led to the link called it “a humorous look at sample error”. I’m hoping the person who tweeted meant bad sampling, because the problem is, the story was not about sampling error.

And that is because sampling error isn’t. Isn’t what? It isn’t error. It doesn’t occur by mistake. It is not caused by bad procedures. There is nothing practical you can do when sampling to avoid sampling error. Sampling error exists because you are taking a sample. The only way to avoid sampling error is to test the entire population – in which case it isn’t a sample, it’s a census.

This is a vivid example of when a word in common use is given a different very specific meaning within a discipline that then confuses the heck out of everyone.

It has been found that even students who get A grades in first year statistics at university, often have serious flaws and gaps in their understanding of statistics. I would predict that the idea of sampling error will be a cavernous hole of misunderstanding for most.

The problem is not sampling error, but bias. Take a perfect random sample, where each object in the population has an equal probability of selection. This will reduce, and perhaps even eliminate bias. But sampling error will remain.

Because of natural variation it is unlikely that all people send the same number of texts in a day.

So how do you teach this? I use the approach of talking about variation*. Variation is inherent in all natural, human and manufacturing processes. We then classify variation into four categories: Natural, Explainable, Sampling and Bias. The term “natural variation” describes the omnipresence of variation in real life. “Explainable variation” is what we are often looking for in statistical analysis – can we use age of a car to help explain some of the variation in prices of cars, for instance. Sampling variation (also known as sampling error) occurs when we take a sample and use it to draw conclusions about the population. We would not expect two samples from the same population to yield exactly the same results. The fourth category is variation due to biased sampling.

This approach is not comprehensive, and can be a bit clunky in the terminology, jumping between variation and error. But it gives a framework for students to identify the difference between sampling error/variation and error due to biased sampling. We do classroom activities where students get different samples from the same population to illustrate sampling variation/error.

This is important. It is important that people in general understand that samples are not going to represent the population exactly. They also need to understand that through the use of theoretical probability models statisticians and analysts do allow for that sampling error. Bias, however, is another story for another day.

You can see how we explain the different kinds of variation in this YouTube video:

By the way – the correct answer to the question at the start of the post is False. No sampling method, no matter how good it is, will eliminate sampling error.

Let’s see if you get it – here are some statements about variation. Classify each of the following as examples of natural variation, explainable variation, sampling variation or variation due to biased sampling. I’ll put the answers in the comments to this blog.

  • When I bike to work, sometimes it takes me longer than other times.
  • When I bike to work with a head wind, it generally takes me longer than with a tail wind.
  • Two students each took random samples of ten students from their class and asked them how many friends they have on Facebook. They got different values for their means.
  • Two students each asked eight of their friends how many friends they have on Facebook. They got different values for their means.

*Note: This approach is based on the thought-provoking work by Wild and Pfannkuch, reported in “Statistical Thinking in Empirical Enquiry” International Statistical Review (1999) p235.


Textbooks and horseless carriages

Why do my students like me and the bookreps don’t? Because I do not require a textbook for either of my large entry level courses in Statistics and Operations Research. I have found that so few students use any prescribed text, that it is pointless prescribing one. I have found other ways to engage students and help them to learn the skills, attitudes and content that I believe are necessary. The other problem was that I never found a text that aimed to develop the same skills, attitudes and content that I wanted them to. Too many of them seemed to smother all the fun and joy in unnecessary computation.

Apple’s big announcement about textbooks on the iPad made me examine my stance regarding textbooks. I’m happy about the iPad being used for educational purposes. I think it is a fabulous medium with amazing potential (not the least example being our AtMyPace: Statistics app). I’m just not sure about the whole electronic book thing. I hope it is not a “legacy” solution – like the ones in upgraded computer packages for people who can’t let go of their old way of doing things.

When cars, or should I say “horseless carriages” were invented they looked remarkably like carriages. The power was measured in horse-power, and terms like trunk and hood were inherited. A modern car bears little resemblance to early vehicles, with streamlining and creature comforts, air conditioning, remote central locking etc. But the transformation was evolutionary.

Textbooks have been around for a while, and make a teacher’s job easier. You can say, “Read Chapter 3 and answer questions 1,3 and 6”, and leave the students to it… Maybe. But the current theories of education, particularly those of constructivism, which I personally embrace, would not support this as an effective learning mechanism. Students need to actively engage with the material in order to have it integrate with (and sometimes replace) their current knowledge.

If we simply take the textbook and turn it into an electronic book, the gain is minimal. We have reduced the weight, and hopefully the price. We have made it a little more “modern” and added the gimmick factor. The wonderful Apple promotional video shows fabulous interactive aspects in the text, with excited and clean-cut students lapping up the content, totally engaged and alert. I have no doubt that there are exciting aspects to some eBook texts, but essentially they are text in a different format. And we have yet to see how much interactivity there is in the standard textbook.

A definite advantage of the electronic format is the potential to keep material current with upgrades rather than having to produce yet another edition of the increasingly bloated familiar text. And there is the potential for instructors to select parts of different texts – though that is already possible with “Made-to-measure” textbooks in paper format.

The thing is, a learning tool on the iPad can be so much more! Our little app uses video that students can control the pace of. The use of multimedia – audio and visual, allows for better learning. Then the question sets take the student through material in an interactive way, endeavoring to expose their current erroneous thinking, in order to make it easier to construct learning correctly. There are two parallel question sets so that students can assess for themselves how their learning is progressing. This is only the beginning. Our aim is to include interactive learning activities and links to deeper material. We will include adaptive testing that assesses the level of understanding and adjusts accordingly. These are not new ideas, and have been shown to facilitate better learning. There are more ideas as well.

iBooks textbooks for iPad are pretty exciting, but let’s get our horseless carriages sleek, exciting and versatile as quickly as possible.

The Importance of Titles

My colleague has an obsession about titles – and it is starting to rub off on me. Any time our graduate students present their work, the first thing that grabs her (and now my) attention is the title – the opening slide on the Powerpoint presentation. She declares that clarity of title indicates clarity of thought. It tells us whether they have mastered what they are talking about themselves. A woolly title indicates woolly thinking.

As a result of her indoctrination I have included as part of a regression write-up, that students are required to provide a suitable title. They are graded on it. Their titles include “Regression Analysis”,“Relationship between mileage and price of used cars”and “Height does not affect salary.” These titles are indicative of their level of understanding.

Titles Tell All
“Regression Analysis”
is a poor title that tells only the technique, without the context. It is popular among students who have not learned that statistical analysis occurs within a context. They are still focussed on the process rather than the purpose. Their write-ups tend to be narratives of what they have done, in the order that they did it. I call it (a little unkindly) a “what I did in the holidays” report. They may have done a considerable amount of work and really want me to know how much they have done. Their reports are often quite repetitious and cover every last detail, without anything to tie it together. The report may be correct, but seldom has insight.

A better title is “Relationship between mileage and price of used cars”. This student has worked out what it is that they are examining. They often have very good reports, though they tend to stick to the “facts” and do not roam into the murky world of implications. There is very little of their own thinking in the report. Again there is a tendency to try to include every last detail. And they like to leave the punchline to last. They want to build up suspense, like a detective novel, until in one dramatic flourish, the outcome is revealed. This will not work in many real world settings. I tell my students that everything important they have to say must be on the first page, or it won’t get read.

Which is why I like titles such as “Height does not affect salary,” “New phone owners text more” and “Men prefer milk chocolate”. These titles are catchy and tell us the outcome. This is not to say that students get away with unsupported assertions. There must be correct statistical evidence for any of their statements. But I like a bit of courage. And a short report that sticks to the important stuff shows the courage to leave out or report only briefly, some of their analysis and trust that I know they did it.

Helen considers a report

Think again before calling a report "Statistical Analysis".

This has been a rather simplistic approach to titles (and reports). In reality the style and content of the title is also influenced by the purpose of the analysis and the intended audience. There is a tendency towards catchy titles in academic papers, for the obvious reason that we want people to remember our paper and cite it. Newspaper titles must be short, and can lack precision.

The purpose of this post, however is to draw instructors‘ attention to the production of titles as a means of assessing understanding, both formally and informally, such as in class. And for students, I hope it makes you think again before you give your analysis the title, “Statistical Analysis”!

Should students calculate?

At an NCTM conference session on teaching statistics I suggested that there was no point in teaching how to calculate a standard deviation. It caused a somewhat heated response, mostly in opposition, but it did get us thinking.

Similarly I have suggested that using the graphical method of Linear Programming is not helpful for most students, with similarly mixed response. The paper was rejected by reviewers.

Each of those issues can have a post all of their own. What I want to discuss here is when calculation is useful, and when it isn’t.

Type of student and purpose of the class

Five students

Students are different

Students are different. Think about who your students are and the purpose for the calculation, before deciding if it is necessary or helpful to their learning.

 Mathematics class

If you are a mathematics teacher, and the aim is for students to engage in purposeful use of mathematics, then calculate away. Statistics and operations research are disciplines that use mathematics in an applied setting. It can be easy to see the purpose of mathematics to find out the optimal product mix, or the number of servers needed at a supermarket, or to decide whether a marketing approach has improved sales. Deriving the EOQ formula is a wonderfully simple application of calculus. Two variable linear programming on a cartesian plane is a great way to practice graphing skills. These are really good for teachers of mathematics who are teaching students of mathematics.

This is being enabled in several US states by the MINDSET project, which, in its own definition, “uses decision making tools from Industrial and Systems Engineering and Operations Research in a fourth-year high school mathematics curriculum. Principal performance related goals of the project are to improve upon the math students’ ability to formulate and solve multi-step problems and interpret results, and to improve students’ attitude toward mathematics.”

When statistics is taught as part of a mathematics curriculum, then there may be some point in the use of calculations and table-reading when the aim is to develop skills that transfer to other areas.

However, apart from that, for most beginning level students of statistics and operations research it is counter-productive to calculate by hand.

Type of student: Business statistics

Helen scream

Some people don't really like to do mathematics

In a business statistics course, there are often students who dislike mathematics, and calculation is something they reluctantly learn to do by rote. But a key to understanding and enjoying statistics is being immersed in the context. Get a data set, and let them use the package to find out what they can from it. Decide what test is needed, interpret the output and apply it to the context. This is exciting and involving. They don’t need to calculate by hand anymore. Researchers don’t – that is what packages were invented for. I’m aware that Excel statistics add-in has some flaws, but it is there and mostly harmless. Let students have the fun of spending the time on the exciting part of statistics, not the hand calculations.

What about an MBA Management Science course? Linear programming is a versatile and powerful tool, and by limiting problems to two decision variables to make plotting possible, we trivialise it. The answer is often self-evident, and there are artefacts of two-variable models that can be generalised erroneously. Let students have something approaching real-life size models and cases and they will enjoy the power of the technique.

Applying a formula repeatedly does not lead to comprehension. If you are a mathematician, you can read the formula and understand what is happening without applying it. If you are not so inclined, repeated application of an algorithm is done automatically and disconnected from understanding. Let the students have real data, and use real methods.

In our video on Confidence intervals, you can see how the formula is used to show what is happening, but then Excel is used for the calculation. What do you think?

The meaning of the mean

Here is an exercise you might like to try on a class or individual, when introducing the mean. I have found it interesting and enlightening for all parties, especially those who think they know everything.

Dr Nic: Tell me what a mean is, as if explaining it to someone who doesn’t know about statistics.

Student: It’s an average.satisfied
Dr Nic: Correct, however you haven’t really increased my understanding with that description.
Student: It is what you get when you add all the numbers together and divide by the number of numbers.

Dr Nic: That is a correct description of how to calculate a mean. Still I’m not getting any idea of what it does.

Student: It’s rather like a middle number.

Dr Nic: 
That has merit, though that description works better for a median. I still don’t think you are getting to the essence of it.


This is harder than I thought!

Student: I give up. This is harder than I thought.

Dr Nic: It really is. The idea of a mean is quite tricky. I like to think of it as a way of summarising a whole lot of numbers, in order to make comparisons.

Student: Huh?

Dr Nic: Say you have a whole lot of numbers that are the times different people take to complete a Rogo puzzle. I can even give you some: 16, 23, 30, 14, 63, 34. Say you want to summarise these numbers in one number, how would you do it?

Student: You could add them up (180 seconds) and say how many there are (six people)– or you could find out what the total is divided by the number of numbers. Which is the mean! (30 seconds per person)

Dr Nic: Very good. However, why would you want to do this?
Student: Because then you could say that… on average it took 30 seconds to solve the Rogo?

Dr Nic: Absolutely, but really why would you want to? Mostly we want a mean in order to compare. (Or in Operations Research we may like to use a mean to provide an input to decision-making.) If we had a second group of people who had a mean of 23 seconds for that Rogo, then we can see that on average the second group took less time. Or we could try another Rogo with the first group of people and find that the mean was 47 seconds. We would probably conclude that the second Rogo was more difficult.

Student: Hmm. So a mean is a way to summarise a set of numerical data that can be used for comparisons.

Dr Nic: Fabulous! I couldn’t have put it better myself.

Student: What’s a Rogo puzzle?

Dr Nic (aka Dr Rogo): Funny you should ask – it’s a puzzle we invented some years ago and made an app for, but it has faded away.

Very happyComment:

Another way to look at a mean is that it is an emergent property of a set of data. One observation can tell us a little bit about a phenomenon, but once you get a set of data, there are emergent properties that can help to explain the phenomenon.

Until I started to think about it, I had thought a mean was a really obvious concept. But it isn’t – and it is worth spending time on to clarify understanding in students. (And unless you wish to baffle them with long words, or have students with a strong mathematical bckground, I’d avoid the terms “measure of central tendency”, and “first moment”, until they have a better grip on the subject.)