Graphs – beauty and truth

Graphs – beauty and truth (with apologies to Keats)

A good graph is elegant

I really like graphs. I like the way graphs turn numbers into pictures. A good graph is elegant. It uses a few well-placed lines to communicate what would take a paragraph of text. And like a good piece of literature or art, a good graph continues to give, beyond the first reading. I love looking at my YouTube and WordPress graphs. These graphs tell me stories. The WordPress analytics tell me that when I put up a new post, I get more hits, but that everyday more than 1000 people read one of my posts. The YouTube analytics tell me stories about when people want to know about different aspects of statistics. It is currently the end of the North American school year, and the demand is for my video on Choosing which statistical test to use. Earlier in the year, the video about levels of measurement is the most popular. And not many people view videos about statistics on the 25th of December. I’m happy to report that the YouTube and WordPress graphs are good graphs.

Spreadsheets have made it possible for anyone and everyone to create graphs. I like that graphs are easier to make. Drawing graphs by hand is a laborious task and fraught with error. But sometimes my heart aches when I see a graph used badly. I suspect that this is when a graphic artist has taken control, and the search for beauty has over-ridden the need for truth.

Three graphs spurred me to write this post.

Graph One: Bad-tasting Donut on house occupation

The first was on a website to find out about property values. I must have clicked onto something to find out about the property values in my area, and was taken to the qv website. And this is the graph that disturbed me.

Graphs named after food are seldom a good idea

Sure it is pretty – uses pretty colours and shading, and you can find out what it is saying by looking at the key – with the numbers beside it. But a pie or donut chart should not be used for data which has inherent order. The result here is that the segments are not in order. Or rather they are ordered from most frequent to least frequent, which is not intuitive. Ordinal data is best represented in a bar or column chart. To be honest, most data is best represented in a bar or column chart. My significant other suggested that bar charts aren’t as attractive as pie charts. Circles are prettier than rectangles. Circles are curvy and seem friendlier than straight lines and rectangles. So prettiness has triumphed over truth.

Graph Two: Misleading pictogram (a tautology?)

It may be a little strong to call bad communication lack of truth. Let’s look at another example. In a way it is cheating to cite a pictogram in a post like this. Pictograms are the lowest form of graph and are so often incorrect, that finding a bad one is easier than finding a good one. In the graph below of fatalities it is difficult to work out what one little person represents.

What does one little person represent?

A quick glance, ignoring the numbers, suggests that the road toll in 2014 is just over half what it was in 2012. However, the truth, calculated from the numbers, is that the relative size is 80%. 2012 has 12 people icons, representing 280 fatalities. One icon is removed for 2013, representing a drop of 9 fatalities. 2011 has one icon fewer again, representing a drop of 2 fatalities. There is so much wrong in the reporting of road fatalities, that I will stop here. Perhaps another day…

Graph Three: Mysterious display on Household income

And here is the other graph that perplexed me for some time. It came in the Saturday morning magazine from our newspaper, as part of an article about inequality in New Zealand. Anyone who reads my blog will be aware that my politics place me well left of centre, and I find inequality one of the great ills of the modern day. So I was keen to see what this graph would tell me. And the answer is…

See how long it takes for you to find where you appear on the graph. (Pretending you live in NZ)

I have no idea. Now, I have expertise in the promulgation of statistics, and this graph stumped me for some time. Take a good look now, before I carry on.

I did work out in the end, what was going on in the graph, but it took far longer than it should. This article is aimed at an educated but not particularly statistically literate audience, and I suspect there will be very few readers who spent long enough working out what was going on here. This graph is probably numerically correct. I had a quick flick back to the source of the data (who, by the way, are not to be blamed for the graph, as the data was presented in a table) and the graph seems to be an accurate depiction of the data. However, the graph is so confusing as to be worse than useless. Please post critiques in the comments. This graph commits several crimes. It is difficult to understand. It poses a question and then fails to help the reader find the answer. And it does not provide insights that an educated reader could not get from a table. In fact, I believe it has obscured the data.

Graphs are the main way that statistical analysts communicate with the outside world. Graphs like these ones do us no favours, even if they are not our fault. We need to do better, and make sure that all students learn about graphs.

Teaching suggestion – a graph a day

Here is a suggestion for teachers at all levels. Have a “graph a day” display – maybe for a month? Students can contribute graphs from the news media. Each day discuss what the graph is saying, and critique the way the graph is communicating. I have a helpful structure for reading graphs in my post: There’s more to reading graphs than meets the eye; 

Here is a summary of what I’ve said and what else I could say on the topic.

Thoughts about Statistical Graphs

  • The choice of graph depends on the purpose
  • The text should state the purpose of the graph
  • There is not a graph for everything you wish to communicate
  • Sometimes a table communicates better than a graph
  • Graphs are part of the analysis as well as part of the reporting. But some graphs are better to stay hidden.
  • If it takes more than a few seconds to work out what a graph is communicating it should either be dumped or have an explanation in the text
  • Truth (or communication) is more important than beauty
  • There is beauty in simplicity
  • Be aware than many people are colour-blind, or cannot easily differentiate between different shades.

Feedback from previous post on which graph to use

Late last year I posted four graphs of the same data and asked for people’s opinions. You can link back to the post here and see the responses: Which Graph to Use.

The interesting thing is not which graph was selected as the most popular, but rather that each graph had a considerable number of votes. My response is that it depends.  It depends on the question you are answering or the message you are sending. But yes – I agree with the crowd that Graph A is the one that best communicates the various pieces of information. I think it would be improved by ordering the categories differently. It is not very pretty, but it communicates.

I recently posted a new video on YouTube about graphs. It is a quick once-over of important types of graphs, and can help to clarify what they are about. There are examples of good graphs in there.

I have written about graphs previously and you can find them here on the Collected Works page.

I’m interested in your thoughts. And I’d love to see some beautiful and truthful graphs in the comments.


The nature of mathematics and statistics and what it means to learn and teach them

I’ve been thinking lately….

Sometimes it pays to stop and think. I have been reading a recent textbook for mathematics teachers, Dianne Siemon et al, Teaching mathematics: foundations to middle years (2011). On page 47 the authors asked me to “Take a few minutes to write down your own views about the nature of mathematics, mathematics learning and mathematics teaching.” And bearing in mind I see statistics as related to, but not enclosed by mathematics, I decided to do the same for statistics as well. So here are my thoughts:

The nature of mathematics

Mathematicians love the elegance of mathematics

Mathematicians love the elegance of mathematics

Mathematics is a way of modelling and making sense of the world. Mathematics underpins scientific and commercial endeavours as well as everyday life. Mathematics is about patterns and proofs and problem structuring and solution finding. I used to think it was all about the answer, but now I think it is more about the process. I used to think that maths was predominantly an individual endeavour, but now I can see how there is a social or community aspect as well. I fear that too often students are getting a parsimonious view of mathematics, thinking it is only about numbers, and something they have to do on their own. I find my understanding of the nature of mathematics is rapidly changing as I participate in mathematics education at different ages and stages. I have also been influenced by the work of Jo Boaler.

To learn mathematics

My original idea of mathematics learning comes from my own successful experience of copying down notes from the board, listening to the teacher and doing the exercises in the textbook. I was not particularly fluent with my times-tables, but loved problem-solving. If I got something wrong, I was happy to try again until I nutted it out. Sometimes I even did recreational maths, like the time I enumerated all possible dice combinations in Risk to find out who had the advantage – attacker or defender. I always knew that it took practice to be good at mathematics. However I never really thought of mathematics as a social endeavour. I feel I missed out, now. From time to time I do have mathematical discussions with my colleague. It was an adventure inventing Rogo and then working out a solution method. Mathematics can be a social activity.

To teach mathematics

When I became a maths teacher I perpetuated the method that had worked for me, as I had not been challenged to think differently. I did like the ideas of mastery learning and personalised system of instruction. This meant that learners progressed to the next step only when they had mastered the previous one. I was a successful enough teacher and enjoyed my work.

Then as a university lecturer I had to work differently, and experimented. I had a popular personalised system of instruction quantitative methods course, relying totally on students working individually, at their own pace. I am happy that many of my students were successful in an area they had previously thought out of their reach. For some students it was the only subject they passed.

What I would do now

If I were to teach mathematics at school level again, I hope I would do things differently. I love the idea of “Number talks” and rich tasks which get students to think about different ways of doing things. I had often felt sad that there did not seem to be much opportunity to have discussions in maths, as things were either right or wrong. Now I see what fun we could have with open-ended tasks. Maths learning should be communal and loud and exciting, not solitary, quiet and routine. I have been largely constructivist in my teaching philosophy, but now I would like to try out social constructivist thinking.


And what about statistics? At school in the 1970s I never learned more than the summary statistics and basic probability. At uni level it was bewildering, but I managed to get an A grade in a first year paper without understanding any of the basic principles. It wasn’t until I was doing my honours year in Operations Research and was working as a tutor in Statistical methods that things stared to come together – but even then I was not at home with statistical ideas and was happy to leave them behind when I graduated.

The nature of statistics

Statistics lives in the real world

Statistics lives in the real world

My views now on the nature of statistics are quite different. I believe statistical thinking is related to mathematical thinking, but with less certainty and more mess. Statistics is about models of reality, based on imperfect and incomplete data. Much of statistics is a “best guess” backed up by probability theory. And statistics is SO important to empowered citizenship. There are wonderful opportunities for discussion in statistics classes. I had a fun experience recently with a bunch of Year 13 Scholarship students in the Waikato. We had collected data from the students, having asked them to interpret a bar chart and a pie chart. There were some outliers in the data and I got them to suggest what we should do about them. There were several good suggestions and I let them discuss for a while then moved on. One asked me what the answer was and I said I really couldn’t say – any one of their suggestions was valid. It was a good teaching and learning moment. Statistics is full of multiple good answers, and often no single, clearly correct, answer.

Learning statistics

My popular Quantitative Methods for Business course was developed on the premise that learning statistics requires repeated exposure to similar analyses of multiple contexts. In the final module, students did many, many hypothesis tests, in the hope that it would gradually fall into place. That is what worked for me, and it did seem to work for many of the students. I think that is not a particularly bad way to learn statistics. But there are possibly better ways.

I do like experiential learning, and statistics is perfect for real life experiences. Perhaps the ideal way to learn statistics is by performing an investigation from start to finish, guided by a knowledgeable tutor. I say perhaps, because I have reservations about whether that is effective use of time. I wrote a blog post previously, suggesting that students need exposure to multiple examples in order to know what in the study is universal and what applies only to that particular context. So perhaps that is why students at school should be doing an investigation each year within a different context.

The nature of understanding

This does beg the question of what it means to learn or to understand anything. I hesitate to claim full understanding. Of anything. Understanding is progressive and multi-faceted and functional. As we use a technique we understand it more, such as hypothesis testing or linear programming. Understanding is progressive. My favourite quote about understanding is from Moore and Cobb, that “Mathematical understanding is not the only understanding.” I do not understand the normal distribution because I can read the Gaussian formula. I understand it from using it, and in a different way from a person who can derive it. In this way my understanding is functional. I have no need to be able to derive the Gaussian function for what I do, and the nature and level of my understanding of the normal distribution, or multiple regression, or bootstrapping is sufficient for me, for now.

Teaching statistics

I believe our StatsLC videos do help students to understand and learn statistics. I have put a lot of work into those explanations, and have received overwhelmingly positive feedback about the videos. However, that is no guarantee, as Khan Academy videos get almost sycophantic praise and I know that there are plenty of examples of poor pedagogy and even error in them. I have recently been reading from “Make it Stick”, which summarises theory based on experimental research on how people learn for recall and retention. I was delighted to find that the method we had happened upon in our little online quizzes was promoted as an effective method of reinforcing learning.

Your thoughts

This has been an enlightening exercise, and I recommend it to anyone teaching in mathematics or statistics. Read the first few chapters of a contemporary text on how to teach mathematics. Dianne Siemon et al, Teaching mathematics: foundations to middle years (2011) did it for me. Then “take a few minutes to write down your own views about the nature of mathematics, mathematics learning and mathematics teaching.” To which I add my own suggestion to think about the nature of statistics or operations research. Who knows what you will find out. Maybe you could put a few of your ideas down in the comments.


Understanding Statistical Inference

Inference is THE big idea of statistics. This is where people come unstuck. Most people can accept the use of summary descriptive statistics and graphs. They can understand why data is needed. They can see that the way a sample is taken may affect how things turn out. They often understand the need for control groups. Most statistical concepts or ideas are readily explainable. But inference is a tricky, tricky idea. Well actually – it doesn’t need to be tricky, but the way it is generally taught makes it tricky.

Procedural competence with zero understanding

I cast my mind back to my first encounter with confidence intervals and hypothesis tests. I learned how to calculate them (by hand  – yes I am that old) but had not a clue what their point was. Not a single clue. I got an A in that course. This is a common occurrence. It is possible to remain blissfully unaware of what inference is all about, while answering procedural questions in exams correctly.

But, thanks to the research and thinking of a lot of really smart and dedicated statistics teachers, we are able put a stop to that. And we must. Help us make great resourcces

We need to explicitly teach what statistical inference is. Students do not learn to understand inference by doing calculations. We need to revisit the ideas behind inference frequently. The process of hypothesis testing, is counter-intuitive and so confusing that it spills its confusion over into the concept of inference. Confidence intervals are less confusing so a better intermediate point for understanding statistical inference. But we need to start with the concept of inference.

What is statistical inference?

The idea of inference is actually not that tricky if you unbundle the concept from the application or process.

The concept of statistical inference is this –

We want to know stuff about a large group of people or things (a population). We can’t ask or test them all so we take a sample. We use what we find out from the sample to draw conclusions about the population.

That is it. Now was that so hard?

Developing understanding of statistical inference in children

I have found the paper by Makar and Rubin, presenting a “framework for thinking about informal statistical inference”, particularly helpful. In this paper they summarise studies done with children learning about inference. They suggest that “ three key principles … appeared to be essential to informal statistical inference: (1) generalization, including predictions, parameter estimates, and conclusions, that extend beyond describing the given data; (2) the use of data as evidence for those generalizations; and (3) employment of probabilistic language in describing the generalization, including informal reference to levels of certainty about the conclusions drawn.” This can be summed up as Generalisation, Data as evidence, and Probabilistic Language.

We can lead into informal inference early on in the school curriculum. The key Ideas in the NZ curriculum suggest that “ teachers should be encouraging students to read beyond the data. Eg ‘If a new student joined our class, how many children do you think would be in their family?’” In other words, though we don’t specifically use the terms population and sample, we can conversationally draw attention to what we learn from this set of data, and how that might relate to other sets of data.

Explaining directly to Adults

When teaching adults we may use a more direct approach, explaining explicitly, alongside experiential learning to understanding inference. We have just completed made a video: Understanding Inference. Within the video we have presented three basic ideas condensed from the Five Big Ideas in the very helpful book published by NCTM, “Developing Essential Understanding of Statistics, Grades 9 -12”  by Peck, Gould and Miller and Zbiek.

Ideas underlying inference

  • A sample is likely to be a good representation of the population.
  • There is an element of uncertainty as to how well the sample represents the population
  • The way the sample is taken matters.

These ideas help to provide a rationale for thinking about inference, and allow students to justify what has often been assumed or taught mathematically. In addition several memorable examples involving apples, chocolate bars and opinion polls are provided. This is available for free use on YouTube. If you wish to have access to more of our videos than are available there, do email me at

Please help us develop more great resources

We are currently developing exciting innovative materials to help students at all levels of the curriculum to understand and enjoy statistical analysis. We would REALLY appreciate it if any readers here today would help us out by answering this survey about fast food and dessert. It will take 10 minutes at a maximum. We don’t mind what country you are from, and will do the currency conversions.  And in a few months I will let you know how we got on. and we would love you to forward it to your friends and students to fill it out also – the more the merrier! It is an example of a well-designed questionnaire, with a meaningful purpose.



Summarising with Box and Whisker plots

In the Northern Hemisphere, it is the start of the school year, and thousands of eager students are beginning their study of statistics. I know this because this is the time of year when lots of people watch my video, Types of Data. On 23rd August the hits on the video bounced up out of their holiday slumber, just as they do every year. They gradually dwindle away until the end of January when they have a second jump in popularity, I suspect at the start of the second semester.

One of the first topics in many statistics courses is summary statistics. The greatest hits of summary statistics tend to be the mean and the standard deviation. I’ve written previously about what a difficult concept a mean is, and then another post about why the median is often preferable to the mean. In that one I promised a video. Over two years ago – oops. But we have now put these ideas into a video on summary statistics. Enjoy! In 5 minutes you can get a conceptual explanation on summary measures of position. (Also known as location or central tendency)


I was going to follow up with a video on spread and started to think about range, Interquartile range, mean absolute deviation, variance and standard deviation. So I decided instead to make a video on the wonderful boxplot, again comparing the shoe- owning habits of male and female students in a university in New Zealand.

Boxplots are great. When you combine them with dotplots as done in iNZIght and various other packages, they provide a wonderful way to get an overview of the distribution of a sample. More importantly, they provide a wonderful way to compare two samples or two groups within a sample. A distribution on its own has little meaning.

John Tukey was the first to make a box and whisker plot out of the 5-number summary way back in 1969. This was not long before I went to High School, so I never really heard about them until many years later. Drawing them by hand is less tedious than drawing a dotplot by hand, but still time consuming. We are SO lucky to have computers to make it possible to create graphs at the click of a mouse.

Sample distributions and summaries are not enormously interesting on their own, so I would suggest introducing boxplots as a way to compare two samples. Their worth then is apparent.

A colleague recently pointed out an interesting confusion and distinction. The interquartile range is the distance between the upper quartile and the lower quartile. The box in the box plot contains the middle 50% of the values in the sample. It is tempting for people to point this out and miss the point that the interquartile range is a good resistant measure of spread for the WHOLE sample. (Resistant means that it is not unduly affected by extreme values.) The range is a poor summary statistic as it is so easily affected by extreme values.

And now we come to our latest video, about the boxplot. This one is four and a half minutes long, and also uses the shoe sample as an example. I hope you and your students find it helpful. We have produced over 40 statistics videos, some of which are available for free on YouTube. If you are interested in using our videos in your teaching, do let us know and we will arrange access to the remainder of them.

Introducing Probability

I have a guilty secret. I really love probability problems. I am so happy to be making videos about probability just now, and conditional probability and distributions and all that fun stuff. I am a little disappointed that we won’t be doing decision trees with Bayesian review, calculating EVPI. That is such fun, but I gave up teaching that some years ago.

The reason probability is fun is because it is really mathematics, and puzzles and logic. I love permutations and combinations too – there is something cool about working out how many ways something can happen.

So why should I feel guilty? Well, in all honesty I have to admit that there is very little need for most of that in a course about statistics at high-school or entry level university. When I taught statistical methods for management, we did some probability, but only from an applied viewpoint, and we never touched intersection and union signs or anything like that. We applied some distributions, but without much theoretical underpinning.

The GAISE (Guidelines for Assessment and Instruction in Statistics Education) Report says, “Teachers and students must understand that statistics and probability are not the same. Statistics uses probability, much as physics uses calculus.”

The question is, why do we teach probability – apart from the fact that it’s fun and makes a nice change from writing reports on time series and bivariate analysis, inference and experiments. The GAISE report also says, “Probability is an important part of any mathematical education. It is a part of mathematics that enriches the subject as a whole by its interactions with other uses of mathematics. Probability is an essential tool in applied mathematics and mathematical modeling. It is also an essential tool in statistics.”

The concept of probability is as important as it is misunderstood. It is vital to have an understanding of the nature of chance and variation in life, in order to be a well-informed, (or “efficient”) citizen. One area in which this is extremely important is in understanding risk and relative risk. When a person is told that their chances of dying of some rare disease have just doubled, it is important that they know that it may be because they have gone from one chance in a million to two chances in a million. Sure it has doubled, but it still is pretty trivial. An understanding of probability is also important in terms of gambling and resistance to the allures of games of chance. And more socially acceptable gambling, such as stockmarket trading, also requires an understanding of chance and variation.

The concept of probability is important, and a few rules of probability may help with understanding, but I suspect the mathematicians get carried away and create problems that are unlikely (probability close to zero) to ever occur in reality. Anything requiring a three-way Venn Diagram has moved from applied problem to logic puzzle.This is in stark contrast to the very applied data-driven approach used in teaching statistics in New Zealand.

Teaching Probability

The traditional approach to teaching probability is to start with the coin and the dice and the balls in the urns. As well as being mind-bogglingly boring and pointless, this also projects an artificial certainty about the probabilities, which is confusing when we start discussing models. If you look at the Khan Academy videos (but don’t) you will find trivial examples about coloured balls or sweets or strangely complex problems involving hitting a circular target. The traditional approach is also to teach probability as truth. “The probability of getting a boy is one-half”. What does that even mean?

I am currently reading the new Springer volume, Probabilistic Thinking, and intend to write a review and post it on this blog, if I can get through enough before my review copy expires. It is inspiring and surprisingly gripping (but I don’t think that is enough of a review to earn me a hard copy to keep.). There are many great ideas for teaching in it, that I hope to pass on in due time.

The New Zealand approach to teaching probability comes from a modelling perspective, right from the start. At level 1, the first two years of schooling, children are exploring chance situations, playing games with a chance element and describing possible outcomes. By years 5 and 6 they are assigning numeric values to the likelihood of an occurrence. They (in the curriculum) are being introduced to model estimates and experimental estimates of probability. Bearing in mind how difficult high school maths teachers are finding the new approach, I don’t have a lot of confidence that the primary teachers are equipped yet to make the philosophical changes, let alone enact them in the classroom.

How to study statistics (Part 1)

To students of statistics

Most of my posts are directed at teachers and how to teach statistics. The blog this week and next is devoted to students. I present principles that will help you to learn statistics. I’m turning them into a poster, which I will make available for you to printing later. I’d love to hear from other teachers as I add to my list of principles.

1. Statistics is learned by doing

One of the best predictors of success in any subject is how much time you spent on it. If you want to learn statistics, you need to put in time. It is good to read the notes and the textbook, and to look up things on the internet and even to watch Youtube videos if they are good ones. But the most important way to learn statistics is by doing. You need to practise at the skills that are needed by a statistician, which include logical thinking, interpretation, judgment and writing. Your teacher should provide you with worthwhile practice activities, and helpful timely feedback. Good textbooks have good practice exercises. On-line materials have many practice exercises.

Given a choice, do the exercises that have answers available. It is very important that you check what you are doing, as it is detrimental to practise something in the wrong way. Or if you are using an on-line resource, make sure you check your answers as you go, so that you gain from the feedback and avoid developing bad habits.

So really the first principle should really be “statistics is learned by doing correctly.

2. Understanding comes with application, not before.

Do not wait until you understand what you are doing before you get started. The understanding comes as you do the work. When we learn to speak, we do not wait until we understand grammatical structure before saying anything. We use what we have to speak and to listen, and as we do so we gain an understanding of how language works.  I have found that students who spent a lot of time working through the process of calculating conditional probabilities for screening tests grew to understand the “why” as well as the “how” of the process. Repeated application of using Excel to fit a line to bivariate data and explaining what it meant, enabled students to understand and internalise what a line means. As I have taught statistics for two decades, my own understanding has continued to grow.

There is a proviso. You need to think about what you are doing, and you need to do worthwhile exercises. For example, mechanically calculating the standard deviation of a set of numbers devoid of context will not help you understand standard deviation. Looking at graphs and trying to guess what the standard deviation is, would be a better exercise. Then applying the value to the context is better still.

Applying statistical principles to a wide variety of contexts helps us to discern what is specific to a problem and what is general for all problems. This brings us to the next principle.

3. Spend time exploring the context.

In a statistical analysis, context is vital, and often very interesting. You need to understand the problem that gave rise to the investigation and collection of the data. The context is what makes each statistical investigation different. Statisticians often work alongside other researchers in areas such as medicine, psychology, biology and geology, who provide the contextual background to the problem. This provides a wonderful opportunity for the statistician to learn about a whole range of different subjects. The interplay between the data and context mean that every investigation is different.

In a classroom setting you will not have the subject expert available, but you do need to understand the story behind the data. These days, finding out is possible with a click of a Google or Wikipedia button. Knowing the background to the data helps you to make more sensible judgments – and it makes it more interesting.

4. Statistics is different from mathematics

In mathematics, particularly pure mathematics, context is stripped away in order to reveal the inner pure truth of numbers and logic.  There are applied areas involving mathematics, which are more like statistics, such as operations research and engineering. At school level, one of the things that characterises the study of maths is right and wrong answers, with a minimum of ambiguity. That is what I loved about mathematics – being able to apply an algorithm and get a correct answer. In statistics, however, things are seldom black-and-white.  In statistics you will need to interpret data from the perspective of the real world, and often the answer is not clear. Some people find the lack of certainty in statistics disturbing. There is considerable room for discussion in statistics. Some aspects of statistics are fuzzy, such as what to do with messy data, or which is the “best” model to fit a time series. There is a greater need for the ability to write in statistics, which makes if more challenging for students for whom English is not their native language.

5. Technology is essential

With computers and calculators, all sorts of activities are available to help learn statistics. Graphs and graphics enable exploration that was not possible when graphs had to be drawn by hand. You can have a multivariate data set and explore all the possible relationships with a few clicks. You should always look at the data in a graphical form before setting out to analyse.

Sometimes I would set optional exercises for students to explore the relationship between data, graphs and summary measures. Very few students did so, but when I led them through the same examples one at a time I could see the lights go on. When you are given opportunities to use computing power to explore and learn – do it!

But wait…there’s more

Here we have the first five principles for students learning statistics. Watch this space next week for some more. And do add some in the comments and I will try to incorporate your ideas as well.

Oh Ordinal data, what do we do with you?

What can you do with ordinal data? Or more to the point, what shouldn’t you do with ordinal data?

First of all, let’s look at what ordinal data is.

It is usual in statistics and other sciences to classify types of data in a number of ways. In 1946, Stanley Smith Stevens suggested a theory of levels of measurement, in which all measurements are classified into four categories, Nominal, Ordinal, Interval and Ratio. This categorisation is used extensively, and I have a popular video explaining them. (Though I group Interval and Ratio together as there is not much difference in their behaviour for most statistical analysis.)

Costing no more than a box of popcorn, our snack-size course will help help you learn all you need to know about types of data.

Costing no more than a box of popcorn, our snack-size course will help help you learn all you need to know about types of data, and appropriate statistics and graphs.

Nominal is pretty straight-forward. This category includes any data that is put into groups, in which there is no inherent order. Examples of nominal data are country of origin, sex, type of cake, or sport. Similarly it is pretty easy to explain interval/ratio data. It is something that is measured, by length, weight, time (duration), cost and similar. These two categorisations can also be given as qualitative and quantitative, or non-parametric and parametric.

Ordinal data

But then we come to ordinal level of measurement. This is used to describe data that has a sense of order, but for which we cannot be sure that the distances between the consecutive values are equal. For example, level of qualification has a sense of order

  • A postgraduate degree is higher than
  • a Bachelor’s degree,which is higher than
  • a high-school qualification, which is higher
  • than no qualification.

There are four steps on the scale, and it is clear that there is a logical sense of order. However, we cannot sensibly say that the difference between no qualification and a high-school qualification is equivalent to the difference between the high-school qualification and a bachelor’s degree, even though both of those are represented by one step up the scale.

Another example of ordinal level of measurement is used extensively in psychological, educational and marketing research, known as a Likert scale. (Though I believe the correct term is actually Likert item – and according to Wikipedia, the pronunciation should be Lick it, not Like it, as I have used for some decades!). A statement is given, and the response is given as a value, often from 1 to 5, showing agreement to the statement. Often the words “Strongly agree, agree, neutral, disagree, strongly disagree” are used. There is clearly an order in the five possible responses. Sometimes a seven point scale is used, and sometimes the “neutral” response is eliminated in an attempt to force the respondent to commit one way or the other.

The question at the start of this post has an ordinal response, which could be perceived as indicating how quantitative the respondent believes ordinal data to be.

What prompted this post was a question from Nancy under the YouTube video above, asking:

“Dr Nic could you please clarify which kinds of statistical techniques can be applied to ordinal data (e.g. Likert-scale). Is it true that only non-parametric statistics are possible to apply?”


As shown in the video, there are the purists, who are adamant that ordinal data is qualitative. There is no way that a mean should ever be calculated for ordinal, data, and the most mathematical thing you can do with it is find the median. At the other pole are the practical types, who happily calculate means for any ordinal data, without any concern for the meaning (no pun intended.)

There are differing views on finding the mean for ordinal data.

There are differing views on finding the mean for ordinal data.

So the answer to Nancy would depend on what school of thought you belong to.

Here’s what I think:

All ordinal data is not the same. There is a continuum of “ordinality” if you like.

There are some instances of ordinal data which are pretty much nominal, with a little bit of order thrown in. These should be distinguished from nominal data, only in that they should always be graphed as a bar chart (rather than a pie-chart)* because there is inherent order. The mode is probably the only sensible summary value other than frequencies. In the examples above, I would say that “level of qualification” is only barely ordinal. I would not support calculating a mean for the level of qualification. It is clear that the gaps are not equal, and additionally any non-integer result would have doubtful interpretation.

Then there are other instances of ordinal data for which it is reasonable to treat it as interval data and calculate the mean and median. It might even be supportable to use it in a correlation or regression. This should always be done with caution, and an awareness that the intervals are not equal.

Here is an example for which I believe it is acceptable to use the mean of an ordinal scale. At the beginning and the end of a university statistics course, the class of 200 students is asked the following question: How useful do you think a knowledge of statistics is will be to you in your future career? Very useful, useful, not useful.

Now this is not even a very good Likert question, as the positive and negative elements are not balanced. There are only three choices. There is no evidence that the gaps between the elements are equal. However if we score the elements as 3,2 and 1, respectively and find that the mean for the 200 students is 1.5 before the course, and 2.5 after the course, I would say that there is meaning in what we are reporting. There are specific tests to use for this – and we could also look at how many students changed their minds positively or negatively. But even without the specific test, we are treating this ordinal data as something more than qualitative. What also strengthens the evidence for doing this is that the test is performed on the same students, who will probably perceive the scale in the same way each time, making the comparison more valid.

So what I’m saying is that it is wrong to make a blanket statement that ordinal data can or can’t be treated like interval data. It depends on meaning and number of elements in the scale.

What do we teach?

And again the answer is that it depends! For my classes in business statistics I told them that it depends. If you are teaching a mathematical statistics class, then a more hard line approach is justified. However, at the same time as saying, “you should never calculate the mean of ordinal data”, it would be worthwhile to point out that it is done all the time! Similarly if you teach that it is okay to find the mean of some ordinal data, I would also point out that there are issues with regard to interpretation and mathematical correctness.

Please comment!

Foot note on Pie charts

*Yes, I too eschew pie-charts, but for two or three categories of nominal data, where there are marked differences in frequency, if you really insist, I guess you could possibly use them, so long as they are not 3D and definitely not exploding. But even then, a barchart is better. – perhaps a post for another day, but so many have done this.

Probability and Deity

Our perception of chance affects our worldview

There are many reasons that I am glad that I majored in Operations Research rather than mathematics or statistics. My view of the world has been affected by the OR way of thinking, which combines hard and soft aspects. Hard aspects are the mathematics and the models, the stuff of the abstract world. Soft aspects relate to people and the reality of the concrete world.  It is interesting that concrete is soft! Operations Research uses a combination of approaches to aid in decision making.

My mentor was Hans Daellenbach, who was born and grew up in Switzerland, did his postgrad study in California, and then stepped back in time several decades to make his home in Christchurch, New Zealand. Hans was ahead of his time in so many ways. The way I am writing about today was his teaching about probability and our subjective views on the likelihood of events.

Thanks to Daniel Kahneman’s publishing and 2002 Nobel prize, the work by him and Amos Tversky is reaching into the popular realm and is even in the high school mathematics curriculum, in a diluted form. Hans Daellenbach required first year students to read a paper by Tversky and Kahneman in the late 1980’s, over a decade earlier. This was not popular, either with the students or the tutors who were trying to make sense of the paper. Eventually we made up some interesting exercises in tutorials, and structured the approach enough for students to catch on. (Sometimes nearly half our students were from a non-English speaking background, so reading the paper was extremely challenging for them.) As a tutor and later a lecturer, I internalised the thinking, and it changed the way I see the world and chance.

People’s understanding of probability and chance events has an impact on how they see the world as well as the decisions they make.

For example, Kahneman introduced the idea of the availability heuristic. This means that if someone we know has been affected by a catastrophic (or wonderful) unlikely event, we will perceive the possibility of such an event as more likely. For example if someone we know has had their house broken into, then we feel less secure, as we perceive the likelihood of that as increased.  Someone we know wins the lottery, and suddenly it seems possible for us. Nothing has changed in the world, but our perception has changed.

There is another easily understood concept of confirmation bias. We notice and remember events and sequences of events that reinforce or confirm our preconceived notions. “Bad things come in threes” is a wonderful example. Something bad or two things bad happen, so we look for or wait for the third, and then stop counting. Similarly we remember the times when our lucky number is lucky, and do not remember the unlucky times. We mentally record the times our hunches pay off, and quietly forget the times they don’t.

So how does this affect us as teachers of statistics? Are there ethical issues involved in how we teach statistics?

I believe in God and I believe that He guides me in my decisions in life. However I do not perceive God as a “micro-manager”. I do not believe that he has time in his day to help me to find carparks, and to send me to bargains in the supermarket. I may be wrong, and I am prepared to be proven wrong, but this is my current belief. There are many people who believe in God (or in that victim-blaming book, “The Secret”), who would disagree with me. When they see good things happen, they attribute them to the hand of God, or karma or The Secret.  There are people in some cultures who do not believe in chance at all. Everything occurs as God’s will, hence the phrase, “ insha’Allah”, or “God willing”. If they are delayed in traffic, or run into a friend, or lose their job, it is because God willed it so. This is undoubtedly a simplistic explanation, but you get the idea.

Now along comes the statistics teacher and teaches probability.  Mathematically there are some things for which the probability is easily modelled. Dice, cards, counters, balls in urns, socks in drawers can all have their probability modelled, using the ratio of number of chosen events over number of possible events. There are also probabilities estimated using historic frequencies, and there are subjective estimates of probabilities. Tversky and Kahnemann’s work showed how flawed humans are at the subjective estimates.

For some (most?) students probability remains “school-knowledge” and makes no difference to their behaviour and view of the world. It is easy to see this on game-shows such as “Deal or No Deal”, my autistic son’s favourite. It is clear that except for the decision to take the deal or not, there is no skill whatsoever in this game. In the Australian version, members of the audience hold the different cases and can guess what their case holds. If they get it right they win $500. When this happens they are praised – well done! When the main player is choosing cases, he or she is warned that they will need to be careful to avoid the high value cases. This is clearly impossible, as there is no way of knowing which cases contain which values. Yet they are praised, “Well done!” for cases that contain low values. Sometimes they even ask the audience members what they think they are holding in the case. This makes for entertaining television – with loud shouting at times to “Take the Deal!”. But it doesn’t imbue me with any confidence that people understand probability.

Having said that, I know that I act irrationally as well. In the 1990s there were toys called Tamagotchis which were electronic pets. To keep your pet happy you had to “play” with it, which involved guessing which way the pet would turn. I KNEW that it made NO difference which way I chose and that I would do just as well by always choosing the same direction. Yet when the pet had turned to the left four times in succession, I would choose turning to the right. Assuming a good random number generator in the pet, this was pointless. But it also didn’t matter!

So if I, who have a fairly sound understanding of probability distributions and chance, still think about which way my tamagotchi is going to turn, I suspect truly rational behaviour in the general populace with regard to probabilistic events is a long time coming! Astrologers, casinos, weather forecasters, economists, lotteries and the like will never go broke.

However there are other students for whom a better understanding of the human tendency to find patterns, and confirm beliefs could provide a challenge. Their parents may strongly believe that God intervenes often or that there is no uncertainty, only lack of knowledge. (In a way this is true, but that’s a topic for another day) Like the child who has just discovered the real source of Christmas bounty, probability models are something to ponder, and can be disturbing.

We do need to be sensitive in how we teach probability. Not only can we shake people’s beliefs, but we can also use insensitive examples. I used to joke about how car accidents are a poisson process with batching, which leads to a very irregular series. Then for the last two and a half years I have been affected by the Christchurch earthquakes.  I have no sense of humour when it comes to earthquakes. None of us do. When I saw in a textbook an example of probability a building falling down as a result of an earthquake, I found that upsetting. A friend was in such a building and, though she physically survived it will be a long time before she will have a full recovery, if ever. Since then I have never used earthquakes as an example of a probabilistic event when teaching in Christchurch. I also refrain as far as possible from using other examples that may stir up pain, or try to treat them in a sober manner. Breast cancer, car accidents and tornadoes kill people and may well have affected our pupils. Just a thought.

Judgment Calls in Statistics and O.R.

The one-armed operations researcher

My mentor, Hans Daellenbach told me a story about a client asking for a one-armed Operations Researcher. The client was sick of getting answers that went, “On the one hand, the best decision would be to proceed, but on the other hand…”

People like the correct answer. They like certainty. They like to know they got it right.

I tease my husband that he has to find the best picnic spot or the best parking place, which involves us driving around considerably longer than I (or the children) were happy with. To be fair, we do end up in very nice picnic spots. However, several of the other places would have been just fine too!

In a different context I too am guilty of this – the reason I loved mathematics at school was because you knew whether you were right or wrong and could get a satisfying row of little red ticks (checkmarks) down the page. English and other arts subjects, I found too mushy as you could never get it perfect. Biology was annoying as plants were so variable, except in their ability to die. Chemistry was ok, so long as we stuck to the nice definite stuff like drawing organic molecules and balancing redox equations.

I think most mathematics teachers are mathematics teachers because they like things to be right or wrong. They like to be able to look at an answer and tell whether it is correct, or if it should get half marks for correct working. They do NOT want to mark essays, which are full of mushy judgements.

Again I am sympathetic. I once did a course in basketball refereeing. I enjoyed learning all the rules, and where to stand, and the hand signals etc, but I hated being a referee. All those decisions were just too much for me. I could never tell who had put the ball out, and was unhappy with guessing. I think I did referee two games at a church league and ended up with an angry player bashing me in the face with the ball. Looking back I think it didn’t help that I wasn’t much of a player either.

I also used to find marking exam papers very challenging, as I wanted to get it right every time. I would agonise over every mark, thinking it could be the difference between passing and failing for some poor student. However as the years went by, I realised that the odd mistake or inconsistency here or there was just usual, and within the range of error. To someone who failed by one mark, my suggestion is not to be borderline. I’m pretty sure we passed more people that we shouldn’t have, than the other way around.

Life is not deterministic

The point is, that life in general is not deterministic and certain and rule-based. This is where the great divide lies between the subject of mathematics and the practice of statistics. Generally in mathematics you can find an answer and even check that it is correct. Or you can show that there is no answer (as happened in one of our national exams in 2012!). But often in statistics there is no clear answer. Sometimes it even depends on the context. This does not sit well with some mathematics teachers.

In operations research there is an interesting tension between optimisers and people who use heuristics. Optimisers love to say that they have the optimal solution to the problem. The non-optimisers like to point out that the problem solved optimally, is so far removed from the actual problem, that all it provides is an upper or lower bound to a practical solution to the actual real-life problem situation.

Judgment calls occur all through the mathematical decision sciences. They include

  • What method to use – Linear programming or heuristic search?
  • Approximations – How do we model a stochastic input in a deterministic model?
  • Assumptions – Is it reasonable to assume that the observations are independent?
  • P-value cutoff – Does a p-value of exactly 0.05 constitute evidence against the null hypothesis?
  • Sample size – Is it reasonable to draw any inferences at all from a sample of 6?
  • Grouping – How do we group by age? by income?
  • Data cleaning – Do we remove the outlier or leave it in?

A comment from a maths teacher on my post regarding the Central Limit Theorem included the following: “The questions that continue to irk me are i) how do you know when to make the call? ii) What are the errors involved in making such a call? I suppose that Hypothesis testing along with p-values took care of such issues and offered some form of security in accepting or rejecting such a hypothesis. I am just a little worried that objectivity is being lost, with personal interpretation being the prevailing arbiter which seems inadequate.”

These are very real concerns, and reflect the mathematical desire for correctness and security. But I propose that the security was an illusion in the first place. There has always been personal interpretation.Informal inference is a nice introduction to help us understand that. And in fact it would be a good opportunity for lively discussion in a statistics class.

With bootstrapping methods we don’t have any less information than we did using the Central Limit Theorem. We just haven’t assumed normality or independence. There was no security. There was the idea that with a 95% confidence interval, for example, we are 95% sure that we contain the true population value. I wonder how often we realised that 1 in 20 times we were just plain wrong, and in quite a few instances the population parameter would be far from the centre of the interval.

The hopeful thing about teaching statistics via bootstrapping, is that by demystifying it we may be able to inject some more healthy scepticism into the populace.

The Central Limit Theorem: To teach or not to teach

The question of whether to teach explicitly the Central Limit Theorem seems to divide instructors along philosophical lines. Let us look first at these lines.

There are at least three different areas of activity within the discipline of statistics. These are

  • Theory of statistics and research into statistics
  • Practice of statistics
  • Teaching statistics and related research

Theory and research in statistics

The theory of statistics is mathematical. It is taught and practised in Mathematics and Statistics Departments of Universities. It is possible to be an expert on the theory and mathematics of statistics while having little contact with real data. The theory provides underpinnings to the practice of statistics. It is vital that some people know this – but not most of us. One would hope that people employed as statisticians would have a sound understanding of both the theoretical and applied aspects of statistics. This relates strongly to the research into statistics, which seems to be very mathematical, from my perusal of journals. This research advances the theory and use of statistical methods and philosophy.

Practice of statistics

The practice of statistics occurs in many, many areas, particularly in universities. Most postgraduate courses require some proficiency in the application of statistical methods. Researchers in areas as diverse as psychology, genetics, market research, education, geography, speech therapy, physiotherapy, mechanics, management, economics and medicine all use statistical methods. Some researchers have a deep understanding of the theory of statistics, but most aim to be safe and competent practitioners. When they get to the tricky bits they know to ask a statistician, but most of the day-to-day data generation, collection and analysis is within their capability.

Teaching of statistics and related research

Then there is the teaching of statistics. The level of applicability and theory taught will depend on the context. An instructor in statistics (in a non-service course) in a Department of Mathematics would tend towards the mathematical aspects, as that is most appropriate to the audience. However in just about every other setting the emphasis will be on the practical aspects of data collection and inference. This treatment of statistics is explicable, accessible and interesting to just about anyone, whereas only the mathematically inclined are likely to get excited about the theory of statistics.

There is another growing area, which is the research into the teaching and learning of statistics. This informs and is informed by the other areas, as well as general educational research and cognitive psychology. Much of my thinking comes from this background. An overview of some of the material relating to college level can be found in this literature review. The general topic of How Students Learn Statistics is introduced in this early paper by Joan Garfield (1995), a leader in the field of statistics education research.

Statistics in the school curriculum

Statistics is gradually making its way into the school curriculum internationally, and in New Zealand has become a separate subject in the final year of schooling. There are philosophical issues arising as most of the teachers of statistics are mathematicians, and some tend towards the beauty and elegance of the formulas, proofs etc. The aim of the curriculum, however, is more towards statistical investigations and statistical literacy. There are fuzzy, dirty, ambiguous, context driven explorations with sometimes extensive write-ups. There is discussion and critique of statistical reports. There are experiments which may or may not produce usable results. Some of this is well into the realms of social science and well away from what mathematicians find appealing or even comfortable. In another life I can hear myself saying, “I didn’t become a maths teacher to mark essay questions!” There is a bit of a mismatch between the skill-set and attitudes of the teachers and the curriculum.

Teaching the Central Limit Theorem

One place where this is particularly evident is in the question of teaching the Central Limit Theorem. Mathematicians like the Central Limit Theorem and it seems that they like to teach it. One teacher states “The fact that the CLT is to be de-emphasised in Yr 13 is a major disappointment to me…” This statement prompted this post. I agree that the CLT is neat. It is really handy. And it makes confidence interval calculation almost trivial. There are cool little exercises you can do to illustrate it. It is the backbone of traditional statistical theory.

However, teaching and learning do not always go hand in hand. I wonder how many students really do internalise the Central Limit Theorem. Evidence says not many. Chance, Delmas and Garfield, in “The challenge of developing statistical literacy reasoning and thinking” (Ben Zvi and Garfield 2004) state: “Sampling distributions is a difficult topic for students to learn. A complete understanding of sampling distributions requires students to integrate and apply several concepts from different parts of a statistics course and to be able to reason about the hypothetical behavior of many samples – a distinct, intangible thought process for most students. The Central Limit Theorem provides a theoretical model of the behavior of sampling distributions, but students often have difficulty mapping this model to applied contexts. As a result students fail to develop a deep understanding of the concept of sampling distributions and therefore often develop only a mechanical knowledge of statistical inference. Students may learn how to compute confidence intervals and carry out tests of significance, but the are not able to understand and explain related concepts, such as interpreting a p-value.”

I have a confession to make. I didn’t teach the Central Limit Theorem. It never seemed as if it were going to help my students understand what was going on. For a few years I made them do a little simulation exercise which helped them to see why the square-root of n occurred in the denominator of the formula for the standard error. That was fun and seemed to help. But the words “Central Limit Theorem” seldom passed my lips in my twenty years of instruction.

What has helped immeasurably have been videos, beginning with “Understanding the p-value” and plenty of different examples and exercises using confidence intervals and hypothesis tests. (Another confession – I taught traditional statistical inference, not resampling. My excuse was that I didn’t know any better, and I had to stay in parallel with the course provided by the maths department.) What I have found from my own experience as a learner and as a teacher is that students learn to understand statistics by DOING statistics.

Definition of the Central Limit Theorem

The Central Limit Theorem states that regardless of the shape of the population distribution, the distribution of sample means is normal if the sample size is large. This was a really brilliant model for when simulation and resampling was impossible. The Central Limit Theorem makes it possible to calculate confidence intervals for population means from sample data. It is the reason why most statistical procedures either assume normality at some point, or take steps to correct for the lack thereof. (See the paper by Cobb I referred to extensively in last week’s post.)

In a curriculum that develops from informal inference to formal inference using resampling, there is no need to call on the Central Limit Theorem. With resampling we use the distribution of the sample as the best estimate of the distribution of the population. True, it is quicker to use the old method of plug the values in the formula. However it isn’t much quicker than using the free iNZight software for resampling.

At high school level we want students to get an understanding of what inference is. (I would suggest my Pinkie Bar lesson as a good way of introducing the rejection part of Cobbs mantra, Randomise, Repeat, Reject.) I’m not convinced that teaching the Central Limit Theorem, and formula-based Confidence intervals for means and proportions lead to understanding. Research suggests that it doesn’t. I agree that statistical theorists, and educators and researchers should all understand the Central Limit Theorem. I just don’t think that it has a vital place in an innovative curriculum based on resampling.

Concern for students

I suspect that teachers fear that if their students are not taught the Central Limit Theorem and traditional confidence intervals at high school they will be at a disadvantage at university. I’d like to reassure them that it just isn’t true. All first year university statistics courses that I know of assume no prior knowledge of statistics. (The same is true of some second year courses as well!) The greatest gift a high school statistics teacher can give their students is an attitude of excitement and success, with a healthy helping of scepticism, and an idea of what inference is – that we can draw conclusions about a population from a sample. If my first year students had started from that point, half our work would have been done.