About Dr Nic

I love to teach just about anything. My specialties are statistics and operations research. I have insider knowledge on Autism through my family. I have a lovely husband, two grown-up sons, a fabulous daughter-in-law and a new adorable grandson. I have four blogs - Learn and Teach Statistics, Never Ordinary Life, Chch Relief Society and StatsLC News.

Proving causation

Aeroplanes cause hot weather

In Christchurch we have a weather phenomenon known as the “Nor-wester”, which is a warm dry wind, preceding a cold southerly change. When the wind is from this direction, aeroplanes make their approach to the airport over the city. Our university is close to the airport in the direct flightpath, so we are very aware of the planes. A new colleague from South Africa drew the amusing conclusion that the unusual heat of the day was caused by all the planes flying overhead.

Statistics experts and educators spend a lot of time refuting claims of causation. “Correlation does not imply causation” has become a catch cry of people trying to avoid the common trap. This is a great advance in understanding that even journalists (notoriously math-phobic) seem to have caught onto. My own video on important statistical concepts ends with the causation issue. (You can jump to it at 3:51)

So we are aware that it is not easy to prove causation.

In order to prove causation we need a randomised experiment. We need to make random any possible factor that could be associated, and thus cause or contribute to the effect. This next video, about experimental design, addresses this concept. It is possible to prove that one factor causes an effect by using randomised design.

There is also the related problem of generalizability. If we do have a randomised experiment, we can prove causation. But unless the sample is also a random representative sample of the population in question, we cannot infer that the results will also transfer to the population in question. This is nicely illustrated in this matrix from The Statistical Sleuth by Fred L. Ramsey and Daniel W Schafer.

The relationship between the type of sample and study and the conclusions that may be drawn.

The relationship between the type of sample and study and the conclusions that may be drawn.

The top left-hand quadrant is the one in which we can draw causal inferences for the population.

Causal claims from observational studies

A student posed this question:  Is it possible to prove a causal link based on an observational study alone?

It would be very useful if we could. It is not always possible to use a randomised trial, particularly when people are involved. Before we became more aware of human rights, experiments were performed on unsuspecting human lab rats. A classic example is the Vipeholm experiments where patients at a mental hospital were the unknowing subjects. They were given large quantities of sweets in order to determine whether sugar caused cavities in teeth. This happened into the early 1950s. These days it would not be acceptable to randomly assign people to groups who are made to smoke or drink alcohol or consume large quantities of fat-laden pastries. We have to let people make those lifestyle choices for themselves. And observe. Hence observational studies!

There is a call for “evidence-based practice” in education to follow the philosophy in medicine. But getting educational experiments through ethics committee approval is very challenging, and it is difficult to use rats or fruit-flies to impersonate the higher learning processes of humans. The changing landscape of the human environment makes it even more difficult to perform educational experiments.

To find out the criteria for justifying causal claims in an observational study I turned to one of my favourite statistics text-books, Chance Encounters by Wild and Seber  (page 27). They cite the Surgeon General of the United States. The criteria for the establishment of a cause and effect relationship in an epidemiological study are the following:

  1. Strong relationship: For example illness is four times as likely among people exposed to a possible cause as it is for those who are not exposed.
  2. Strong research design
  3. Temporal relationship: The cause must precede the effect.
  4. Dose-response relationship: Higher exposure leads to a higher proportion of people affected.
  5. Reversible association: Removal of the cause reduces the incidence of the effect.
  6. Consistency: Multiple studies in different locations producing similar effects
  7. Biological plausibility: there is a supportable biological mechanism
  8. Coherence with known facts.

Teaching about causation

In high school, and entry-level statistics courses, the focus is often on statistical literacy. This concept of causation is pivotal to correct understanding of what statistics can and cannot claim. It is worth spending some time in the classroom discussing what would constitute reasonable proof and what would not. In particular it is worthwhile to come up with alternative explanations for common fallacies, or even truths in causation. Some examples for discussion might be drink-driving and accidents, smoking and cancer, gender and success in all number of areas, home game advantage in sport, the use of lucky charms, socks and undies. This also ties nicely with probability theory, helping to tie the year’s curriculum together.

Absolute and Relative Risk

It is important that citizens can make sense out of the often outrageous claims of advertisers and pro-screening advocates.  It isn’t what they say, but how they say it. What looks like a very large and scary increase in risk, can in fact make very little practical difference. Conversely a large risk can be made to look smaller through the manner in which it is communicated.

I found a wonderful set of notes on the Census at School site, presented as a powerpoint file.

I also found several very interesting and educational sites about risk.

This first one explains about risk and relative risk: Science blog on Cancer Research UK

This one also includes Number needed to treat. Patient Health UK.

And a here is a great summary and set of exercises at the Auckland Maths Association website. You need to scroll down to “Relative Risk Resources”. (I found this after writing the rest of the blog, and it pretty much says what I say, but more succinctly!)

Teaching about Risk

Risk is a great topic for teaching about probability, percentages and perception.

It’s what’s on the bottom that counts!

In exploring risk, there are several distinct processes needed. Depending on the format in which the information is given, students may need to construct their own frequency table, or interpret the one provided. From the frequency table they must calculate the probability, making sure that they choose the correct denominator. Then if they are looking for relative risk, they need to make sure that they again choose the correct denominator. For some reason, the numerator is usually easier. But what can be tricky is the denominator.

We can use as an example the increase in probability of passing a particular statistics course if students use our Statistics Learning Centre materials to help them. We haven’t collected any data yet, so these figures are aspirational (as in a work of fiction!). Because we are talking about risk, we have to frame the outcome in negative terms. We would not talk about the risk of passing a course, but rather of failing one. So we will say that students who use StatsLC materials reduce their risk of failing by 66.7% percent. That is pretty impressive, but how much better it sounds if we frame it in terms of how much their risk will increase if they decide not to use the wonderful materials from StatsLC. Their risk of failure increases by 200%. That sounds pretty drastic.

But what we have failed to mention is the absolute risk, which is the proportion of students who fail their stats courses with and without the help of StatsLC. Here are some pairs of absolute risks that will give the results given:

All of the following sets of numbers show a 200% increase in risk of failure for students who do not use StatsLC materials.


Risk of failing, when using StatsLC materials

Risk of failing when they don’t use StatsLC materials

Actual increase in risk of failing.













In Scenario A, the pass-rate for the statistics course has gone from 97% to 99%. In scenario B, the pass-rate has gone from 70% to 90%, and in Scenario C, the pass-rate has gone from 40% to 80%. All of these scenarios could accurately be described by the same change in relative risk. They all double the risk of failing if the student does not use StatsLC.

This is really at the end of the story, based on what is reported. But if we wish to find out what is really going on, the best idea is to build a table of natural frequencies. These are great for calculating conditional probabilities by stealth.

Here is a table of natural frequencies for Scenario C above, using 1000 as our total number of people. Before we fill it out, we also need to know how many people used Statistics Learning Centre materials. 30% of students did NOT use StatsLC materials.



Total in category

Use StatsLC

80% of 700 = 560

20% of 700 = 140


Do not use StatsLC

40% of 300 = 120

60% of 300 = 180


Total pass or fail




From this table, all manner of statistics can be computed.

What proportion of students who passed, used the StatsLC materials?

The answer is (the number of people who passed AND used StatsLC materials)/( the number of people who passed) = 560/680 =82%. It is important to find the correct denominator.

Then when people calculate relative risk, it is important to be careful about choosing the baseline.

Another question might be, by how much does your risk of failure decrease, in relative terms, if you use the StatsLC materials?

The first step is to find the decrease in absolute terms. The risk of failure, not using StatsLC = 0.6. The risk of failure when using StatsLC has decreased to 0.2. That is an absolute decrease in risk of 0.4. Then we need to express this relative to the baseline. As we talked about the decrease in risk, it will be compared with the larger number, or 0.6, the risk of failing when using the StatsLC materials. So 0.4/0.6 = 0.667 or 66.7%. However, if we were talking about the increase in risk for NOT using StatsLC materials, then we would find 0.4/0.2 = 200%.

A great way to develop interaction and group discussion would be to give individuals in the group different information that is needed for the computation. Later on you could include one wrong “fact”, which they would need to ferret out. Another possibility would be to give students information about different scenarios that they need to present in the best or worst possible light.

These are great teaching opportunities, and worthwhile for everyday life.  It is a good thing they have been included in the NZ curriculum for year 12.

A note to regular readers – I will probably be posting less frequently for a while, but feel free to read back over some of my previous 95 posts if you miss the weekly rant. ;)

Those who can, teach statistics

The phrase I despise more than any in popular use (and believe me there are many contenders) is “Those who can, do, and those who can’t, teach.” I like many of the sayings of George Bernard Shaw, but this one is dismissive, and ignorant and born of jealousy. To me, the ability to teach something is a step higher than being able to do it. The PhD, the highest qualification in academia, is a doctorate. The word “doctor” comes from the Latin word for teacher.

Teaching is a noble profession, on which all other noble professions rest. Teachers are generally motivated by altruism, and often go well beyond the requirements of their job-description to help students. Teachers are derided for their lack of importance, and the easiness of their job. Yet at the same time teachers are expected to undo the ills of society. Everyone “knows” what teachers should do better. Teachers are judged on their output, as if they were the only factor in the mix. Yet how many people really believe their success or failure is due only to the efforts of their teacher?

For some people, teaching comes naturally. But even then, there is the need for pedagogical content knowledge. Teaching is not a generic skill that transfers seamlessly between disciplines. You must be a thinker to be a good teacher. It is not enough to perpetuate the methods you were taught with. Reflection is a necessary part of developing as a teacher. I wrote in an earlier post, “You’re teaching it wrong”, about the process of reflection. Teachers need to know their material, and keep up-to-date with ways of teaching it. They need to be aware of ways that students will have difficulties. Teachers, by sharing ideas and research, can be part of a communal endeavour to increase both content knowledge and pedagogical content knowledge.

There is a difference between being an explainer and being a teacher. Sal Khan, maker of the Khan Academy videos, is a very good explainer. Consequently many students who view the videos are happy that elements of maths and physics that they couldn’t do, have been explained in such a way that they can solve homework problems. This is great. Explaining is an important element in teaching. My own videos aim to explain in such a way that students make sense of difficult concepts, though some videos also illustrate procedure.

Teaching is much more than explaining. Teaching includes awakening a desire to learn and providing the experiences that will help a student to learn.  In these days of ever-expanding knowledge, a content-driven approach to learning and teaching will not serve our citizens well in the long run. Students need to be empowered to seek learning, to criticize, to integrate their knowledge with their life experiences. Learning should be a transformative experience. For this to take place, the teachers need to employ a variety of learner-focussed approaches, as well as explaining.

It cracks me up, the way sugary cereals are advertised as “part of a healthy breakfast”. It isn’t exactly lying, but the healthy breakfast would do pretty well without the sugar-filled cereal. Explanations really are part of a good learning experience, but need to be complemented by discussion, participation, practice and critique.  Explanations are like porridge – healthy, but not a complete breakfast on their own.

Why statistics is so hard to teach

“I’m taking statistics in college next year, and I can’t wait!” said nobody ever!

Not many people actually want to study statistics. Fortunately many people have no choice but to study statistics, as they need it. How much nicer it would be to think that people were studying your subject because they wanted to, rather than because it is necessary for psychology/medicine/biology etc.

In New Zealand, with the changed school curriculum that gives greater focus to statistics, there is a possibility that one day students will be excited to study stats. I am impressed at the way so many teachers have embraced the changed curriculum, despite limited resources, and late changes to assessment specifications. In a few years as teachers become more familiar with and start to specialise in statistics, the change will really take hold, and the rest of the world will watch in awe.

In the meantime, though, let us look at why statistics is difficult to teach.

  1. Students generally take statistics out of necessity.
  2. Statistics is a mixture of quantitative and communication skills.
  3. It is not clear which are right and wrong answers.
  4. Statistical terminology is both vague and specific.
  5. It is difficult to get good resources, using real data in meaningful contexts.
  6. One of the basic procedures, hypothesis testing, is counter-intuitive.
  7. Because the teaching of statistics is comparatively recent, there is little developed pedagogical content knowledge. (Though this is growing)
  8. Technology is forever advancing, requiring regular updating of materials and teaching approaches.

On the other hand, statistics is also a fantastic subject to teach.

  1. Statistics is immediately applicable to life.
  2. It links in with interesting and diverse contexts, including subjects students themselves take.
  3. Studying statistics enables class discussion and debate.
  4. Statistics is necessary and does good.
  5. The study of data and chance can change the way people see the world.
  6. Technlogical advances have put the power for real statistical analysis into the hands of students.
  7. Because the teaching of statistics is new, individuals can make a difference in the way statistics is viewed and taught.

I love to teach. These days many of my students are scattered over the world, watching my videos (for free) on YouTube. It warms my heart when they thank me for making something clear, that had been confusing. I realise that my efforts are small compared to what their teacher is doing, but it is great to be a part of it.

On-line learning and teaching resources

Twenty-first century Junior Woodchuck Guidebook

I grew up reading Donald Duck comics. I love the Junior Woodchucks, and their Junior Woodchuck Guidebook. The Guidebook is a small paperback book, containing information on every conceivable subject, including geography, mythology, history, literature and the Rubaiyat of Omar Khayyam.  In our family, when we want to know something or check some piece of information, we talk about consulting the Junior Woodchuck Guidebook. (Imagine my joy when I discovered that a woodchuck is another name for a groundhog, the star of my favourite movie!) What we are referring to is the internet, the source of all possible information! Thanks to search engines, there is very little we cannot find out on the internet. And very big thanks to Wikipedia, to which I make an annual financial contribution, as should all who use it and can afford to.

You can learn just about anything on the internet. Problem is, how do you know what is good? And how do you help students find good stuff? And how do you use the internet wisely? And how can it help us as learners and teachers of statistics and operations research? These questions will take more than my usual 1000 words, so I will break it up a bit. This post is about the ways the internet can help in teaching and learning. In a later post I will talk about evaluating resources, and in particular multimedia resources.


Both the disciplines in which I am interested, statistics and operations research, apply mathematical and analytic methods to real-world problems. In statistics we are generally trying to find things out, and in operations research we are trying to make them better. Either way, the context is important. The internet enables students to find background knowledge regarding the context of the data or problem they are dealing with. It also enables instructors to write assessments and exercises that have a degree of veracity to them even if the actual raw data proves elusive. How I wish people would publish standard deviations as well as means when reporting results!


Which brings us to the second use for on-line resources. Real problems with real data are much more meaningful for students, and totally possible now that we don’t need to calculate anything by hand. Sadly, it is more difficult than first appears to find good quality raw data to analyse, but there is some available. You can see some sources in a previous post and the helpful comments.


If you are struggling to understand a concept, or to know how to teach or explain it, do a web search. I have found some great explanations, and diagrams especially, that have helped me. Or I have discovered a dearth of good diagrams, which has prompted me to make my own.


Videos can help with background knowledge, with explanations, and with inspiring students with the worth of the discipline. The problem with videos is that it takes a long time to find good ones and weed out the others. One suggestion is to enlist the help of your students. They can each watch two or three videos and decide which are the most helpful. The teacher then watches the most popular ones to check for pedagogical value. It is great when you find a site that you can trust, but even then you can’t guarantee the approach will be compatible with your own.

Social support

I particularly love Twitter, from which I get connection with other teachers and learners, and ideas and links to blogs. I belong to a Facebook group for teachers of statistics in New Zealand, and another Facebook group called “I love Operations Research”. These wax and wane in activity, and can be very helpful at times. Students and teachers can gain a lot from social networking.


There is good open-source software available, and 30-day trial versions for other software. Many schools in New Zealand use the R-based iNZight collection of programmes, which provide purpose-built means for timeseries analysis, bootstrapping and line fitting.

Answers to questions

The other day I lost the volume control off my toolbar. (Windows Vista, I’m embarrassed to admit). So I put in the search box “Lost my volume control” and was directed to a YouTube video that took me step-by-step through the convoluted process of reinstating my volume control! I was so grateful I made a donation. Just about any computer related question can be answered through a search.

Interactive demonstrations

I love these. There are two sites I have found great:

The National Library of Virtual Manipulatives, based in Utah.

NRich – It has some great ideas in the senior statistics area. From the UK.

A problem with some of these is the use of Flash, which does not play on all devices. And again – how do we decide if they are any good or not?

On-line textbooks

Why would you buy a textbook when you can get one on-line. I routinely directed my second-year statistical methods for business students to “Concepts and Applications of Inferential Statistics”. I’ve found it just the right level. Another source is Stattrek. I particularly like their short explanations of the different probability distributions.

Practice quizzes

There aren’t too many practice quizzes  around for free. Obviously, as a provider of statistical learning materials, I believe quizzes and exercises have merit for practice with immediate and focussed feedback. However, it can be very time-consuming to evaluate practice quizzes, and some just aren’t very good. On the other hand, some may argue that any time students spend learning is better than none.

Live help

There are some places that provide live, or slightly delayed help for students. I got hooked into a very fun site where you earned points by helping students. Sadly I can’t find it now, but as I was looking I found vast numbers of on-line help sites, often associated with public libraries. And there are commercial sites that provide some free help as an intro to their services. In New Zealand there is the StudyIt service, which helps students preparing for assessments in the senior high school years. At StatsLC we provide on-line help as part of our resources, and will be looking to develop this further. From time to time I get questions as a result of my YouTube videos, and enjoy answering them ,unless I am obviously doing someone’s homework! I also discovered “ShowMe” which looks like a great little iPad app, that I can use to help people more.

This has just been a quick guide to how useful the internet can be in teaching and learning. Next week I will address issues of quality and equity.

How to learn statistics (Part 2)

Some more help (preaching?) for students of statistics

Last week I outlined the first five principles to help people to learn and study statistics.

They focussed on how you need to practise in order to be good at statistics and you should not wait until you understand it completely before you start applying. I sometimes call this suspending disbelief. Next I talked about the importance of context in a statistical investigation, which is one of the ways that statistics is different from pure mathematics. And finally I stressed the importance of technology as a tool, not only for doing the analysis, but for exploring ideas and gaining understanding.

Here are the next five principles (plus 2):

6. Terminology is important and at times inconsistent

There are several issues with regard to statistical terminology, and I have written a post with ideas for teachers on how to teach terminology.

One issue with terminology is that some words that are used in the study of statistics have meanings in everyday life that are not the same. A clear example of this is the word, “significant”. In regular usage this can mean important or relevant, yet in statistics, it means that there is evidence that an effect that shows up in the sample also exists in the population.

Another issue is that statistics is a relatively young science and there are inconsistencies in terminology. We just have to live with that. Depending on the discipline in which the statistical analysis is applied or studied, different terms can mean the same thing, or very close to it.

A third language problem is that mixed in with the ambiguity of results, and judgment calls, there are some things that are definitely wrong. Teachers and examiners can be extremely picky. In this case I would suggest memorising the correct or accepted terminology for confidence intervals and hypothesis tests. For example I am very fussy about the explanation for the R-squared value in regression. Too often I hear that it says how much of the dependent variable is explained by the independent variable. There needs to be the word “variation” inserted in there to make it acceptable. I encourage my students to memorise a format for writing up such things. This does not substitute for understanding, but the language required is precise, so having a specific way to write it is fine.

This problem with terminology can be quite frustrating, but I think it helps to have it out in the open. Think of it as learning a new language, which is often the case in new subject. Use glossaries, to make sure you really do know what a term means.

7. Discussion is important

This is linked with the issue of language and vocabulary. One way to really learn something is to talk about it with someone else and even to try and teach it to someone else. Most teachers realise that the reason they know something pretty well, is because they have had to teach it. If your class does not include group work, set up your own study group. Talk about the principles as well as the analysis and context, and try to use the language of statistics. Working on assignments together is usually fine, so long as you write them up individually, or according to the assessment requirements.

8. Written communication skills are important

Mathematics has often been a subject of choice for students who are not fluent in English. They can perform well because there is little writing involved in a traditional mathematics course. Statistics is a different matter, though, as all students should be writing reports. This can be difficult at the start, but as students learn to follow a structure, it can be made more palatable. A statistics report is not a work of creative writing, and it is okay to use the same sentence structure more than once. Neither is a statistics report a narrative of what you did to get to the results. Generous use of headings makes a statistical report easier to read and to write. A long report is not better than a short report, if all the relevant details are there.

9. Statistics has an ethical and moral aspect

This principle is interesting, as many teachers of statistics come from a mathematical background, and so have not had exposure to the ethical aspects of research themselves. That is no excuse for students to park their ethics at the door of the classroom. I will be pushing for more consideration of ethical aspects of research as part of the curriculum in New Zealand. Students should not be doing experiments on human subjects that involve delicate subjects such as abuse, or bullying. They should not involve alcohol or other harmful substances. They should be aware of the potential to do harm, and make sure that any participants have been given full information and given consent. This can be quite a hurdle, but is part of being an ethical human being. It also helps students to be more aware when giving or withholding consent in medical and other studies.

10. The study of statistics can change the way you view the world

Sometimes when we learn something at school, it stays at school and has no impact on our everyday lives. This should not be the case with the study of statistics. As we learn about uncertainty and variation we start to see this in the world around us. When we learn about sampling and non-sampling errors, we become more critical of opinion polls and other research reported in the media. As we discover the power of statistical analysis and experimentation, we start to see the importance of evidence-based practice in medicine, social interventions and the like.

11. Statistics is an inherently interesting and relevant subject.

And it can be so much fun. There is a real excitement in exploring data, and becoming a detective. If you aren’t having fun, you aren’t doing it right!

12. Resources from Statistics Learning Centre will help you learn.

Of course!

How to study statistics (Part 1)

To students of statistics

Most of my posts are directed at teachers and how to teach statistics. The blog this week and next is devoted to students. I present principles that will help you to learn statistics. I’m turning them into a poster, which I will make available for you to printing later. I’d love to hear from other teachers as I add to my list of principles.

1. Statistics is learned by doing

One of the best predictors of success in any subject is how much time you spent on it. If you want to learn statistics, you need to put in time. It is good to read the notes and the textbook, and to look up things on the internet and even to watch Youtube videos if they are good ones. But the most important way to learn statistics is by doing. You need to practise at the skills that are needed by a statistician, which include logical thinking, interpretation, judgment and writing. Your teacher should provide you with worthwhile practice activities, and helpful timely feedback. Good textbooks have good practice exercises. On-line materials have many practice exercises.

Given a choice, do the exercises that have answers available. It is very important that you check what you are doing, as it is detrimental to practise something in the wrong way. Or if you are using an on-line resource, make sure you check your answers as you go, so that you gain from the feedback and avoid developing bad habits.

So really the first principle should really be “statistics is learned by doing correctly.

2. Understanding comes with application, not before.

Do not wait until you understand what you are doing before you get started. The understanding comes as you do the work. When we learn to speak, we do not wait until we understand grammatical structure before saying anything. We use what we have to speak and to listen, and as we do so we gain an understanding of how language works.  I have found that students who spent a lot of time working through the process of calculating conditional probabilities for screening tests grew to understand the “why” as well as the “how” of the process. Repeated application of using Excel to fit a line to bivariate data and explaining what it meant, enabled students to understand and internalise what a line means. As I have taught statistics for two decades, my own understanding has continued to grow.

There is a proviso. You need to think about what you are doing, and you need to do worthwhile exercises. For example, mechanically calculating the standard deviation of a set of numbers devoid of context will not help you understand standard deviation. Looking at graphs and trying to guess what the standard deviation is, would be a better exercise. Then applying the value to the context is better still.

Applying statistical principles to a wide variety of contexts helps us to discern what is specific to a problem and what is general for all problems. This brings us to the next principle.

3. Spend time exploring the context.

In a statistical analysis, context is vital, and often very interesting. You need to understand the problem that gave rise to the investigation and collection of the data. The context is what makes each statistical investigation different. Statisticians often work alongside other researchers in areas such as medicine, psychology, biology and geology, who provide the contextual background to the problem. This provides a wonderful opportunity for the statistician to learn about a whole range of different subjects. The interplay between the data and context mean that every investigation is different.

In a classroom setting you will not have the subject expert available, but you do need to understand the story behind the data. These days, finding out is possible with a click of a Google or Wikipedia button. Knowing the background to the data helps you to make more sensible judgments – and it makes it more interesting.

4. Statistics is different from mathematics

In mathematics, particularly pure mathematics, context is stripped away in order to reveal the inner pure truth of numbers and logic.  There are applied areas involving mathematics, which are more like statistics, such as operations research and engineering. At school level, one of the things that characterises the study of maths is right and wrong answers, with a minimum of ambiguity. That is what I loved about mathematics – being able to apply an algorithm and get a correct answer. In statistics, however, things are seldom black-and-white.  In statistics you will need to interpret data from the perspective of the real world, and often the answer is not clear. Some people find the lack of certainty in statistics disturbing. There is considerable room for discussion in statistics. Some aspects of statistics are fuzzy, such as what to do with messy data, or which is the “best” model to fit a time series. There is a greater need for the ability to write in statistics, which makes if more challenging for students for whom English is not their native language.

5. Technology is essential

With computers and calculators, all sorts of activities are available to help learn statistics. Graphs and graphics enable exploration that was not possible when graphs had to be drawn by hand. You can have a multivariate data set and explore all the possible relationships with a few clicks. You should always look at the data in a graphical form before setting out to analyse.

Sometimes I would set optional exercises for students to explore the relationship between data, graphs and summary measures. Very few students did so, but when I led them through the same examples one at a time I could see the lights go on. When you are given opportunities to use computing power to explore and learn – do it!

But wait…there’s more

Here we have the first five principles for students learning statistics. Watch this space next week for some more. And do add some in the comments and I will try to incorporate your ideas as well.

Open Letter to Khan Academy about Basic Probability

Khan academy probability videos and exercises aren’t good either

Dear Mr Khan

You have created an amazing resource that thousands of people all over the world get a lot of help from. Well done. Some of your materials are not very good, though, so I am writing this open letter in the hope that it might make some difference. Like many others, I believe that something as popular as Khan Academy will benefit from constructive criticism.

I fear that the reason that so many people like your mathematics videos so much is not because the videos are good, but because their experience in the classroom is so bad, and the curriculum is poorly thought out and encourages mechanistic thinking. This opinion is borne out by comments I have read from parents and other bloggers. The parents love you because you help their children pass tests.  (And these tests are clearly testing the type of material you are helping them to pass!) The bloggers are not so happy, because you perpetuate a type of mathematical instruction that should have disappeared by now. I can’t even imagine what the history teachers say about your content-driven delivery, but I will stick to what I know. (You can read one critique here)

Just over a year ago I wrote a balanced review of some of the Khan Academy videos about statistics. I know that statistics is difficult to explain – in fact one of the hardest subjects to teach. You can read my review here. I’ve also reviewed a selection of videos about confidence intervals, one of which was from Khan Academy. You can read the review here.

Consequently I am aware that blogging about the Khan Academy in anything other than glowing terms is an invitation for vitriol from your followers.

However, I thought it was about time I looked at the exercises that are available on KA, wondering if I should recommend them to high school teachers for their students to use for review. I decided to focus on one section, introduction to probability. I put myself in the place of a person who was struggling to understand probability at school.

Here is the verdict.

First of all the site is very nice. It shows that it has a good sized budget to use on graphics and site mechanics. It is friendly to get into. I was a bit confused that the first section in the Probability and Statistics Section is called “Independent and dependent events”. It was the first section though. The first section of this first section is called Basic Probability, so I felt I was in the right place. But then under the heading, Basic probability, it says, “Can I pick a red frog out of a bag that only contains marbles?” Now I have no trouble with humour per se, and some people find my videos pretty funny. But I am very careful to avoid confusing people with the humour. For an anxious student who is looking for help, that is a bit confusing.

I was excited to see that this section had five videos, and two sets of exercises. I was pleased about that, as I’ve wanted to try out some exercises for some time, particularly after reading the review from Fawn Nguyen on her experience with exercises on Khan Academy. (I suggest you read this – it’s pretty funny.)

So I watched the first video about probability and it was like any other KA video I’ve viewed, with primitive graphics and a stumbling repetitive narration. It was correct enough, but did not take into account any of the more recent work on understanding probability. It used coins and dice. Big yawn. It wastes a lot of time. It was ok. I do like that you have the interactive transcript so you can find your way around.

It dawned on me that nowhere do you actually talk about what probability is. You seem to assume that the students already know that. In the very start of the first video it says,

“What I want to do in this video is give you at least a basic overview of probability. Probability, a word that you’ve probably heard a lot of and you are probably just a little bit familiar with it. Hopefully this will get you a little deeper understanding.”

Later in the video there is a section on the idea of large numbers of repetitions, which is one way of understanding probability. But it really is a bit skimpy on why anyone would want to find or estimate a probability, and what the values actually mean. But it was ok.

The first video was about single instances – one toss of a coin or one roll of a die. Then the second video showed you how to answer the questions in the exercises, which involved two dice. This seemed ok, if rather a sudden jump from the first video. Sadly both of these examples perpetuate the common misconception that if there are, say, 6 alternative outcomes, they will necessarily be equally likely.


Then we get to some exercises called “Probability Space” , which is not an enormously helpful heading. But my main quest was to have a go at the exercises, so that is what I did. And that was not a good thing. The exercises were not stepped, but started right away with an example involving two dice and the phrase “at least one of”. There was meant to be a graphic to help me, but instead I had the message “scratchpad not available”. I will summarise my concerns about the exercises at the end of my letter. I clicked on a link to a video that wasn’t listed on the left, called Probability Space and got a different kind of video.

This video was better in that it had moving pictures and a script. But I have problems with gambling in videos like this. There are some cultures in which gambling is not acceptable. The other problem I have is with the term  “exact probability”, which was used several times. What do we mean by “exact probability”? How does he know it is exact? I think this sends the wrong message.

Then on to the next videos which were worked examples, entitled “Example: marbles from a bag, Example: Picking a non-blue marble, Example: Picking a yellow marble.” Now I understand that you don’t want to scare students with terminology too early, but I would have thought it helpful to call the second one, “complementary events, picking a non-blue marble”. That way if a student were having problems with complementary events in exercises from school, they could find their way here. But then I’m not sure who your audience is. Are you sure who your audience is?

The first marble video was ok, though the terminology was sloppy.

The second marble video, called “Example: picking a non-blue marble”, is glacially slow. There is a point, I guess in showing students how to draw a bag and marbles, but… Then the next example is of picking numbers at random. Why would we ever want to do this? Then we come to an example of circular targets. This involves some problem-solving regarding areas of circles, and cancelling out fractions including pi. What is this about? We are trying to teach about probablity so why have you brought in some complication involving the area of a circle?

The third marble video attempts to introduce the idea of events, but doesn’t really. By trying not to confuse with technical terms, the explanation is more confusing.

Now onto some more exercises. The Khan model is that you have to get 5 correct in a row in order to complete an exercise. I hope there is some sensible explanation for this, because it sure would drive me crazy to have to do that. (As I heard expressed on Twitter)

What are circular targets doing in with basic probability?

The first example is a circular target one.  I SO could not be bothered working out the area stuff so I used the hints to find the answer so I could move onto a more interesting example. The next example was finding the probability of a rolling a 4 from a fair six sided die. This is trivial, but would have been not a bad example to start with. Next question involve three colours of marbles, and finding the probability of not green. Then another dart-board one. Sigh. Then another dart board one. I’m never going to find out what happens if I get five right in a row if I don’t start doing these properly. Oh now – it gave me circumference. SO can’t be bothered.

And that was the end of Basic probability. I never did find out what happens if I get five correct in a row.

Venn diagrams

The next topic is called “Venn diagrams and adding probabilities “. I couldn’t resist seeing what you would do with a Venn diagram. This one nearly reduced me to tears.

As you know by now, I have an issue with gambling, so it will come as no surprise that I object to the use of playing cards in this example. It makes the assumption that students know about playing cards. You do take one and a half minutes to explain the contents of a standard pack of cards.  Maybe this is part of the curriculum, and if so, fair enough. The examples are standard – the probability of getting a Jack of Hearts etc. But then at 5:30 you start using Venn diagrams. I like Venn diagrams, but they are NOT good for what you are teaching at this level, and you actually did it wrong. I’ve put a comment in the feedback section, but don’t have great hopes that anything will change. Someone else pointed this out in the feedback two years ago, so no – it isn’t going to change.

Khan Venn diagram

This diagram is misleading, as is shown by the confusion expressed in the questions from viewers. There should be a green 3, a red 12, and a yellow 1.

Now Venn diagrams seem like a good approach in this instance, but decades of experience in teaching and communicating complex probabilities has shown that in most instances a two-way table is more helpful. The table for the Jack of Hearts problem would look like this:

Jacks Not Jacks Total
Hearts 1 12 13
Not Hearts 3 36 39
Total 4 48 52

(Any teachers reading this letter – try it! Tables are SO much easier for problem solving than Venn diagrams)

But let’s get down to principles.

The principles of instruction that KA have not followed in the examples:

  • Start easy and work up
  • Be interesting in your examples – who gives a flying fig about two dice or random numbers?
  • Make sure the hardest part of the question is the thing you are testing. This is particularly violated with the questions involving areas of circles.
  • Don’t make me so bored that I can’t face trying to get five in a row and not succeed.

My point

Yes, I do have one. Mr Khan you clearly can’t be stopped, so can you please get some real teachers with pedagogical content knowledge to go over your materials systematically and make them correct. You have some money now, and you owe it to your benefactors to GET IT RIGHT. Being flippant and amateurish is fine for amateurs but you are now a professional, and you need to be providing material that is professionally produced. I don’t care about the production values – keep the stammers and “lellows” in there if you insist. I’m very happy you don’t have background music as I can’t stand it myself. BUT… PLEASE… get some help and make your videos and exercises correct and pedagogically sound.

Dr Nic

PS – anyone else reading this letter, take a look at the following videos for mathematics.

And of course I think my own Statistics Learning Centre videos are pretty darn good as well.

Other posts about concerns about Khan:

Another Open Letter to Sal ( I particularly like the comment by Michael Paul Goldenberg)

Breaking the cycle (A comprehensive summary of the responses to criticism of Khan

Statistics is not beautiful (sniff)

Statistics is not really elegant or even fun in the way that a mathematics puzzle can be. But statistics is necessary, and enormously rewarding. I like to think that we use statistical methods and principles to extract truth from data.

This week many of the high school maths teachers in New Zealand were exhorted to take part in a Stanford MOOC about teaching mathematics. I am not a high school maths teacher, but I do try to provide worthwhile materials for them, so I thought I would take a look. It is also an opportunity to look at how people with an annual budget of more than 4 figures produce on-line learning materials. So I enrolled and did the first lesson, which is about people’s attitudes to math(s) and their success or trauma that has led to those attitudes. I’m happy to say that none of this was new to me. I am rather unhappy that it would be new to anyone! Surely all maths teachers know by now that how we deal with students’ small successes and failures in mathematics will create future attitudes leading to further success or failure. If they don’t, they need to take this course. And that makes me happy – that there is such a course, on-line and free for all maths teachers. (As a side note, I loved that Jo, the teacher switched between the American “math” and the British/Australian/NZ “maths”).

I’ve only done the first lesson so far, and intend to do some more, but it seems to be much more about mathematics than statistics, and I am not sure how relevant it will be. And that makes me a bit sad again. (It was an emotional journey!)

Mathematics in its pure form is about thinking. It is problem solving and it can be elegant and so much fun. It is a language that transcends nationality. (Though I have always thought the Greeks get a rough deal as we steal all their letters for the scary stuff.) I was recently asked to present an enrichment lesson to a class of “gifted and talented” students. I found it very easy to think of something mathematical to do – we are going to work around our Rogo puzzle, which has some fantastic mathematical learning opportunities. But thinking up something short and engaging and realistic in the statistics realm is much harder. You can’t do real statistics quickly.

On my run this morning I thought a whole lot more about this mathematics/statistics divide. I have written about it before, but more in defense of statistics, and warning the mathematics teachers to stay away or get with the programme. Understanding commonalities and differences can help us teach better. Mathematics is pure and elegant, and borders on art. It is the purest science. There is little beautiful about statistics. Even the graphs are ugly, with their scattered data and annoying outliers messing it all up. The only way we get symmetry is by assuming away all the badly behaved bits. Probability can be a bit more elegant, but with that we are creeping into the mathematical camp.

English Language and English literature

I like to liken. I’m going to liken maths and stats to English language and English literature. I was good at English at school, and loved the spelling and grammar aspects especially. I have in my library a very large book about the English language, (The Cambridge encyclopedia of the English Language, by David Crystal) and one day I hope to read it all. It talks about sounds and letters, words, grammar, syntax, origins, meanings. Even to dip into, it is fascinating. On the other hand I have recently finished reading “The End of Your Life Book Club” by Will Schwalbe, which is a biography of his amazing mother, set around the last two years of her life as she struggles with cancer. Will and his mother are avid readers, and use her time in treatment to talk about books. This book has been an epiphany for me. I had forgotten how books can change your way of thinking, and how important fiction is. At school I struggled with the literature side of English, as I wanted to know what the author meant, and could not see how it was right to take my own meaning from a book, poem or work of literature.  I have since discovered post-modernism and am happy drawing my own meaning.

So what does this all have to do with maths and statistics? Well I liken maths to English language. In order to be good at English you need to be able to read and write in a functional way. You need to know the mechanisms. You need to be able to DO, not just observe. In mathematics, you need to be able to approach a problem in a mathematical way.  Conversely, to be proficient in literature, you do not need to be able to produce literature. You need to be able to read literature with a critical mind, and appreciate the ideas, the words, the structure. You do need to be able to write enough to express your critique, but that is a different matter from writing a novel.  This, to me is like being statistically literate – you can read a statistical report, and ask the right questions. You can make sense of it, and not be at the mercy of poorly executed or mendacious research. You can even write a summary or a critique of a statistical analysis. But you do not need to be able to perform the actual analysis yourself, nor do you need to know the exact mathematical theory underlying it.

Statistical Literacy?

Maybe there is a problem with the term “statistical literacy”. The traditional meaning of literacy includes being able to read and write – to consume and to produce – to take meaning and to create meaning. I’m not convinced that what is called statistical literacy is the same.

Where I’m heading with this, is that statistics is a way to win back the mathematically disenfranchised. If I were teaching statistics to a high school class I would spend some time talking about what statistics involves and how it overlaps with, but is not mathematics. I would explain that even people who have had difficulty in the past with mathematics, can do well at statistics.

The following table outlines the different emphasis of the two disciplines.

Mathematics Statistics
Proficiency with numbers is important Proficiency with numbers is helpful
Abstract ideas are important Concrete applications are important
Context is to be removed so that we can model the underlying ideas Context is crucial to all statistical analysis
You don’t need to write very much. Written expression in English is important

Another idea related to this is that of “magic formulas” or the cookbook approach. I don’t have a problem with cookbooks and knitting patterns. They help me to make things I could not otherwise. However, the more I use recipes and patterns, the more I understand the principles on which they are based. But this is a thought for another day.

The importance of being wrong

We don’t like to think we are wrong

One of the key ideas in statistics is that sometimes we will be wrong. When we report a 95% confidence interval, we will be wrong 5% of the time. Or in other words, about 1 in 20 of 95% confidence intervals will not contain the population parameter we are attempting to estimate. That is how they are defined. The thing is, we always think we are part of the 95% rather than the 5%. Mostly we will be correct, but if we do enough statistical analysis, we will almost definitely be wrong at some point. However, human nature is such that we tend to think it will be someone else. There is also a feeling of blame associated with being wrong. The feeling is that if we have somehow missed the true value with our confidence interval, it must be because we have made a mistake. However, this is not true. In fact we MUST be wrong about 5% of the time, or our interval is too big, and not really a 95% confidence interval.

The term “margin of error” appears with increasing regularity as elections approach and polling companies are keen to make money out of sooth-saying. The common meaning of the margin of error is half the width of a 95% confidence interval. So if we say the margin of error is 3%, then about one time in twenty, the true value of the proportion will actually be more than 3% away from the reported sample value.

What doesn’t help is that we seldom do know if we are correct or not. If we knew the real population value we wouldn’t be estimating it. We can contrive situations where we do know the population but pretend we don’t. If we do this in our teaching, we need to be very careful to point out that this doesn’t normally happen, but does in “classroom world” only. (Thanks to MD for this useful term.) General elections can give us an idea of being right or wrong after the event, but even then the problem of non-sampling error is conflated with sampling error. When opinion polls turn out to miss the mark, we tend to think of the cause as being due to poor sampling, or people changing their minds, or all number of imaginative explanations rather than simple, unavoidable sampling error.

So how do we teach this in such a way that it goes beyond school learning and is internalised for future use as efficient citizens?

Teaching suggestions

I have two suggestions. The first is a series of True/False statements that can be used in a number of ways. I have them as part of on-line assessment, so that the students are challenged by them regularly. They could be well used in the classroom as part of a warm-up exercise at the start of a lesson. Students can write their answers down or vote using hands.

Here are some examples of True/False statements (some of which could lead to discussion):

  1. You never know if your confidence interval contains the true population value.
  2. If you make your confidence interval wide enough you can be sure that you contain the true population value.
  3. A confidence interval tells us where we are pretty sure the sample statistic lies.
  4. It is better to have a narrow confidence interval than a wide one, as it gives us more certain information, even though it is more likely to be wrong.
  5. If your study involves twenty confidence intervals, then you know that exactly one of them will be wrong.
  6. If a confidence interval doesn’t contain the true population value, it is because it is one of the 5% that was calculated incorrectly.

You can check your answers at the end of this post.

Experiential exercise

The other teaching suggestion is for an experiential exercise. It requires a little set up time.

Make a set of cards for students with numbers on them that correspond to the point estimate of a proportion, or a score that will lead to that. (Specifications for a set of 35 cards representing the results from a proportion of 0.54 and 25 trials is given below).

Introduce the exercise as follows:
“I have a computer game, and have set the ratio of wins to losses at a certain value. Each of you has played 25 times, and the number of wins you have obtained will be on your card. It is really important that you don’t look at other people’s cards.”

Hand them out to the students. (If you have fewer than 35 in your class, it might be a good idea to make sure you include the cards with 8 and 19 in the set you use – sometimes it is ok to fudge slightly to teach a point.)
“Without getting information from anyone else, write down your best estimate of the true proportion of wins to losses in the game. Do you think you are correct? How close do you think you are to the true value?”

They will need to divide the number of wins by 25, which should not lead to any computational errors! The point is that they really can’t know how close their estimate is to the true value – and what does “correct” mean?

Then work out the margin of error for a sample of size 25, which in this case is estimated at 20%. Get the students to calculate their 95% confidence intervals, and decide if they have the interval that contains the true population value. Get them to commit one way or the other.

Now they can talk to each other about the values they have.

There are several ways you can go from here. You can tell them what the population proportion was from which the numbers were drawn (0.54). They can then see that most of them had confidence intervals that included the true value, and some didn’t. Or you can leave them wondering, which is a better lesson about real life. Or you can do one exercise where you do tell them and one where you don’t.

This is an area where probability and statistics meet. You could make a nice little binomial distribution problem out of being correct in a number of confidence intervals. There are potential problems with independence, so you need to be a bit careful with the wording. For example: Fifteen  students undertake separate statistical analyses on the topics of their choice, and construct 95% confidence intervals. What is the probability that all the confidence intervals are correct, in that they do contain the estimated population parameter? This is well modelled by a binomial distribution with n =15 and p=0.05. P(X=0)=0.46. And another interesting idea – what is the probability that two or more are incorrect? 0.17 is the answer. So there is a 17% chance that more than one of the confidence intervals does not contain the population parameter of interest.

This is an area that needs careful teaching, and I suspect that some teachers have only a sketchy understanding of the idea of confidence intervals and margins of error. It is so important to know that statistical results are meant to be wrong some of the time.

Answers: T,T,F, debatable, F,F.

Data for the 35 cards:

Number on card













Number of cards













Teaching with School League tables

NCEA League tables in the newspaper

My husband ran for cover this morning when he saw high school NCEA (National Certificates of Educational Achievement)  league tables in the Press. However, rather than rave at him yet again, I will grasp the opportunity to expound to a larger audience. Much as I loathe and despise league tables, they are a great opportunity to teach students to explore data rich reports with a critical and educated eye.  There are many lessons to learn from league tables. With good teaching we can help dispell some of the myths the league tables promulgate.

When a report is made short and easy to understand, there is a good chance that much of the ‘truth’ has been lost along with the complexity. The table in front of me lists 55 secondary and area schools from the Canterbury region. These schools include large “ordinary” schools and small specialist schools such as Van Asch Deaf Education Centre and Southern Regional Health School. They include single-sex and co-ed, private, state-funded and integrated. They include area schools which are in small rural communities, which cover ages 5 to 21. The “decile” of each of the schools is the only contextual information given, apart from the name of the school.  (I explain the decile, along with misconceptions at the end of the post.) For each school is given percentages of students passing at the three levels. It is not clear whether the percentages in the newspaper are of participation rate or school roll.

This is highly motivating information for students as it is about them and their school. I had an argument recently with a student from a school which scores highly in NCEA. She was insistent that her friend should change schools from one that has lower scores. What she did not understand was that the friend had some extra learning difficulties, and that the other school was probably more appropriate for her. I tried to teach the concept of added-value, but that wasn’t going in either. However I was impressed with her loyalty to her school and I think these tables would provide an interesting forum for discussion.

Great context discussion

You could start with talking about what the students think will help a school to have high pass rates. This could include a school culture of achievement, good teaching, well-prepared students and good resources. This can also include selection and exclusion of students to suit the desired results, selection of “easy” standards or subjects, and even less rigorous marking of internal assessment. Other factors to explore might be single-sex vs co-ed school, the ethnic and cultural backgrounds of the students, private vs state-funded schools.  All of these are potential explanatory variables. Then you can point out how little of this information is actually taken into account in the table. This is a very common occurrence, with limited space and inclusion of raw data. I suspect at least one school appears less successful because some of the students sit different exams, either Cambridge or International Baccalaureate. These may be the students who would have performed well in NCEA.

Small populations

It would be good to look at the impact of small populations, and populations of very different sizes in the data. Students should think about what impact their behaviour will have on the results of the school, compared with a larger or smaller cohort. The raw data provided by the Ministry of Education does give a warning for small cohorts. For a small school, particularly in a rural area, there may be only a handful of students in year 13, so that one student’s success or failure has a large impact on the outcome. At the other end of the scale, there are schools of over 2000, which will have about 400 students in year 13. This effect is important to understand in all statistical reporting. One bad event in a small hospital, for instance, will have a larger percentage effect than in a large hospital.

Different rules

We hear a lot about comparing apples and oranges. School league tables include a whole fruit basket of different criteria. Schools use different criteria for allowing students into the school, into different courses, and whether they are permitted to sit external standards. Attitudes to students with special educational needs vary greatly. Some schools encourage students to sit levels outside their year level.

Extrapolating from a small picture

What one of the accompanying stories points out is that NCEA is only a part of what schools do. Sometimes the things that are measurable get more attention because it is easier to report in bulk. A further discussion with students could be provoked using statements such as the following, which the students can vote on, and then discuss. You could also discuss what evidence you would need to be able to refute or support them.

  • A school that does well in NCEA level 3 is a good school.
  • Girls’ schools do better than boys’ schools at NCEA because girls are smarter than boys.
  • Country schools don’t do very well because the clever students go to boarding school in the city.
  • Boys are more satisfied with doing just enough to get achieved.

Further extension

If students are really interested you can download the full results from the Ministry of Education website and set up a pivot table on Excel to explore questions.

I can foresee some engaging and even heated discussions ensuing. I’d love to hear how they go.

Short explanation of Decile – see also official website.

The decile rating of the school is an index developed in New Zealand and is a measure of social deprivation. The decile rating is calculated from a combination of five values taken from census data for the meshblocks in which the students reside. A school with a low decile rating of 1 or 2 will have a large percentage of students from homes that are crowded, or whose parents are not in work or have no educational qualifications. A school with a decile rating of 10 will have the fewest students from homes like that. The system was set up to help with targeted funding for educational achievement. It recognises that students from disadvantaged homes will need additional resources in order to give them equal opportunity to learn. However, the term has entered the New Zealand vernacular as a measure of socio-economic status, and often even of worth. A decile 10 school is often seen as a rich school or a “top” school. The reality is that this is not the case.  Another common misconception is that one tenth of the population of school age students is in each of the ten bands. How it really works is that one tenth of schools is in each of the bands. The lower decile schools are generally smaller than other schools, and mostly primary schools. In 2002 there were nearly 40,000 secondary students in decile 10 schools, with fewer than 10,000 in decile 1 schools.