Statistical software for worried students

Statistical software for worried students: Appearances matter

Let’s be honest. Most students of statistics are taking statistics because they have to. I asked my class of 100 business students who choose to take the quantitative methods course if they did not have to. Two hands went up.

Face it – statistics is necessary but not often embraced.

But actually it is worse than that. For many people statistics is the most dreaded course they are required to take. It can be the barrier to achieving their career goals as a psychologist, marketer or physician. (And it should be required for many other careers, such as journalism, law and sports commentator.)

Choice of software

Consequently, we have worried students in our statistics courses. We want them to succeed, and to do that we need to reduce their worry. One decision that will affect their engagement and success is the choice of computer package. This decision rightly causes consternation to instructors. It is telling that one of the most frequently and consistently accessed posts on this blog is Excel, SPSS, Minitab or R. It has been  viewed 55,000 times in the last five years.

The problem of which package to use is no easier to solve than it was five years ago when I wrote the post. I am helping a tertiary institution to re-develop their on-line course in statistics. This is really fun – applying all the great advice and ideas from ”
Guidelines for Assessment and Instruction in Statistics” or GAISE. They asked for advice on what statistics package to use. And I am torn.


Here is what I want from a statistical teaching package:

  • Easy to use
  • Attractive to look at (See “Appearances Matter” below)
  • Helpful output
  • Good instructional materials with videos etc (as this is an online course)
  • Supports good pedagogy

If I’m honest I also want it to have the following characteristics:

  • Guidance for students as to what is sensible
  • Only the tests and options I want them to use in my course – not too many choices
  • An interpretation of the output
  • Data handling capabilities, including missing values
  • A pop up saying “Are you sure you want to make a three dimensional pie-chart?”

Is this too much to ask?


Overlapping objectives

Here is the thing. There are two objectives for introductory statistics courses that partly overlap and partly conflict. We want students to

  • Learn what statistics is all about
  • Learn how to do statistics.

They probably should not conflict, but they require different things from your software. If all we want the students to do is perform the statistical tests, then something like Excel is not a bad choice, as they get to learn Excel as well, which could be handy for c.v. expansion and job-getting. If we are more concerned about learning what statistics is all about, then an exploratory package like Tinkerplots or iNZight could be useful.

Ideally I would like students to learn both what statistics is all about and how to do it. But most of all, I want them to feel happy about doing statistical analysis.

Appearances matter

Eye-appeal is important for overcoming fear. I am confident in mathematics, but a journal article with a page of Greek letters and mathematical symbols, makes me anxious. The Latex font makes me nervous. And an ugly logo puts me off a package. I know it is shallow. But it is a thing, and I suspect I am far from alone. Marketing people know that the choice of colour, word, placement – all sorts of superficial things effect whether a product sells. We need to sell our product, statistics, and to do that, it needs to be attractive. It may well be that the people who design software are less affected by appearance, but they are not the consumers.

Terminal or continuing?

This is important: Most of our students will never do another statistical analysis.

Think about it :

Most of our students will never do another statistical analysis.

Here are the implications: It is important for the students to learn what statistics is about, where it is needed, potential problems and good communication and critique of statistical results. It is not important for students to learn how to program or use a complex package.

Students need to experience statistical analysis, to understand the process. They may also discover the excitement of a new set of data to explore, and the anticipation of an interesting result. These students may decide to study more statistics, at which time they will need to learn to operate a more comprehensive package. They will also be motivated to do so because they have chosen to continue to learn statistics.


In my previous post I talked about Excel, SPSS, Minitab and R. I used to teach with Excel, and I know many of my past students have been grateful they learned it. But now I know better, and cannot, hand on heart recommend Excel as the main software. Students need to be able to play with the data, to look at various graphs, and get a feel for variation and structure. Excel’s graphing and data-handling capabilities, particularly with regard to missing values, are not helpful. The histograms are disastrous. Excel is useful for teaching students how to do statistics, but not what statistics is all about.

SPSS and Minitab

SPSS was a personal favourite, but it has been a while since I used it. It is fairly expensive, and chances are the students will never use it again. I’m not sure how well it does data exploration. Minitab is another nice little package. Both of these are probably overkill for an introductory statistics course.

R and R Commander

R is a useful and versatile statistical language for higher level statistical analysis and learning but it is not suitable for worried students. It is unattractive.

R Commander is a graphical user interface for R. It is free, and potentially friendlier than R. It comes with a book. I am told it is a helpful introduction to R. R Commander is also unattractive. The book was formatted in Latex. The installation guide looks daunting. That is enough to make me reluctant – and I like statistics!

The screenshot displayed on the front page of R Commander

iNZight and iNZight Lite

I have used iNZight a lot. It was developed at the University of Auckland for use in their statistics course and in New Zealand schools. The full version is free and can be installed on PC and Mac computers, though there may be issues with running it on a Mac. The iNZight lite, web-based version is fine. It is free and works on any platform. I really like how easy it is to generate various plots to explore the data. You put in the data, and the graphs appear almost instantly. IiNZIght encourages engagement with the data, rather than doing things to data.

For a face-to-face course I would choose iNZight Lite. For an online course I would be a little concerned about the level of support material available. The newer version of iNZight, and iNZight lite have benefitted from some graphic design input. I like the colours and the new logo.


I’ve heard about Genstat for some time, as an alternative to iNZight for New Zealand schools, particularly as it does bootstrapping. So I requested an inspection copy. It has a friendly vibe. I like the dialog box suggesting the graph you might like try. It lacks the immediacy of iNZight lite. It has the multiple window thing going on, which can be tricky to navigate. I was pleased at the number of sample data sets.


NZGrapher is popular in New Zealand schools. It was created by a high school teacher in his spare time, and is attractive and lean. It is free, funded by donations and advertisements. You enter a data set, and it creates a wide range of graphs. It does not have the traditional tests that you would want in an introductory statistics course, as it is aimed at the NZ school curriculum requirements.


Statcrunch is a more attractive, polished package, with a wide range of supporting materials. I think this would give confidence to the students. It is specifically designed for teaching and learning and is almost conversational in approach. I have not had the opportunity to try out Statcrunch. It looks inviting, and was created by Webster West, a respected statistics educator. It is now distributed by Pearson.


I recently had my attention drawn to this new package. It is free, well-supported and has a clean, attractive interface. It has a vibe similar to SPSS. I like the immediate response as you begin your analysis. Jasp is free, and I was able to download it easily. It is not as graphical as iNZight, but is more traditional in its approach. For a course emphasising doing statistics, I like the look of this.

Data, controls and output from Jasp


So there you have it. I have mentioned only a few packages, but I hope my musings have got you thinking about what to look for in a package. If I were teaching an introductory statistics course, I would use iNZight Lite, Jasp, and possibly Excel. I would use iNZight Lite for data exploration. I might use Jasp for hypothesis tests, confidence intervals and model fitting. And if possible I would teach Pivot Tables in Excel, and use it for any probability calculations.

Your thoughts

This is a very important topic and I would appreciate input. Have I missed an important contender? What do you look for in a statistical package for an introductory statistics course? As a student, how important is it to you for the software to be attractive?


The Myth of Random Sampling

I feel a slight quiver of trepidation as I begin this post – a little like the boy who pointed out that the emperor has  no clothes.

Random sampling is a myth. Practical researchers know this and deal with it. Theoretical statisticians live in a theoretical world where random sampling is possible and ubiquitous – which is just as well really. But teachers of statistics live in a strange half-real-half-theoretical world, where no one likes to point out that real-life samples are seldom random.

The problem in general

In order for most inferential statistical conclusions to be valid, the sample we are using must obey certain rules. In particular, each member of the population must have equal possibility of being chosen. In this way we reduce the opportunity for systematic error, or bias. When a truly random sample is taken, it is almost miraculous how well we can make conclusions about the source population, with even a modest sample of a thousand. On a side note, if the general population understood this, and the opportunity for bias and corruption were eliminated, general elections and referenda could be done at much less cost,  through taking a good random sample.

However! It is actually quite difficult to take a random sample of people. Random sampling is doable in biology, I suspect, where seeds or plots of land can be chosen at random. It is also fairly possible in manufacturing processes. Medical research relies on the use of a random sample, though it is seldom of the total population. Really it is more about randomisation, which can be used to support causal claims.

But the area of most interest to most people is people. We actually want to know about how people function, what they think, their economic activity, sport and many other areas. People find people interesting. To get a really good sample of people takes a lot of time and money, and is outside the reach of many researchers. In my own PhD research I approximated a random sample by taking a stratified, cluster semi-random almost convenience sample. I chose representative schools of different types throughout three diverse regions in New Zealand. At each school I asked all the students in a class at each of three year levels. The classes were meant to be randomly selected, but in fact were sometimes just the class that happened to have a teacher away, as my questionnaire was seen as a good way to keep them quiet. Was my data of any worth? I believe so, of course. Was it random? Nope.

Problems people have in getting a good sample include cost, time and also response rate. Much of the data that is cited in papers is far from random.

The problem in teaching

The wonderful thing about teaching statistics is that we can actually collect real data and do analysis on it, and get a feel for the detective nature of the discipline. The problem with sampling is that we seldom have access to truly random data. By random I am not meaning just simple random sampling, the least simple method! Even cluster, systematic and stratified sampling can be a challenge in a classroom setting. And sometimes if we think too hard we realise that what we have is actually a population, and not a sample at all.

It is a great experience for students to collect their own data. They can write a questionnaire and find out all sorts of interesting things, through their own trial and error. But mostly students do not have access to enough subjects to take a random sample. Even if we go to secondary sources, the data is seldom random, and the students do not get the opportunity to take the sample. It would be a pity not to use some interesting data, just because the collection method was dubious (or even realistic). At the same time we do not want students to think that seriously dodgy data has the same value as a carefully collected random sample.

Possible solutions

These are more suggestions than solutions, but the essence is to do the best you can and make sure the students learn to be critical of their own methods.

Teach the best way, pretend and look for potential problems.

Teach the ideal and also teach the reality. Teach about the different ways of taking random samples. Use my video if you like!

Get students to think about the pros and cons of each method, and where problems could arise. Also get them to think about the kinds of data they are using in their exercises, and what biases they may have.

We also need to teach that, used judiciously, a convenience sample can still be of value. For example I have collected data from students in my class about how far they live from university , and whether or not they have a car. This data is not a random sample of any population. However, it is still reasonable to suggest that it may represent all the students at the university – or maybe just the first year students. It possibly represents students in the years preceding and following my sample, unless something has happened to change the landscape. It has worth in terms of inference. Realistically, I am never going to take a truly random sample of all university students, so this may be the most suitable data I ever get.  I have no doubt that it is better than no information.

All questions are not of equal worth. Knowing whether students who own cars live further from university, in general, is interesting but not of great importance. Were I to be researching topics of great importance, such safety features in roads or medicine, I would have a greater need for rigorous sampling.

So generally, I see no harm in pretending. I use the data collected from my class, and I say that we will pretend that it comes from a representative random sample. We talk about why it isn’t, but then we move on. It is still interesting data, it is real and it is there. When we write up analysis we include critical comments with provisos on how the sample may have possible bias.

What is important is for students to experience the excitement of discovering real effects (or lack thereof) in real data. What is important is for students to be critical of these discoveries, through understanding the limitations of the data collection process. Consequently I see no harm in using non-random, realistic sampled real data, with a healthy dose of scepticism.

On-line learning and teaching resources

Twenty-first century Junior Woodchuck Guidebook

I grew up reading Donald Duck comics. I love the Junior Woodchucks, and their Junior Woodchuck Guidebook. The Guidebook is a small paperback book, containing information on every conceivable subject, including geography, mythology, history, literature and the Rubaiyat of Omar Khayyam.  In our family, when we want to know something or check some piece of information, we talk about consulting the Junior Woodchuck Guidebook. (Imagine my joy when I discovered that a woodchuck is another name for a groundhog, the star of my favourite movie!) What we are referring to is the internet, the source of all possible information! Thanks to search engines, there is very little we cannot find out on the internet. And very big thanks to Wikipedia, to which I make an annual financial contribution, as should all who use it and can afford to.

You can learn just about anything on the internet. Problem is, how do you know what is good? And how do you help students find good stuff? And how do you use the internet wisely? And how can it help us as learners and teachers of statistics and operations research? These questions will take more than my usual 1000 words, so I will break it up a bit. This post is about the ways the internet can help in teaching and learning. In a later post I will talk about evaluating resources, and in particular multimedia resources.


Both the disciplines in which I am interested, statistics and operations research, apply mathematical and analytic methods to real-world problems. In statistics we are generally trying to find things out, and in operations research we are trying to make them better. Either way, the context is important. The internet enables students to find background knowledge regarding the context of the data or problem they are dealing with. It also enables instructors to write assessments and exercises that have a degree of veracity to them even if the actual raw data proves elusive. How I wish people would publish standard deviations as well as means when reporting results!


Which brings us to the second use for on-line resources. Real problems with real data are much more meaningful for students, and totally possible now that we don’t need to calculate anything by hand. Sadly, it is more difficult than first appears to find good quality raw data to analyse, but there is some available. You can see some sources in a previous post and the helpful comments.


If you are struggling to understand a concept, or to know how to teach or explain it, do a web search. I have found some great explanations, and diagrams especially, that have helped me. Or I have discovered a dearth of good diagrams, which has prompted me to make my own.


Videos can help with background knowledge, with explanations, and with inspiring students with the worth of the discipline. The problem with videos is that it takes a long time to find good ones and weed out the others. One suggestion is to enlist the help of your students. They can each watch two or three videos and decide which are the most helpful. The teacher then watches the most popular ones to check for pedagogical value. It is great when you find a site that you can trust, but even then you can’t guarantee the approach will be compatible with your own.

Social support

I particularly love Twitter, from which I get connection with other teachers and learners, and ideas and links to blogs. I belong to a Facebook group for teachers of statistics in New Zealand, and another Facebook group called “I love Operations Research”. These wax and wane in activity, and can be very helpful at times. Students and teachers can gain a lot from social networking.


There is good open-source software available, and 30-day trial versions for other software. Many schools in New Zealand use the R-based iNZight collection of programmes, which provide purpose-built means for timeseries analysis, bootstrapping and line fitting.

Answers to questions

The other day I lost the volume control off my toolbar. (Windows Vista, I’m embarrassed to admit). So I put in the search box “Lost my volume control” and was directed to a YouTube video that took me step-by-step through the convoluted process of reinstating my volume control! I was so grateful I made a donation. Just about any computer related question can be answered through a search.

Interactive demonstrations

I love these. There are two sites I have found great:

The National Library of Virtual Manipulatives, based in Utah.

NRich – It has some great ideas in the senior statistics area. From the UK.

A problem with some of these is the use of Flash, which does not play on all devices. And again – how do we decide if they are any good or not?

On-line textbooks

Why would you buy a textbook when you can get one on-line. I routinely directed my second-year statistical methods for business students to “Concepts and Applications of Inferential Statistics”. I’ve found it just the right level. Another source is Stattrek. I particularly like their short explanations of the different probability distributions.

Practice quizzes

There aren’t too many practice quizzes  around for free. Obviously, as a provider of statistical learning materials, I believe quizzes and exercises have merit for practice with immediate and focussed feedback. However, it can be very time-consuming to evaluate practice quizzes, and some just aren’t very good. On the other hand, some may argue that any time students spend learning is better than none.

Live help

There are some places that provide live, or slightly delayed help for students. I got hooked into a very fun site where you earned points by helping students. Sadly I can’t find it now, but as I was looking I found vast numbers of on-line help sites, often associated with public libraries. And there are commercial sites that provide some free help as an intro to their services. In New Zealand there is the StudyIt service, which helps students preparing for assessments in the senior high school years. At StatsLC we provide on-line help as part of our resources, and will be looking to develop this further. From time to time I get questions as a result of my YouTube videos, and enjoy answering them ,unless I am obviously doing someone’s homework! I also discovered “ShowMe” which looks like a great little iPad app, that I can use to help people more.

This has just been a quick guide to how useful the internet can be in teaching and learning. Next week I will address issues of quality and equity.

Teaching with School League tables

NCEA League tables in the newspaper

My husband ran for cover this morning when he saw high school NCEA (National Certificates of Educational Achievement)  league tables in the Press. However, rather than rave at him yet again, I will grasp the opportunity to expound to a larger audience. Much as I loathe and despise league tables, they are a great opportunity to teach students to explore data rich reports with a critical and educated eye.  There are many lessons to learn from league tables. With good teaching we can help dispell some of the myths the league tables promulgate.

When a report is made short and easy to understand, there is a good chance that much of the ‘truth’ has been lost along with the complexity. The table in front of me lists 55 secondary and area schools from the Canterbury region. These schools include large “ordinary” schools and small specialist schools such as Van Asch Deaf Education Centre and Southern Regional Health School. They include single-sex and co-ed, private, state-funded and integrated. They include area schools which are in small rural communities, which cover ages 5 to 21. The “decile” of each of the schools is the only contextual information given, apart from the name of the school.  (I explain the decile, along with misconceptions at the end of the post.) For each school is given percentages of students passing at the three levels. It is not clear whether the percentages in the newspaper are of participation rate or school roll.

This is highly motivating information for students as it is about them and their school. I had an argument recently with a student from a school which scores highly in NCEA. She was insistent that her friend should change schools from one that has lower scores. What she did not understand was that the friend had some extra learning difficulties, and that the other school was probably more appropriate for her. I tried to teach the concept of added-value, but that wasn’t going in either. However I was impressed with her loyalty to her school and I think these tables would provide an interesting forum for discussion.

Great context discussion

You could start with talking about what the students think will help a school to have high pass rates. This could include a school culture of achievement, good teaching, well-prepared students and good resources. This can also include selection and exclusion of students to suit the desired results, selection of “easy” standards or subjects, and even less rigorous marking of internal assessment. Other factors to explore might be single-sex vs co-ed school, the ethnic and cultural backgrounds of the students, private vs state-funded schools.  All of these are potential explanatory variables. Then you can point out how little of this information is actually taken into account in the table. This is a very common occurrence, with limited space and inclusion of raw data. I suspect at least one school appears less successful because some of the students sit different exams, either Cambridge or International Baccalaureate. These may be the students who would have performed well in NCEA.

Small populations

It would be good to look at the impact of small populations, and populations of very different sizes in the data. Students should think about what impact their behaviour will have on the results of the school, compared with a larger or smaller cohort. The raw data provided by the Ministry of Education does give a warning for small cohorts. For a small school, particularly in a rural area, there may be only a handful of students in year 13, so that one student’s success or failure has a large impact on the outcome. At the other end of the scale, there are schools of over 2000, which will have about 400 students in year 13. This effect is important to understand in all statistical reporting. One bad event in a small hospital, for instance, will have a larger percentage effect than in a large hospital.

Different rules

We hear a lot about comparing apples and oranges. School league tables include a whole fruit basket of different criteria. Schools use different criteria for allowing students into the school, into different courses, and whether they are permitted to sit external standards. Attitudes to students with special educational needs vary greatly. Some schools encourage students to sit levels outside their year level.

Extrapolating from a small picture

What one of the accompanying stories points out is that NCEA is only a part of what schools do. Sometimes the things that are measurable get more attention because it is easier to report in bulk. A further discussion with students could be provoked using statements such as the following, which the students can vote on, and then discuss. You could also discuss what evidence you would need to be able to refute or support them.

  • A school that does well in NCEA level 3 is a good school.
  • Girls’ schools do better than boys’ schools at NCEA because girls are smarter than boys.
  • Country schools don’t do very well because the clever students go to boarding school in the city.
  • Boys are more satisfied with doing just enough to get achieved.

Further extension

If students are really interested you can download the full results from the Ministry of Education website and set up a pivot table on Excel to explore questions.

I can foresee some engaging and even heated discussions ensuing. I’d love to hear how they go.

Short explanation of Decile – see also official website.

The decile rating of the school is an index developed in New Zealand and is a measure of social deprivation. The decile rating is calculated from a combination of five values taken from census data for the meshblocks in which the students reside. A school with a low decile rating of 1 or 2 will have a large percentage of students from homes that are crowded, or whose parents are not in work or have no educational qualifications. A school with a decile rating of 10 will have the fewest students from homes like that. The system was set up to help with targeted funding for educational achievement. It recognises that students from disadvantaged homes will need additional resources in order to give them equal opportunity to learn. However, the term has entered the New Zealand vernacular as a measure of socio-economic status, and often even of worth. A decile 10 school is often seen as a rich school or a “top” school. The reality is that this is not the case.  Another common misconception is that one tenth of the population of school age students is in each of the ten bands. How it really works is that one tenth of schools is in each of the bands. The lower decile schools are generally smaller than other schools, and mostly primary schools. In 2002 there were nearly 40,000 secondary students in decile 10 schools, with fewer than 10,000 in decile 1 schools.

Parts and whole

The whole may be greater than the sum of the parts, but the whole still needs those parts. A reflective teacher will think carefully about when to concentrate on the whole, and when on the parts.


If you were teaching someone golf, you wouldn’t spend days on a driving range, never going out on a course. Your student would not get the idea of what the game is, or why they need to be able to drive straight and to a desired length. Nor would it be much fun! Similarly if the person only played games of golf it would be difficult for them to develop their game. Practice driving and putting is needed.  A serious student of golf would also read and watch experts at golf.


Learning music is similar. Anyone who is serious about developing as a musician will spend a considerable amount of time developing their technique and their knowledge by practicing scales, chords and drills. But at the same time they need to be playing full pieces of music so that they feel the joy of what they are doing. As they play music, as opposed to drill, they will see how their less-interesting practice has helped them to develop their skills. However, as they practice a whole piece, they may well find a small part that is tripping them up, and focus for a while on that. If they play only the piece as a whole, it is not efficient use of time. A serious student of music will also listen to and watch great musicians, in order to develop their own understanding and knowledge.

Benefits of study of the whole and of the parts

In each of these examples we can see that there are aspects of working with the whole, and aspects of working with the parts. Study of the whole contributes perspective and meaning to study, and helps to tie things together. It helps to see where they have made progress. Study of the parts isolates areas of weakness, develops skills and saves time in practice, thus being more efficient.

It is very important for students to get an idea of the purpose of their study, and where they are going. For this reason I have written earlier about the need to see the end when starting out in a long procedure such as a regression or linear programming model.

It is also important to develop “statistical muscle memory” by repeating small parts of the exercise over and over until it is mastered. Practice helps people to learn what is general and what is specific in the different examples.

Teaching conditional probability

We are currently developing a section on probability as part of our learning materials. A fundamental understanding of probability and uncertainty are essential to a full understanding of inference. When we look at statistical evidence from data, we are holding it up against what we could reasonably expect to happen by chance, which involves a probability model. Probability lies in the more mathematical area of the study of statistics, and has some fun problem-solving aspects to it.

A popular exam question involves conditional probability. We like to use a table approach to this as it avoids many of the complications of terminology. I still remember my initial confusion over the counter-intuitive expression P(A|B) which means the probability that an object from subset B has the property of A. There are several places where students can come unstuck in Bayesian review, and the problems can take a long time. We can liken solving a conditional probability problem to a round of golf, or a long piece of music. So what we do in teaching is that first we take the students step by step through the whole problem. This includes working out what the words are saying, putting the known values into a table, calculating the unknown values in the table, and the using the table to answer the questions involving conditional probability.

Then we work on the individual steps, isolating them so that students can get sufficient practice to find out what is general and what is specific to different examples. As we do this we endeavour to provide variety such that students do not work out some heuristic based on the wording of the question, that actually stops them from understanding. An example of this is that if we use the same template each time, students will work out that the first number stated will go in a certain place in the table, and the second in another place etc. This is a short-term strategy that we need to protect them from in careful generation of questions.

As it turns out students should already have some of the necessary skills. When we review probability at the start of the unit, we get students to calculate probabilities from tables of values, including conditional probabilities. Then when they meet them again as part of the greater whole, there is a familiar ring.

Once the parts are mastered, the students can move on to a set of full questions, using each of the steps they have learned, and putting them back into the whole. Because they are fluent in the steps, it becomes more intuitive to put the whole back together, and when they meet something unusual they are better able to deal with it.

Starting a course in Operations Research/Management Science

It is interesting to contemplate what “the whole” is, with regard to any subject. In operations research we used to begin our first class, like many first classes, talking about what management science/operations research is. It was a pretty passive sort of class, and I felt it didn’t help as first-year university students had little relevant knowledge to pin the ideas on. So we changed to an approach that put them straight into the action and taught several weeks of techniques first. We started with project management and taught critical path. Then we taught identifying fixed and variable costs and break-even analysis. The next week was discounting and analysis of financial projects. Then for a softer example we looked at multi-criteria decision-making, using MCDM. It tied back to the previous week by taking a different approach to a decision regarding a landfill. Then we introduced OR/MS, and the concept of mathematical modelling. By then we could give real examples of how mathematical models could be used to inform real world problems. It was helpful to go from the concrete to the abstract. This was a much more satisfactory approach.

So the point is not that you should always start with the whole and then do the parts and then go back to the whole. The point is that a teacher needs to think carefully about the relationship between the parts and the whole, and teach in a way that is most helpful.

Oh Ordinal data, what do we do with you?

What can you do with ordinal data? Or more to the point, what shouldn’t you do with ordinal data?

First of all, let’s look at what ordinal data is.

It is usual in statistics and other sciences to classify types of data in a number of ways. In 1946, Stanley Smith Stevens suggested a theory of levels of measurement, in which all measurements are classified into four categories, Nominal, Ordinal, Interval and Ratio. This categorisation is used extensively, and I have a popular video explaining them. (Though I group Interval and Ratio together as there is not much difference in their behaviour for most statistical analysis.)

Costing no more than a box of popcorn, our snack-size course will help help you learn all you need to know about types of data.

Costing no more than a box of popcorn, our snack-size course will help help you learn all you need to know about types of data, and appropriate statistics and graphs.

Nominal is pretty straight-forward. This category includes any data that is put into groups, in which there is no inherent order. Examples of nominal data are country of origin, sex, type of cake, or sport. Similarly it is pretty easy to explain interval/ratio data. It is something that is measured, by length, weight, time (duration), cost and similar. These two categorisations can also be given as qualitative and quantitative, or non-parametric and parametric.

Ordinal data

But then we come to ordinal level of measurement. This is used to describe data that has a sense of order, but for which we cannot be sure that the distances between the consecutive values are equal. For example, level of qualification has a sense of order

  • A postgraduate degree is higher than
  • a Bachelor’s degree,which is higher than
  • a high-school qualification, which is higher
  • than no qualification.

There are four steps on the scale, and it is clear that there is a logical sense of order. However, we cannot sensibly say that the difference between no qualification and a high-school qualification is equivalent to the difference between the high-school qualification and a bachelor’s degree, even though both of those are represented by one step up the scale.

Another example of ordinal level of measurement is used extensively in psychological, educational and marketing research, known as a Likert scale. (Though I believe the correct term is actually Likert item – and according to Wikipedia, the pronunciation should be Lick it, not Like it, as I have used for some decades!). A statement is given, and the response is given as a value, often from 1 to 5, showing agreement to the statement. Often the words “Strongly agree, agree, neutral, disagree, strongly disagree” are used. There is clearly an order in the five possible responses. Sometimes a seven point scale is used, and sometimes the “neutral” response is eliminated in an attempt to force the respondent to commit one way or the other.

The question at the start of this post has an ordinal response, which could be perceived as indicating how quantitative the respondent believes ordinal data to be.

What prompted this post was a question from Nancy under the YouTube video above, asking:

“Dr Nic could you please clarify which kinds of statistical techniques can be applied to ordinal data (e.g. Likert-scale). Is it true that only non-parametric statistics are possible to apply?”


As shown in the video, there are the purists, who are adamant that ordinal data is qualitative. There is no way that a mean should ever be calculated for ordinal, data, and the most mathematical thing you can do with it is find the median. At the other pole are the practical types, who happily calculate means for any ordinal data, without any concern for the meaning (no pun intended.)

There are differing views on finding the mean for ordinal data.

There are differing views on finding the mean for ordinal data.

So the answer to Nancy would depend on what school of thought you belong to.

Here’s what I think:

All ordinal data is not the same. There is a continuum of “ordinality” if you like.

There are some instances of ordinal data which are pretty much nominal, with a little bit of order thrown in. These should be distinguished from nominal data, only in that they should always be graphed as a bar chart (rather than a pie-chart)* because there is inherent order. The mode is probably the only sensible summary value other than frequencies. In the examples above, I would say that “level of qualification” is only barely ordinal. I would not support calculating a mean for the level of qualification. It is clear that the gaps are not equal, and additionally any non-integer result would have doubtful interpretation.

Then there are other instances of ordinal data for which it is reasonable to treat it as interval data and calculate the mean and median. It might even be supportable to use it in a correlation or regression. This should always be done with caution, and an awareness that the intervals are not equal.

Here is an example for which I believe it is acceptable to use the mean of an ordinal scale. At the beginning and the end of a university statistics course, the class of 200 students is asked the following question: How useful do you think a knowledge of statistics is will be to you in your future career? Very useful, useful, not useful.

Now this is not even a very good Likert question, as the positive and negative elements are not balanced. There are only three choices. There is no evidence that the gaps between the elements are equal. However if we score the elements as 3,2 and 1, respectively and find that the mean for the 200 students is 1.5 before the course, and 2.5 after the course, I would say that there is meaning in what we are reporting. There are specific tests to use for this – and we could also look at how many students changed their minds positively or negatively. But even without the specific test, we are treating this ordinal data as something more than qualitative. What also strengthens the evidence for doing this is that the test is performed on the same students, who will probably perceive the scale in the same way each time, making the comparison more valid.

So what I’m saying is that it is wrong to make a blanket statement that ordinal data can or can’t be treated like interval data. It depends on meaning and number of elements in the scale.

What do we teach?

And again the answer is that it depends! For my classes in business statistics I told them that it depends. If you are teaching a mathematical statistics class, then a more hard line approach is justified. However, at the same time as saying, “you should never calculate the mean of ordinal data”, it would be worthwhile to point out that it is done all the time! Similarly if you teach that it is okay to find the mean of some ordinal data, I would also point out that there are issues with regard to interpretation and mathematical correctness.

Please comment!

Foot note on Pie charts

*Yes, I too eschew pie-charts, but for two or three categories of nominal data, where there are marked differences in frequency, if you really insist, I guess you could possibly use them, so long as they are not 3D and definitely not exploding. But even then, a barchart is better. – perhaps a post for another day, but so many have done this.

Difficult concepts in statistics

Recently someone asked: “I don’t suppose you’d like to blog a little on the pedagogical knowledge relevant to statistics teaching, would you? A ‘top five statistics student misconceptions (and what to do about them)’ would be kind of a nice thing to see …”

I wish it were that easy. Here goes:

Things that I have found students find difficult to understand and what I have done about them.


When I taught second year regression we would get students to collect data and fit their own multiple regressions. The interesting thing was that quite often students would collect unrelated data. The columns of the data would not be of the same observations. These students had made it all the way through first year statistics without really understanding about multivariate data.

So from them on when I taught about regression I would specifically begin by talking about observations (or data points) and explain how they were connected. It doesn’t hurt to be explicit. In the NZ curriculum materials for high school students are exercises using data cards which correspond to individuals from a database. This helps students to see that each card, which corresponds to a line of data, is one person or thing. In my video about Levels of measurement, I take the time to show this.

First suggestion is “Don’t assume”.  This applies to so much!

And this is also why it is vital that instructors do at least some of their own marking (grading). High school teachers are going, “Of course”. College professors – you know you ought to! The only way you find out what the students don’t understand, or misunderstand, or replicate by rote from your own notes, is by reading what they write. This is tedious, painful and sometimes funny in a head-banging sort of way, but necessary. I also check the prevalence of answers to multiple choice questions in my on-line materials. If there is a distracter scoring highly it is worthwhile thinking about either the question or the teaching that is leading to incorrect responses.


Well duh! Inference is a really, really difficult concept and is the key to inferential statistics. The basic idea, that we use information from a sample to draw conclusions about the population seems straight-forward. But it isn’t. Students need lots and lots of practice at identifying what is the population and what is the sample in any given situation. This needs to be done with different types of observations, such as people, commercial entities, plants or animals, geographical areas, manufactured products, instances of a physical experiment (Barbie bungee jumping), and times.

Second suggestion is “Practice”. And given the choice between one big practical project and a whole lot of small applied exercises, I would go with the exercises. A big real-life project is great for getting an idea of the big picture, and helping students to learn about the process of statistical analysis. But the problem with one big project is that it is difficult to separate the specific from the general. Context is at the core of any analysis in statistics, and makes every analysis different. Learning occurs through experiencing many different contexts and from them extracting what is general to all analysis, what is common to many analyses and what is specific to that example. The more different examples a student is exposed to, the better opportunity they have for constructing that learning. An earlier post extols the virtues of practice, even drill!


One of the most difficult things is for students to make connections between parts of the curriculum. A traditional statistics course can seem like a toolbox of unrelated but confusingly different techniques. It takes a high level of understanding to link the probability, data and evidence aspects together in a meaningful way. It is good to have exercises that hep students to make these connections. I wrote about this with regard to Operations Research and Statistics. But students need also to be making connections before they get to the end of the course.

The third suggestion is “get students to write”

Get students to write down what is the same and what is different between chi-sq analysis and correlation. Get them to write down how a poisson distribution is similar to and different from a binomial distribution. Get them to write down how bar charts and histograms are similar and different. The reason students must write is that it is in the writing that they become aware of what they know or don’t know. We even teach ourselves things as we write.

Graphs and data

Another type of connection that students have trouble with is that between the data and the graph, and in particular identifying variation and distribution in a histogram or similar. There are many different graphs, that can look quite similar, and students have problems identifying what is going on. The “value graph” which is produced so easily in Excel does nothing to help with these problems. I wrote a full post on the problems of interpreting graphs.

The fourth suggestion is “think hard”. (or borrow)

Teaching statistics is not for wusses. We need to think really hard about what students are finding difficult, and come up with solutions. We need to experiment with different ways of explaining and teaching. One thing that has helped my teaching is the production of my videos. I wish to use both visual and text (verbal) inputs as best as possible to make use of the medium. I have to think of ways of representing concepts visually, that will help both understanding and memory. This is NOT easy, but is extremely rewarding. And if you are not good at thinking up new ideas, borrow other people’s ideas. A good idea collector can be as good as or better than a good creator of ideas.

To think of a fifth suggestion I turned to my favourite book , “The Challenge of Developing Statistical Literacy, Reasoning and Thinking”, edited by Dani Ben-Zvi and Joan Garfield. I feel somewhat inadequate in the suggestions given above. The book abounds with studies that have shown areas of challenge or students and teachers. It is exciting that so many people are taking seriously the development of pedagogical content knowledge regarding the discipline of statistics. Some statisticians would prefer that the general population leave statistics to the experts, but they seem to be in the minority. And of course it depends on what you define “doing statistics” to mean.

But the ship of statistical protectionism has sailed, and it is up to statisticians and statistical educators to do our best to teach statistics in such a way that each student can understand and apply their knowledge confidently, correctly and appropriately.

The flipped classroom

Back in the mid1980s I was a trainee teacher at a high school in Rotorua. My associate teacher commented that she didn’t like to give homework much of the time as the students tended to practise things wrong, thus entrenching bad habits away from her watchful gaze. She had  a very good point! Bad habits can easily be developed when practising solving equations, trigonometry, geometry.

Recently the idea of the “flipped classroom” has gained traction, particularly enabled by near universal access to internet technology in some schools or neighbourhoods. When one “flips” the classroom, the students spend their homework time learning content – watching a video or reading notes. Then the classroom time is used for putting skills to practice, interactive activities, group work, problem-solving – all active things that are better with the teacher around. Having a teacher stand at the front of the room and lecture for a large percentage of the time is not effective teaching practice.

I ws surprised at a teaching workshop to find that many of the teachers were not even aware of the concept of “flipping”. To me this is a case for Twitter as a form of professional development. To address this gap, I am writing about the flipped classroom, especially with regard to statistics and mathematics.

There are two important aspects to flipping – what the students do when they are not in class, and what students do when they are in class.

Work away from class

In theory, classroom “flipping” has always been possible. You could set students notes to read or sections of the textbook to study. In some schools and cultures this is successful, though it does presuppose a high level of literacy. Universities expect students to read, though my experience is that they avoid it if possible – unless they are taking Law, which of course means they can’t avoid it.

Technology has changed the landscape for flipping. With ready access to the internet it is feasible for video and other work to be set remotely for students. Sometimes teachers prepare the material themselves, and sometimes they may specify a YouTube video or similar to watch. This is not as easy as it may sound. As you can see from my critique of videos about confidence intervals, there is a lot of dross from which to extract the gold. And Khan Academy is no exception.

One big advantage of video over a live lecture, even if the video is merely a talking head, is that the student can control the pace and repeat parts that aren’t clear. My experience of lecturing to classes of several hundred students was that the experience was far from personal. I would set the pace to aim at the middle, as I’m sure most do. In later years I put all my lectures into short audio files with accompanying notes. Students could control the pace and repeat parts they didn’t understand. They could stop and think for a bit and do the exercises as I suggested, sometimes using Excel in parallel. They could quickly look through the notes to see if they even needed to listen to the audio. It was much more individualised.

Another advantage was that you can remove errors, stumbles, gaps and tighten up the experience. I’ve found a fifty minute lecture can be reduced to about half the time, in terms of the recording.

Despite this much more individual approach I was still expected to give lectures (that’s what lecturers do isn’t it?) until the Christchurch earthquakes made my mode of delivery expedient and we were able to stop physical lectures. The students could view the delivery of the material without coming to the university. They could then do exercises, also set up on the LMS, with instant feedback.

Work in the classroom

People tend to focus on what happens away from the classroom, when talking about flipped classroom. It is equally important to think hard about what happens in class. Having the teacher and peers there to help when working through problems in mathematics is better than being at a dead-end at home, with no one to help.  But week after week of turning up to class, working on numbered exercises from the textbooks doesn’t sound like much fun.

Taking the content delivery out of the classroom frees up the teacher for all sorts of different activities. It can be a challenge for teachers to change how they think about how to use the time. There are opportunities for more active learning, based on the grounding done on-line. In a mathematics or statistics classroom there is room for creativity and imagination. Debates, group work, competitions, games, looking for errors, peer review and peer-grading are all possibilities. If anyone thinks there is no room for imagination in the teaching of mathematics, they should take a look at the excellent blog by Fawn Nyugen, Finding ways to Nguyen students over.  I wish she had taught my sons. Or me. (Nguyen is pronounced “Win”)

I am currently working with teachers on teaching statistical report-writing. This is something that benefits from peer review and discussion. Students can work separately to write up results, and then read each other’s work. This is done in English and Social Science classes, and language classes. There is much we can learn from teachers in other disciplines.

Potential Problems

Students can also be resistant to change, and some coaching may be needed at the start of the year.

There is a big investment by teachers if they wish to create their own materials. Finding suitable materials on line can take longer than making your own. A team approach could help here, where teachers pool their resources and provide the “at home” resources and links for each other’s classes. I would be cautious not to try to do too much at once in implementing “flipped classroom”. It would probably be wise to start with just one class at a time.

Where internet access is not universal, there needs to be adaptations. It may be that the students can use school resources out of school time. Or students could take the material home on a memory stick.

Special needs

One issue to consider is the students who have special learning needs. In one Twitter discussion it was suggested that the flipped classroom is great because the student can learn the content when they have a helper (parent!) to assist. This is an admirable theory and I might have agreed had I not been on the other side. As a mother of a son with special needs, the thought of homework was often too much for me. The daily battle of life was enough without adding further challenge. In addition my son had been full-on all day and had little capacity for homework even if I had been willing. We need to avoid assuming ideal circumstances.

Try it!

Overall though, in appropriate circumstances, the concept of flipping has a lot going for it. It is always good to try new things.

If you never have a bad lesson or a failed new idea, you aren’t being daring enough!

Less is more

“Less is More” is a bit of a funny title for a mathematical blog!

Garlic bread and Ice Cream Sundaes

Back in the seventies, garlic bread became very popular in our household. I loved its buttery, salty, garlicky goodness, and made it quite often. One time I decided that if a little bit of garlic was yummy, then lots of garlic would be even more delicious. I was wrong! The garlic bread was barely edible, and the house and its occupants gave off a distinctive aroma for several days. More garlic did not mean “better”. From then on whenever I used garlic, I would recite in my head “More is not always better.”

Similarly it is fun to see children given a whole range of ice cream flavours, sauces and toppings and watch them create a dessert with EVERYTHING. From experience we know that there are only so many different forms of sugar and fat that should be added to ice cream at one time. If we are smart, we have several bowls, one with chocolate and nuts, one with caramel and crunchy toffee, one with raspberry and biscuit crumbs. That way we can appreciate the different flavours, without having them overridden. Having said that, we then discover that there comes a point of diminishing or even negative returns on investment. The final bowl of ice cream is often regretted.

Enough of food!

“Less is more” applies to teaching

The statement “Less is more” applies to teaching, and particularly subjects like Statistics and Operations Research.

As I learned with the garlic bread, we need to be careful not to give students too much. It is tempting, when developing on-line resources to keep including every possible activity, video and link that is relevant. However we have found that too many activities become overwhelming. It is tempting, as instructors to want to give plenty of practice and every possible resource. We assume that students can pick which items are useful for them. Instead we found that conscientious students want to complete EVERYTHING, and get discouraged when there is so much to do. They possibly don’t need to do all the activities, and waste their time on the easy ones.

We need to be selective about how we use our students’ time. Unless the homework or activity is going to help them learn and accomplish the goals of the course, it should not be there. I am reminded of the hell that was homework for my older son and me when he was going through middle-school. The teacher believed that more homework was better, and the result was misery in our family. Eventually I cried, “Enough!” and arranged an interview with her. I asked her for the specific learning objectives of the “worksheet”, which I know was an unfair question. Clearly the objective of worksheets is to keep the parents of conscientious girls (and the very uncommon conscientious boys) happy because their children were getting homework to do. She never did come up with learning objectives that satisfied me, so William (or rather, I) ceased to do her homework sheets, concentrating instead on times-tables, reading and handwriting. (Or generally nothing at all!)

But I digress. The point is – don’t waste student time on “busy” work. If students understand the process and internalise a skill after ten examples, then they do not need another ten. I DO believe in drill or practice, but it needs to be well developed and practising the skills we wish students to develop. For example there is no need for students to calculate by hand the standard deviation of ten sets of numbers devoid of context. However there is great value in large numbers of questions getting students to determine which test is appropriate in a given scenario.

If you really want to make more resources, rather than making more tests, provide a larger question bank for the current quizzes. That way students can do the quiz multiple times to achieve mastery, but those who have mastered the material immediately can move on.

We should not teach all we know

And as with the ice cream sundaes, when choosing content, what we leave out is as important than what we put in. We should not attempt to teach all we know. When writing the scripts for my videos I find it is important to stick to the main ideas and get them well explained. Sometimes total accuracy is sacrificed in the interests of comprehensibility. I come back to the dreaded question, “Where do babies come from?”, the answer to which depends enormously on the source of the question and context. Seldom is a full biological explanation required or even desirable.

Leonardo Da Vinci is purported to have said, “Simplicity is the ultimate sophistication.” It is the art of the true teacher to be able to reduce complex ideas into a simple form. Bill Bryson is the master of this. In his book, “A Short History of Nearly Everything”, Bryson puts forth complex ideas in ways that a layperson can understand. This is a skill I seek after as a teacher, and try to use in my videos and resources.

Choosing the statistical test – in simple terms

I was unhappy with the branching diagrams commonly used to teach how to choose a statistical test. I felt that there was a more integrative way to express this that would also help peoples understanding. I came up with quite a different diagram that is featured in our most popular video to date.

The students love it. But there are aspects about the diagram which could be looked at a different way. For example I ask “How many samples?”, and say that an independent samples t-test is used on two separate samples. Really it could also be defined as one sample with two variables – the measurement variable and another variable for group membership. When people are just coming to grips with new ideas, they don’t need to see multiple ways of doing things. If they are at the stage to see the other way of looking at it, they aren’t going to need the diagram.

Another very cool thing Da Vinci said was “Art is never finished, only abandoned.” On that note, I will stop now.

Statistical muscle memory

I am forever grateful to the teachers at my convent high school. In my first year I was required to take thirteen different subjects, one of which was typing. At the time computers were still objects mainly occurring in science fiction and operated by punch-cards, but the nuns thought we should get a wide exposure to different subjects (just in case I decided to be a typist/linguist/artist/scientist… instead of a maths teacher). Consequently I can touch-type, a skill which has been invaluable in my career as an academic. I don’t think about where my fingers are going – in fact when I do think about it, it tends to slow me down, and I make more errors. I’m grateful for all the subjects to which I was exposed, but typing is the one I use most often. When I am writing, the words go from my brain to the screen almost without effort.

Muscle memory comes from practice

Part of how this works is called “muscle memory” which “involves consolidating a specific motor task into memory through repetition”. (Wikipedia)  It is a combination of stuff happening in the brain in the conscious and unconscious mind. (You can tell I am not a neuro-scientist!). We all have physical and mental skills that we use without thought. Much of the time when we are walking, riding a bicycle, driving, playing an instrument, playing sport, swimming, the processes involved do not require intervention from our conscious thought. The way we get this “muscle memory” is through repetition. A toddler learning to walk spends a lot of time practising until walking becomes fluent and unconscious. Until then they have to sit down to concentrate on something else.  My son is a pianist, and spent hours on the piano as a young child, playing the same thing over and over until he could play as he wished. Now the piano is more like an extension of him, and he does not have to think about where his fingers are going. This is even more remarkable as he is totally blind and has been from birth.

Now wouldn’t it be nice if we could somehow replicate this level of competence in the area of learning mathematics, statistics and operations research. It isn’t so much “muscle memory” as mastery – becoming an expert.

I recently helped to organise a six or twelve hour Rogaine – a sporting event requiring cross-country running/hiking and navigation. At the first meeting of the committee, we were given maps of the area and asked to come up with about seventy potential checkpoints. I spent quite a bit of time studying the map, and the satellite view on Google, and managed to have some ideas. I realised what a novice I am when we met again, and Pete, one of the grand old men of rogaining, led us through the map. He could look at it and tell right away whether slopes went up or down, were steep or flat, likely to have swamps etc. I was in awe. By the end of the evening’s work my brain was mush.
This was the difference between a novice and an expert.

Novices and Experts

One of my favourite books about learning, “How People Learn: Brain, Mind, Experience and School” introduced me some years ago to the concept of novice and expert. It is a very helpful way to describe the process of mastering a skill or subject. All of us are experts in some areas. I know my times-table up to 12 automatically. Someone says 56 and automatically I think 7 times 8. Or 42, and I think 6 times 7. (Not 6 times 8 as in the Hitchhikers’ Guide to the Galaxy). I don’t have to think about it.

Similarly if I look at a histogram, it talks to me. I can see what is happening in the data, I can tell if the bin sizes are causing strange anomalies. I can tell the difference between two samples. Time series graphs are also no mystery. Trend and seasonality pop out at me. I can look at a scatter plot and have a rough idea of the correlation shown. In operations research I can hear of a problem situation, and right away I have a fair idea what techniques are going to be most appropriate in dealing with the problem. Many teachers are like this. And so we should be – as we are experts in our fields, compared to our students.

But our being experts doesn’t help students to become that way. Well not alone anyway. My experience with teaching statistics is that it takes a large number of different instances in order to see patterns. And the ability to see patterns is one of the main ways that an expert differs from a novice. It is only as I have engaged in statistical analysis and operations research that I have gained this knowledge. And it is as I have taught these subjects that I have become aware of what students find difficult.

Well-designed repetitive practice is needed

I have previously written on the need for repetition, calling the post Drill and Rote in teaching LP and hypothesis testing.  Rather than using the unpopular word “drill”, I like to talk about “well-designed repetitive practice”. I related it to two specific topics. I am currently revisiting this idea as I have been studying the NZ statistics curriculum and developing support materials for learning statistics. I have also been following the progress of a blogger who is taking a coursera course in statistics and blogging about his experiences. Though my experience of the coursera course is second-hand, I have noted a major flaw in it. (Several actually, but I’ll write a whole post about that when my friend has finished the course.) The major flaw is that there are no practice exercises, only graded assessments. This is really really bad, and I suspect that this is not an isolated example.

Too often the teaching/learning and the assessing are conflated in a way that means that neither is happening well. I suspect this happens often in NZ NCEA, but I would love to be proven wrong. Certainly it is endemic in many university courses at all levels. It’s a bit of a vicious cycle. Time-starved students tend only to put effort into activities which will affect their final grade directly. Many students do not seem to make the connection between learning in a low-stakes activity and the grade in the final assessment. Consequently they engage only in high-jeopardy activities, which behaviour is not conducive to understanding, and discourages risk-taking. Students take the “safe” road of finding out what they think the instructor wants them to say and then saying it. Instructors are torn regarding how much help they can give the students if the piece of work is meant to be the students’ own work. Consequently opportunities for teaching, coaching, questionning, discussion, and – dare I say it – learning are lost.

Unbundle learning and summative assessment

So where am I going with this? Teachers need to look at their curriculum plan and make sure that the students have enough learning opportunities before they are given a summative assessment activity. In the coursera course this could include providing on-line exercises and self-tests so that students can practice their knowledge. Students in turn need to avail themselves of non-assessed exercises in order to learn. It all sounds rather obvious, and idealistic.

In my “blended learning” course we found a solution that helped many of our students. It was based on the Personalised System of Instruction method, as developed by Fred Keller in the mid 1960s. There were many opportunities for students to repeat the practice tests until they gained mastery. The only way to fail the course was to fail to complete all of the assessments, correct to 80% before the end of the course. We had about the same pass/fail rate as traditional courses, a fact which I found interesting. I wonder if this could be adapted to school assessments. Students could be required to pass a “qualifiying” assessment, with which they can get help, before they are allowed to attempt the high-stakes assessment. I’d be interested to hear from teachers who may be already implementing this.

The style of question matters.

Whatever type of question is asked, is the kind of learning the students will do. For example we did not ask students to calculate standard deviations by hand, but rather got them to identify which graph showed the greatest or least variation (or consistency). We got them to critique graphs, identify flawed thinking in reports, and decide which test should be used in a specific instance. It can be tricky to assess higher order thinking, but it is possible, and over the years our questions have grown in sophistication to meet the needs we identified.

There must be multiple contexts, preferably using real data. When discovering patterns, students need to be able to tell what is general from what is specific to a given example or exercise.

Immediate feedback is very important so that students do not learn incorrect things. The on-line medium is ideal for this, and feedback can become very specifically targetted as we recognise common errors students make.

None of us like to do unpleasant tasks, and we wish to help students want to spend more time learning statistics. We continue to study motivational theory and try out different ways to help students engage. Success is a great motivator, and getting instant feedback and improving marks for repeated tries on tests works well. We are also looking at introducing some game elements.

Warning – advertising message ahead!

These are the principles we are using as we develop our Statistics Learning Centre. At present we are providing materials for Level 3 NCEA Statistics in New Zealand (NZStat3), and introductory business statistics just about anywhere (See AtMyPace: Statistics)! We hope to use our experiences providing these materials to develop and fine tune them, thus providing high quality resources for all students of statistics. In this way they will be able to “train” in a safe, fun and challenging environment as they develop their statistical muscle memory.