Dr Nic goes to ICOTS9

I had a great time at ICOTS9. Academic conferences are a bit of a lottery, but ICOTS is two for two for me. Both ICOTS8 and ICOTS9 were winners – enjoyable, interesting and inspiring.  I’ve just returned from ICOTS9 in Flagstaff, Arizona, several kg heavier, with lots of ideas for teaching and our videos, and feeling supported in the work I am doing on this blog, and with our resources and videos. I have met smart, good people who are genuinely trying to make things better in the world, by helping people learn about statistics.

Most times when I go to an academic conference, I feel happy if I get to one good, understandable and inspiring session per day. ICOTS conferences are different. I attended every session and just about all of the papers and presentations gave me something to take home.

Some aspects of conferences are communal, and some are more individual. The Keynote speakers provide a shared experience to discuss. The keynote speakers at ICOTS9 were all interesting and inspiring, and the highlight was Sir David Spiegelhalter. I’m a bit disappointed I didn’t get a photo with him, but I was hurrying off on the Grand Canyon expedition. (I was going to call it “Dr Nic meets Sir David”!) Monday’s keynote was with Pedro Silva, giving advice on how to maintain “fitness” as a professional statistician. I was impressed at how he saved up his own money to attend conferences and learn things. I am putting into place my own development plan. Then on Tuesday Sir David Spiegelhalter gave us a modified version of what he gives to schoolkids. Bacon and breast-cancer screening provided examples of risk interpretation. He reiterated the necessity of using frequencies rather than probabilities in communicating and working with questions of risk. The highlight for me, which I also tweeted, was Sir David’s statement that “combinatorics has no place in a course on probability.” I also appreciated his analysis of the PISA results, which are given far too much weight, when changes may nearly all be attributable to chance.

Wednesday’s keynote speaker, Rachel Fewster, was from New Zealand and gave some great examples of team-based learning at post-secondary level schooling. In fact the course she was talking about was second-year uni, equivalent to Junior year in the USA. I liked the idea of baking dice, and seeing if it changed the probabilities. Her students’ video on sample size effect was particularly engaging. Another idea that appealed to me was to randomly assign students to be spies and agents, and then use their results based on different criteria to detect how many spies and agents there were in each group. I’m sure there are many applications for such an activity.

Zalman Usiskin was the keynote on Thursday and discussed the integration of statistics into the whole school curriculum. I was fascinated by his epic effort to count all the numbers in a newspaper. There were 13,518 in the 64 pages of the main six sections of the newspaper. This included interesting problems of definition, which would provide some good discussion in a class activity. And finally, Friday’s keynote address was by Ronald Wasserstein, of the American Statistical Association. He teased us with the promise of a new website to be launched in August, thisisstatistics.org. This site is designed to give students a better idea of the prospects of a career in statistics. He also stressed that the most important skills for statisticians are not technical. We need to be able to communicate, collaborate, plan our career and develop leadership capacity.

In the parallel session, for me there were two main themes, which is probably because I chose sessions with these topics! I learned how younger students learn and understand probability, and the use of simulations and bootstrapping in teaching inference.

Highlights

  • The food, the excursion, the people – all excellent and memorable.
  • I was very excited that the paper I found most inspiring was also chosen for a prize.  Christoph Till did some interesting and rigorous experimentation to see how younger students understand ideas of uncertainty and risk, and if an intervention could improve that understanding. You can see the paper here. http://icots.net/9/proceedings/pdfs/ICOTS9_8I3_TILL.pdf One idea that appealed to me, was getting students to come up with their own scenarios around risk and probability.
  • The work by the people at Ludwigsburg is innovative and important and looks like fun.
  • I found out more about AP statistics, and was enticed by the idea of being an AP reader.
  • Tea and toast: Statisticians seem to be obsessed by dropping toast and seeing if the butter side goes down and by a lady who thinks she can detect if the milk was added before or after the tea. Many of the risk examples involve screening for different forms of cancer, and it would be nice if we can move as well to other scenarios such as lie-detectors and recruitment strategies.
  • The Island is still available for use as a teaching tool at RMIT, and we may be able to work with researchers to explore ways in which the virtual world can be used to teach different concepts. We have access to willing subjects, and they have the tools to assess and develop understanding.
  • I met lots of great people, including many who read this blog, and fellow tweeters. Hi! There is a wonderful atmosphere of cooperation at an ICOTS Conference.
  • One thing I was dying to find out, was where the next one will be held. ICOTS10 in 2018 is to be held in Kyoto, Japan. For once I won’t be too far from my time zone. I hope I see many of you there!

Big thanks to all the team, led by Roxy Peck and Roy St Laurent.

Roy, Dr Nic and Roxy at ICOTS9

It is so random! Or is it? The meaning of randomness

The concept of “random” is a tough one.

First there is the problem of lexical ambiguity. There are colloquial meanings for random that don’t totally tie in with the technical or domain-specific meanings for random.

Then there is the fact that people can’t actually be random.

Then there is the problem of equal chance vs displaying a long-term distribution.

And there is the problem that there are several conflicting ideas associated with the word “random”.

In this post I will look at these issues, and ask some questions about how we can better teach students about randomness and random sampling. This problem exists for many domain specific terms, that have colloquial meanings that hinder comprehension of the idea in question. You can read about more of these words, and some teaching ideas in the post, Teaching Statistical Language.

Lexical ambiguity

First there is lexical ambiguity. Lexical ambiguity is a special term meaning that the word has more than one meaning. Kaplan, Rogness and Fisher write about this in their 2014 paper “Exploiting Lexical Ambiguity to help students understand the meaning of Random.” I recently studied this paper closely in order to present the ideas and findings to a group of high school teachers. I found the concept of leveraging lexical ambiguity very interesting. As a useful intervention, Kaplan et al introduced a picture of “random zebras” to represent the colloquial meaning of random, and a picture of a hat to represent the idea of taking a random sample. I think it is a great idea to have pictures representing the different meanings, and it might be good to get students to come up with their own.

Representations of the different meanings of the word, random.

Representations of the different meanings of the word, random.

So what are the different meanings for random? I consulted some on-line dictionaries.

Different meanings

Without method

The first meaning of random describes something happening without pattern, method or conscious decision. An example is “random violence”.
Example: She dressed in a rather random faction, putting on whatever she laid her hand on in the dark.

Statistical meaning

Most on-line dictionaries also give a statistical definition, which includes that each item has an equal probability of being chosen.
Example: The students’ names were taken at random from a pile, to decide who would represent the school at the meeting.

Informal or colloquial

One meaning: Something random is either unknown, unidentified, or out of place.
Example: My father brought home some random strangers he found under a bridge.

Another colloquial meaning for random is odd and unpredictable in an amusing way.
Example: My social life is so random!

People cannot be random

There has been considerable research into why people cannot provide a sequence of random numbers that is like a truly randomly generated sequence. In our minds we like things to be shared out evenly and the series will generally have fewer runs of the same number.

Animals aren’t very random either, it seems. Yesterday I saw a whole lot of sheep in a paddock, and while they weren’t exactly lined up, there was a pretty similar distance between all the sheep.

Equal chance vs long-term distribution

In the paper quoted earlier, Kaplan et al used the following definition of random:

“We call a phenomenon random if individual outcomes are uncertain, but there is nonetheless a regular distribution of outcomes in a large number of repetitions.” From Moore (2007) The Basic Practice of Statistics.

Now to me, that does not insist that each outcome be equally likely, which matches with my idea of randomness. In my mind, random implies chance, but not equal likelihood. When creating simulation models we would generate random variates following all sorts of distributions. The outcomes would be far from even, but in the long run they would display a distribution similar to the one being modelled.

Yet the dictionaries, and the later parts of the Kaplan paper insist that randomness requires equal opportunity to be chosen. What’s a person to do?

I propose that the meaning of the adjective, “random” may depend on the noun that it is qualifying. There are random samples and random variables. There is also randomisation and randomness.

A random sample is a sample in which each object has an equal opportunity of being chosen, and each choice of object is by chance, and independent of the previous objects chosen. A random variable is one that can take a number of values, and will generally display a pattern of outcomes similar to a given distribution.

I wonder if the problem is that randomness is somehow equated with fairness. Our most familiar examples of true randomness come from gambling, with dice, cards, roulette wheels and lotto balls. In each case there is the requirement that each outcome be equally likely.

Bearing in mind the overwhelming evidence that the “statistical meaning” of randomness includes equality, I begin to think that it might not really matter if people equate randomness with equal opportunity.

However, if you think about medical or hazard risk, the story changes. Apart from known risk increasing factors associated with lifestyle, whether a person succumbs to a disease appears to be random. But the likelihood of succumbing is not equal to the likelihood of not succumbing. Similarly there is a clear random element in whether a future child has a disability known to be caused by an autorecessive gene. It is definitely random, in that there is an element of chance, and that the effects on successive children are independent. But the probability of a disability is one in four. I suppose if you look at the outcomes as being which children are affected, there is an equal chance for each child.

But then think about a “lucky dip” containing many cheap prizes and a few expensive prizes. The choice of prize is random, but there is not an even chance of getting a cheap prize or an expensive prize.

I think I have mused enough. I’m interested to know what the readers think. Whatever the conclusion is, it is clear that we need to spend some time making clear to the students what is meant by randomness, and a random sample.

 

Introducing Probability

I have a guilty secret. I really love probability problems. I am so happy to be making videos about probability just now, and conditional probability and distributions and all that fun stuff. I am a little disappointed that we won’t be doing decision trees with Bayesian review, calculating EVPI. That is such fun, but I gave up teaching that some years ago.

The reason probability is fun is because it is really mathematics, and puzzles and logic. I love permutations and combinations too – there is something cool about working out how many ways something can happen.

So why should I feel guilty? Well, in all honesty I have to admit that there is very little need for most of that in a course about statistics at high-school or entry level university. When I taught statistical methods for management, we did some probability, but only from an applied viewpoint, and we never touched intersection and union signs or anything like that. We applied some distributions, but without much theoretical underpinning.

The GAISE (Guidelines for Assessment and Instruction in Statistics Education) Report says, “Teachers and students must understand that statistics and probability are not the same. Statistics uses probability, much as physics uses calculus.”

The question is, why do we teach probability – apart from the fact that it’s fun and makes a nice change from writing reports on time series and bivariate analysis, inference and experiments. The GAISE report also says, “Probability is an important part of any mathematical education. It is a part of mathematics that enriches the subject as a whole by its interactions with other uses of mathematics. Probability is an essential tool in applied mathematics and mathematical modeling. It is also an essential tool in statistics.”

The concept of probability is as important as it is misunderstood. It is vital to have an understanding of the nature of chance and variation in life, in order to be a well-informed, (or “efficient”) citizen. One area in which this is extremely important is in understanding risk and relative risk. When a person is told that their chances of dying of some rare disease have just doubled, it is important that they know that it may be because they have gone from one chance in a million to two chances in a million. Sure it has doubled, but it still is pretty trivial. An understanding of probability is also important in terms of gambling and resistance to the allures of games of chance. And more socially acceptable gambling, such as stockmarket trading, also requires an understanding of chance and variation.

The concept of probability is important, and a few rules of probability may help with understanding, but I suspect the mathematicians get carried away and create problems that are unlikely (probability close to zero) to ever occur in reality. Anything requiring a three-way Venn Diagram has moved from applied problem to logic puzzle.This is in stark contrast to the very applied data-driven approach used in teaching statistics in New Zealand.

Teaching Probability

The traditional approach to teaching probability is to start with the coin and the dice and the balls in the urns. As well as being mind-bogglingly boring and pointless, this also projects an artificial certainty about the probabilities, which is confusing when we start discussing models. If you look at the Khan Academy videos (but don’t) you will find trivial examples about coloured balls or sweets or strangely complex problems involving hitting a circular target. The traditional approach is also to teach probability as truth. “The probability of getting a boy is one-half”. What does that even mean?

I am currently reading the new Springer volume, Probabilistic Thinking, and intend to write a review and post it on this blog, if I can get through enough before my review copy expires. It is inspiring and surprisingly gripping (but I don’t think that is enough of a review to earn me a hard copy to keep.). There are many great ideas for teaching in it, that I hope to pass on in due time.

The New Zealand approach to teaching probability comes from a modelling perspective, right from the start. At level 1, the first two years of schooling, children are exploring chance situations, playing games with a chance element and describing possible outcomes. By years 5 and 6 they are assigning numeric values to the likelihood of an occurrence. They (in the curriculum) are being introduced to model estimates and experimental estimates of probability. Bearing in mind how difficult high school maths teachers are finding the new approach, I don’t have a lot of confidence that the primary teachers are equipped yet to make the philosophical changes, let alone enact them in the classroom.

We are developing a whole series of videos, teaching probability from a modelling perspective. I am particularly pleased with the second one, which introduces model estimates of probability with an example with clear and logical assumptions, rather than the contrived “We assume the coin is fair”. I am hoping that with these videos we can help students and teachers embrace a more model-based approach – and no one will ever say “The weights of the lemons follow a normal distribution.” I also hope I can do this and still leave the fun in there.


Support Dr Nic and Statistics Learning Centre videos

This is a short post, sometimes called e-begging!
I had been toying with the idea of a Kickstarter project, as a way for supporters of my work to help us keep going. Kickstarter is a form of crowd-sourcing, which lets a whole lot of people each contribute a little bit to get a project off the ground.

But we don’t really have one big project, but rather a stream of videos and web-posts to support the teaching and learning of statistics. Patreon provides a more incremental way for appreciative fans to support the work of content creators.

You can see a video about it here:

And here is a link to the Patreon page: Link to Patreon

Rather than producing for one big publishing company, who then hold the rights to our material, we would love to keep making our content freely available to all. You can help, with just a few dollars per video.

A helpful structure for analysing graphs

Mathematicians teaching English

“I became a maths teacher so I wouldn’t have to mark essays”
“I’m having trouble getting the students to write down their own ideas”
“When I give them templates I feel as if it’s spoon-feeding them”

These are comments I hear as I visit mathematics teachers who are teaching the new statistics curriculum in New Zealand. They have a point. It is difficult for a mathematics teacher to teach in a different style. But – it can also be rewarding and interesting, and you never get asked, “Where is this useful?”

The statistical enquiry cycle shown in this video provides a structure for all statistical investigations and learning.

We start with a problem or question, and undergo an investigation, either using extant data, an experiment or observational study to answer the question. Writing skills are key in several stages of the cycle. We need to be able to write an investigative question (or hypotheses). We need to write down a plan, and sometimes an entire questionnaire. We need to write down what we find in the analysis and we need to write a conclusion to answer the original question. That’s a whole heap of writing!

And for teachers who may not be all that happy about writing themselves, and students who chose mathematical subjects to avoid writing, it can be a bridge too far.
In previous posts on teaching report writing I promote the use of templates, and give some teaching suggestions.

In this post I am concentrating on analysing graphs, using a handy acronym, OSEM. OSEM was developed by Jeremy Brocklehurst from Lincoln High School near Christchurch NZ. There are other acronyms that would work just as well, but we like this one, not the least for its link with kiwi culture. We think it is awesome (OSEM). You could Google “o for awesome”, to get the background. OSEM stands for Obvious, Specific, Evidence and Meaning. It is a process to follow, rather than a checklist.

The following video takes you a step at a time through analysing a dotplot/boxplot output from iNZight (or R). Through the example, students see how to apply OSEM when examining position, spread, shape and special features of a graph. This helps them to be thorough in their analysis. For the example we use real data. Often the examples in textbooks are too neat, and when students are confronted with the messiness of reality, they don’t know what to say.

I like the use of O for obvious. I think students can be scared to say what they think might be too obvious, and look for tricky things. By including “obvious” in the process, it allows them to write about the important, and usually obvious features of a graph. I also like the emphasis on meaning, Unless the analysis of the data links back to the context and purpose of the investigation, it is merely a mathematical exercise.

Is this spoon-feeding? Far from it. We are giving students a structure that will help them to analyse any graph, including timeseries, scatter plots, and histograms, as well as boxplots and dotplots. It emphasises the use of quantitative information, linked with context. There is nothing revolutionary about it, but I think many statistics teachers may find it helpful as a way to breakdown and demystify the commenting process.

Class use of OSEM

In a class setting, OSEM is a helpful framework for students to work in groups. Students individually (perhaps on personal whiteboards) write down something obvious about the graph. Then they share answers in pairs, and decide which one to carry on with. In the pair they specify and give evidence for their “obvious” statement. Then the pairs form groups of four, and they come up with statements of meaning, that are then shared with the class as a whole.

Spoon feeding has its place

On a side-note – spoon-feeding is a really good way to make sure children get necessary nutrition until they learn to feed themselves. It is preferable to letting them starve before they get the chance to develop sufficient skills and co-ordination to get the food to their mouths independently.

Teaching Confidence Intervals

If you want your students to understand just two things about confidence intervals, what would they be?

What and what order

When making up a teaching plan for anything it is important to think about whom you are teaching, what it is you want them to learn, and what order will best achieve the most important desired outcomes. In my previous life as a university professor I mostly taught confidence intervals to business students, including MBAs. Currently I produce materials to help teach high school students. When teaching business students, I was aware that many of them had poor mathematics skills, and I did not wish that to get in the way of their understanding. High School students may well be more at home with formulas and calculations, but their understanding of the outside world is limited. Consequently the approaches for these two different students may differ.

Begin with the end in mind

I use the “all of the people, some of the time” principle when deciding on the approach to use in teaching a topic. Some of the students will understand most of the material, but most of the students will only really understand some of the material, at least the first time around. Statistics takes several attempts before you approach fluency. Generally the material students learn will be the material they get taught first, before they start to get lost. Therefore it is good to start with the important material. I wrote a post about this, suggesting starting at the very beginning is not always the best way to go. This is counter-intuitive to mathematics teachers who are often very logical and wish to take the students through from the beginning to the end.

At the start I asked this question – if you want your students to understand just two things about confidence intervals, what would they be?

To me the most important things to learn about confidence intervals are what they are and why they are needed. Learning about the formula is a long way down the list, especially in these days of computers.

The traditional approach to teaching confidence intervals

A traditional approach to teaching confidence intervals is to start with the concept of a sampling distribution, followed by calculating the confidence interval of a mean using the Z distribution. Then the t distribution is introduced. Many of the questions involve calculation by formula. Very little time is spent on what a confidence interval is and why we need them. This is the order used in many textbooks. The Khan Academy video that I reviewed in a previous post does just this.

A different approach to teaching confidence intervals

My approach is as follows:
Start with the idea of a sample and a population, and that we are using a sample to try to find out an unknown value from the population. Show our video about understanding a confidence interval. One comment on this video decried the lack of formulas. I’m not sure what formulas would satisfy the viewer, but as I was explaining what a confidence interval is, not how to get it, I had decided that formulas would not help.

The new New Zealand school curriculum follows a process to get to the use of formal confidence intervals. Previously the assessment was such that a student could pass the confidence interval section by putting values into formulas in a calculator. In the new approach, early high school students are given real data to play with, and are encouraged to suggest conclusions they might be able to draw about the population, based on the sample. Then in Year 12 they start to draw informal confidence intervals, based on the sample. This uses a simple formula for the confidence interval of a median and is shown in the following video:

Then in Year 13, we introduce bootstrapping as an intuitively appealing way to calculate confidence intervals. Students use existing data to draw a conclusion about two medians. This video goes through how this works and how to use iNZight to perform the calculations.

In a more traditional course, you could instead use the normal-based formula for the confidence interval of a mean. We now have a video for that as well.

You could then examine the idea of the sampling distribution and the central limit theorem.

The point is that you start with getting an idea of what a confidence interval is, and then you find out how to find one, and then you start to find out the theory underpinning it. You can think of it as successive refinement. Sometimes when we see photos downloading onto a device, they start off blurry, and then gradually become clearer as we gain more information. This is a way to learn a complex idea, such as confidence intervals. We start with the big picture, and not much detail, and then gradually fill out the details of the how and how come of the calculations.

When do we teach the formulas?

Some teachers believe that the students need to know the formulas in order to understand what is going on. This is probably true for some students, but not all. There are many kinds of understanding, and I prefer a conceptual and graphical approaches. If formulas are introduced at the end of the topic, then the students who like formulas are satisfied, and the others are not alienated. Sometimes it is best to leave the vegetables until last! (This is not a comment on the students!)

For more ideas about teaching confidence intervals see other posts:
Good, bad and wrong videos about confidence intervals
Confidence Intervals: informal, traditional, bootstrap
Why teach resampling

The silent dog – null results matter too!

Recently I was discussing the process we use in a statistical enquiry. The ideal is that we start with a problem and follow the statistical enquiry cycle through the steps Problem, Plan, Data collection, Analysis and Conclusion, which then may lead to other enquiries. We have recently published a video outlining this process.

I have also previously written a post suggesting that the cyclical nature of the process was overstated.

The context of our discussion was another video I am working on, that acknowledges that often we start, not at the beginning, but in the middle, with a set of data. This may be because in an educational setting it is too expensive and time consuming to require students to collect their own data. Or it may be that as statistical consultants we are brought into an investigation once the data has been collected, and are needed to make some sense out of it. Whatever the reason, it is common to start with the data, and then loop backwards to the Problem and Plan phases, before performing the analysis and writing the conclusions.

Looking for relationships

We, a group of statistical educators, were suggesting what we would do with a data set, which included looking at the level of measurement, the origins of the data, and the possible intentions of the people who collected it. One teacher suggests to her students that they do exploratory scatter plots of all the possible pairings, as well as comparative dotplots and boxplots. The students can then choose a problem that is likely to show a relationship – because they have already seen that there is a relationship in the data.

I have a bit of a problem with this. It is fine to get an overview of the relationships in the data – that is one of the beauties of statistical packages. And I can see that for an assignment, it is more rewarding for students to have a result they can discuss. If they get a null result there is a tendency to think that they have failed. Yet the lack of evidence of a relationship may be more important than evidence of one. The problem is that we value positive results over null results. This is a known problem in academic journals, and many words have been written about the problems of over-occurrence of type 1 errors, or publication bias. Let me illustrate. A drug manufacturer hopes that drug X is effective in treating depression. In reality drug X is no more effective than a placebo. The manufacturer keeps funding different tests by different scientists. If all the experiments use a significance level of 0.05, then about 5% of the experiments will produce a type 1 error and say that there is an effect attributable to drug X. The (false) positive results are able to be published, because academic journals prefer positive results to null-results. Conversely the much larger number of researchers who correctly concluded that there is no relationship, do not get published and the abundance of evidence to the contrary is invisible. To be fair, it is hoped that these researchers will be able to refute the false positive paper.

Let them see null results

So where does this leave us as teachers of statistics? Awareness is a good start. We need to show null effects and why they are important. For every example we give that ends up rejecting the null hypothesis, we need to have an example that does not. Text books tend to over-include results that reject the null, so that when a student meets a non-significant result he or she is left wondering whether they have made a mistake. In my preparation of learning materials, I endeavour to keep a good spread of results – strongly positive, weakly positive, inconclusive, weakly negative and strongly negative.  This way students are accepting of a null result, and know what to say when they get one.

Another example is in the teaching of time series analysis. We love to show series with strong seasonality. It tells a story. (see my post about time series analysis as storytelling.) Retail sales nearly all peak in December, and various goods have other peaks. Jewellery retail sales in the US has small peaks in February and May, and it is fun working out why. Seasonal patterns seem like magic. However, we need also to allow students to analyse data that does not have a strong seasonal pattern, so that they can learn that they also exist!

My final research project before leaving the world of academia involved an experiment on the students in my class of over 200. It was difficult to get through the human ethics committee, but made it in the end. The students were divided into two groups, and half were followed up by tutors weekly if they were not keeping up with assignments and testing. The other half were left to their own devices, as had previously been the case. The interesting result was that it made no difference to the pass rate of the students. In fact the proportion of passes was almost identical. This was a null result. I had supposed that following up and helping students to keep up would increase their chances of passing the course. But they didn’t. This important result saved us money in terms of tutor input in following years. Though it felt good to be helping our students more, it didn’t actually help them pass, so was not justifiable in straitened financial times.

I wonder if it would have made it into a journal.

By the way, my reference to the silent dog in the title is to the famous Sherlock Holmes story, Silver Blaze, where the fact that the dog did not bark was important as it showed that the person was known to it.

Teach students to learn to fish

There is a common saying that goes roughly, “Give a person a fish and you feed him for a day. Teach a person to fish and you feed her for a lifetime.”

Statistics education is all about teaching people to fish. In a topic on questionnaire design, we choose as our application the consumption of sugar drinks, the latest health evil. We get the students to design questionnaires to find out drinking habits. Clearly we don’t want to focus too much on the sugar drink aspect, as this is the context rather than the point of the learning. What we do want to focus on is the process, so that in future, students can transfer their experience writing a questionnaire about sugar drinks to designing a questionnaire about another topic, such as chocolate, or shoe-buying habits.

Questionnaire design is part of the New Zealand school curriculum, and the process includes a desk-check and a pilot survey. When the students are assessed, they must show the process they have gone through in order to produce the final questionnaire. The process is at least as important as the resulting questionnaire itself.

Here is our latest video, teaching the process of questionnaire design.

Examples help learning

Another important learning tool is the use of examples. When I am writing computer code, I usually search on the web or in the manual for a similar piece of code, and work out how it works and adapt it. When I am trying to make a graphic of something, I look around at other graphics, and see what works for me and what does not. I use what I have learned in developing my own graphics. Similarly when we are teaching questionnaire design, we should have examples of good questionnaires, and not so good questionnaires, so that students can see what they are aiming for. This is especially true for statistical report-writing, where a good example can be very helpful for students to see what is required.

Learning how to learn

But I’d like to take it a step further. Perhaps as well as teaching how to design a questionnaire, or write a report, we should be teaching how to learn how to design a questionnaire. This is a transferable skill to many areas of statistics and probability as well as operations research, mathematics, life… This is teaching people to be “life-long learners”, a popular catchphrase.

We could start the topic by asking, “How would you learn how to design a questionnaire?” then see what the students come up with. If I were trying to learn how to design a questionnaire, I would look at what the process might entail. I would think about the whole statistical process, thinking about similarities and differences. I would think about things that could go wrong in a questionnaire. I would also spend some time on the web, and particularly YouTube, looking at lessons on how to design a questionnaire. I would ask questions. I would look at good questionnaires. I would then try out my process, perhaps on a smaller problem. I would evaluate my process by looking at the end-result. I would think about what worked and what didn’t, and what I would do next time.

This gives us three layers of learning, Our students are learning how to write a questionnaire about sugar drinks, and the output from that is a questionnaire. They are also learning the general process of designing a questionnaire, that can be transferred to other questionnaire contexts. Then at the next level up, they are learning how to learn a process, in this case the process of designing a questionnaire. This skill can be transferred to learning other skills or processes, such as writing a time series report, or setting up an experiment or critiquing a statistical report.

Levels of learning in the statistics classroom

Levels of learning in the statistics classroom

I suspect that the top layer of learning how to learn is often neglected, but is a necessary skill for success at higher learning. We are keen as teachers to make sure that students have all the materials and experiences they need in order to learn processes and concepts. Maybe we need to think a bit more about giving students more opportunities to be consciously learning how to learn new processes and concepts.

We can liken it a little to learning history. When a class studies a certain period in history, there are important concepts and processes that they are also learning, as well as the specifics of that topic. In reality the topic is pretty much arbitrary, as it is the tool by which the students learn history skills, such as critical thinking, comparing, drawing parallels and summarising. In statistics the context, though hopefully interesting, is seldom important in itself. What matters is the concepts, skills and attitudes the student develops through the analysis. The higher level in history might be to learn how to learn about a new philosophical approach, whereas the higher level in statistics is learning how to learn a process.

The materials we provide at Statistics Learning Centre are mainly fishing lessons, with some examples of good and bad fish.  It would be great if we could also use them to develop students’ ability to learn new things, as well as to do statistics. Something to work towards!

Why I am going to ICOTS9 in Flagstaff, Arizona

I was a university academic for twenty years. One of the great perks of academia is the international conference. Thanks to the tax-payers of New Zealand I have visited Vancouver, Edinburgh, Melbourne (twice), San Diego, Fort Lauderdale, Salt Lake City and my favourite, Ljubljana. This is a very modest list compared with many of my colleagues, as I didn’t get full funding until the later years of my employ.

Academic conferences enable university researchers and teachers from all over the world to gather together and exchange ideas and contacts. They range from fun and interesting to mind-bogglingly boring. My first conference was IFORS in Vancouver in 1996, and I had a blast. It helped that my mentor, Hans Daellenbach, was also there, and I got to meet some of the big names in operations research. I have since attended two other IFORS conferences, and it is amazing how connected you can feel to people whom you meet only every few years. I always try to go to most sessions of the conference as I feel an obligation to the people who have paid to have me there. It is unethical to be paid to go to a conference, and then turn up only for a couple of sessions and the banquet. Sometimes sessions that I have only limited connection with can turn out to be interesting. I found I could always listen for the real world application that I could then include in my teaching. That would usually take up the first few minutes of the talk. Once the formulas appeared I would glaze over and go to my happy place. Having said that, I also think mental health breaks are important, and would take time out to reflect. I get more out of conferences if I leave my husband at home. The quiet time in my hotel room was also important for invigorating my teaching and research.

Most academic conferences focus on research, though they often have a teaching stream, which I frequent. ICOTS is different though as it is mostly about teaching, with a research stream! ICOTS stands for International Conference on Teaching Statistics, and runs every four years. I attended my first ICOTS in Slovenia in 2010. What surprised me was how many people there were from New Zealand! At the welcome reception I wandered around introducing myself to people and more often than not found they were also from New Zealand. How ironic to spend 40 hours getting to this amazing place and meet large numbers of fellow kiwis! (Named for the bird, not the fruit!). Ljubljana is a wonderful city, with fantastic architecture and lots of bike routes and geocaches. I made good use of my spare time. The conference itself was inspiring too. I attended just about every session, and gave a paper about making videos to teach statistics. I saw the dance of the p-value, and learned about statistics teaching in some African countries. I was impressed by the keynote by Gerd Gigerenzer, and went home and cancelled my mammogram. I put faces to some of the names in statistics education, though I was sad not to see George Cobb there, or Joan Garfield. What struck me was how nice everyone was. I loved my trip to some caves on the half-day excursion.

The point of this post is to encourage readers to go to ICOTS 9 in July this year. I admit I was a little disappointed when they announced the venue. I was hoping for somewhere a little more exotic. However the great benefit is that it is going to cost considerably less to get there than to many countries, and take less time. (For people from New Zealand and Australia, a trip of less than 24 hours is a bonus.) Now that I am no longer paid by a university to go to conferences, the cost is a big consideration. If necessary I will sell our caravan. Another benefit of the venue is it is very convenient for teachers from the US to attend. I am hoping to find out more about AP statistics, and other US statistics teaching.

I am currently reviewing an edited book published by Springer, Probabilistic Thinking. As I read each chapter I am increasingly excited that most of the authors will be attending ICOTS9. This is a great opportunity to discuss with them their ideas, and how to apply them in the classroom and in our resources. I am particularly interested in the latest research on how children and adults learn statistics and probability. This ICOTS I am doing a presentation about setting up a blog, Twitter and YouTube. In four years’ time I hope to be able to add to the research using what we have learned from students’ responses on our on-line resources.

I am a little apprehensive about the altitude and temperature, but have planned to arrive a few days early in Phoenix to acclimatise myself. In the interests of economy I will be staying at the university dorms, and just found out there is no air-conditioning in the bedrooms. My daughter-in-law from Utah tells me to buy a fan. I’m pretty happy about a trip to the Grand Canyon on the afternoon off.  The names of presenters and their abstracts are now available on the ICOTS9 website, so you can see what interesting times await.

I really hope I see a lot of you there – and not just New Zealanders.

 

The Myth of Random Sampling

I feel a slight quiver of trepidation as I begin this post – a little like the boy who pointed out that the emperor has  no clothes.

Random sampling is a myth. Practical researchers know this and deal with it. Theoretical statisticians live in a theoretical world where random sampling is possible and ubiquitous – which is just as well really. But teachers of statistics live in a strange half-real-half-theoretical world, where no one likes to point out that real-life samples are seldom random.

The problem in general

In order for most inferential statistical conclusions to be valid, the sample we are using must obey certain rules. In particular, each member of the population must have equal possibility of being chosen. In this way we reduce the opportunity for systematic error, or bias. When a truly random sample is taken, it is almost miraculous how well we can make conclusions about the source population, with even a modest sample of a thousand. On a side note, if the general population understood this, and the opportunity for bias and corruption were eliminated, general elections and referenda could be done at much less cost,  through taking a good random sample.

However! It is actually quite difficult to take a random sample of people. Random sampling is doable in biology, I suspect, where seeds or plots of land can be chosen at random. It is also fairly possible in manufacturing processes. Medical research relies on the use of a random sample, though it is seldom of the total population. Really it is more about randomisation, which can be used to support causal claims.

But the area of most interest to most people is people. We actually want to know about how people function, what they think, their economic activity, sport and many other areas. People find people interesting. To get a really good sample of people takes a lot of time and money, and is outside the reach of many researchers. In my own PhD research I approximated a random sample by taking a stratified, cluster semi-random almost convenience sample. I chose representative schools of different types throughout three diverse regions in New Zealand. At each school I asked all the students in a class at each of three year levels. The classes were meant to be randomly selected, but in fact were sometimes just the class that happened to have a teacher away, as my questionnaire was seen as a good way to keep them quiet. Was my data of any worth? I believe so, of course. Was it random? Nope.

Problems people have in getting a good sample include cost, time and also response rate. Much of the data that is cited in papers is far from random.

The problem in teaching

The wonderful thing about teaching statistics is that we can actually collect real data and do analysis on it, and get a feel for the detective nature of the discipline. The problem with sampling is that we seldom have access to truly random data. By random I am not meaning just simple random sampling, the least simple method! Even cluster, systematic and stratified sampling can be a challenge in a classroom setting. And sometimes if we think too hard we realise that what we have is actually a population, and not a sample at all.

It is a great experience for students to collect their own data. They can write a questionnaire and find out all sorts of interesting things, through their own trial and error. But mostly students do not have access to enough subjects to take a random sample. Even if we go to secondary sources, the data is seldom random, and the students do not get the opportunity to take the sample. It would be a pity not to use some interesting data, just because the collection method was dubious (or even realistic). At the same time we do not want students to think that seriously dodgy data has the same value as a carefully collected random sample.

Possible solutions

These are more suggestions than solutions, but the essence is to do the best you can and make sure the students learn to be critical of their own methods.

Teach the best way, pretend and look for potential problems.

Teach the ideal and also teach the reality. Teach about the different ways of taking random samples. Use my video if you like!

Get students to think about the pros and cons of each method, and where problems could arise. Also get them to think about the kinds of data they are using in their exercises, and what biases they may have.

We also need to teach that, used judiciously, a convenience sample can still be of value. For example I have collected data from students in my class about how far they live from university , and whether or not they have a car. This data is not a random sample of any population. However, it is still reasonable to suggest that it may represent all the students at the university – or maybe just the first year students. It possibly represents students in the years preceding and following my sample, unless something has happened to change the landscape. It has worth in terms of inference. Realistically, I am never going to take a truly random sample of all university students, so this may be the most suitable data I ever get.  I have no doubt that it is better than no information.

All questions are not of equal worth. Knowing whether students who own cars live further from university, in general, is interesting but not of great importance. Were I to be researching topics of great importance, such safety features in roads or medicine, I would have a greater need for rigorous sampling.

So generally, I see no harm in pretending. I use the data collected from my class, and I say that we will pretend that it comes from a representative random sample. We talk about why it isn’t, but then we move on. It is still interesting data, it is real and it is there. When we write up analysis we include critical comments with provisos on how the sample may have possible bias.

What is important is for students to experience the excitement of discovering real effects (or lack thereof) in real data. What is important is for students to be critical of these discoveries, through understanding the limitations of the data collection process. Consequently I see no harm in using non-random, realistic sampled real data, with a healthy dose of scepticism.