In its everyday use, ‘significant’ means ‘having a meaning or importance’. But a difference that is statistically significant might actually have very little importance in a practical sense. Suppose I’d actually done my headache pill experiment on a huge group of people. The experiment might indicate that my new pill only cures, let’s say, one more headache in a thousand than aspirin does, but even a small difference like this may be statistically significant if the number of people in the experiment is large enough. If my new pills cost, let’s say, a hundred times as much as aspirin, this very slightly increased performance may be of no practical significance at all, even if the difference is statistically significant. It could also happen that a result is not statistically significant but still has practical significance — we can’t rule out the possibility that the result is just due to chance, but it might indicate the need for a bigger experiment.

Now let’s turn to ‘reliable’. Suppose I have a way of measuring something. Many measuring techniques won’t always give you exactly the same result if you repeat the measurement again. But, in the statistical sense, a measuring method is said to be reliable if it tends to give similar numbers when you repeat the measurement. At home I’ve got a very accurate balance for weighing things, and I also have a set of bathroom scales that is rather old. If I weigh an object on the balance, and then weigh it again, I might not get exactly the same result, but I know that the two results will vary by only a small amount.. However, if I weigh something twice on my old bathroom scales, the results will differ more, on average. So, in the statistical sense, the balance is more reliable than the bathroom scales.

However, that doesn’t necessarily mean that the balance will give more accurate results. On both weighing devices you can set the zero; in other words you can see what reading they give when nothing is on the weighing surface, and adjust this reading so it is zero. Suppose I forgot to do this with the balance, and it actually read 200 grams with nothing on its weighing surface. Then I weigh the same thing several times. I would still get more or less the same result every time, but the recorded weights would all be about 200g too big. Now suppose I do the same thing with the bathroom scales, but adjusting the zero properly before I began. The results of repeated weighings will still differ more on the bathroom scales than on the balance, so the balance is still more reliable in the statistical sense. But, on average, the results from the bathroom scales might be nearer to the true weight of the object.

The clear message is that reliability, in the statistical sense, is not the only thing we should be concerned about. Validity, or lack of bias, are important too. These terms refer to how close the average result of the measuring process is to the true value of what’s being measured. With the zero set wrongly on the balance and right on the bathroom scales, measurements made on the balance are more reliable but less valid than those on the bathroom scales. These ideas of reliability and validity are used with social and psychological measurements rather more than with physical measurements, but the basic ideas are the same.

The upshot is that a measurement which is reliable in the statistical sense might not be reliable in the everyday sense — you might not want to rely on it or trust it, if its validity is not great enough.

So, if you hear someone talking about a result being ‘significant’ or ‘reliable’, you must always try to clarify the sense in which the words are being used. Maybe they aren’t saying quite what it sounds as if they are saying!

  < previous   Page 2 of 2

Comments