GUILDENSTERN: ...Four: a spectacular vindication of the principle that each individual coin spun individually is as likely to come down heads as tails and therefore should cause no surprise each individual time it does.—"Rosencrantz & Guildenstern Are Dead" by Tom Stoppard

There has been a lot of bru-ha-ha over the last few days about the much anticipated discovery of what looks to be the Higgs Boson at CERN. Among many other things that you have probably read is the statement that the confidence that the signal is real is 99.9999%. You might be wondering, why so many 9's? That is, they had a signal a while back that was already 99% or thereabouts certain. If I had 99% confidence in winning the lottery I would go out right now and spend $1000 on lottery tickets. Why was a 99% confidence limit not good enough to indicate discovery? Indeed, the announced discovery, with 99.9999%, is at the statistical confidence level that is considered the *minimum* for a particle physicist to announce a discovery. Why do they have to be so damn confident?

Rather than talking about the energy spectra of interaction cross sections, let's talking about flipping coins. At the opening of Tom Stoppard's play *Rosencrantz & Guildenstern Are Dead*, the two courtiers are flipping coins (and have been doing so for some time). They are approaching a streak of 100 flips of heads in a row. Rosencrantz (who wins a coin each time it comes up heads) is not concerned about this, but Guildenstern is so disturbed by the seeming violation of the laws of probability that he philosophizes at length about what it is that's going on. (The real thing that's going on is that he's a character in a play, not a real person.) Let's keep it more modest, though.

Suppose I were to walk up to you with a quarter, and flip it six times in a row. If the quarter is normal, and if I'm not cheating, the probability that all six flips of the quarter will come down heads is about 1.5%. In other words, if I *do* flip six heads in a row, you can be 98.5% sure that it was not due to random chance, that I must have been cheating somehow. (Ask me to show you this sometime.) You're not 100% confident, because there *is* a small chance that six heads will come up in a row just randomly, but it is a very small chance... and so you would be well within your rights to think that something was probably up. It may not be good enough to convict somebody in a courtroom, but it's certainly good enough to bet on.

Suppose instead, however, that 30 people come up to you, and each one of them flips six coins in a row. The probability that *at least one* of those people will flip six heads in a row is 38%. So, while it won't happen every time this crowd of coin-flippers accosts you, you shouldn't be particularly surprised that *somebody* flipped six heads in a row if a whole bunch of people tried it. Even though it's extremely unlikely that *any given* coin flipper will flip the coin six times, the probably that somebody somewhere will is entirely reasonable. Lightning has to strike somewhere. (See Randall Munroe's much more concise take on this, and on overreactions to it.)

This same principle applies to particle physics. The particle physicists looking for the Higgs Boson were not sure at exactly what energy the particle would show up. Here's one of the plots from the CMS collaboration:

From the 2012 July 4 CMS Higgs Seminar; (c) CERN

The signature of the Higgs Boson is the extra bump of events at an energy of 125 GeV. There are lots and lots of events at all energies in the plot; there's a little something extra there, which indicates that something is going on there, and that something is probably the production of a short-lived Higgs boson. But they didn't know before they found it to look right at 125 GeV; it could have been at other energies, too. If all they were after was finding something that was "a little extra" at 95% confidence, they could have found it lots of places; indeed, there's a data point hanging out there at a bit over 135 GeV that is that far away from the background. But since there's 30 data points in the plot, I'm not the least bit surprised to see that. Randomly, you'd expect to see at least one of those more than 3/4 of the time somebody showed you a plot like this with 30 data points, even if there are no new particles.

The physicists in these collaborations were doing the equivalent of looking at a whole bunch of people flipping coins, and trying to find somebody who was flipping more heads than tails. If you look at 30 people who flip 6 coins and you find one person who has flipped 6 heads in a row, you have no right to declare that you've found a person who is cheating at flipping coins; the chances of that happening randomly are too high. Similarly, if you look at a whole bunch of different energies, and you see a single place where more is going on to 99% confidence than you'd expect from random fluctuations, you don't have much confidence that you've really found anything... because if you look at enough different energies, you will eventually find the unlikely random fluctuation. This is why for a particle physicist to be confident that she really has discovered something, she needs six nines in her confidence.

As for why the Higgs field (the "same thing" as the Higgs Boson... it's complicated) gives particles mass... *that* I *really* don't understand.

[...] The Higgs Boson and Statistics | Galactic Interactions [...]

[...] The Higgs Boson and Statistics | Galactic Interactions [...]

"Indeed, the announced discovery, with 99.9999%, is at the statistical confidence level that is considered the minimum for a particle physicist to announce a discovery. "

Your 99.9999% statement is wrong....

http://understandinguncertainty.org/higgs-it-one-sided-or-two-sided

Yeah, OK, although only a little bit wrong. It's 99.99994% confidence if they were allowing for either a high or low excursion from the background. If they went in assuming that what they were looking for was a high excursion (i.e. something extra going on, not something suppressed), then it should be 99.99997% confidence. Obviously, though,that small error in the last digit doesn't undermine the rest of my post.

(Also, when you're that far out, the assumption that all of your errors are normally distributed is probably not perfect. Yeah, there's the Central Limit Theorem, but we're far enough out on the tails that small deviations, correlations, or systematics can mean that the actual confidence limit isn't exactly that.)

What I mean is that you commit to the fallacy of the inversion of the p-values. More specifically

P(|x|>5σ | H_0 = True) is not the same as P( H_0 = False ||x|>5σ)

Those probabilities can actually be very dissimilar. 1 in 3.5 million is the likelihood of finding a false positive (conditional on us using perfect measurement devices, making the right statistical assumptions and using the right statistical model). Is not the likelihood of the alternative hypothesis being true!

See also this

http://itsastatlife.blogspot.co.uk/2012/07/sigma-tised.html

And yes we can not apply the CLT when our errors are not bounded in probability.

I apologize for my typing errors. Thank you very much for taking the time to reply to my comments...

Well, the post you linked to was all about one-sided vs. two sided.

Re: "confidence", the "99.99997% confidence" is the probability that *something* is going on, i.e. it's just not a random fluctuation. In your language, that's just saying that's the probability that the null hypothesis (i.e. it's a random fluctuation) is false. I don't think anybody is saying that this is definitely the Higgs boson even given that the signal is real. This confidence is the no-prior confidence, rather than the confidence that "it *IS* the Higgs boson or some other confidence where you come in looking for something other than "an extra bump in the spectrum". (I would argue that coming in with a prior like that is usually going to be based on guesswork or estimations that undermines quoting probabilities to this level of precision).

What I say is that the probability "that is something is going on" is not equal to 1-p_value. That is not so much a matter of language (even though it may sound like it) as it is a matter of using the right base rates. What you refer as the "no-prior confidence" it is just the probability that a set of observations generated **from the hypothesized model** (your Η_0) will fall into the +-5σ region.

Let me give you an example. We want to check whether a coin is fair (hence: H_0: p=0.5) . We throw the coin 1000 times and we observe 570 tails and 430 heads. The (one-sided) p-value of this observation conditional on the H_0: fair coin is 0.000004 and the probability of having less than 570 tails *by using a fair coin* is 99.9996%. As you can see the p-values and the confidence regions depend on what you choose as your null (which is assumed to be correct). The only way to derive the posterior probability ( the probability the the H_0 is false given the data) is by assuming (by using all knowledge prior knowledge available) a prior probability for the parameter. If, for example, you know the probability distribution of the parameter p (coin example ) you can certainly derive a posterior probability

"I would argue that coming in with a prior like that is usually going to be based on guesswork or estimations that undermines quoting probabilities to this level of precision"

In any case I believe that quoting probabilities to this level of precision is not a good thing to do. That is because you condition on everything else (measurement devices, theory, statistical assumptions etc.) being absolutely correct.

Let me just give you another example

The chance of a non-psychic being a lottery jackpot winner is 1 in 13,983,816. The chance of a lottery jackpot winner being a psychic (i.e. that *something* is going on) is definitely not 1 in 13,983,816!

Correction :

The chance of a lottery jackpot winner being a psychic (i.e. that *something* is going on) is definitely not 13,983,815 in 13,983,816!

However, the chance of the non-psychic NOT being a lottery winner is in fact 13,983,815 in 13,983,816 (assuming we're talking about a particular non-psychic).

"he chance of the non-psychic NOT being a lottery winner is in fact 13,983,815 in 13,983,816 "

Yes but what we are examining here is NOT the chances a particular non-psychic has. Our objective is to discover whether somebody has supernatural abilities based on the outcome of the lottery (a lady-tasting-tea kind of experiment).

[...] Edinburgh und FermiLab und Artikel hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier, hier und ein Videoclip 6., [...]

Interesting to see what a discovery needs in order to be called 'confirmed'

Perhaps my current blogpost on misunderstanding Higgs statistics and p-values would be of interest: errorstatistics.com Comments welcome.