A Tale of Two Studies

While looking up information on how ammonia takes the sting out of stings, I ran across two studies, both double-blinded, placebo-controlled trials of a sting relief formulation. The one that mentioned ammonia was the one I read first, because that's what I was looking for. The other one named a product brand name I'd never heard of before; I read it by accident, clicking on the wrong link in the search results. These two trials came up as the top two results when I searched google scholar for ammonia mosquito bite relief.

The two studies are: Effectiveness of Ammonium Solution in Relieving Type I Mosquito Bite Symptoms: A Double-blind, Placebo-controlled Study and The efficacy of Prrrikweg® gel in the treatment of insect bites: a double-blind, placebo-controlled clinical trial.

Go ahead and read only the abstract; those are all I'm going to talk about, not the rest of the papers. The abstracts say it all.

In both trials, volunteers were bitten on both arms by lab mosquitoes then the bites were treated with either the test solution or a placebo, one on each arm, and nobody knew which arm got which treatment. (In the ammonia trial, because ammonia has such a distinctive smell, the researchers made the whole room stink of ammonia then everybody wore nose clips, all so they couldn't smell which treatment was which.)

In both trials, the volunteers were asked to rate how itchy the bites were after treatment. In the Prrrikweg trial, the welts were also measured for size. The ammonia trial was 90 minutes long because they were testing only immediate relief of itching, while the Prrrikweg trial was 48 hours long. I'm not sure how the latter stopped people from scratching the bites while they weren't under observation in the lab, which I've found makes them seem to get bigger. I've also had some mosquito bites disappear on their own in less than 48 hours while others hang on and itch for a week, so I would expect with a trial this long Prrrikweg would see a fair bit of placebo effect.

At any rate, the results and conclusions are where things get hilarious.

In the ammonia trial, there's a pretty clear result: no reduction (0% - not even a placebo effect) in itching on placebo-treated bites, while 64% of ammonia-treated bites either itched less or stopped itching entirely. Conclusion: ammonia isn't 100% (nothing is) but works very well at relieving the itch of mosquito bites for most people.

In the Prrikweg trial, there's also a pretty clear result: to quote directly from the abstract, "There was not a statistically significant difference between the itch relief provided by the two treatments." Conclusion, again quoted directly from the abstract: "an effect on itching is not inconceivable."

Um...... What?

The mental gymnastics required to draw that conclusion involved the extra measurements in the Prrrikweg trial, namely, welt size. They said they found a statistically significant difference in welt size with the test solution compared to placebo, and also a statistically significant correlation between welt size and itching intensity.

I won't comment on the p-values quoted in the abstract; my command of statistics isn't good enough for me to say anything about them confidently and I wouldn't want to be wrong. The site Science-based Medicine has a recent post on the relationship between p-values and the matrix of true or false; positive or negative results which I am still digesting, and which may provide something to think about in this context.

Even absent statistics, and accepting that if they said something was statistically significant it was, saying that a) there was no effect on itching, b) there was an effect on welt size, c) there is a correlation between welt size and itching, therefore despite (a), this treatment has a potential effect on itching... is just wishful thinking. And yet it is listed for sale as insect bite relief at various vitamin and supplement stores online, even though there is no evidence that it works.