Critique of: Patterns of Drug Use and the Influence of Gender on Self-Reports of Memory Ability in Ecstasy Users: A Web-Based Study

Critique of: Patterns of Drug Use and the Influence of Gender on Self-Reports of Memory Ability in Ecstasy Users: A Web-Based Study

Author(s): Rodgers J, Buchanan T, Scholey AB, Heffernan TM, Ling J, Parrott AC (2003) Journal of Psychopharmacology 17(4): 389-396

Full Text of Research Report in PDF Format Michael Diehr

This article, just released, is getting some press.

Oddly, the university has issued a Press Release along with this article, a practice which smacks a bit of self-promotion. As usual, the PR is being picked up with little or no change by various media outlets.

The press report makes claims as follows:

"People who take the recreational drug ecstasy risk impairing their memory, according to an international study which surveyed users in places including the UK, other European countries, the USA and Australia"

But the article itself is much more modest, concluding simply that :

"Recreational drug users perceive their memory ability to be impaired compared to non-users. The type of memory difficulties reported varies depending upon the drug of choice. These difficulties are exacerbated in ecstasy users. Individuals reporting average levels of use of ecstasy are more likely to report memory problems than non-ecstasy drug users or drug free individuals."

(Emphasis mine).

Methodological Flaws:

This study is rife with methodological flaws -- to their credit the authors do acknowledge most of them, but they are present nonetheless.

The data was collected anonymously without proper controls:

The data was collected solely via self report forms on the web. Thus, there was absolutely no ability to know that the data was in fact entered accurately or seriously or carefully. In fact (as discussed below) they come close to accusing one of the websites involved in stacking the data, and therefore they removed the data form consideration.

The data was heavily massaged:

They started with 1199 responses.
They discarded 435 of the responses because they were "suspect" (fraudalent, duplicates, etc.), leaving 763 responses, of which 282 were from supposed MDMA users.
Then the authors discarded 84 of the MDMA users' data because they felt it might be "biased".

The data came from users...

"...(recruited through a harm-reduction orientated website, which hosted much information and discussion of possible negative effects of ecstasy and possible ways to mitigate them) had different response patterns from the rest of the sample"

Later on they hypothesize:

"The website through which these people were recruited presents a very large amount of information about ecstasy, advice on possible techniques to protect oneself against harmful effects of drug use, and discussion forums where extensive (and wellinformed) debates on these topics occur. There was also discussion of our research on these forums, with people who had just participated posting comments about it. Taken together, all these observations suggest that there might be something unusual about this group of participants. One possibility is that their self-reports were biased (e.g. keen ecstasy users might wish to downplay any harmful effects it might have). Another is that they were implementing some of the potentially neuroprotective strategies discussed on the website, and that these were actually working."

After creating this highly biased and sub-sampled data set, they were left with a core sample of 199 MDMA users.

The measures were weak:

Data collected were all self-report, and included The Everyday Memory Questionnaire (EMQ) and the Prospective Memory questionnaire (PM), as well as substance use and demographic questions.
Strangely, no actual tests of memory were given. The authors apparently tried to cobble one together by looking at the number of errors made while filling out the EMQ, but this is not a standard, validated test.

The Effects were small

For the sake of argument, put aside all the methodological flaws and just look at the data, what conclusions can be drawn?

Reported MDMA users report more memory problems
Reported MDMA users make slightly more errors filling out the forms.

In every study, one must ask two questions about any given statistical test.

(A) is it statistically significant (meaning "Am I reasonably sure this result didn't come about from chance") and

(B) how large (or clinically significant, meaningful, or relevant) is the effect.

Because of the fairly large sample size (~200) the answer to (A) is going to be "yes" for most tests in this study if there is even the smallest difference between groups.

However, the answer to (B) is not guaranteed.

In fact, the two main claims they make are not terribly strong, so the answer to question (B) is closer to "no" than "yes" for both claims.

The evidence for claim 2 (MDMA users make more errors) is quite weak: The R-Squared value (the percent of variation in the measure explained) is only 1%. In other words, the difference between self-reported MDMA users and non-users is detectable, but is tiny.

Polysubstance use is not controlled for

The majority of the subjects who claimed to do MDMA also claimed to use other substances, and these were correlated with the questionnaires as well. The authors note this problem but do little to account for it.

Interpretation

A big problem with this study is that it is measuring not memory, but reported memory. It is well known in the field of neuropsychology that reports of memory typically correlate more highly with mood and depression than with actual memory performance. It is also well known that people's responses to questionnaires are biased by their expectations. In fact, the authors specifically excluded data from analysis because they worried it was biased in the direction of not showing the finding they expected.

In addition, there is the logical impossibility of using self-report measures to measure memory performance: The EMQ asks participants to answer questions about everyday memory difficulties over the past six months! A person with poor memory is expected to remember all of their memory failures and accurately report them?

Conclusion

At it's best, this study shows that after you remove people who don't believe MDMA causes memory problems, the remaining people claiming to be MDMA users also claim to have memory problems.

At it's worst, the authors have gone out of their way to massage and manipulate the data to fit their conclusion (and even then, the results are mild and unimpressive). The accompanying press release (author unknown) greatly exaggerates the scope, quality, and nature of the findings.