Those worried about the cognitive bias modification field being affected by ever-moving goal posts may have thought their concerns confirmed by Grafton and colleagues’ re-analysis of the meta-analysis by Cristea and colleagues[1,2]. The paper concludes with the suggestion that we should only call CBM CBM if it is successful. To provide a treatment-inspired analogue: “This? No, this is just water, it’s only homoeopathy if it works”.
It seems that we witness an almost prototypical disagreement between experimentalists and treatment-evaluationists about which question to ask and which data to include. Importantly, the two author groups appear quite agreed that the answer to the question ‘whether assigning an anxious individual to engage in a CBM procedure will result in direct symptom reduction’ would be ‘not likely’.
Perhaps Grafton and colleagues had better directed their critical attention towards the work by ‘field-insiders’ in which CBM is quite consistently touted as a treatment, not to mention the apparent push for clinical dissemination and premature commercial exploitation. Thus, the question meta-analysed by Cristea and colleagues, authors specialising in meta-analytical evaluation of (proposed) treatments, appears perfectly legitimate.
Grafton and colleagues’ exposé on the correct question to meta-analyse, reads uncomfortably like a perceived-damage-containing mission. The discomfort is aggravated by the presented re-analysis, applying dichotomising and partly mystifying criteria to distil a subset of eligible studies from those selected for the original meta-analysis. Specifically, the requirements for a study to pass criterion 3, “effect size computed by Cristea et al reflects legitimate bias assessed emotional vulnerability assessment” (p.268), remains unknown, as do the rules governing the final dividing criterion “intended CBM procedure successfully induced the process of bias modification” (p.268).
One could attempt to re-construct the criterion-rules from the tables provided, but it matters little. The analysis by Grafton and colleagues is flawed in a manner that must have escaped the attention of authors, reviewers, and editors alike, even after Cristea and colleagues pointed it out in their commentary[3]. To be very explicit: Grafton et al. meta-analysed the study effect-size estimates calculated by Cristea et al.
In their original paper[1], Cristea and colleagues state clearly that a) for studies reporting multiple symptom outcomes, these were averaged into a single effect-size estimate (p.8), and b) effect-size estimates reflect symptoms assessed post-training, excluding assessments following a stressor-procedure (p.9).
Based on the narrative, it appears that criterion 3 has to do with each study either a) assessing symptoms on trait (rather than state) measures, yet effect-size estimates averaging across state and trait measures were analysed, or b) employing a post-training stressor-procedure, yet symptoms assessed preceding such stressor-procedures were analysed. Surely, we are not to assume reliable retro-active impact of unannounced stressors, nor that excluding studies with state measures only, results in adjustment of state measures retained for other studies. Therefore, we must conclude that this small yet crucial detail has gone unnoticed.
A meta-analysis by Grafton and colleagues, assessing evidence for their hypotheses, could perfectly exist alongside the meta-analysis by Cristea and colleagues. The currently presented re-analysis, however, does not convince.
Text: Anne-Wil Kruijt (postdoctoral fellow Stockholm University)
& Per Carlbring. Originally published in British Journal of Psychiatry as an eLetter.
Photo: Benjamin Reay
References:
1 Cristea IA, Kok RN, Cuijpers P. Efficacy of cognitive bias modification interventions in anxiety and depression: meta-analysis. Br J Psychiatry 2015; 206: 7–16.
2 Grafton B, MacLeod C, Rudaizky D, Holmes EA, Salemink E, Fox E, et al. Confusing procedures with process when appraising the impact of cognitive bias modification (CBM) on emotional vulnerability: A response to Cristea et al. (2015). Br J Psychiatry 2017; 211: 266–71.
3 Cristea IA, Kok RN, Cuijpers P. Invited commentary on … Confusing procedures with process in cognitive bias modification research†. Br J Psychiatry 2017; 211: 272–3.
Link to our published eLetter:
Kruijt, A.-W. & Carlbring, P. (2017). eLetter: Processing confusing procedures in the recent re-analysis of a cognitive bias modification (CBM) meta-analysis, British Journal of Psychiatry.