Memory Games, Part IV: "Interpreted With Caution"
Open-Label Results Aren't All They're Cracked Up To Be
At this point in my ongoing series devoted to Cassava Sciences I’ve covered the investigations by major news outlets, a specific case study in how to implicate oneself in a coverup, and the orchestrated use of activist investors as the first line of defense against research misconduct allegations. A bit of a whirlwind, I admit. In the meantime, if you arrived here via a public forum like Twitter, you’ve probably also noticed (or are the source of) some, er, commentary, or perhaps questions from the audience or even allusions to a conspiratorial cabal accompanied by an advertisement for Cassava’s phase 3 clinical trials. And yes—in case you’re unaware, in the 14 months or so since the FDA citizen petition kicked off a cascade of ongoing investigations by “certain government agencies” (their words), hundreds of patients in multiple clinical trials have been actively taking simufilam. On the spectrum of how quickly things on this earth should be adjudicated, I’d personally put “potential fraud in experimental medications currently being ingested by humans” a ways down on the pretty damn quickly end. Granted, there don’t appear to be any safety concerns thus far, but that should hardly be any comfort if it does indeed turn out that all of the patients’ time, energy, opportunity cost, and money were squandered on a counterfeit.
I bring up these devoted fans and their chittering commentary because this edition will be a bit more freeform and informal—in case this wasn’t apparent already—and will dissect a few of the Cassava apologists’ favorite talking points. I’ll start with some of the simplest ones and work my way towards the more nuanced sleights of hand.
Who cares if Wang et al. touched up a few stupid blots? Well, I’m hoping that you do, dear reader. The record of concerning blots, overlapping tissue, incorrect antibodies, questionable methods, and downright physically impossible experiment descriptions among Dr. Wang’s papers is overwhelming and sits utterly unrefuted. In the modern jungle of crowdsourced post-publication scrutiny, it has become abundantly clear that we’re mired in a bit of a fraud epidemic (and, relatedly, a replication crisis) in scientific publishing. A review of biomedical literature by Bik et al. estimates that 4% of papers have image duplication issues, up from less than 1% in the 1990s—and that’s just for pure duplication, which is perhaps the easiest potential fraud to identify! Maybe this is too much to ask, but I’d posit that we, the diligent and responsible public, should care about strong signals of fraud whenever we’re confronted by them. That doesn’t mean jumping to immediate conclusions, but it does mean treating these kinds of issues with just a tab bit more wariness.
Yeah, but who cares about the science if the drug works? Ah, yes, the “fraud in the service of a greater good” argument. Look, we all want to help Alzheimer’s and dementia patients, and at the risk of sending Cassava stalwarts into a preemptive frenzy I’d even add that if there were clear and persuasive evidence that the drug was likely working, then it could absolutely make sense to devise measures in order to keep the trials running—even if the fraud allegations prove true. Such evidence does not currently exist, but before backing up that claim in more detail, I’ll remark that what I mean by “devise measures in order to keep the trials running” is not equivalent to “just pretend it’s all good.” Cassava’s patents for simufilam rely on data from papers that have since been retracted. If Cassava and their shareholders are inclined to continue the clinical trials without the necessary apparatus for maintaining IP or exclusivity rights, and a signal of reliable and statistically significant efficacy existed then sure, full steam ahead! Something tells me the prospect of simply becoming an altruistic benefactor would hasten the end of Cassava’s trials faster than a clinical hold, but we’ll almost certainly never know. As intriguing as this outcome would be for its pure novelty (has something like this ever happened before? I honestly have no idea), it feels like an obscure outcome. Plus, the data thus far is, well, let’s just get to that now…
But what about their open-label data? Aren’t they curing Alzheimer’s as we speak? Almost certainly not. Even if Cassava steers the ship through what looks like an impossibly stormy sea of fraud, there are still very compelling reasons to doubt much of what we’ve seen from their open-label data thus far. Let’s set aside momentarily the reasonable notion that an outfit which doesn’t appear to take the quality of their science very seriously may not take the quality of their clinical practices very seriously either and take the results at face value—what should we make of them?
We’ll start with what Cassava itself says about them, which is that the open-label results do “not constitute, and should not be interpreted as, evidence of therapeutic benefit for simufilam.” Why include this disqualifying statement immediately after slicing up your study statistics every which way so as to suggest resounding success? Well, for starters it’s an uncontrolled, unblinded study, which means that it’s not really set up to detect much aside from long-term safety signals—nice as it was of them to give us all these distribution details (we’ll get to them later), they don’t really mean anything in the context of showing that their drug works. ADAS-Cog scores are highly susceptible to noise effects, and the lack of a control group means that the noise is essentially uncorrected. They compare ADAS-Cog score improvement with the “expected decline” in order to compute their p-value, but reading over the meta-analysis they cite we can see that expected rates of decline depend on the initial ADAS-Cog score, and range from 3 to 6 points (they use the provided 5.5 point population average) for each patient given the MMSE scores reported at baseline (16 to 26). Slicing up the population into “improved” and “declined less than 5 points” is an exercise in describing a noisy distribution and nothing more.
Let’s take a closer look at the Ito et al. meta-analysis, because there’s more there than just a population decline estimate. They also review a few common treatments, the best of which was galantamine, which they conclude provides a mean of 4.88 points benefit compared to placebo at 52 weeks. For comparison, Cassava is claiming a 7 point benefit, but again, it’s from an uncontrolled and uncorrected open-label study versus a large meta-analysis of controlled galantamine studies, and Cassava’s results still fall well within the 90% confidence interval for galantamine (not to mention the other two drugs as well, and honestly not even particularly far outside of placebo).
It gets more suspicious from there, however, because these numbers are for the first 100 patients, and Cassava had reported numbers from the first 50 patients last year in which they claimed a 3.2 point improvement, meaning the “next 50” actually declined by 0.2 points. Because Cassava volunteered so much information about their score distributions, I was able to run a basic stochastic optimization (basically just a directed random walk until all criteria are satisfied) to back out sets of valid scores for every patient in each of the sets of 50. An example set is provided below, and it tells us with pretty high confidence (p = 0.01) that the two populations are not statistically identical. It’s really not that surprising of an outcome, but it’s emblematic of everything that’s wrong with hanging your hat on uncontrolled, unblinded trial results.
Even if the data is really shaky, isn’t there at least something there? Here’s where we come full circle, because if the preclinical research looked clean and thorough then a bit of cautious optimism might be warranted. However, if the basic science is indeed a mirage, then simufilam is merely a shot in the dark. In that case, everything we’ve seen out of Cassava’s trials so far is prone to the classic base rate neglect—which is to say that the probability of choosing a chemical at random and discovering that it improves Alzheimer’s is so low that even a “statistically significant” result is far more likely to be a false positive than a true discovery. It magnifies all of the doubts to the point where we can feel pretty certain that an uncontrolled, unblinded study should have a negligible effect on our prior, which is to say that a random chemical will almost certainly not cure Alzhemier’s.
In conclusion. It’s unfortunate that so many peoples’ hopes and dollars have been staked on a poorly conceived exploratory trial, and that Cassava has found ways to nudge these people on despite knowing full well that nothing they’ve presented so far should been taken as a reliable signal of efficacy. Mix in the appearance of unrefuted fraud in the preclinical research and selective data disclosure practices and things really start to crumble fast—and that’s even if you’re inclined to believe that the data collection practices were kosher in the first place! Meanwhile, Cassava’s CEO continues to engage in deceptive statistical equivocations when describing their data, pushing a narrative of “responders vs. non-responders” to describe an inherently noisy outcome measure without any fundamental reason to expect that drug response should differ among the treatment population. Let’s be clear about this, Cassava is engaging in a responder analysis without any justification or clear methodology, and they are doing so in order to obfuscate the shortcomings of their trial design. Do not fall for it. Be vigilant, and as always, continue to ask for the data—all of it, not just the portions that fit the company’s preferred narrative.