Almost every day, it seems, someone
somewhere is announcing a new breakthrough that deepens
our understanding of the genetic roots of cancer. Each
opens "promising new avenues of research" and tantalizing
prospects of "new targeted therapies," but with the
major exception of the BRCA breast cancer genes, few
of these discoveries have led to revolutions in risk
prediction, diagnosis or treatment.
A new study in the Journal of
the National Cancer Institute may explain why —
a lot of these studies' results and conclusions may
be simply wrong. While the science of DNA microarray
expression profiling has raced ahead, the basic discipline
of sound statistical analysis has too often been left
behind, according to National Cancer Institute experts
Drs Richard Simon and Alain Dupuy.
Of the 42 studies they reviewed,
21 contained basic analytical flaws that rendered at
least some of their findings untenable, the authors
say. And even though most would probably only be comprehensible
to a geneticist, they were often widely read —
two-thirds of the studies reviewed appeared in journals
with an impact factor of more than six.
FATAL
FLAWS
The reviewers found three basic types of flaw. First,
outcome-related gene finding studies, which look for
differential gene expression between two populations
with different clinical outcomes, frequently applied
an unstated, unclear or inadequate control for multiple
testing.
Class discovery studies seek to
categorize patient groups based on similarities in particular
gene sequences, which are then compared to predict clinical
outcome. In these, the claim of correlation between
clusters and clinical outcome was often spurious, the
reviewers concluded, because researchers failed to use
available statistical tools for evaluating the robustness
of their findings.
The third type of analysis commonly
used in DNA microarray studies is supervised prediction.
The study seeks to build a "classifier" that can be
used to predict outcomes in similar patients. BRCA is
an example of this. But several studies made statistical
errors while validating their classifiers. And almost
incredibly, some cancer survival outcome measures simply
classified patients as "alive" or "dead", without considering
survival time, a surefire recipe for meaningless results.
STOPPED
AT THE PASS
It would be nice to think that this sort of statistical
shoddiness was confined to the pioneering frontier of
genetics, but another study has delivered an equally
damning verdict on phase II cancer drug trials. A lot
of new cancer drugs fall at this final hurdle, and the
new findings might help to explain why.
Researchers from Memorial Sloan-Kettering
Cancer Center reviewed 70 such trials that appeared
in either the Journal of Clinical Oncology or
Cancer, both leading publications in the field.
They concluded that only nine of the 70 clearly defined
measures by which an experimental drug could be judged
to offer benefit. What's missing from these studies,
says lead author Dr Andrew Vickers, is a reliable benchmark
of standard treatment against which to compare the experimental
drug.
"When a novel agent is added to
an existing standard in the hope of increasing response
rates over and above those expected from the standard
treatment alone, historical data on the response rates
to the standard treatment are required," he said. "Similarly,
some agents are thought to slow disease progression,
rather than lead to rapid tumour regression, necessitating
an endpoint such as progression-free survival or overall
survival at one year. That survival target clearly needs
to be developed by reference to historical data," he
writes.
But Dr Vickers found that 32 of
the 70 reviewed studies did not give any justification
for their historical bar or target. And of the studies
that did refer clearly to prior data, only nine did
so properly.
|
