The outrageous proposals of statisticians
Last week I gave a course about design of experiments at a major life science research institute. The audience was mainly young PhD students. There were hardly any post-docs or research leaders. I assume because post-docs and research leaders think they don’t need training in design of experiments anymore. Anyhow. I just want to give a picture of the audience to indicate that they are not at the top of the food chain. They need to struggle for a piece of a bench or a growth chamber. They get the poor corner of the greenhouse that nobody else wants to use. They have limited resources and are under time pressure. Unfortunately, they also seem to have least access to a statistician as well. Helping them out with their statistical questions is probably the most efficacious way a statistician can spend her or his time. This institute is in that respect not different from most other organisations I came across.
Typically, I ask in these courses whether they have good access to statistical advice when designing experiments. Last week I received again a common answer: “I have seldom the possibility to consult a statistician, but I also stopped looking for one, because they anyhow come up with outrageous proposals about replicate numbers.” This is for me the perfect intro to my talk on the need for more frequent and more meaningful interaction between scientists and statistician, and on the need to be clear and realistic about the research objective.
Nowadays, in the presence of modern and far more flexible analysis methods, we can also be more flexible on the design of the experiments. Statistics is not about cookbook recipes anymore. We can do more by using our resources wiser, and by incorporating our knowledge in design and analysis. The prerequisite however is that the statistician is aware of what the scientist really wants, that he understands what the priorities of the research are, that he understands the risks of obtaining a wrong conclusion, that he knows about the long-term objectives of the research program. There needs to be a tighter involvement of the statistician in the research program. It is the duty of both researcher and statistician to ensure that the statistician is part of the scientific team.
So how would a better involvement make the statistician’s proposal less outrageous?
Let’s first sketch the situation without involvement. A detached statistician is asked to provide the number of replicates needed to fulfil a list of goals. He carries out a power analysis assuming that all these goals are equally important and have to be reached. He bases the calculations on a conservative estimate of variances, to make sure he will not be blamed for an inconclusive experiment. He uses a simplistic model for the power calculations because otherwise he would need to invest more time programming extensive simulation routines for a project he does not feel a bond with. The result is a high, not to say outrageous, number of replicates. When he communicates this to the scientist he gets in the best case negative feedback, but usually the scientist shrugs her shoulders, goes back to the bench or growth chamber and does a smaller experiment as planned before.
A better communication would reveal that some of the experimental factors are actually not factors of which the levels need to be formally compared but are just needed to create variation in the test conditions. Some factors may be confounded. A simplification of the goals may reduce the need of replicates dramatically while maintaining the more important goals. At least a more elaborate power analysis could indicate that the initially hoped for difference requires many replicates, but a slightly bigger differences could be found with far less replicates.
Power studies are usually given as a single number (the power for n replicates to detect a delta d as significant). Sometimes you find graphs that provide this power to find a fixed delta in function of a varying number of replicates. Seldom you see this power graphs for varying delta, as shown below. The red line gives the power with 10 replicates. Ten was chosen because it leads to a power of 80% to find differences of 3 to be significant at the set alpha level. The graph shows that a difference of 4 has actually a power very close to 1 in this experiment. If 4 is actually a more realistic delta, but the scientist was a bit overoptimistic, the trial will be overpowered. If she see this graph she may settle for 7 replicates (the blue line) because those have enough power to find a delta of 4. Even 6 may do. Statistician and scientist will only come to these compromises if they discuss the broader research goals rather than the narrow talk about replicates. They need to study this sort of graphs.
A closer look at the proposed experiment may reveal some unnecessary and avoidable sources of variability. Measurements could be made more accurate. All this lead to the use of smaller and more realistic variance estimates in the power analysis, it would reduce the targeted number of replicates even further.
In the end, the compromise wil not be that outrageous anymore, the scientist will understand better why such an experiment is needed and will not venture in a pointless experiment or will have adapted her objectives to more realistic levels. An hopefully the gap between scientist and statistician closes further which ultimately leads to better science.
