Benchmark hub
What independent, neutral comparisons of tools have found. Benchmarks age quickly and can be biased by their choice of data and metrics, so read them for patterns and for what they measured, not only for the winner.
How to read a benchmark
Who ran it?
Neutral benchmarks (by groups that did not develop any of the tools) are more trustworthy than the comparison table in a new tool's own paper, where the new tool almost always wins (the "self-assessment trap").
What is the ground truth?
Simulations (controllable, but may not match reality), spike-ins and mixtures (realistic, limited), reference materials (GIAB, Quartet), orthogonal validation, or consistency only (weakest).
Which metric?
Precision, recall, F1, AUROC, AUPRC, FDR calibration, runtime, memory. A tool can win on one and lose on another. Check whether the metric matches your use.
Does it match your data?
Species, platform, depth, sample size and tissue all change rankings. A method that wins on cell lines may not win on tumour biopsies.
Guidelines for running a good benchmark: Weber et al. 2019, Genome Biology. Continuous, community-maintained benchmarks (Open Problems in Single-Cell Analysis, OpenEBench, precisionFDA challenges, CAMI, CASP) are replacing one-off papers.