Study design, workflows and reproducibility
Most failed omics projects fail in design or data management, not in algorithms. This chapter covers the practical layer: design, batch effects, file formats, public data, workflow managers and reporting.
Design before data
- Power: estimate from pilot or public data (RNA-seq: RNASeqPower, PROPER, ssizeRNA; scRNA-seq: scPower, hierarchicell; GWAS: genpwr). Biological replicates beat sequencing depth.
- Randomise and block: spread conditions across extraction days, library batches, flow cells and MS runs. Record everything in sample metadata.
- Controls: spike-ins (ERCC, SIRVs, sequins), pooled QC samples, reference materials (GIAB cell lines, Quartet multi-omics reference materials, NIST plasma), negative controls (blanks).
- Matched multi-omics: split the same aliquot across assays where possible; time from collection to freezing matters most for RNA and metabolites.
Batch effects
If batch is not confounded with the variable of interest, include it as a covariate in the model (preferred) or correct it: ComBat (empirical Bayes on continuous data), ComBat-seq (counts), limma removeBatchEffect (for visualisation only), RUVSeq/RUVg (using control genes), SVA (surrogate variables for unknown batches), and for single-cell, Harmony or scVI. Check before and after with PCA coloured by batch and by biology, kBET or LISI scores, and silhouette widths.
Correcting batch and then testing for group differences with the corrected data can exaggerate significance, especially when the design is unbalanced (Nygaard et al. 2016, Biostatistics). Model batch in the test instead of pre-correcting when possible.
File formats you will meet
| Format | Layer | Contains |
|---|---|---|
| FASTQ | Sequencing | Reads + per-base Phred qualities |
| POD5 / BAM (unaligned) | Long-read | Nanopore raw signal / PacBio reads with kinetics or modification tags (MM/ML) |
| SAM / BAM / CRAM | Alignment | Aligned reads, CIGAR strings, MAPQ, tags; CRAM is reference-compressed |
| VCF / BCF / gVCF | Variants | Variant records, genotypes, annotations |
| BED / narrowPeak / bigWig / bedGraph | Intervals, signal | Regions (0-based, half-open), peaks, coverage tracks |
| GTF / GFF3 | Annotation | Genes, transcripts, exons (1-based, inclusive) |
| .hic / .cool / .mcool / pairs | 3D genome | Contact matrices at multiple resolutions |
| h5ad / h5Seurat / RDS / MuData / SpatialData (zarr) | Single-cell, spatial | Count matrices + cell and gene metadata + embeddings |
| mzML / mzXML / raw | Mass spectrometry | Spectra; vendor raw files need conversion |
| mzTab / mzIdentML | Proteomics, metabolomics results | Identifications and quantities |
| BIOM / phyloseq / TreeSummarizedExperiment | Microbiome | Feature tables with taxonomy and trees |
BED is 0-based half-open; GTF, VCF and SAM are 1-based. Off-by-one errors between them are among the most common bugs in genomics code.
Public data you should know
Raw and processed data
- GEO, SRA, ENA, DDBJ (sequencing)
- EGA, dbGaP (controlled human data)
- ArrayExpress / BioStudies
- PRIDE / ProteomeXchange, MassIVE (proteomics)
- MetaboLights, Metabolomics Workbench, GNPS (metabolomics)
- CELLxGENE Census, Human Cell Atlas, Single Cell Portal
Reference cohorts
- TCGA / GDC, ICGC/PCAWG, CPTAC (cancer multi-omics)
- GTEx (tissue expression and QTLs)
- UK Biobank, All of Us, FinnGen, Biobank Japan
- gnomAD, 1000 Genomes, TOPMed, HPRC
- ENCODE, Roadmap, IHEC, 4DN (regulation)
- DepMap, LINCS/CMap (perturbations)
Knowledge bases
- Ensembl, GENCODE, RefSeq, UCSC
- UniProt, AlphaFold DB, PDB
- GO, Reactome, KEGG, WikiPathways, MSigDB
- ClinVar, ClinGen, OMIM, HPO, Orphanet
- HMDB, ChEBI, LIPID MAPS
- GWAS Catalog, Open Targets, STRING, OmniPath
Reproducible computation
- Workflow managers: Nextflow with the community-curated nf-core pipelines (rnaseq, sarek, atacseq, methylseq, scrnaseq, mag, ampliseq and more), Snakemake, WDL/Cromwell (Terra, GATK), CWL.
- Environments: conda/mamba (bioconda), Docker, Apptainer/Singularity for HPC; renv (R) and uv/poetry (Python) for analysis code.
- Version control: git for code; record reference genome build, annotation release, tool versions and parameters. DVC or data registries for large data.
- Notebooks and reports: Quarto, R Markdown, Jupyter. Keep exploratory notebooks separate from the final, scripted analysis.
- FAIR data: findable, accessible, interoperable, reusable. Use community metadata standards (MINSEQE, MIAPE, MIxS, Metabolomics Standards Initiative).
- Compute: WGS alignment and calling ≈ hours per genome on a server (minutes on GPU with Parabricks or DRAGEN); scRNA-seq with ~106 cells needs tens of GB of RAM or out-of-core tools; deep models need GPUs.