DNA promoter transformer

Problem

Promoter recognition is a sequence-classification problem: given a short stretch of DNA, decide whether it is a promoter. The signal is a localized motif at a fixed window position (the -10 and -35 boxes in bacteria), which is exactly the kind of local pattern a convolution detects well and a plain attention model, with no locality prior, struggles to learn from few examples.

The project pairs the two. A convolution stem over the embedded sequence picks up motif features, and a transformer encoder reads those features and classifies from a prepended CLS token. The aim is a clean pipeline: real labelled data, a model sized to the data, and evaluation that does not cut corners.

How it works

Each nucleotide is embedded, passed through a small 1D convolution stem that encodes local motif structure, then through a transformer encoder; classification reads the CLS token after a layer norm. The same architecture serves both datasets, with vocabulary and sequence length configured per task.

The small E. coli set is evaluated with 10-fold stratified cross-validation, so every sequence is held out once and the result sits alongside the published leave-one-out baselines as a close comparison. The genome-scale human set uses the authors’ own train/test split, with a validation slice carved off the training data for early stopping, and trains on the Apple MPS GPU when present (falling back to CUDA, then CPU).

DNA sequence -> nucleotide embedding -> conv stem (motif detector)
   -> prepend CLS token + positional embedding -> transformer encoder
   -> layer norm on CLS -> linear head -> promoter / non-promoter

Hard parts

• Learning from 106 examples: a plain transformer overfits badly on the E. coli set (31/106 errors). Adding the conv stem’s locality prior and shrinking the model drops that to 9/106, between the published backprop (8) and nearest-neighbour (13) baselines.
• Evaluation on a tiny set: 10-fold stratified cross-validation holds class balance in every fold and reports pooled error over all folds, rather than a single lucky split.
• Device portability: the trainer selects MPS, CUDA, or CPU automatically, so the same code runs on the workstation GPU and in a CPU-only CI environment.
• Reproducibility: both datasets are fetched from their public sources with no account required, and the tests assert the data parses to its known shape before any training runs.

Results

• E. coli promoters (UCI, 106 sequences, 10-fold CV): 9/106 errors. Published baselines on the same set: KBANN 4, backprop 8, nearest-neighbour 13, ID3 19.
• Human non-TATA promoters (genomic-benchmarks, 36,131 sequences): 0.889 accuracy, 0.916 precision, 0.875 recall, 0.895 F1 on the 9,034-sequence held-out test split, trained on the MPS GPU.
• The human figures come from a single early-stopped run and vary slightly by seed; they sit in the range the benchmark authors report for a CNN on this dataset.
• Test suite covers the data loaders, model wiring, fold stratification, and a cross-validation check that must clear the ID3 baseline.

Artifacts

A Python project with separate loaders for the E. coli and human datasets, a shared conv+transformer model, cross-validation and GPU training entry points, device selection, and a pytest suite. Datasets: UCI DOI 10.24432/C5S01D (CC BY 4.0) and Grešová et al. (2023), BMC Genomic Data 24:25.