This paper introduces Motif-Vocab, a new way to split DNA sequence for machine learning that explicitly finds short regulatory patterns called motifs and labels them with the transcription factor (TF) that recognizes them. DNA here means deoxyribonucleic acid. Motifs are short, partly variable sequences where TF proteins bind to control gene activity. The authors’ goal is to give a language model tokens that map directly to known biological signals, rather than only to letters or fixed-length substrings.

The authors build Motif-Vocab to scan both strands of DNA with position-weight matrices (PWMs). A PWM is a simple table that scores how well each short sequence matches a TF motif. Because raw PWM scores are not comparable across motifs of different lengths or information content, the method calibrates each motif against a null score distribution computed from the training background. Matches that pass a significance threshold become TF-identity tokens. Regions that do not match any motif are represented by fallback tokens: single bases, fixed-length k-mers, or byte-pair encoding (BPE), a common subword method from natural-language processing. The tokenizer also resolves overlapping motif calls deterministically and stores token spans so models and users can trace tokens back to specific nucleotide positions.

To test the idea, the authors pretrained a BERT (Bidirectional Encoder Representations from Transformers) model with masked language modeling on two billion base pairs, keeping the encoder and training data fixed while changing only the tokenization. Real motif libraries outperformed randomized-motif controls on 54 of 55 evaluated downstream tasks. In a motif-disjoint recognition task derived from DART-Eval Task 2, TF-specific tokens improved macro-F1 by 0.040 over a position-matched generic motif token and by 0.033 over a matched no-motif tokenizer (95% bootstrap confidence interval: 0.027–0.038). The motif tokens also received stronger attribution and caused larger occlusion effects, which supports the idea that the model uses the TF labels when making predictions. At the same time, dense no-motif tokenizers (those without motif labels) remain strong baselines and even came close on a five-task BERT-base panel.