NexusBPE
Adaptive Multi-Domain Tokenizer Pipeline
NexusBPE is a controlled tokenization pipeline designed for training tokenizers on carefully balanced multi-domain corpora. It provides precise control over data composition, enabling reproducible and high-coverage tokenizers for research models.
View on HuggingFaceTraining Data Composition
Balanced multi-domain corpus
The tokenizer is trained on a carefully curated mixture of five public datasets, each representing a distinct linguistic domain. This composition ensures broad vocabulary coverage across code, mathematics, and natural language.
Programming languages, algorithms, software documentation
Encyclopedic knowledge, structured academic writing
Mathematical reasoning, proofs, formulas
Grade school math problems, step-by-step reasoning
Natural language, storytelling, conversational text
Configuration
Key training parameters
# NexusBPE standard configuration
tokenizer_config = {
"vocab_size": 32000,
"character_coverage": 0.9999,
"byte_fallback": True,
"split_digits": True,
"model_type": "bpe",
"pad_id": 0,
"bos_id": 1,
"eos_id": 2,
"unk_id": 3
}character_coverage: 0.9999Ensures near-complete coverage of all characters in target domains
byte_fallback: TrueHandles out-of-vocabulary characters via byte-level fallback encoding
split_digits: TrueSeparates digits for better mathematical expression tokenization
Synthetic data generator
The pipeline includes an integrated synthetic data generator that augments training data with programmatically generated examples, improving coverage for edge cases in mathematical and code domains.
# Synthetic data augmentation
generator = SyntheticDataGenerator(
domains=["math", "code"],
samples_per_domain=10000,
seed=42
)
# Merge with real corpus
corpus = merge_corpora(
real=load_datasets(config),
synthetic=generator.generate(),
weights=[0.85, 0.15]
)Integration
Used in SparseMind v14 and v15
NexusBPE is currently an internal research tool used to train tokenizers for our own models. The trained tokenizer for SparseMind v15 is available on HuggingFace for research purposes.
NexusBPE is a support tool rather than a standalone research contribution. Its value lies in providing a reproducible, controlled pipeline for our own training workflows, not in algorithmic novelty.
