AmeforgeAmeforge
Internal ToolUsed in SparseMind v14 / v1532k Vocabulary

NexusBPE

Adaptive Multi-Domain Tokenizer Pipeline

NexusBPE is a controlled tokenization pipeline designed for training tokenizers on carefully balanced multi-domain corpora. It provides precise control over data composition, enabling reproducible and high-coverage tokenizers for research models.

View on HuggingFace
32k
Vocabulary Size
BPE tokens
99.9%
Character Coverage
Across target domains
5
Training Domains
Code, Math, Academic, Reasoning, Narrative
v14/v15
Used In
SparseMind architecture

Balanced multi-domain corpus

The tokenizer is trained on a carefully curated mixture of five public datasets, each representing a distinct linguistic domain. This composition ensures broad vocabulary coverage across code, mathematics, and natural language.

30%
CodeParrotCode

Programming languages, algorithms, software documentation

25%
WikiText-103Academic

Encyclopedic knowledge, structured academic writing

25%
OpenWebMathMathematics

Mathematical reasoning, proofs, formulas

10%
GSM8KReasoning

Grade school math problems, step-by-step reasoning

10%
TinyStoriesNarrative

Natural language, storytelling, conversational text

Key training parameters

python
# NexusBPE standard configuration
tokenizer_config = {
    "vocab_size": 32000,
    "character_coverage": 0.9999,
    "byte_fallback": True,
    "split_digits": True,
    "model_type": "bpe",
    "pad_id": 0,
    "bos_id": 1,
    "eos_id": 2,
    "unk_id": 3
}
Parameter notes
character_coverage: 0.9999

Ensures near-complete coverage of all characters in target domains

byte_fallback: True

Handles out-of-vocabulary characters via byte-level fallback encoding

split_digits: True

Separates digits for better mathematical expression tokenization

Synthetic data generator

The pipeline includes an integrated synthetic data generator that augments training data with programmatically generated examples, improving coverage for edge cases in mathematical and code domains.

python
# Synthetic data augmentation
generator = SyntheticDataGenerator(
    domains=["math", "code"],
    samples_per_domain=10000,
    seed=42
)

# Merge with real corpus
corpus = merge_corpora(
    real=load_datasets(config),
    synthetic=generator.generate(),
    weights=[0.85, 0.15]
)

Used in SparseMind v14 and v15

Vocabulary size32 000 tokens
Character coverage> 99.9%
Domains coveredCode, Math, Academic, Reasoning, Narrative
Evaluation datasetTinyStories (perplexity: 1.05)
Availability

NexusBPE is currently an internal research tool used to train tokenizers for our own models. The trained tokenizer for SparseMind v15 is available on HuggingFace for research purposes.

Note on differentiation

NexusBPE is a support tool rather than a standalone research contribution. Its value lies in providing a reproducible, controlled pipeline for our own training workflows, not in algorithmic novelty.

Download tokenizer on HuggingFace