carbon_pilot_corpus → carbon_tokenized_corpus → carbon_gpu_enrichment

GPU pipeline group · resumable, token-budgeted, bucketed inference

Click any block for details

carbon_cpu_enriched_sequences
CPU-enriched Parquet, full corpus (~32M rows)
carbon_pilot_corpus — sampling.py
Stratified ~25% subset, single streaming pass
carbon_tokenized_corpus — tokenize_and_tag.py · ProcessPoolExecutor (fork)
<dna> tag + QC filter
qc_flag == clean, uppercase
context truncate + 6-mer align
≤ 32,768 native tokens
Carbon 6-mer tokenizer
token_ids, token_mask, token_length
carbon_gpu_enrichment — embeddings.py · dg.multi_asset (group "gpu")
Bucket by token length
7 buckets, 512 → 32,768 tokens
Base-model forward + chunked lm_head
bf16, flash-attn2, torch.compile per bucket
OOM retry cascade
full → 1/2 → 1/4 → singleton → quarantine
carbon_embeddings
mean-pooled hidden state, per record_id
carbon_likelihood_stats
log-prob / perplexity, from the same logits
quarantine shard
rows no batch size could fit
Detail
Click any block above to see how it works.
Upstream input Sampling Tokenization GPU forward pass Embeddings output Likelihood output Quarantine