BitNet-style ternary LLM. Latent FP32 + STE absmean quantization. 31M params, perplexity 5.0 in 10 minutes on a laptop GPU. 1.2 MB packed.
TernY trains all weights in latent FP32 with STE quantization, then hardens to ternary for inference. No FP32 math at deployment — just add and subtract.
TernY follows the BitNet b1.58 paradigm with key innovations: SubLN normalization, ReLU² activation, and per-channel FP16 scales.
Latent FP32 weights with absmean STE quantization to ternary {-1,0,+1}. Per-channel FP16 scale absorbs magnitude. No multiplications — just sign flips and zero-skips.
STE absmean · Per-channel scaleDouble LayerNorm (pre-attention + pre-FFN). Squared ReLU activation removes negative values entirely — natural fit for ternary computation.
SubLN · ReLU²Grouped Query Attention (6Q/3KV) with Rotary Position Embeddings. Tied embedding matrix (FP32). No bias terms. Clean, lean decoder-only.
6Q/3KV · RoPE · Tied embeddingsUnconditional generations from the 31M model after 10 minutes of training on a laptop RTX 3050.
Head-to-head comparison between the frozen random ternary approach (GOT v2) and the BitNet-style trained approach (TernY v3).
| Metric | TernY v3 (BitNet) | GOT v2 (Frozen) |
|---|---|---|
| Approach | Train all weights | 97% frozen random |
| Trainable params | 31.1M (100%) | 83K (2.3%) |
| Perplexity | 5.0 | ~28 |
| Loss (2K steps) | 1.61 | 6.90 |
| Training time | 10 min | - |
| Packed size | 1.2 MB | 1.1 MB |
| Text generation | ✓ Coherent | None |
| Scaling laws | Work | ✗ Plateau |
| Domain layer (DL) | ✓ Confirmed | ✓ Confirmed |
* TernY v3 trains all weights in latent FP32 with STE quantization. The insight: frozen ternary + per-neuron scale cannot escape the loss plateau. BitNet-style training unlocks true language generation.
TernY was built from scratch in under two weeks. Every iteration uncovered something that didn't work — and one thing that did.
97% weights random and frozen. Only per-neuron int8 scale trains. 3.6M params, loss 5.28→2.40. Works on curve fitting, not on language.
14M params, BPE 32k tokenizer. Loss 4063→823. But perplexity plateaus at ~28 — frozen ternary hits a fundamental ceiling.
Abandon frozen random. Train all weights in latent FP32, use absmean STE quantization, per-channel FP16 scale, SubLN, ReLU². Architecture reset.
31M model achieves perplexity 5.0 in 10 minutes on RTX 3050. Generates grammatically correct English. The approach is validated.
63.6M params (d=768, L=6, 8Q/4KV). Training on 82M tokens. Expected ~2 hours on RTX 3050. Scaling laws confirmed. 8Q/4KV is for the 64M model only.
From research prototype to practical deployment. The path forward is clear.
Complete training of the 64M parameter model on 82M tokens. Target perplexity < 4.0.
In ProgressStandard evaluation on MMLU, HellaSwag, and other LM benchmarks. Measure real capability.
PlannedPort ternary transformer inference to Cortex-M0+. Packed weights, no DSP, no FPU.
PlannedScale-delta domain modules (200 binary deltas per domain, ~2.7 KB each). Proven concept — now productize.
ConfirmedFormal publications and codebase references.
Full paper covering BitNet-style training, absmean STE quantization, SubLN architecture, domain-layer fine-tuning, and scaling laws for ternary transformers.
Concept DOIComplete codebase including training pipeline, tokenizer, packing, domain modules. Restricted access — available upon request.
Code Repository (Restricted)TernY is an independent research project exploring whether ternary transformers can generate coherent language — and they can.
TernY is a BitNet-style ternary transformer where every weight is {-1, 0, +1}. No FP32 inference. No matrix multiplication — just additions and subtractions. The model is trained end-to-end with latent FP32 weights and Straight-Through Estimator quantization, then hardened to ternary at inference.
At 31M parameters, TernY v3 achieves perplexity 5.0 after 10 minutes of training on a laptop RTX 3050. The 64M model is currently training. This is the starting line for practical ternary language models.
