Introducing Infall-TT: fine-tuning for Tenstorrent
Note: Infall-TT is not available to the public yet. The code, commands and results in this post are examples of what we are building and may change before release.
We’re excited to introduce Infall-TT, our first open-source project. It is being built to bring quantization, fine-tuning and, later, inference to Tenstorrent Blackhole and Galaxy Neural Accelerators. Infall-TT is in developer preview and is not available to download yet. Some highlights:
- Native BFP4 and BFP8 quantization. 4.5 and 8.5 bits per weight, bit for bit what the chip holds.
- Accuracy that holds up. On Qwen3-0.6B, the perplexity loss of plain 4-bit rounding drops from 30.9% to 4.6% (5.3 bits per weight). At BFP8 the change is 0.06%.
- Matches the CPU reference. GPTQ with its heaviest step on a Tenstorrent device lands within 0.02–0.31% of the CPU result.
- Fine-tuning on a 4-bit base. LoRA on a frozen BFP4 base is in development.
- Built on Tenstorrent’s open stack, TT-NN and tt-train.
- Open source under Apache 2.0, with a free edition for a single Neural Accelerator, when it is released.
There is nothing to install yet. The package and the source code are not published, so the commands below are previews of how it will work. To be reminded the day the beta opens, join the waitlist.
Why we built it
Tools like Unsloth made fine-tuning open models fast and approachable on NVIDIA GPUs. Tenstorrent’s Blackhole chips are a different machine: they compute on tiles of 32×32 values and keep weights in their own block-float formats. A port of a GPU library leaves that on the table, so Infall-TT is written for Blackhole from the first line.
Native BFP4
In BFP4, every 16 consecutive weights share one 8-bit exponent and each keeps a sign and 3 magnitude bits. That is 4.5 bits per weight, read directly by the chip’s matrix engine. Infall-TT reproduces Tenstorrent’s packer bit for bit, so every quantized weight we produce is exactly what the chip will hold.
In the preview, quantizing a model is a single command:
# one commandinfall-tt quantize Qwen/Qwen3-8B-Base --out qwen3-8b-bfp4 --layout mixedTo keep the model the same model, we add GPTQ on the block-float grid, exponent search, low-rank corrections and dynamic per-layer layouts, and we measure every change against the original BF16 model.
Accuracy
The hard part of 4-bit is not making the model smaller. It is keeping it the same model. Each technique takes another bite out of the loss:
Each technique takes another bite out of the 4-bit loss
Qwen3-0.6B-Base, WikiText-2 perplexity increase vs BF16. Lower is better.
Calibration on C4, evaluation on WikiText-2 test (held out). Bits per weight count the stored 4-bit and 8-bit weights, their shared exponents and the BF16 corrections. 0.6B is the hardest case for 4-bit; larger models lose less.
And every recipe, at every bit budget, measured against the original BF16 model:
Accuracy for every bit budget
Each point is one quantization recipe. Down and to the left is better.
Qwen3-0.6B-Base, WikiText-2. Same setup as the chart above.
Fine-tuning
Fine-tuning on a 4-bit base is our QLoRA for Tenstorrent. The adapter layer already trains correctly on the Blackhole simulator, and PEFT export already loads in the official peft library. This API is a preview and still in development:
import infall_ttmodel, tokenizer = infall_tt.load_model("Qwen/Qwen3-8B-Base", load_in_4bit=True) # frozen BFP4 basemodel = infall_tt.lora.apply(model, rank=16, alpha=32)dataset = [{"prompt": q, "response": a} for q, a in my_examples] # your own data codetrainer = infall_tt.Trainer(model, tokenizer, dataset, max_steps=500, learning_rate=2e-4)trainer.train()model.push_to_hub("your-org/qwen3-8b-finetuned", kind="adapter")Free for one card, enterprise for the cluster
We plan to release infall-tt as open source under Apache 2.0, free and covering everything for a single Neural Accelerator. infall-tt-enterprise will be under a commercial license and will add multi-accelerator training on P300, QuietBox and Galaxy, with support from the team that builds it.
What’s next
State-of-the-art 4-bit accuracy on larger models, then GGUF export for llama.cpp and Ollama, reinforcement learning (GRPO and DPO), inference on a single card, and Galaxy-scale training.
Work with us
We are always seeking sponsorships to accelerate our research and grow Infall-TT, for the open-source community and for enterprises. If you would like to sponsor us, join us, contribute or build with Infall-TT, email us at hello@syntheticmindslab.com.
hello@syntheticmindslab.comTo be reminded when the beta opens, join the waitlist.
A note on results: device results so far come from ttsim, Tenstorrent’s bit-exact Blackhole simulator. It reproduces the chip’s arithmetic exactly but not its timing, so we publish no speed or throughput numbers until we measure on silicon. Accuracy is measured against the original BF16 model on held-out text the method never saw.
Fine-tuning for Neural Accelerators is coming.
Waitlist now open