Thinking Machines Lab logo

Inkling-2-Mini-Infall_TT

GGUF quantization of thinkingmachines/Inkling-2-Mini by Thinking Machines Lab · Updated November 21, 2026

GGUFVisionReasoningToolsMoEApache 2.0
Inkling-2-Mini Infall-TT GGUF

What is Inkling-2-Mini-Infall_TT?

Inkling-2-Mini-Infall_TT is an 8-bit GGUF quantization of thinkingmachines/Inkling-2-Mini, released by Thinking Machines Lab. Inkling-2-Mini is a multimodal autoregressive looped transformer with 26.4B total parameters and 3.4B active per token, built as a mixture of experts. It accepts images and text, and supports reasoning and tool use.

The GGUF files are standard and run on any machine with llama.cpp. We publish seven files, from 18.7 GB (Q4_K_M) to 27.8 GB (Q8_0), so you can trade file size against fidelity to fit your memory.

Specification

Original modelthinkingmachines/Inkling-2-Mini
Model providerThinking Machines Lab, Inc.
Date of releaseNovember 20, 2026
LanguagesEnglish, general multilingual support
Model typeMultimodal autoregressive looped transformer
Parameters26.4B total, 3.4B active
Context length1,048,576 tokens (GGUF metadata)
FormatGGUF, importance-matrix quantization
LicenseApache 2.0
Quantized withInfall-TT (Synthetic Minds), for Tenstorrent hardware
DeveloperSynthetic Minds
Target hardwareTenstorrent Blackhole P150A/B, P300C, Galaxy Blackhole, or any machine running llama.cpp

Available quantizations

Each file is a single GGUF. File sizes are the exact sizes on Hugging Face. Quality figures are measured against the Q8_0 reference (see below).

QuantSizeTop-1 vs Q8_0Mean KLFile
Q4_K_MSmallest18.7 GB~96.8%~0.0059Download
Q5_K_S19.2 GB~97.3%~0.0041Download
Q5_K_M21.2 GB~97.5%~0.0037Download
Q6_K_S24.0 GB~97.7%~0.0028Download
Q6_K24.7 GB~97.8%~0.0027Download
Q6_K_L25.4 GB~97.9%~0.0026Download
Q8_0Reference27.8 GB100%0Download

Quantization quality

We compared every quant against the Q8_0 reference on wikitext-2. Smaller files cost a little fidelity, but every quant stays close to Q8_0. The numbers are approximate values read from the charts.

Output distortion (KL divergence to Q8_0)

Output distortion versus model size: KL divergence to Q8_0 logits, wikitext-2, 65,536 tokens
Mean KL divergence to the Q8_0 logits over 65,536 tokens. Lower means closer to Q8_0.

Distortion drops fastest between Q4_K_M and Q5_K_S. From Q6_K_S upward the gains are small, with Q6_K_L only marginally better than Q6_K_S.

Top-1 token agreement

Top-1 token agreement versus model size: how often each quant picks the same next token as Q8_0, wikitext-2
How often each quant picks the same next token as Q8_0. Higher is better.

Even the smallest quant, Q4_K_M, makes the same next-token choice as Q8_0 about 97 times in 100.

Perplexity

Perplexity versus model size on the full wikitext-2 test set, error bars shown
Perplexity on the full wikitext-2 test set. Lower is better. Error bars are shown on the chart.

Perplexity differences between all quants (about 0.01) are far smaller than the error bars (about ±0.035), so on this measure they are statistically indistinguishable from Q8_0.

QuantSize (GB)Mean KLTop-1Perplexity
Q4_K_M18.7~0.0059~96.8%~5.943
Q5_K_S19.2~0.0041~97.3%~5.936
Q5_K_M21.2~0.0037~97.5%~5.935
Q6_K_S24.0~0.0028~97.7%~5.931
Q6_K24.7~0.0027~97.8%~5.934
Q6_K_L25.4~0.0026~97.9%~5.933
Q8_027.80 (reference)100% (reference)~5.931

Download

Using the Hugging Face CLI:

hf download synthetic-minds/Inkling-2-Mini-Infall_TT-GGUF --include "Inkling-2-Mini-Infall_TT-Q8_0.gguf" --local-dir ./

Change the file name in --include to download a different quantization.

Run with llama.cpp

Install llama.cpp on macOS or Linux:

curl -LsSf https://llama.app/install.sh | sh

Or on Windows (PowerShell):

irm https://llama.app/install.ps1 | iex

Then start a server straight from the Hugging Face repo:

llama-server -hf synthetic-minds/Inkling-2-Mini-Infall_TT-GGUF:Q8_0

Or run the file you downloaded:

llama-server -m ./Inkling-2-Mini-Infall_TT-Q8_0.gguf

Run with Ollama

Install Ollama on macOS or Linux:

curl -fsSL https://ollama.com/install.sh | sh

On Windows, use the installer from ollama.com/download. Then run the model directly from the repo:

ollama run hf.co/synthetic-minds/Inkling-2-Mini-Infall_TT-GGUF:Q8_0

Evaluations

Selected results from the original model card, as reported by Thinking Machines Lab. They were measured on the original model, not on this quantization, so results from the 8-bit GGUF may differ slightly.

Inkling 2 MiniGemma 4 31BMuse GlimmerQwen3.8-27BNemotron 3.5 Super VL
Params (B) activated / total5/2631/3130/3027/2712/125
Overall
AA Index (v5.0)20.14.46.320.113.6
Coding
SWEBench Verified (bash only)79.2%46.1%76.0%77.8%71.1%
SWEBench Pro v2 (Hard)77.9%14.2%44.3%73.8%44.1%
Terminal Bench v2.171.6%43.4%51.7%73.0%60.7%
Agentic search & tool use
BrowseComp79.0%37.0%63.9%61.6%78.9%
DeepSearchQA85.3%61.7%74.6%79.7%77.7%
MCP Atlas82.7%54.2%75.5%77.3%67.7%
Vision
MMMU Pro (Standard 10)74.9%73.4%74.3%76.3%72.3%
OmniDocBench v1.692.8%82.5%90.4%91.5%77.9%
ScreenSpot Pro (original / with Python)76.5 / 87.7%5.9% / –75.4% / –57.6 / 78.9%43.5% / –
Reasoning & long context
HLE-Diamond15.2%7.2%9.1%15.3%8.8%
ARC-AGI (v1 / v2)84.5 / 49.3%––87.5 / 42.4%22.7 / 1.5%
LongBench v266.9%52.9%33.1%52.3%62.1%
Safety
FORTRESS (adversarial)89.5%67.8%90.3%78.8%79.6%
Misbehavior (internal) ↓9.8%27.6%46.4%57.7%52.8%

Bold marks the best reported score in each row (lowest where ↓ is shown). See the original model card for the full set of benchmarks and methodology.

Frequently asked questions

What is Inkling-2-Mini-Infall_TT?

Inkling-2-Mini-Infall_TT is a set of GGUF quantizations of Inkling-2-Mini, a multimodal autoregressive looped transformer from Thinking Machines Lab with 26.4B total and 3.4B active parameters (mixture-of-experts). Synthetic Minds quantized it with the beta preview of Infall-TT. The files are standard GGUF and run on any machine with llama.cpp.

How do I run Inkling-2-Mini-Infall_TT locally?

Install llama.cpp and run llama-server -hf synthetic-minds/Inkling-2-Mini-Infall_TT-GGUF:Q8_0, or run ollama run hf.co/synthetic-minds/Inkling-2-Mini-Infall_TT-GGUF:Q8_0. You can also download a file with the Hugging Face CLI and start it with llama-server -m.

Which quantization should I download?

Q8_0 (27.8 GB) is the reference. Q4_K_M (18.7 GB) is the smallest and still picks the same next token as Q8_0 about 97 times in 100 on wikitext-2 (96.8%). Distortion drops fastest between Q4_K_M and Q5_K_S; from Q6_K_S upward the gains are small.

Does quantization hurt quality?

Smaller files cost a little fidelity, but every quant stays close to Q8_0. Perplexity differences between all quants are about 0.01, far smaller than the error bars of about ±0.035, so on that measure they are statistically indistinguishable from Q8_0. Values are approximate, read from the charts on the model card.

Do the benchmark results apply to these GGUF files?

No. The benchmark table comes from the original model card published by Thinking Machines Lab. It was measured on the original model, not on this quantization, so results from the 8-bit GGUF may differ slightly.

Can I use it commercially?

The original model is released by Thinking Machines Lab, Inc. under the Apache 2.0 license, and these quantized weights follow the same license.

Does it run on Tenstorrent hardware?

The target hardware is Tenstorrent Blackhole P150A/B, P300C and Galaxy Blackhole, or any machine running llama.cpp. The GGUF files themselves are standard and run on any machine with llama.cpp. Infall-TT is in beta preview and is not publicly available yet.

Support our work

If this quant is useful to you, you can support Synthetic Minds on Buy Me a Coffee, or join the Infall-TT waitlist and follow us on X and Hugging Face.

License

The original model is released by Thinking Machines Lab, Inc. under the Apache 2.0 license, and these quantized weights follow the same license. Copyright 2026 Synthetic Minds LTD.