A GGUF File Is Not a Download, It Is a Binary: Sandboxing Local Model Loading

I pull my models from the community. A 20 GB file shows up on a repo I trust, I download it, I point the server at it, and the first prompt comes back clean. That last part is the problem. The file parsed. Parsing a 20 GB binary of someone else's bytes is code doing work on my machine, and on the Malwlab box the process doing that work can also read my API key and my conversation history. "It worked" is not a safety result. It is the incident report for a very quiet compromise.

A GGUF is a container format for tensors and metadata. The loader has to walk that structure, allocate memory for what it describes, and hand the result to a runtime. Every local AI tool I run does this: Ollama, llama.cpp server, LM Studio, Jan. They all take a file that a stranger made and parse it in a process that is already sitting next to credentials, a model store, and, on most homelabs, the rest of the host's data. That is a supply chain execution path. It looks like a file download because the interface is a download bar, but the parser is the code that runs.

The reason I treat that code as hostile now is a May 2026 advisory on oss-security that documented six flaws in the llama.cpp GGUF parser. None of the six received a CVE. As of this week the two most serious are still open in the C++ loader on master. The critical one is an unbounded alignment field in the header: a crafted file can drive the parser's seek arithmetic out of range. It has a public proof of concept on Hugging Face. The second allocates up to a gigabyte per string and a gigabyte per array before it notices. The Python loader picked up partial guards in August. The C++ loader did not. I checked the source at three points in history and the headline flaws are still there in the current release.

That last fact is the whole reason this lab note exists. You cannot pin your way out of it. There is no build that fixes the two flaws that matter, because the fixes are not written yet. A version floor protects you from what has been CVE'd. It says nothing about what is still open, and for the loader that reads a stranger's file, the open flaws are the point. The control that covers that gap is not a number in a release tag. It is a boundary the file cannot cross. That boundary is a sandbox, and the rest of this note is how I built it.

One more incident from this quarter makes the operational point. Ollama shipped a fix in release 0.17.1 on February 25, 2026 that added tensor-bounds validation to its GGUF loader, via pull request #14406. The CVE for the flaw (CVSS 3.1 9.1, a memory leak through the unauthenticated model-creation endpoint that exposes process memory, including API keys and conversation data) was not published until May. The fix landed two months before the CVE existed. The 0.17.1 changelog does not mention it. Ollama did not issue a security advisory. If your only signal for what changed in a loader is a changelog entry or a CVE feed, you will run the vulnerable version into the public report and never know the safe one was available. I cite it not to score Ollama but because the same pattern shows up across this stack: the fix ships, quietly, and the version you are on may already be the wrong one without anyone telling you.

The exposure is broader than my rack. In a September 2025 Shodan-based study, Cisco Talos found 1,139 publicly reachable Ollama instances, with more than 1,000 of them found in the first ten minutes of scanning, and 214 actively hosting models and answering requests, roughly 19 percent of the total. By the May 2026 CVE disclosure, Cyera put the count of internet-exposed Ollama deployments at around 300,000. Two different studies, two different dates, and the number only went up. The local AI stack is a real attack surface that people run on machines they also trust with their data. My inference box is not exposed to the internet. It does not need to be. The threat model that matters here is a hostile file, not a hostile neighbor.

The environment

My build floor is met. Build 752 is well past the marker that closed the CVE'd parser flaws. But as I noted, the two open flaws from the May advisory have no safe version to pin to, so a clean build-floor check did not change what I needed to do.

The exposure, measured

I read the loader's own process environment. It is clean. No API key, no token, nothing secret. That is a good hygiene result and I would not pass it up. It is also not isolation. A clean process environment means the secret is not sitting in the variables the parser inherits. It does not mean the parser cannot reach the secret. The key lives as a plaintext file on the same machine, owned by the server user, readable by anything running as that user. The conversation data sits in a database on the host. The realistic compromise of a hostile GGUF is a memory corruption in the parser that becomes code execution in a process that owns the key file and can open the database. The sandbox's job is to make that read fail. Not to hope the parser is careful.

The recipe

Two machines. The split is the design, and everything else hangs off it.

1. A separate intake machine

The intake box is a small VM whose only job is to fetch and vet model files. It has the egress the registries need: Hugging Face, GitHub, whatever mirrors I pull from. It has no path to the inference host's data and no credentials for it. A compromise of the intake box hands an attacker a downloader, not my keys. On the intake box each file gets a SHA-256, a pass through YARA and a file-type check, and a slot in a manifest that records the hash, the source, and the date.

2. A loader with no egress

The inference host runs the loader in a locked-down VM, or a container with the same properties. The properties that matter, in order:

• No egress: llama-server does not need the network once the GGUF is on local disk. I disable outbound networking on the loader entirely. It is the control most homelabs skip because it is the least convenient, and the one that turns a "file" back into a file.
• Read-only root filesystem: the loader gets a read-only root and a small tmpfs for scratch. It should not need to write anything permanent, and if a corruption tries to persist something, it has nowhere to put it.
• Non-root user: the loader runs as a dedicated account that owns nothing except the model directory. No shell, no sudo, and no path to the key file or the database, which live on a separate partition with 0600 on the key.
• Bind mount, read-only: the file arrives as a single read-only mount. The loader reads tensors. It does not get the rest of the box.

Concretely, the loader runs with a bind-mounted, read-only model directory, no network, and the model user:

docker run -d \
  --name llama-sandbox \
  --read-only --tmpfs /tmp:size=256m \
  --network none \
  --user 1000:1000 \
  --cap-drop ALL --security-opt no-new-privileges \
  -v /srv/models:/models:ro \
  llama-rocm:752 \
  /opt/llama/bin/llama-server \
    --model /models/model-35b-a3b.Q4_K_M.gguf \
    --n-gpu-layers 99 --ctx-size 200000

3. Verified promotion, one direction

Files move intake to inference on a one-way channel I control. On the inference host a small job verifies the arriving file's hash against the manifest before the loader is allowed to mount it. The loader never pulls from the internet and never sees the intake box's trust. If a file fails the hash or the scan, it is quarantined on the intake side and never promoted.

What I tested, and what I have not

I am going to be precise about this part because a lab note that pretends the drill is done is worse than one that says it is not.

What works: the loader runs with egress disabled and serves the 20 GB model. GPU inference is unchanged. The key file is on a separate partition the loader cannot read. The build floor is met for the CVE'd flaws.

What does not fit the sandbox yet: my second inference lane runs vLLM, and it cannot run offline. It pulls weights from Hugging Face at startup, and the proxy in front of it falls back to external providers. There is no air-gapped path for that lane without pre-stashing the weights and stripping the fallbacks. I have not done that yet. The recipe in this note is proven on the llama.cpp lane only.

What I have not done: the crash drill. I have not taken a known-bad file and watched the sandbox contain it. That is the next session, and it will be run on the intake side, where a failure is a quarantine event instead of a host compromise.

What I would change next

• A weekly job that checks upstream for a fix to the two open flaws, so the first build that actually closes them gets pulled on its own.
• A YARA signature for the known alignment-overflow pattern, so the intake scanner can name it when it sees it.
• The crash drill with a poisoned file, intake side only.

A GGUF is the first thing that executes on the machine that loads it, and the machine that loads it is the machine that holds the keys. The fix is not a smarter parser and it is not a version number. It is a loader that has no egress, no keys to read, and nowhere to write. The file can be hostile. It just cannot do anything here.

Sources

• oss-security advisory, GGUF parser flaws, May 15 2026: https://seclists.org/oss-sec/2026/q2/546
• GHSA-96jg-mvhq-q7q7 (CVE-2026-33298): https://github.com/ggml-org/llama.cpp/security/advisories/GHSA-96jg-mvhq-q7q7
• GHSA-8947-pfff-2f3c (CVE-2026-21869): https://github.com/ggml-org/llama.cpp/security/advisories/GHSA-8947-pfff-2f3c
• Ollama PR #14406, tensor-bounds validation: https://github.com/ollama/ollama/pull/14406
• Ollama v0.17.1 release: https://github.com/ollama/ollama/releases/tag/v0.17.1
• Cisco Talos, Shodan case study on Ollama, Sept 2025: https://blogs.cisco.com/security/detecting-exposed-llm-servers-shodan-case-study-on-ollama
• V-01 public proof of concept: https://huggingface.co/k4wwak/gguf-alignment-overflow-poc