> ## Content Index
> Fetch the complete content index at: https://mistercyber.lt/llms.txt
> Use this file to discover other available public pages before exploring further.

# Fitting a 27B Model Into 16 GB, and the Four Tuning Ideas That Lost
- URL: https://mistercyber.lt/fitting-a-27b-model-into-16-gb-and-the-four-tuning-ideas-that-lost/
- Published: 2026-10-26T08:43:38.000Z
- Updated: 2026-10-26T08:57:40.000Z
- Description: A 12.1 GB uncensored coder model on an RTX 5070 Ti, 40,960 tokens of context, and four tuning attempts that measured slower than the defaults.
- Author: Aurimas
- Tags: AI, Linux

The model file is 12.1 GB. The card has 16,303 MiB of memory. That leaves about four gigabytes for everything else, which sounds like plenty until you remember that context is also stored in the same place.

This is the setup I ended up with after a day of trying things and deleting most of them: one 27B model, fully resident on an RTX 5070 Ti, serving 40,960 tokens of context at around 50 tokens per second. Everything below was measured on this machine, and the parts that did not work are in here too, because they cost more time than the parts that did.

## What the model actually is

Qwen3.8 27B, a dense coder model, quantised to IQ3\_S. The GGUF I run is [on Hugging Face](https://huggingface.co/huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-GGUF?ref=mistercyber.lt), published by huihui-ai. One file, no shards, 12.1 GB on disk. The variant I run is abliterated, which means the refusal behaviour was edited out of the weights themselves rather than argued away with a prompt. That is the whole trick: no jailbreak text, no system prompt gymnastics, just a model that answers the question it was given.

The practical difference matters less than it sounds. A model with refusals removed does not become smarter or more accurate, and it will still be wrong about things. What changes is that a request I consider legitimate does not come back with a paragraph about how it cannot help.

Why run it locally at all? The reasons are practical. Local inference has no per-token bill, so a long refactor costs electricity instead of credit. It works with the network cable out. And the prompts stay on my disk, which matters for anything that touches private code.

## What the uncensored part is for

Nothing exotic. Four examples, all of them things I did this month:

- **Modding a game I own.** Patching a single-player Unity game on my own copy, offline, so its own achievements stop being locked. A hosted model usually answers that with a paragraph about cheating.
- **Reverse engineering my own files.** Reading a game's .NET assembly to find where an achievement flag gets set, then writing a small patch. Same kind of work as debugging anything else, and this is what the game's modding scene does openly.
- **Obfuscated code from my own machine.** Shell from a test VM, a crash dump full of encoded strings, "explain what this does" where the code is mine and the box is a throwaway.
- **Practice targets.** CTF boxes and a lab network I built for the purpose, where the only thing at risk is my own time.

None of that is a request for a weapon. What changes with the refusals gone is smaller than it sounds: I stop paying tokens for a safety paragraph, and I stop rewording ordinary engineering questions so a classifier does not trip on them. The model does not become more capable at security work, or at anything else.

Which is the honest limit. A model that will not say no is not a model you can trust. It still invents APIs, still gets shell syntax wrong, still needs the same review before anything runs. The refusals were an interruption, not a safety net I had been leaning on.

## The arithmetic that decides everything

![Bar chart showing 14,211 MiB of 16,303 MiB VRAM used, with 12.1 GB for model weights and the rest for KV cache and compute buffers](https://mistercyber.lt/content/images/2026/10/01-vram.png)

The budget after loading. Weights are fixed; everything left of the free space is context.

Weights take 12,387 MiB and do not move. Whatever remains has to hold the KV cache, the compute buffers, and the desktop sitting behind the window. At 40,960 tokens with a q4\_0 cache, the measured total lands at 14,211 MiB, which leaves roughly 2 GB. A browser with a few tabs will eat into that. A second model will not fit at all.

This is why the interesting decisions are all about context size. The model is not the variable; it either fits or it does not. Context is the dial, and every turn of it costs memory on a card that is already mostly full.

## One command, three sizes

![Three llama.cpp presets: max at 40,960 tokens, mid at 32,768, fast at 16,384 with their VRAM and speed tradeoffs](https://mistercyber.lt/content/images/2026/10/03-presets.png)

The whole interface: llm up, then pick a size.

A shell script called `llm` wraps llama.cpp. `llm up max` loads the native 40,960 context, `mid` drops to 32,768, `fast` to 16,384\. Default KV cache type is q4\_0, which halves the cache size and buys back context. `llm down` stops the server and frees the card.

Nothing is registered with systemd. That is deliberate. This machine is also my gaming PC, and a model that starts on boot would be holding 14 GB of VRAM every time I wanted to play something. Manual start and manual stop, with one PID file and one log, leaves the desktop as a desktop.

The mid preset turned out to be pointless. It costs the same memory as max and gives the same speed, so there is no reason to pick it unless 40k is genuinely unavailable for some reason. Fast is the one with a real trade: about 5 more tokens per second in exchange for 24,576 fewer tokens of context. I do not take that trade.

## Four ideas that lost

![Table of four rejected tuning attempts with their measured results: bigger batches, speculative decoding, 80k context and 128k context with KV on CPU](https://mistercyber.lt/content/images/2026/10/02-rejected.png)

Every row looked reasonable before testing. None of them survived contact with a measurement.

**Bigger compute batches.** Raising `-ub` to 2048 and `-b` to 4096 gave about 4% more prefill throughput and then spilled past the card. Generation collapsed to 31 tokens per second. Reverted.

**Speculative decoding.** A 0.6B model as the draft should, in theory, let the big model skip ahead. On this pairing it produced no speedup whatsoever and consumed roughly 2 GB of VRAM that the context needed. Uninstalled.

**80k and 96k context.** Both loaded, both crawled. Generation fell somewhere between 10 and 21 tokens per second depending on the setting, which is slower than reading comfortably. Dropped.

**128k with the KV cache on the CPU.** This one worked, in the sense that the server started and answered. It ran at 18.8 tokens per second: a third of the speed, paid for context I was not using. Dropped.

The lesson from all four is the same. llama.cpp has an enormous number of flags and most of them are not free on a card that is already 87% full. The defaults, plus a smaller KV cache type, beat everything I tried to add to them.

## The 40,960 wall

There is no 64k option, and the card is not the reason. The limit is the model's training context. Ask llama.cpp for more and it does not degrade quietly or truncate; it refuses with `training context overflow` and the server never loads. The number in the model card is the number you get, and I spent a while assuming a bigger card would change that. It would not. More VRAM buys a bigger KV cache for the same 40,960 tokens, not a longer conversation.

## Vision, on the wrong processor

The same model ships with a multimodal projector, a separate 0.9 GB file that lets it look at screenshots. Loading it on the GPU works and costs VRAM that the context needs more. So the projector runs on the CPU instead, via `--no-mmproj-offload`. A screenshot takes noticeably longer to process than a text prompt, which is the entire price of keeping the other 0.9 GB for tokens.

## What the speeds mean in practice

A 1,931-token prompt prefills at 1,664 tokens per second on this card, and generation sits at 49 to 50 tokens per second. Those are numbers on a benchmark, so here is the translation: a medium file plus instructions arrives, gets read in a bit over a second, and the answer streams faster than I read it. The bottleneck stops being the model and becomes how long I take to check the diff.

## What still runs in the cloud

Not everything moved. My editor's default agent still points at a hosted model, because a local 27B at 50 tokens per second is a different tool from a frontier model, and pretending otherwise would be dishonest. The local one is configured as a separate agent with a deliberately small prompt, used for work where the answer is checkable and the code cannot leave the machine.

One detail worth writing down: this model reasons before answering, and if the token limit is set too low, the reasoning consumes the whole budget and the content can come back empty. Anything under roughly 900 output tokens was blank for me. That cost an hour of blaming the server.

## How to install it, step by step

Everything below assumes an NVIDIA card with about 13 GB of free VRAM, 13 GB of disk, and a recent driver. The model is the same on either platform; only the binary and the shell change. I run Linux, so that path is tested. The Windows steps use the same flags and the project's own prebuilt binaries, but I have not run them myself.

Grab the binaries from the [llama.cpp releases page](https://github.com/ggml-org/llama.cpp/releases?ref=mistercyber.lt). Pick the newest build and match the CUDA version to your driver: 12.8 assets work on most current cards, 13.4 needs a very recent driver.

### Linux

**1\. Check what the card reports.** If this fails, fix the driver before anything else.

```
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv
```

**2\. Unpack llama.cpp.** Two archives, not one. The `cudart-` file contains the CUDA runtime libraries and has to sit next to the binaries.

```
VER=b11889
BASE=https://github.com/ggml-org/llama.cpp/releases/download/$VER
mkdir -p ~/Apps/llama.cpp && cd ~/Apps/llama.cpp
wget $BASE/llama-$VER-bin-ubuntu-cuda-12.8-x64.tar.gz
wget $BASE/cudart-llama-$VER-bin-ubuntu-cuda-12.8-x64.tar.gz
tar -xzf llama-$VER-bin-ubuntu-cuda-12.8-x64.tar.gz
tar -xzf cudart-llama-$VER-bin-ubuntu-cuda-12.8-x64.tar.gz
```

If you would rather compile it, the whole build is three lines and needs the CUDA toolkit:

```
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)
```

**3\. Download the weights.** The CLI is part of `huggingface_hub`; install it with pipx or your package manager, then pull the one file you need. The [CLI docs](https://huggingface.co/docs/huggingface%5Fhub/guides/cli?ref=mistercyber.lt) list the other install paths.

```
hf download \
  huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-GGUF \
  --include "*IQ3_S1.gguf" --local-dir ~/Models
```

Add `mmproj-model-bf16.gguf` from the same repo if you want image input. It is another 0.9 GB.

**4\. Start the server.** This is the command I actually run, minus the wrapper script.

```
MODEL=~/Models/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-IQ3_S1.gguf

~/Apps/llama.cpp/llama-server \
  -m $MODEL -ngl 99 -c 40960 -fa on \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --mmproj ~/Models/mmproj-model-bf16.gguf --no-mmproj-offload \
  --cache-reuse 256 --parallel 1 --jinja -a local \
  --host 127.0.0.1 --port 8080
```

**5\. Confirm it is alive.** The health endpoint returns 503 while the model loads, then flips to ok. On a cold start that takes a few seconds on an NVMe drive.

```
curl 127.0.0.1:8080/health
```

**6\. Point a client at it.** Anything that accepts an OpenAI base URL works. The endpoint is `http://127.0.0.1:8080/v1`, the model name is whatever you passed as `-a`, or just `local`.

### Windows

**1\. Same check, different shell.** Open PowerShell and run `nvidia-smi`. A driver from the last year will do.

**2\. Two zips, extracted into one folder.** `llama-b11889-bin-win-cuda-12.4-x64.zip` holds the binaries, `cudart-llama-bin-win-cuda-12.4-x64.zip` holds the runtime DLLs. Skipping the second one produces a missing-DLL error that looks like a broken install.

```
mkdir C:\llama
# extract both zips into C:\llama
```

**3\. Weights.** Python is the easy path. If you do not have it, use the HF web interface and click download on the single GGUF.

```
hf download `
  huihui-ai/Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-GGUF `
  --include "*IQ3_S1.gguf" --local-dir C:\Models
```

**4\. Start it.** Same flags, PowerShell backticks for line breaks, and `-c 32768` as a starting point. Windows keeps more VRAM for the desktop, so the 40k setting that fits on Linux can come up short here. Raise it once you see how much headroom is left in Task Manager.

```
$M = "C:\Models\Huihui-Qwen3.8-27B-Coder390-EfficientThink-abliterated-IQ3_S1.gguf"

C:\llama\llama-server.exe `
  -m $M -ngl 99 -c 32768 -fa on `
  --cache-type-k q4_0 --cache-type-v q4_0 `
  --parallel 1 --jinja -a local --host 127.0.0.1 --port 8080
```

### Tuning, in the order that matters

Context size is the only dial that changes much. Drop `-c` to 32768 or 16384 before you change anything else, because that is what decides whether the model fits at all.

Next is the cache type. `q4_0` on both K and V roughly halves the cache and is what makes 40k possible on a 16 GB card. `q8_0` is more accurate and costs the context you just bought.

Then flash attention, `-fa on`. Leave it on.

Then `--cache-reuse 256`, which helps when you send nearly the same long prompt twice, as editors and agents tend to do.

What does not help, measured on this machine: bigger batches, speculative decoding with a small draft model, and any context beyond 40,960\. All of that is in the table above with the numbers attached.

Measure before you believe anything, including this post. The server prints prompt and generation throughput per request, so a single `curl` tells you more than a forum thread.

## What I have not measured

No evals. I have not compared this quantisation against a larger one on real tasks, so the quality claim is "good enough for the work I gave it", not "as good as the full model". The speeds are one machine with one driver version. And the abliterated variant has not been checked for degraded performance on ordinary tasks, which is exactly the kind of thing people assume away.

It runs, it is quick enough to use, and it costs nothing per token. That is a lower bar than a benchmark, and it is the one that actually mattered here.

*Header image: the Qwen logo, from Alibaba’s Qwen project model cards.*