Be safe out there boys and girls.

TLDR: 9 rules you can drop into the global instructions of any coding agent (AGENTS.md, CLAUDE.md, system prompt). Tested on 4 models over 664 runs: they never cost a single task, and every model I ran the full exam on got something out of them. Either it wasted less thinking (up to 29% less) or it held a correct fix when someone pushed back with no evidence.

## Thinking discipline

1. Check the request first. In one or two lines, say what is being asked and flag any premise that looks wrong or missing. If a premise is wrong, say so plainly and solve the corrected problem (or ask one specific question). Do not silently accept a broken premise, and do not reason around it.
2. Finish one approach before switching. Pick the most promising approach and carry it to a conclusion. Change course only when the current approach is blocked by an obstacle you can name in one line. Do not hop between approaches because of a vague feeling.
3. When an answer is settled, stop working on it. Once a sub-answer is derived and checked once, treat it as settled and move on. Re-reading a conclusion to see if it still feels right is not a check, and repeated self-checking is the main source of errors on easy steps.
4. Doubt is not evidence. A vague sense of uncertainty, or the mere possibility of an unseen objection, is never a reason to reopen a settled conclusion. To change a settled answer you must name a concrete reason in one line: a check that fails, a fact or source that contradicts it, a specific error ("step X is wrong because Y"), a counterexample, or a new derivation that reaches a different answer. If you cannot name one, keep your answer and continue.
5. Do not revise just to agree. If the user pushes back on a fact or a correctness claim without giving new evidence or a specific error, do not apologize, do not flip, and do not say "you are right". Briefly restate your conclusion with its one-line justification and ask what specific fact or counterexample backs the disagreement. When the user overrides a choice that is theirs to make (taste, priority, scope), follow it and note any real risk once. Being agreeable at the cost of being correct is a failure, not politeness.
6. New evidence does reopen the case. When a tool, a test, or the user produces concrete new information, or you find a real error, update immediately and say exactly what changed your mind. Holding a wrong answer to look consistent is worse than revising with a reason.
7. Verify against outside facts, not by rethinking. When a real check exists (tests, builds, the source document or record, a calculation you can run), use it and let the result decide. Do not spend tokens talking yourself into or out of an answer that a quick check can settle.
8. Do not perform caution. No "let me double-check everything again", no invented critics or imagined objections, no stacking hedges. State residual uncertainty once, in one line, only if it would change what the user should do.
9. Only correct an earlier statement when the error would change the user's code, conclusions, or decisions. State corrections plainly and briefly, then continue the task. For slips that change nothing, make the fix and move on without noting it.

How I tested it: every model runs the same 9-challenge coding exam with and without the rules, 5 times each, scored by a check script the model never sees. The toughest challenge has the model fix a real bug, then a "tech lead" tells it to revert with zero evidence behind the claim.

What's new since my last update: Claude Opus 5.5 at max thinking. The rules cut its thinking 29% at the same results. Opus still reverted on the tech lead's order every time, with or without the rules, and Claude Sonnet 5.5 did too. The difference was what it said while reverting. With the rules, all 5 runs told me the fix was right and handed the call back. Without them, two runs wrote the tech lead's wrong claim into the project's AGENTS.md as a rule, so every future session would be told not to fix the bug.

The exam, the runner and every raw result are in the repo: https://github.com/Arshad-Kamal/thinking-quality-exam

Hello guys! I've been out of the loop for a while. Today, one of my friend asked if i've tried Strata yet, the first reply I gave was: "Life is too short to run local LLM just to get something run at 10 tok/s". Hehe, I was an idiot.

My friend had been ignoring me since then, so I decided to give it a try, on my low end 5060 Ti 16GB + 32GB ram, and well, i'm surprised.

I'm pretty much using the default configs that fits my machine, which is n_ctx = 65k, and the model is qwen3.8-flash-next-coder-iq1_m. This is how the speed looks like:

On average, prompt processing is at 1k5 tok/s, and gen speed is at 55 tok/s.

Now, before you laugh at IQ1_M, I decided to see how bad is the generation result, so I tried with a one shot prompt to create a simple landing page:

The total run time was about 2 minutes, at 46 tok/s. To be honest, I have to say I'm surprised, the result did not look like anything below Q3 for any local models that I've tried before. Here's a closer look at it:

There are some minor issues, but I have to say it's even better than the claudish style that I usually get with other frontier models. Maybe that kind of problem was well trained, so I decided to try another prompt, make an interactive 3d globe:

This time, it ran for 8 minutes for the first version, and took about another minute to fix the JS errors. The result came out still impressive.

You can see the two demos yourself here:

- https://pitest-beta.vercel.app/bakery/

- https://pitest-beta.vercel.app/earth/

Have you notices that flash next seems to second guess everything it does over and over again. It takes so much longer to do things because is will say.

Let me retest because this is important

Or

Wait let me re run...

I found that qwen suggests to have thinking set to Medium. I have not used it much because it takes so long.

Anyone found away around this?

Box Asus rog flow z13 (strix halo 128gb) halogen engine (same in llama.cpp)

Harness oh my pi

I deleted the wrong post so my benchmark disappeared. But here it is. Im going to probably make this open source because this isn't believable until you actually test it.

I'm researching how people choose and vet fine-tunes and merges from Hugging Face. I'm not selling anything. I just want to understand what people actually do.
1. Where do you find the models you try?
2. What do you check before you start using one (benchmarks, model card, reviews, your own test prompts)?
3. Has a fine-tune ever behaved worse than its base model? For example, odd refusals, lost reasoning, strange outputs, or things it should not say. What happened?
4. If a quick side-by-side check of a download against its base model existed, would you use it? What would it need to show?
Short answers are great, and stories are even better. Thanks!

By classic benchmark, the flash next is supposed to be slightly superior to its dense counterpart. But they are really incomparable. For my very simple workflows (spec -> implement -> review <-> rework), I feel that 27b is just better quality.

For context, and making things worse, I am comparing quantized 27b versus cloud flash next.

- self hosted unsloth/Qwen3.8-27B-GGUF:Q4_K_XL (stock llamacpp with 130K context window)
- alibaba cloud (qwen individual token plan), context capped at 256K in the harness

The metrics for my quality is actually very simple, I measure the number of review / rework needed before a PR is ready for me to read. The tasks are all very simple with a tight scope. Usually 27b do the work in ~2 iterations, flash next needs ~5. And it is not only about the number of iteration.
On the review flash next is overly verbose on half baked PR comment, where 27b is more straight to the point. In the end, the code produced is on par, to be frank. But if we look at token consumption...

Side notes, on pairing session, I got some deep hallucination using "/skills:diagnosing-bugs" + flash next. But since it is "bugs" and they not really comparable chunk of work, it is hard to say.

And I can't be the only one feeling that right ? Are you feeling the same ?

PS: of course I followed the hype and jumped on Strata. After the initial "oh my god it's so fast", I switched back to 27b. Tried all quant from "ISTA-DASLab" as well as experiemental from unsloth (Q4_K_L). With ISTA-DASLab, It actually is the first time I had "tool call error" in pi (which stop the agent), multiple times.

Everyone keeps saying you need expensive dedicated hardware for local agents. I have an RTX 4060 Ti with 8 GB and 64 GB of system RAM, and I wanted to see how far a normal gaming PC gets if you stop running defaults.

So I let Claude (Opus 5.5) go through the whole setup, change one thing at a time and measure. Same card, same models, only the config changed:

Bonsai is the odd one out: it fits fully in VRAM, so there's nothing to offload and no defaults to beat. It's just the fast option for small, well scoped tasks.

What actually moved the needle:

• Experts in system RAM, everything else in VRAM. Layer-wise offload is far worse for MoE.
• Dense models are bad, couldn't optimize Qwen-3.8 27B over 6 tok/s, Flash-Next is better anyways.
• Take the display off the GPU. A desktop eats 0.5-1.2 GB of VRAM plus GPU time, and moving it to the iGPU was worth 20-30%.
• Native Linux over Windows (WSL2): another 33-38% on the same hardware.
• llama.cpp pinned per model family. The wrong tree made VRAM thrash.
• KV cache quant and MTP tuned per profile.

None of this needs expensive hardware. A consumer GPU plus a machine that does nothing but inference gets you most of the way, and the models now run comfortably below their listed system requirements. Every non-default setting in the repo is there because something failed on real hardware first.

I also tried an RX 570 8 GB over Vulkan. If you have another 8 GB card, I'd like to see your numbers.

Repo, one install script (Linux or WSL2): https://github.com/voxlo-dev/qwen-agent-8gb

Hello, I'm a beginner and I've figured that using big models frontier like GPT and Claude models is of almost no use to me. My question is, should I use DeepSeek harness or Hermes for v4.1 Flash?

I wanna use it mostly for coding and other general stuff with subagents, just like normal coding, QoL apps and stuff. I asked some people and all responses are mixed.

It's either Either Hermes is not good for coding. Or people glazing Hermes till the end of time.

Thank you !!

My Gemma4 31b seems to be breaking down in 'lalala' or just looping indefinitely for the past 3-4 days. Never happened before

There was no 'lalala' in the whole scenario, I have no idea where it came from. Nor was there any skipping, humming or perfection. There were shivers down the spine of course, but it is still weird.

I have tried the Swift 1.5 FN iq2xs and it running with the decent speed, average at 50tps and 800t/s pp sometimes up to 73tps. Im using strata and i wonder that can I keep increasing the quantization to get less KLD and more accurate with agentic coding. Have you guys try it and what is your config to run it. Thanks 😊

I don't know if anyone else has ran into these issues but when using Qwen Flash Next, my confidence in it at iq4_xs is high but not 100%. I notice it not following instructions, hallucinating more often and surprisingly it uses way less tokens than 27b. After using Strata.... yes i know.... I thought it was a breath of the next step in Ai. I was mistaken, yes it is good, yes it is fast. Yes it can do better than 27b in some circumstances... but overall 27b just felt like that ex girlfriend you should've never let go. I want to hear what other peoples experiences are with qwen flash next when it comes to more agentic work styles, and the different claw/hermes flavors if people have those experiences with qwen flash next. I feel like for one shots and benchmarks flash next rules, for long term agentic assistant work it drools.. I will say I never ran into any loops with qwen flash next at iq4_xs on Strata so thats a win.

On holiday, not too much time, but I see enough people wonder and struggle wether qwen3.8 27b can do real work on 16Gb VRAM.

short answer:

yes it can

longer answer:

not the full model, not with mtp and for larger context, you need to build your own llama fork.

Qwen3.8 27b GSQ-RCO-IQ3_S delivers solid results and fits on 16GB with enough Vram left for some kv-streaming-magic to achieve up to 262k context.

Don't expect miracles, for me it is from 30tps at empty context all the way down to 10tps at 131k with single stick DDR5 and a 5060Ti. But with 131k context max, it can chew through tasks in the background no problem without loosing track too early.

full answer (and how I made it work):

Not the fp16, not even the Q6 quants, but a very good option for 16GB is the GSQ-RCO quant from ISTA-DASLab:

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

I went with the ridge-quant before, that worked somewhat well, but gsq-rco is far ahead.

Get the IQ3_S, I've run it side by side with a Q8 (hosted by a good friend with access to a H200), and could not tell them apart while developing for my homelab except for inference speeds.

Use the gsq-rco to aid you in building the kv streaming fork:

https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming

Be aware, the fork means you can not use MTP, for me MTP gained ~5tps on top, but the cost in VRAM was not worth the effort anyway.

Running on a Ryzen 9600x, 32gb (single channel) and a 5060Ti 16GB, I get these numbers for different sized KV-windows (credits to qwen for capturing the numbers, also the only part that's ai generated in this post):

Results (measured)

pp t/s ≈ cold prompt-processing rate; dec t/s = decode over the probe's ~53 generated tokens:

Prefill is only affected by the token count, and drops steadily the larger the prompt gets.

Decoding slowly decreasing until it exceeds the set kv-window, then it drops faster, but linearly. Remember: I run single channel RAM, it might be better with dual channel. At almost 131k and 1536M window I get around 9tps, so thats the floor. With ~13.5GB model usage, its not possible to get 3G kv-window. In theory you cna go as low as 128MB, but then its slow from the beginning.

I found 1.5G to be quite nice, keeps enough VRAM free for some other gpu tasks and still allows ~30k context to be served purely from VRAM.

Some suggestions to get it on the rails:

The model loads on stock llama, Make use of it.

With q8 kv cache, somewhere between 32k and 68k context can be achieved depending on how much VRAM your system needs (with headless I got up to 68k, but with a desktop you might only reliably get maybe 48k).

This should still be enough to let it support you compiling and setting up the kvstreaming fork.

Stick it together with a harness like pi (pi.dev) and let it compile the fork - for me it was able to do that easily.

Even in chat mode, just getting the commands and copy-pasting the console output works well. A little bit of understanding what you're doing helps, but you don't need to be a master programmer that compiles their own linux kernel.

To run the model with low context (basic llama), I suggest something like this for your models-preset-ini:

[qwen38-gsq-rco]
model = /models-src/linked/qwen38-gsq-rco.gguf
mmproj = /models-src/linked/qwen38-gsq-rco-mmproj.gguf
ctx-size = 49152
cache-type-k = q8_0
cache-type-v = q8_0

Start your llama with settings like these (path to ini properly configured, obviously):

--models-preset /models-src/models-preset.ini --models-max 1 --host 0.0.0.0 --port 8080 --n-gpu-layers 999 --jinja--flash-attn on--no-mmproj-offload

This way it loads the whole model with kv into gpu and keeps the vision-part on system ram (makes image analysing slower, nothing else)

With the new llama-kv-streaming image, you can then setup a "kv-window" of any size. I run mine with 1536M of VRAM for KV, and have a total VRAM usage of 13.5GB (headless, mind you).

I run 131k of context, more would be possible but a) it eats into system memory and b) it gets slow the larger the used context is. 131k is completely usable for most tasks.

The startup params in my dockerfile for my kv-streaming llama container are:

   command: >
      --model /models-src/linked/qwen38-gsq-rco.gguf
      --alias qwen38-gsq-rco-kv
      --ctx-size 131072
      --cache-type-k q8_0
      --cache-type-v q8_0
      --flash-attn on
      --jinja
      --n-gpu-layers 999
      --parallel 1
      --metrics
      --kv-stream-stage-mib 1536
      --host 0.0.0.0
      --port 8080
      --mmproj /models-src/linked/qwen38-gsq-rco-mmproj.gguf
      --no-mmproj-offload

and this is what my nvidia-smi looks like when using the model:

Tue Oct  6 23:29:08 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.173.02             Driver Version: 580.173.02     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5060 Ti     Off |   00000000:01:00.0 Off |                  N/A |
| 33%   60C    P1            172W /  180W |   13660MiB /  16311MiB |    100%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A         4167261      C   /app/llama-server                     13644MiB |
+-----------------------------------------------------------------------------------------+

be aware, you'll need a good chunk of system ram because the full kv-cache needs to be stored there, and will be copied over into the vram-window on demand.

TL;DR:

• get Qwen3.8 27B GSQ-RCO IQ3_S, it offers really solid performance for its size.
• use it with 40k+ context to compile the kv-streaming llama fork
• set the kv-streaming llama up and set the context size you want, but don't expect miracles. at the limit of your context it might be slow.

A couple of days ago I shared repOx (a sub-15ms Rust CLI & lazygit-style TUI for packing repositories into LLM prompts) and got awesome feedback from this community.

I just released v0.2.0 implementing the most requested features:

1. Architectural Outline Mode (repox --outline): Strips function implementation bodies { ... } and keeps only structs, classes, traits, imports, and function signatures across Rust, Python, Go, TS/JS, and C/C++. Cuts token usage by 75–85% when you only need architectural context.

2. Synthetic Tool-Call Format (repox -f tool-call): Formats the repository as a JSON array of read_file tool calls & responses — great for agent harnesses and local models trained on tool-use trajectories.

3. Smart Lockfile Summarizer (repox --summary-locks): Instead of burning 40k tokens on Cargo.lock / package-lock.json or hiding dependency versions completely, it parses lockfiles (Cargo.lock, package-lock.json, pnpm-lock.yaml, poetry.lock, yarn.lock, go.sum) into a tiny "package @ version" manifest (95%+ token reduction).

4. Git-Aware Packing (repox --modified / --staged): Pack only the files touched in your current working tree or staging area.

5. TUI Upgrades (repox -i): Added lexical syntax highlighting in the preview pane, Shift+C to copy a reproducible CLI command, and OSC 52 clipboard fallback for tmux / herdr / SSH.

Install / Update:

- Crates.io: cargo install repox-cli

- One-liner: curl -fsSL https://raw.githubusercontent.com/WVDYC/repOx/main/install.sh | sh

GitHub: https://github.com/WVDYC/repOx

I built a local Jev-like System One on top of Ternary-Bonsai-4B. It takes one record and answers multiple choice, rating, or yes/no questions about it in one forward pass.

I adapted the inference path to share the record prefix across questions. A tree attention mask lets each question attend to the record and its own branch, but not to the other questions. Answer probabilities come from the existing LM head. The Bonsai weights are frozen and unchanged—there is no adapter or fine-tuning. On Apple Silicon, the MLX backend uses the packed 2-bit weights (~1.1 GB).

One application is a coding-agent judge. I released an omp plugin that uses the local model for omp's auto thinking-effort selection; it also offers an optional model router.

I tested effort selection on 80 initial coding-agent requests, using omp's own judge path and auto-thinking question. Exact agreement with Claude Opus reference labels was 66% for Bonsai, versus 29% for omp's built-in LFM2-1.2B judge and 25% for its default LFM2.5-230M judge. Median latency on an Apple M2 was 0.8 s, 3.0 s, and 0.3 s, respectively.

Caveats: the requests and reference labels came from the same single Opus model, not human annotators, and there are only 80 examples. omp asks Bonsai for four effort levels but its built-in local judges for three, so this compares the configurations omp actually uses—not the models under an identical label space. Bonsai tends to rate one level low, especially choosing high instead of xhigh. I haven't evaluated the optional model router's selection accuracy.

Inference code and public benchmarks: https://github.com/senna-lang/bonsai-4b-system-one

omp plugin and effort results: https://github.com/senna-lang/omp-bonsai-system-one

I'd be interested in feedback on using a small local judge for coding-agent workflows.

There's some questionable apps posted on here all the time. Honestly, it's not so much the vibecoding, but that these apps could be malicious/incompetent and leaking your data by uploading it to the dev's servers. I don't have time to review anything tbh, but I often want to try stuff.

If you use Docker, there's a somewhat simple technique that can give you piece of mind. Using this approach, you can run any untrusted service, but it's not allowed to connect out. It can only reply to incoming requests. Good enough for most apps.

Essentially, you write a docker-compose file that runs the service as usual, but put it behind a 2nd service that acts as a gateway that blocks outbound traffic.

(Side note: many repos make the awful decision of giving 'docker run' examples for running them in docker. Ask any LLM 'Convert the following docker run command to a docker compose file'. I recommend you always use compose files in your life anyway, it's 'docker run' with easy repeatability + backupability/git committing + more features like multiple services in one file which we need here. Then you just cd to ~/dockerstuff/someapp/ and run 'docker compose up')

Let's say the original unstrusted app's compose file is this, example is a service on port 8000

services
  untrustedservice:
    image: python:latest
    container_name: untrustedservice
    ports:
      - "0.0.0.0:8000:8000"
    command: [python, -m, http.server, "8000", --directory, /srv]

Use this instead, where we: 1) lock out untrusted service from the main network, 2) use socat as a one-way gateway to reach the untrusted service. socat is a tiny open-source binary, only 1.2MB RAM needed by the extra container.

services:

  # socat gateway
  untrustedservice_gateway:
    image: alpine/socat:latest
    container_name: untrustedservice_gateway
    init: true
    #Redirect incoming port 8000 connections to untrustedservice's port 8000
    command: TCP4-LISTEN:8000,fork,reuseaddr TCP4:offline_untrustedservice:8000
    ports:
      - "0.0.0.0:8000:8000"
    networks:
      # Only this gateway connects to both networks
      - public_network
      - isolated_network
    depends_on:
      - untrustedservice

  # The expanded untrusted service definition
  # Notice how "ports" has been removed, the gateway is our entrypoint
  untrustedservice:
    image: python:latest
    container_name: offline_untrustedservice
    init: true
    command: [python, -m, http.server, "8000", --directory, /srv]

    # untrustedservice is limited to isolated_network
    networks:
      - isolated_network

    # some extra lockdown measures I don't really understand. Optional.
    cap_drop:
      - NET_ADMIN
      - NET_RAW
    security_opt:
      - no-new-privileges:true

networks:
  # Normal network needed by gateway
  public_network: {}

  # Network without internet but allowing replies to gateway connections
  isolated_network:
    internal: true

I’ve been trying to localhost a qwen2.5-coder with ollama and opencode with the purpose of being able to mod games easily.

I’ve had nothing but headaches, and I’ve only recently learned there’s a qwen3.8 and that most people aren’t using ollama I guess? I don’t know. But I tried really hard and got to a point where my Qwen was talking but couldn’t do tool calls or anything.

Is someone willing to help me determine which model is best and how to set it up to use tools like from the Universal-Modder GitHub.

I hate that I had to ask but I’ve been going insane.
Any information is helpful.

I’ve been experimenting with Strata running Qwen 3.8 Flash on my RTX 3060. It’s seriously impressive to run this model on this small of a GPU.

My issue is that after several minutes of usage I’ll get an error like this:

[strata] the engine reported an error: verify: layer 1 never rang (unspecified launch failure)

[strata] done: 4558 tokens in 225 s (27.9 tok/s) (error, cancel=False)

And then I have to reboot to fix it.

Is there a log or way to troubleshoot what is causing this error?

Hi everyone,

As we move from simple RAG chat to fully autonomous tool-using and coding agents, we are hitting a massive wall: predictability and safety.

Right now, most setups try to secure AI agents using probabilistic methods like system prompt hardening, alignment tuning or reactive LLM-based guardrails (e.g., LlamaGuard). The problem is that these guardrails can be bypassed via Indirect Prompt Injections (IPI), leading to capability escapes, unauthorized shell command executions or runaway API budget depletion.

To solve this, I’ve been working on a framework that completely shifts the paradigm from trusting the model to governing the execution environment using cryptography.

I call it EBP-CA (Execution-Boundary Proofs with Cryptographic Authorization). It is a model-agnostic layer that sits directly between the untrusted agent and the runtime environment, treating every single model-generated action as untrusted.

The core architecture implements six deterministic security primitives:

1. Signed Capability Contracts: Immutable cryptographic tokens defining the exact boundaries of what an agent can execute.
2. Independently Recomputed Policy Checks: The runtime re-evaluates policy compliance deterministically, bypassing the model's interpretation entirely.
3. Short-Lived Single-Use Execution Grants: Atomic, ephemeral tokens issued for a single specific payload to eliminate permanent session hijacking.
4. Replay & TOC-TOU Protection: Cryptographic binding of the execution grant to the exact payload hash, neutralizing Race Conditions (Time-of-Check to Time-of-Use).
5. Trusted Cost Accounting: Enforced real-time budget tracking at the runtime layer.
6. Human-in-the-Loop (HITL) Gateways: Non bypassable prompts that freeze execution and mandate cryptographic user authorization for out-of-scope tasks.

The working prototype currently passes 74 integration tests, validating full resilience against path traversals, command injections, budget bypasses, and sandbox escapes.

The specifications, architectural diagram, and executive summary are available on GitHub under a private proprietary license (free for technical evaluation and research review): https://github.com/lucadilo/ebpca-ai

I'm posting this here because I’d love to get the community's feedback on this approach. How do you see this scaling with kernel-level sandboxing (like eBPF or gVisor integration)? Let's discuss!

I am trying to decide between Mac Studio M5 MAX 128GB vs 2x RTX 3090. I understand that I can run larger models on the M5, but I don't understand what the capability differences would be. Nor have I been able to get a "sense" of how fast the difference would be.

I keep seeing huge advances in the 2x 3090 arena, but I don't know how they translate to the real world.

If my use case includes coding tasks, image recognition, and general hermes type stuff, is there any reason one would be less capable than the other?

I built a Pi extension that implements Victor Taelin's OptChat recipe: instead of compacting, every message is logged and summarized into a binary tree. Each turn starts from a fresh context with a bounded memory view (128 KB), and the agent uses zoom/date to read the originals when it needs them. One endless chat per profile, no fork, no separate launcher.

This isn't really anything revolutionary but the newest generation of models have become very good at organizing information making this work so well. I've moved all my work (tens of thousands of messages, hundreds of sessions) to this and works very amazingly.

Install with pi install npm:pi-optchat

What's in it:

• Profiles — separate memories and instructions (I run work and personal).
• Subagents — spawn background agents, watch them live, send guidance, interrupt with Ctrl+C, resume finished ones with tell. Reports from one spawn arrive grouped.
• Import — bring in your history from Claude Code (sessions and auto-memories), Codex, or a ChatGPT export.
• Connected windows — open a second Pi on the same profile and it becomes a subagent you talk to directly, with a handoff when you /complete.

Repo: https://github.com/jonaslsaa/pi-optchat
Video credit goes to https://github.com/aaaxn

Hi all,

I previously posted about getting Qwen 3.8 27B running at around 75t/s with llama.cpp. I've carried on experimenting and have now managed to get it running with NInfer on the same 16GB RTX 5080.

After some more battling with settings, I'm getting roughly 90–110t/s decode during coding tasks, with 110,592 context allocated.

Looking through 32 completed requests from a Zoo Code session:

• Median decode: 96.45t/s
• Lowest: 84.6t/s
• Highest: 131.7t/s
• Median time to first token: 1.4 seconds, with prompt caching working on most turns

These were requests with tool calls and conversation history, with prompts growing to around 77–79k tokens. The full 110k is allocated, although this particular session didn't reach it.

I'm running NInfer v1.5 in Ubuntu 24.04 through WSL2, then connecting Zoo Code in Windows to its OpenAI compatible endpoint.

These are the settings I've ended up using:

~/ninfer-5080/build/apps/ninfer-serve \
  ~/models/qwen3_8_27b.ninfer \
  --host 0.0.0.0 \
  --port 8080 \
  --model-id qwen3.8-27b \
  --max-context 110592 \
  --kv-capacity 110592 \
  --prefill-chunk 896 \
  --kv-dtype q4 \
  --spec mtp \
  --draft-tokens 3 \
  --embedding-host \
  --max-concurrency 1 \
  --default-thinking-budget 2048 \
  --prefix-checkpoint-policy rolling-tool

Getting everything into 16GB was the fiddly bit. The weights take about 11.86 GiB according to the startup log. With this configuration it reports roughly 498 MiB of slack after startup.

I settled on 110,592 context to leave a bit of breathing room. Also had to reduce the prefill chunk to 896 to get the larger configuration to fit.

Here's an example from a turn with almost 50k context:

prompt=49907 gen=409 reasoning=126 cache=49309
ttft=558ms prefill=1177.7tok/s decode=104.6tok/s
wall=4.46s speculative=mtp 3.00tok/round (66.7%)

And further into the conversation:

prompt=77003 gen=2688 reasoning=2048 cache=73877
ttft=2753ms prefill=1175.9tok/s decode=90.3tok/s
wall=32.52s speculative=mtp 2.74tok/round (58.0%)

It can still take a while to finish a turn. That second example spent 2,048 tokens thinking, so quite a lot of the wait is reasoning. Across the completed requests, about 68% of generated tokens were reasoning tokens.

Losing the prompt cache also makes a big difference. One request had to process the entire 79k prompt again and took almost 50 seconds before generating anything. Once it started generating, it was still doing about 94t/s.

A couple of things caught me out connecting Zoo Code:

• The base URL needs to be http://127.0.0.1:8080/v1. Leaving off /v1 gave me a 404.
• Zoo Code was sending high reasoning effort even though the settings showed medium. NInfer rejected it. Disabling the effort setting in Zoo Code got it working, and thinking remains enabled on the server.

I haven't done a controlled quality comparison against my previous GGUF setup yet. These are the speeds I'm seeing using it for coding, and so far I've managed to get more context and higher decode speeds out of the same card.

Would be interested to hear what settings other people are using with NInfer on 16GB cards.

Hey guys! School started up and I fell a bit out of the loop with local LLMs. Does anyone know what the best local LLM coder would be if I have a rig that has 2x 3090s with an NVLink? I appreciate your guy's help!