Yeah I was thinking open router but I will look around at options, I genuinely think this model is good enough for like 90+% of my use cases, and the top frontier models are still not that good at architecture so I have to do that myself still so I won't be losing out.

100k is all the context I have ram for, this is with any auto-compact turned off. This is using Cline in vs code. I’m sure I could tune the system prompt and mode switching more to work better with this specific model, but I haven’t gone down the custom harness rabbit hole yet.

And this is also specifically for the 1bit quant version. I don’t think the fp8 or even fp4 versions have this issue, but I haven’t tried those much

nope just my normal m2 mac mini. I bought it just as a normal computer to do my taxes and whatever, so it’s mind blowing that I can run this kind of AI workload on it. Well, “run” might be generous, it gets like 3-5tok/s

I’m working on a setup that’s more geared towards running tasks overnight so the slow tok/s doesn’t matter as much

[dead]

9B Qwen models are good and fast for local python coding tasks.

they are all overlapping but:

categorization, information retrieval, semantic search, image description

also with the model as part of an agentic system with tool calling

(edit: it is quite impressive what a small model in a feedback loop can do)

My use cases try to avoid accessing world knowledge in the model (I give it access to web search for some adhoc RAG), and ya, I'm just focused on coding so that's the only place I'm looking at right now.

> RTX 3090 for each and every employee

That’s quite inefficient. Get 2x RTX 6000 Pro and run DeepSeek v4 Flash for the whole company. This may or may not be cost effective compared to cheap third party providers, but it lets you own your own data, is not actually that expensive, and reports suggest that you can get 2k tok/s or so from a setup like this (at respectable batch size, but that’s exactly what you want in this situation).

I’m not sure I believe that the unquantized performance is as good as people are claiming, but even light quantization would make it fit easily.

whats the tok/s you get on that. I have heard a few claims of around 30-50 with mtp, but for how cheap the setup is I am surprised I don't hear more about 3060 stacks so I assume there has to be some catch.

Yes, I was fixing issues piecemeal until I found the froggeric template, I've had to fix I think one issues with that one but it's better.

The chat templates are usually the first thing that every major release bork on, and all new model architectures end up having a ~2 week initial window of small fixes before they’re not DoA

Yes yes, oh god yes. They also spread FUD in the form of terrible recommended sampler settings.

If you're using llamacpp, turn on top-n-sigma with sigma of 1, turn off top-p/top-k. You'll thank me later.

Yes for the first question. Google of all companies didn't even get it right with Gemma for a while until recently. For some reason it doesn't seem like people can actually get these templates right.

They usually do include fixes yes.

@danielhanchen may be able to answer this

[dead]

Given that it apparently defaults to 'xhigh', this is probably the answer.

Granted, it's still much lower tokens/s than you'll get out of many MoE models.

Edit: Even set to medium or low there's still a lot of second guessing, less consistency, lower 'acceptable response' rate, and slower/more token churn vs gemma4:26b-a3b. I think gemma4 is just a better 'general purpose' model.

In my experience with Qwen3.6 35B-A3B, disabling thinking made the model generate inaccurate replies. Ask it for the recipe of egg salad and it gives you the recipe of an omelette.

Did I miss something, is it possible to have that model be reliable without thinking?

Thank you, this is exactly what I needed.

Asking it factual information[1]. You just can't compress the entire human knowledge into a 30GB file.

[1] Without searching the internet. And even if you allow it, you'll get much worse results because search means browsing and parsing the top results, and search results are horrible, whereas internal knowledge from training encompasses the entire internet plus all books including very niche stuff.

IME using 5.6 Luna and DS V4 Flash, I notice that although they are excellent at programming, even Opus-like in the way they try to debug, the thing they are worst at is inferring user intent and making good decisions with little information. They are absolutely terrible at that, will misinterpret small wording ambiguities. I suspect that's an ability you can't add with RL training, that it requires the depth of understanding from vast pre-training.

Similar to the way they asked Sol to solve Erdos problems, that's what I want my model to do for programming.

I don't want to try to take my best educated guess at what the best design is BEFORE implementation - especially if you're designing a feature for a codebase you're not an expert in, you don't know like the back of your hand (i.e. one that is mostly or entirely LLM generated).

What sounds good on paper - often times becomes unideal in practice when you get to the reality of implementation.

It may not be worth re-architecting your entire system to get to a "pure" design that would be the best - all things considered.

Instead, I'd like the model to independently design many plausible and coherent good solutions, then implement each of them, then intelligently pick the few winners (after its fixed any bugs that could be causing promising solutions to look artificially bad) - unless there's an obvious one - and then give me the data I need to make an informed decision on which one to go with, all before I even look at the design or implementation.

You're not getting this from a one shot prompt from a 30B model today. You can't even really get it from Sol or Fable - IME. But you can get somewhat close.

The "a3b" refers to its active parameters -- unlike 27b it is a mixture of experts model, so it runs much faster, about as fast as a 3b model, but needs as much memory as a 35b model! So good for unified memory systems like macs :)

MoE models have less active parameters in play at any particular moment, so they perform much faster on lower bandwidth memories (like Strix Halo or DGX Spark) and also use less memory generally.

Dedicated GPU memory tends to be much faster (either DDR6 or HBM). So if you can fit a whole dense model in it, you're probably better off with that.

anecdotes: 35B-A3B does want more memory, bigger model. But if you get it running it will be faster and more enjoyable to use -- text will fly by -- due to only 3B params being active, in my experience at least.

Runs faster on bad hardware.

It's better for VRAM poor people. I get 4-5 t/s with 27B and 20-30 t/s with 35B A3B.

You can run these on CPUs at a somewhat reasonable speed.

Total param count decides how much vram you need to run it. Active param count decides how fast it runs. My 10 year old GPU can load quantized 35B or 27B, but it can’t process 27B parameters per token faster than 2-4tok/s, while it can do A3B at >40tok/s

Did you try the claude reasoning traces finetune for qwen3.6? I find that it works muuch better. I assume the same 3.8 finetune will be released at some pointas well.

Edit: link - https://huggingface.co/rico03/Qwen3.6-27B-Claude-Opus-Reason...

"inclined to overthinking," holy cats you're not kidding! On a Mac mini M4 Pro 64GB I prompted it with "svg owl" and it thought for 17m12s, outputting 36.3KiB of thinking chatter. It did end up producing a 20.2KiB HTML+JS+SVG file with a very nice owl, including cursor-tracking animation, but it ran for more than a half hour!

The MoE models are stupendously faster.

Reduce or turn down thinking:

https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

yeah, that's the A3B part; going up to A5B would probably also feel comfortable.

on the 395+ AI MAX w/128GB, the A10B qwen 3.5 can do a lot of long running work if you don't need to baby sit it. deer-flow works well like that.

[self-reply because comment edit window ended]: I now have a head-to-head benchmark. On my wheezy laptop (specs in sibling comment), this new dense model, Qwen 3.8 27B, gets ~4 tokens/second on generation. The older mixture-of-experts model, Qwen 3.6-35B-A3B, gets ~20 tokens/s.

MoE is literally 5x times faster (on CPU) than comparable dense Qwen!

Tested:

Qwen3.6-35B-A3B-UD-Q4_K_XL

Qwen3.8-27B-UD-Q4_K_XL

Mind sharing your laptops specs? Just interested to see what is needed to locally run Qwen3.6-35B-A3B

They're comparing Qwen 3.8 Max to Gemma 31B, fundamental mistake.

DeepSeek V4 Flash is natively FP4 MoE with very compact KV cache. Say 8 GB/s. Qwen 27B is about 60 GB/s at full FP16 precision.

The MTP is available, but I'm definitely not seeing 18 t/s on the Strix Halo from the 8-bit quantization, even with MTP (more like ~10 with full context on long tasks). This is a slow model (but so was 3.6). What's your exact llama-server command that gets 18 t/s?

Seems like MTP is available immediately!

How are you using email despamming?

I am looking to automatically recover mail from junk folder due to bad icloud mail filtering.

This is the way if you need speed. It costs a little bit in smarts, but compare the MTPLX option listed above with the oQ4e-mtp quant using oMLX. The good cacheing layer in oMLX will help things feel faster for some classes of tasks in my experience.

What is bpw?

Also whats your cutoff for 'acceptable' speed? I would have said 25tok/s.

Damn That's very good

Gemini is worse

I see! Thanks.

I don't see an MTP entry on Unsloth though. Maybe it's not available in a lower quant I need for my poor GPU.

And yet it's also a laptop you can basically take anywhere unlike a giant video card with 1000 watt power supply requirements.

Yes they run in parallel via LMStudio (250k context)

Yeah make sure you're using MTP and potentially tensor parallelism.

yeah, i consider that slow.

Whats your setup? I have a single 3090 and am struggling to get it purring

Same, with a multimodal DSv4 Flash I would just stop paying attention to things. Very smart, and at 260 tok/s it's too fast to care about anything else. If you ever graft something like that I would love to hear about it.

Yes, I have a very dumb flow. The harness has a describe_image tool that takes an image and a prompt and so DSv4 Flash uses it to get an idea of what it's looking at.

So much potential for that channel. He's got a nice range of tests and a no nonsense presentation style.

However, watching tests of heavily quantized models that weren't designed for it (non-QAT) is frustrating. There's no way to tell if the actual model fails because it's dumb or if the lobotomy made it that way.

thaaanks man, this channel seems really informative, although < 10K subs only !

Since the 3060s would have to communicate constantly using pcie, I think I'd go with intel

32GB is perfect for models around 30B parameters. Since qwen has really hit the spot with their 27B dense models, I think it's a good bet. Also, 32GB is enough for other tasks such as image/video generation and loading multiple smaller specialized models

You can buy two b60s for $1300 right now (650 each) if you want a total of 48gb. Intel recently raised the price on all of their gpu's except the b60 series, so they are currently the best deal per gb I think.

Could you elaborate please? Genuinely interested

Go on… what is this $1000 system?

Sorry, I wrote autocompletion by force of habit. I simply meant it can complete code you have already created a structure for, which personally is very nice

Why are you convinced of that? Pretty much every time I see statements like that online, I can find plenty of organic traffic supporting it, not everything is a bot.

I just wouldn't bias myself that way, most people haven't really used local models. This stuff is pretty much all subjective evaluation, there's plenty of reasons for people to favor certain models or disfavor others.

I'm generally extremely skeptical about a lot of the model hype that show up in comments. Except when there is an extreme mismatch the performance, quirks and quality of these things are difficult to nail down. You wouldn't know that from the comment section of every single release.

I think some of these are excited, eager users always ready to hype up the new thing. The same crowd that previously would constantly push for a rewrite from angular->react->svelt->god knows what. Instead now it is on a 6 week cycle and about models/harnesses.

I think most of it is bot driven spam by the various labs. It's hard not to notice 3 month old accounts with very strong opinions about various frontier labs and little else.

Ultimately, I think some of it is legitimate shifts in who's in lead and what is the best. You gotta dig through a lot of crap to get to that, and I don't really know how to to.

Ultimately I'm saying is that I always applied a fair amount of skepticism about what I see in comment sections but these day it is extreme amounts.

hence the "former deepseek v4 pro". I tried it out this morning and have had no complaints. I already liked glm 5.2

Luna is twice the price of Deepseek V4 Flash 0731, and less capable :/

I think it's very clear that someone who has checked out all the models but the one that called itself mechahitler and is explicitly being fine tuned to support far-right politics is making the choice for reasons other than performance and cost. It's not like all the other models even had plausible claims to those metrics.

[dead]

[flagged]

Yeah Opus 5 is almost unusable as a daily driver without making me go insane from excessive claude babble.

If it's as good as Sonnet 4.6 for most things I'd be happy.

It's very agentic coding focused; and I'd say a good executor but certainly not Opus in scale; overall knowledge; long-horizon work and recovery; etc.

e.g. If you try to chat to it about something philosophical for example, or maybe a debate / creative writing, then you'll very quickly see how it is still a much smaller model at the end of the day.

Still, it's such a relatively accessible model to run, and I find a big part of leveraging smaller models is to give it well-scoped tasks; not too high level or ambitious ones. Very impressive for its size and the ability to run locally :)

Qwen small models are heavily coding focused, whereas Opus is everything to everybody (even if code is their bread and butter). The downside is they'll frequently hallucinate world knowledge so they need to be RL'd to double check their knowledge against sources and verify facts/library names/etc.

It comes with MTP already, they just don't mention it in the filename as it's now the default.

When I tested various eval benchmarks on Qwen3.5/3.6 27B with Unsloth's quants, the scores usually dropped 0-5% between UD-Q6 and UD-Q3 depending on the eval.

Unsloth publishes KL divergence numbers which measures how much the quantised probability distribution changes vs unquantised: https://unsloth.ai/docs/models/qwen3.8#quantization-analysis

It's a bit bare at the moment, I assume they are going to add further detail later (eg comparison to other quants), similar to their other releases.

Depends on what software/hardware you'll run it. GGUFs from Unsloth can run on pretty much every single potato; full weights need beefy gpus

Unsloth one is gguf for llama.cpp (and some other on-device engines).

So advantage is not having to produce your own quantisation / gguf from .safetensors you've linked.

Run the unsloth if you are using llama.cpp (GGUF)

Run the one you linked if you are running vllm (safetensors)

if you have the VRAM, use offical release. quantized model lose focus after long context and can do damages or thinking loop

Unsloth usually also fixes the models when they bork something, which always happens. For Gemma for example the tool calling wasn't working for the longest time.

We also made NVFP4 ones if that helps! https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4

There is 0 shot you can make that claim about this model you have not used or downloaded yet

How can you say this when you haven't even tried it yet? Is it just hypothetical vibes?

"Benchmark is stupid" and "model beats model on benchmark" are two different things, though. The second one is objectively true regardless of your views on the first one, right? To expect everyone to share your opinion that benchmarks are stupid is pretty weird, and just saying "no" to an objective truth is the definition of delusion.

> ...but no. They do not beat opus on real-world usage.

I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.

Yep. These small models are actually worse than GPT 3.5 at some tasks (like recalling facts). You can definitely make models smarter at specific tasks (like tool calling, coding) but you can't compress the entire human knowledge into a 30GB file. It's just not enough bits.

> They do not beat opus on real-world usage

We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.

For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.

This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.

As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.

You could run the 2-bit quantization at about 10GB for the weights. You'd probably also need to quantize context to get a decent-sized context. The Unsloth model card shows the 1-bit quantization doing seemingly shockingly well on a task (but there is notable loss in model quality for each bit you give up below about 6 bits). An MoE model can partially reside in system RAM without catastrophic performance loss, but a dense model like this one cannot. You have to run it entirely out of VRAM, or it'll slow to a crawl.

Yes. It won't be as fast as fitting the whole model into VRAM but llama.cpp defaults are pretty smart about GPU/CPU splits these days. Just YOLO it with `llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL` and it'll definitely at least run.

You may be able to run a quantized version. One strategy I've found effective is to set Claude or Codex loose on the problem and they'll do a decent job setting up the best possible model revision.

"--context-shift, --no-context-shift ... whether to use context shift on infinite text generation (default: disabled)"

From: https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...

pretty sure this exists already...

[dead]

Problem with that is I think that it quickly devolves into cargo culting, nonsense and noise.

Arguably, what I am doing is also very very close to that, with the only difference being that I am somewhat less of an idiot than the average internet dweller you'd get on such a site. Or rather a different flavor of idiot.

Ideally, the people building the tools build them in a way that just does the right thing - which I am confident that llama.cpp does or will do in the future.

So you encode that knowledge not in language and online comments but in code and with a filter for actual expertise.

And, frankly, there's really not all that much to it. It's like maybe 3 parameters to play around with.

The valid solution space is pretty small, but people will want to make it "theirs" regardless, so you get non-solutions just so that everyone could also be a part of it. The usual social dynamics foo.

As a MBP and DGX spark owner, I would love such a site... (Feel like it would be a low effort feature of hugging face).

Searching through Reddit and forums for best commands is annoying.

What stackoverflow should have become.

It depends on what model you’re running, and for what workload. For personal use (one or two convos at a time) with models that fit in gpu memory, pcie bandwidth doesn't really matter. Just try and be on gen 3 x8 or higher.

Llama is decent at auto optimizing it if you let it use both gpus. It’ll split the workload so the contiguous layers are all on one gpu. Once the model is loaded, you only transfer weights between gpus once (per token?), at the layer boundary.

I run on an 8gb 3070 and 12 gb 3060, and the only weird thing is that the weaker card gets more layers (and therefore work) because it has more ram.

Oh, if you’re barely fitting the models into your vram, you may need to explicitly adjust the layer balance between cards — sometimes it fails to realize it should have put certain things (like draft models) on the other card so you can fit one more layer in.

As opposed to loading it up in RAM + VRAM? Pretty much always better to split it up to multiple GPUs. My priorities are load up all available VRAM, then offload MoE experts to RAM (if possible), then offload other layers.

I use an RTX 3090 (24GB) and a GTX 1080ti (11GB). Just the 3090 for 3.8 I get maybe 60tg/s (UD-Q4 quant), for both I get around 40tg/s (UD-Q6). Not apples to apples though considering it's different quants.

Here's my config: https://gitea.va.reichard.io/evan/nix/src/branch/master/modu...

Yes. I can split a model like this across 3 GPUs (a 1080 with 8GB and two Titan Vs with 12GB), and it's much faster than running it on 36 CPU cores. As long as it fits in aggregate VRAM, it seems very advantageous to do so.

That's a very deep rabbit hole involving PCIe topology on both the hardware and software (NCCL) side, among other things. It's too system-specific to answer directly, but the entrance to said hole can be found at https://github.com/local-inference-lab/rtx6kpro/blob/master/... .

Disregard references to RTX 6000 cards, most of it is generally applicable to all multiple-GPU boxes.

I haven't tried this either but I'm guessing if you could pool the GPU memory over whatever the kids are using these days, I think it was SLI back in my day. The GPU memory should still be faster than the RAM?

A hetero-GPU setup is definitely cost-effective if you don't strictly require the raw speed of a top-tier card like 5090. Just keep in mind that the total throughput will also be bottlenecked by the slower card.

To provide some anecdotal data, here is how my 5090 + 3060 setup performs with Qwen 3.8 27B (Unsloth's UD-Q4 with MTP):

  Single 5090: 101 t/s (TG), 2650 t/s (PP)
  5090 + 3060: 53 t/s (TG), 1700 t/s (PP)

[0]: https://news.ycombinator.com/item?id=48700091

Depends on your pcie connection. If they're both x16 then it's pretty low overhead, x8 is ok, but x4 is too slow. Also it's a bit tricky getting an optimal setups with mismatched vram, I think you could probably still make use of the full vram if you're clever but it's trickier.

Thanks for this! I got the same card, but not the same CPU nor RAM, but this is a great starting point to start tweaking

For the things it does, what tool is better than ffmpeg? Really struggling to see the cargo cult angle. Similar for llama.cpp, as it is literally the only framework I can get to run on my multi-Radeon rig. It is the most portable runtime out there.

-funroll-loops

The person above is running the OS from a separate GPU, so the command can use all of the VRAM of the 4090.

If you're trying to use the 4090 for your desktop at the same time, you will need smaller context size. You could remove the '-c 170000' and try turning fit on to see what fits.

>But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?

That was just the last buffer allocation request that failed, it didn't tell you by how much it failed by. It could have failed it by a few kilobytes, it could have failed it by 910MB. One would guess it probably failed it by a couple hundred megabytes in the end judging by your results.

There has to be a better way of sharing config , a profile of sorts you can upload and share

If you do not want to tweak parameters you can just run

    llama-server -m model.gguf

There are easier ways to run it. OP seemed to enjoy tinkering and customizing the command to run it exactly the way they want. When I don’t want to tinker Unsloth Studio is probably closest to pick a model and voila.

You know free LLMs can help you understand that command line or design your own...

Start by copying the command line from the Unsloth guides.

You don’t need to fine tune all of those parameters to get started.

It’s really easy to ask an LLM to adjust the command line if you can’t be bothered to read the help out. Copy the help output into the LLM and tell it your goal.

> Ollama is confusing and doesn't seem to support Qwen3?

Typing “Ollama qwen3” into Google takes you right to this page:

https://ollama.com/library/qwen3

If even Googling for basic Ollama support is too hard, there might come a point where you have to acknowledge that local LLMs are not for you. None of this is really that hard with some basic Google bootstrap skills or by asking an LLM to help with the command.

just tell claude/codex "set this up on my system $huggingfacelink"

I'll actually defend you on this one, but not only the command, that's more of a fine tuning option. On the hugging face link for example, there's no flat 'download' area. It's completely unintuitive for people to know how to use it. Reminds me of my first time trying to make sense of torrents when they first arrived.

There are some clients that will index the models and allow you to do that but I'm no expert, I've used OLama studio but it always seems to go weird for me.

Even this command above, it's not clear where op got the model from. So I'm with yah.

For example, op uses : Qwen3.8-27B-IQ4_NL.gguf.. But I cant see where to download it. It's not tagged on hugging face at least..

Well there's a lot of knobs to turn if you want to improve performance. You can always point an LLM at the model card, give it your info, and have it write up the command.

I never install this stuff manually anymore. Just tell your LLM of choice to download model X from URL Y, build the latest inference engine of choice E, and then create batch files or shell scripts to run instruct and/or reasoning models in accordance with instructions at URL Z.

> LM Studio doesn't work behind proxies.

Woa, is that still a thing? You mean like SOCKS5 stuff that you have to manually configure in every application that uses the internet?

I mean maybe I'm just living under a rock but I feel like that's a rather niche situation you got there.

What makes you say that it would be hard to do that?

It's long, I guess, but not cryptic.

You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it.

Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away.

You could also just run -m <modelfile> and let llama-server do the right-ish thing. The defaults are probably fine, but not how you squeeze out these exact numbers. I think at least. I've never tried. My hubris stopped me from trying auto configs.

According to this shitty vibecoded thing "I" built https://hypfer.github.io/will-it-fit-llama-cpp/ (and I guess according to math too), FP16 K/V would give me something like 90k context at the same model quant, which doesn't really fit my usage.

But maybe someone else has experience to share there

I have found that q8 kv quantisation is really bad in anything requiring some kind of literal retrieval even at relatively shorter contexts (5-10k). Eg it would mess up names, titles of papers etc that were deeper in the context.

Okay, so, I did some actual coding with qwen3.8 in the last 2 hours, and if I (let the agent) take the average of all those "mean len" loglines, and let them get weighted by how much happened leading up to that logline, it's an average of 3.33

The unweighted average is 4.1

So with qwen3.8 as well, I think I would not go below 3 in this setup with this workload. Meaning 4, and, to push it a bit further, 5.

Also, cool that it carries over from qwen3.6 like that.

Yes to both.

The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though.

As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4090 has so much compute.

Last logline I saw scroll by right now had 47% acceptance rate for 4th and 28% for 5th, but not sure how representative that is. I think when tuning 3.6, I saw more like 33%? But not 100% sure.