Running a 35B param LLM on a 10 y/o PC - Introducing microFlare v1

10 Aug, 2026

I'd like to introduce you to microFlare. A custom compressed version of Qwen 3.6 35B-A3B I have made that is designed to be ran on low end hardware. If you have a 4GB GPU and 16GB of system RAM or better, you can run this 35B parameter model locally, on your own PC.

Note that even though this model should be able to fit on a 4GB GPU thanks to CPU offloading for experts, I have only tested it on an 8GB card. Similarly, the entire model should theoretically fit onto a 16GB card, but I do not have access to one to test with at the moment. However I'm fairly sure it should work, as with all experts offloaded the model only takes up ~2546MB of VRAM. And the entire model file is only 13.9GiB and I am able to run it 100% on CPU so it should be fine. Someone please email me if you test it in such scenarios and let me know how it runs.

I became interested in local AI a few months ago. The idea of running an LLM with open weights on my own machine that I own and fully control is quite appealing to me. With local AI there are no caps or limits, no chat logs on a website to be leaked, no servers potentially logging your sessions for new training data, and of course there's no cost per token fees beyond the couple of pennies in electricity you use. None of those problems really exist* when you run your own self hosted LLM. It's similar as to why I prefer using Linux and open source software in most cases.

*caveat: you could still be open to risk of leaks and whatnot depending on how you configure your setup. If you expose your local server to the web now many of those same security concerns are on you to maintain against. If you want to be truly safe, don't allow anyone outside your local network to connect to an AI host. Business across multiple locations? Require a VPN for access.

About 3 weeks or so ago I learned about the DwarfStar project, where a developer shrank Deepseek v4 Flash down to a size that could be ran on consumer hardware while still maintaining relatively similar quality of output. They leverage asymmetric quantization where most of the important pieces stay at full precision, as while the "expert" layers get crushed down to roughly just 2 bits of data per weight. This is a trick that can't really be performed the same way on dense models, but for MoE it would seem the amount of memory you need to run them can be far lower than expected.

Unfortunately, Deepseek v4 is a massive model, with 284 billion parameters. So even though DwarfStar allows you to run it without having hundreds of gigabytes of RAM available, it still requires 96 GB of VRAM (or shared memory pool) and ideally wants 128GB. All my old, aging PC has is 16GB of system RAM and 8GB on the GPU. There's no way I could run DwarfStar (DS4) on my home computer.

So about 2 weeks ago I got the idea... what if I could do the same thing with a smaller model that I could actually fit onto my own machine? I began researching what my options were and what tools were out there to help me accomplish my quest. Initial results seemed very promising and it looked like the tools I needed were already freely available. I then decided at the size class of LLMs I was considering that Qwen 3.6 35B-A3B looked like the winner, with the best all around features and test scores even though not the top scorer of every individual category. (I originally wanted to use the 122B-A10B model, but even with extreme compression that version was too large. I might revisit that one when I have beefier hardware available one day.)

I didn't know much about the inner workings of LLMs and how they are stored at this point, so I leveraged some help from Deepseek myself just to be fully transparent. I learned Qwen 3.6 had 41 layers and that some people online had found that the first and last few layers are the most more important, as while the inner DeltaNet could handle more compression. And the 41st layer was actually a separate piece called the MTP that is like a second tiny LLM that tries to speed up the main model by predicting a few tokens at a time for the main model to approve rather than generating them one by one itself.

I downloaded llama.cpp and compiled it, then began preparing the long command to tell it exactly how I wanted to quantize certain layers and pieces at specific sizes. After a few tries, and 12+ hours of generating an iMatrix file, I finally had it running. A couple hours later I had my first custom asymmetrically quantized LLM ready to try out. It worked! I could fit the whole thing into my 24GB of RAM by splitting across the CPU and GPU's pools. But there was a problem. It was slow. Only giving me 4-5 tokens per second. Which made it technically functional, but a bit too slow to practically use very often. I tried to squeeze more layers onto my GPU in an attempt to speed it up, but I hit a wall. My GGUF file had turned out to be around ~18GB and there was just no way I was seemingly ever going to fit enough layers into my 8GB of VRAM to make it fast enough for me to be satisfied with.

So next I started on a new quest. To see if I could squeeze the model down even smaller and get it down below 7.5GB so it could 100% fit into my VRAM even with overhead. I tried this path for about a week, generating 6 more attempts. With files ranging between 6.8GB and 7.4GB, but no matter which configuration I tried the models were over compressed. Upon prompting they would end up in thinking loops, repeatedly saying the same thing over and over again. Often times talking about garbage that had nothing to do with my prompt or the conversation history. One version would go nuts any time it encountered a number or asterisk, and would then just repeat the number over and over again. I had learned a hard lesson through much time and effort, getting a 35B parameter model below 8GB and still functioning coherently just is not possible, or at least not with these techniques. I dropped my quixotic pursuit and regrouped.

So now I had a new goal, increase the precision more but try to keep the expert layers as small as possible. I couldn't fit the entire model into my GPU's VRAM, but I could still try to fit a large portion of it. It was around this time I decided to make the entire model below 15GB total so that it could be useful to other people with a 16GB GPU. So I wanted to develop a new version that would be bigger than 7.5, but under 15. And I went back to iterating.

At this point in the process I started comparing my different versions to see how many layers would fit and how fast the token generation rates were. In this process I learned that thanks to my old hardware with bandwidth deprived buses that the MTP predictor was just not going to be of any use to me. So I decided to strip the MTP out and save some more space. I finally decided I was happy with attempt number 13. I was able to fit up to 20 layers (half the model) into GPU and could now hit speeds of 5-7 tokens/sec.

I was now ready to share my results with the world. But first, I would need to test the model to see how it compared to other versions. Could my asymmetric model score as high as a flat quantized one? Then I learned another hard truth. I was not treading the new ground that I thought I was. At least two other groups had beaten me to it. I learned of the APEX builds being very similar in design to what I was making. Then I found out about Unsloth's Ultra-Dynamic builds (UD) and how they have been doing this for well over a year and have now started doing the optimizations via a special program rather than by hand like I've been doing. Well that really burst my bubble a bit, up until this point I thought I was creating something novel. I was falsely under the impression that antirez had originated this technique with DwarfStar. And I think this ties back to the dangers of not double checking the research an AI has done for you. But ultimately this whole project was a bit over my head when I started, and I'm still proud of what I've built.

So the most basic test of the quality in a compressed AI is its perplexity score. I ran v1-prerelease-13 against llama's built in tester and it came back with a 6.0972. That was better than Unsloth's UD-IQ3_XXS small build, which scored 6.253 (lower is better)! But then I realized no, my score was inflated because I used a 4k context, and the other tests online appeared to be using just a 2k context. So I ran it again, and there was disappointment yet again. My score dropped to a 6.3. I must have gotten something wrong. So I tried one more time, one more hour of rebudgetting bit allocations followed by another 2.5 hours of waiting for the build to complete. I now had my 14th iteration.

When I tested try 14, it came back with a perplexity score of 6.2761. Almost just as good as Unsloth's version, but my file size had grown. I was no longer closest to the IQ3_XXS build, but the IQ3_S version. I'd need to benchmark against that version now (as perplexity scores on my machine do not match up with published scores due to something in my hardware or configuration). It scored 6.2496, only a gap of 0.0265 points between my new version and its closest competitor.

Here's where microFlare compares to the others (lower is better):

Note how once the file size goes under 10GB the Perplexity score plummets (higher is worse). The Q4 evenly quantized version from Bartowski is the biggest I could run, so that is my baseline to aim for. Presumably the full F32 version would score even better. But I was pleasently surprised with how well microFlare scored. It has a score slightly lower than Unsloth's UD-IQ3_S version, but it bested all the others! I can live with a close second.

Next I needed to do more tangible testing. I downloaded the lm-eval tool and ended up deciding to use the GSM8K, high school math test first. I ran the test with a 2k context and at 0 shots (formatting hints) and only the first 20 questions to get a feel for where I was at. Attempt 13 scored a 0.4, but try 14, my final version that I am releasing today scored a 0.6. A major improvement! Although still not as good as Unsloth's IQ3 build which got a ~0.65 in the limited run tests. When I ran ran the full test suite with a 5 shot, microFlare scored:

Lastly I ran microFlare against the MMLU Pro test suite (MMLU regular ran into issues with my configuration, but pro worked so I just went with the upgrade). I did a quick test of just 20 questions from it as the entire benchmark includes 14,000 questions.

I attempted to do a still limited but bigger run for more accuracy with 200 questions per topic. I set it up to run over night and it was still going when I woke up the next day. It had been running for over 9 hours and was estimating it would take 9 more days to complete. I decided to cancel the run and go with a smaller set of just 100 questions per topic. I let that run for another 12 hours or so and then it had a hiccup and failed out. Clearly testing my model was going to be a much more arduous task than I expected. I decided at this point I had enough data to publish my initial findings. Note, I'll probably do a follow up post with more data sometime soon.

Partial MMLU Pro results (8k context, 5 shot, 20 questions per topic)

From my manual prompt testing everything's looking pretty good! All its responses make sense and seem to be functioning correctly. And it's one of the fastest models I tested! Simply using llama-cli with a couple prompts (very low sample rate), I found that the flat quantized Q2_K_L build is the fastest, as iQuants require more processing. And it also allowed the most layers to fit into the GPUs VRAM at once, with a full 20 layers. It can even fit 21 layers on GPU when the kv cache is offloaded to CPU, but on every single model I tested offloading the kv cache slowed performance significantly. The following tests were done with a 32k context and the same settings except for the number of experts that live in VRAM. MicroFlare once again comes in second place.

Ultimately, I'm pretty happy with how microFlare turned out. I proved that I could compress my own version of Qwen down to a size that would fit onto my ancient, low spec PC. I can get almost half the model's layers into GPU and achieve 7-10 tok/s now by prioritizing the inner near edge layers (5-9, 30-34). I can actually squeeze up to 17 layers if using llama-cli with a moderately sized context, but llama-server has more overhead so I can't get it past 16 usually. I call this "inner rim" prioritization, and it's given me the best performance as compared to many other permutations like stacking all from the front or end, or just the heavier outer edge layers. Loading the inner most 20 layers provided the least uplift. Another tip I've found is that offloading the token_embd weight to system RAM really makes little reduction to performance, and that frees up space for more layers (at least on a heavily bandwidth constrained system like mine). Also note, to specify specific layers to offload, you need to utilize the -ot flag. This is how I've been running it lately for example:

./llama-server -m /path-to-models/microFlare-v1a.gguf -ot "^token_embd\.weight=CPU,blk\.[5-9]\.ffn.*exps=CUDA0,blk\.1[0-2]\.ffn.*exps=CUDA0,blk\.2[7-9]\.ffn.*exps=CUDA0,blk\.3[0-4]\.ffn.*exps=CUDA0,exps=CPU" -t 2 -fa on --mlock --no-mmap -ctk q8_0 -ctv q5_1 -c 32000 -b 2048 -ub 2048 -ngl all --host 127.0.0.1 --port 8080

If you need even more speed, there are experimental forks and alternative runtimes that handle layer offloading for you dynamically, prioritizing a cache of "hot experts" that are being used more frequently than others and loading them in and out as needed. But unfortunately both llama-quik and ik_llama would not work with my old PC. You need a CPU that supports AVX2+ to run those, and llama-quik kept running out of memory despite me being able to fit the entire model into my system ram with vanialla llama.cpp. So maybe you can get even better performance than I did using these tools. Someone will have to let me know if they find anything interesting.

But if you really want maximum performance speed, in terms of token generation stick with a model small enough to fit completely onto your GPU's VRAM. With a Q4_K version of Qwen 3.5 9B and no optimization, I was able to get 44 tok/s with everything on my seven year old RTX 2070 Super. Much faster than a CPU/GPU split will ever provide you.

Note the version of microFlare I am releasing today is called microFlare-v1a. This means it is microFlare version 1, revision A. If I decided to make another, even better version of this build, it will be released as ver.1b. However, one day down the road when I make a microFlare v2, it will probably not be based on Qwen. With Qwen 3.7 and 3.8, they have not released any MoE models that are small enough to run on a machine like mine. Their smallest new version is a 27B dense model. And more importantly Qwen may not stay the top dog in this size category, Minimax or someone else might have the crown at that point in the future. And whichever one seems to be the best all around at that point in time will be the basis for v2 of microFlare. The idea is I am providing a simple choice for users on low spec hardware, don't worry about which model or configuration is best, I've already taken care of that for you. Also note, I will probably add a variant with MTP included if anyone's interested in that too.

In conclusion, microFlare has not broken any records or achieved anything that has not already been done I'm afraid. It's hard for me to beat the whole team that makes Unsloth's UD builds, they're who you should go with if you want the absolute best in output quality. As while flat, symmetrically quantized builds appear to be the fastest, especially when using K quants instead of I quants. However, if you're looking for a balance of the two, microFlare appears to be the second fastest AND the second most accurate. For my needs, I think it's my winner.

Another project to keep an eye on is PrismML's Bonsai model. It applies heavy quantization to a dense model. I will probably make another blog post with more extensive testing down the road, including Bonzai. But I've learned a lot along this journey, and hopefully someone out there will find that microFlare is the best open LLM to suite their needs too.