NanoFlare Fits Qwen 27B On An 8GB GPU

08 Sep, 2026

I'd like to introduce you to NanoFlare. Version 1 is an asymmetrically quantized version of Qwen 3.8 27B. It is designed to run on an 8 GB GPU or higher (it can also be run on smaller GPUs with portions offloaded to CPU, or entirely on CPU but it's going to be slow in those scenarios).

I've taken a 102GB model and compressed it down to just 7GB. By quantizing different tensors at different precisions I was able to put the bits into the weights that mattered and greatly reduce the ones that could handle it better or were less important.

Today I'm making the formal announcement of NanoFlare's existence, but it has actually been available on HuggingFace since the same day Qwen 3.8 27B's open weights were first released on August 14th. I'll tell you the story of how I got here in a moment. But late on this past Sunday evening I uploaded version 1.c of NanoFlare, which is the first version I'm really content to release out into the world. It's taken me 42 revisions behind the scenes to get here.

Qwen 3.8 27B is a highly capable model and a new leader in the double digit parameter space for LLMs. Unlike my prior build, microFlare, this is a dense model rather a mixture of experts, which means you're going to get way worse performance if any of the model has to be offloaded to CPU. This means even though NanoFlare's parameter count is only 23% smaller than microFlare's, the total file has to be half the size.

I was unable to get a working version of Qwen 3.6 35B-A3B compressed down to 7GB or less without it getting easily confused and outputting garbage when I was making microFlare, but luckily with the 27B model I was able to pull it off! However, it's not perfect. Some weights are compressed as low as IQ1_M (less than 2 bits) and this comes at a cost. NanoFlare will sometimes stop mid thought, or fail to recall certain things. Make sure you always fact check any information it gives you from the memories of its training data. Recall becoming a little hazy seems to have been the biggest drawback to quantizing down this small. If you want more reliable, high quality output I would advise you use microFlare instead of nanoFlare.

But with that said, NanoFlare works pretty well! I did not feel like v1.a and v1.b were actually usable, more like a toy/experiment. But with v1.c it finally feels like a tool I can actually use. It's pretty decent at writing code, although sometimes it will fail to complete its full output. Generally if you re-prompt it, on the second try it will give you what you wanted.

On my old dual-core PC with 16GB of DDR3 RAM + an RTX 2070 Super (8GB) I am able to run NanoFlare 100% on GPU with up to an 8k context at k: q5_1, v: iq4_nl. If I make it any bigger or higher precision than that I will run out of memory (OOM) eventually. So if I want 100k or the full 262k that this model is capable of, I'll have to offload my kv cache to CPU and system memory. I also have Gnome/Wayland taking up 160~200 MB of my VRAM, so a lighter weight desktop environment may give you space for a bigger context.

If you have a 10GB GPU or higher, this thing should run great for you and have plenty of room for bigger contexts. Also with that use case in mind, I will probably eventually release a version that has the MTP included. It was stripped out to save space.

So how did I build this thing? Three and a half weeks ago I had just wrapped up microFlare and its first release. After working on it pretty hard for a few weeks I finally took a break. Two days later I saw that Qwen was about to release the open weights for 3.8 27B and I decided it was time to try again. On the 22nd I downloaded the raw weights to Qwen 3.5 27B and began work on quantizing it. From what I could tell Qwen 3.8 was going to use the same architecture so hopefully what worked for 3.5 would work for 3.8 as well. I made 3 revisions on the 23rd till I had something that looked decent. Then the next day as soon as I woke up I downloaded the raw weights for 3.8. By that evening I had my first build of NanoFlare based on Qwen 3.8 and I quietly released it as Beta 1.

With my older machine it takes hours to build new revisions and process certain files, so I was pretty proud that despite that I was able to release my first build the same day Qwen released the weights to the public. But I still wasn't happy with the quality. I would then spend the next few days making new revisions trying to fix the issues I was seeing with reasoning getting into loops or other quality flaws. I would eventually upload Release Candidate 1 where I had really focused on quality, but the file size was too big. I couldn't fit even a small kv cache onto the GPU at the same time as the model. So I tried to reduce the file size and find a balance between quality and the real limits of what can fit onto an 8GB GPU. By the time I hit my eighteenth iteration I finally had a 7GB GGUF file that felt like it was giving mostly good quality outputs and so I decided it was finally time to call this version 1.a.final. It was now roughly a week since Qwen 3.8 was released and none of the big players on HuggingFace had any builds smaller than 10GB. I had actually pulled off something that should have been impossible it seemed.

I hadn't spent a whole lot of time testing each individual build manually. I relied a lot on perplexity scores and simple prompts to help me keep track of where I was on quality from build to build. I was rushing to get something out into the world, but now that I had the bandwidth to spend more time with v1.a I quickly determined I was still not happy. It just barely worked and flaked out quite frequently. I was embarrassed at the state of NanoFlare as it stood tbh.

And so I continued on. I was determined I could increase the quality and get this LLM to a usable state. But each bit reallocation I tried seemed to fail me for a while at this point. I finally decided to call uncle and admit a minor defeat. At the cost of performance for quality I would have to give up on making the model small enough to fit on GPU along side a kv cache. So now I was aiming to grow the file as big as I could while still fitting everything else into the GPU's VRAM. The context would just have to always be offloaded and this was going to be the cost of getting such a large model running on an older, now low-end GPU. I had already made a similar concession by deciding to strip the MTP predictor out of NanoFlare the same way I did with MicroFlare. It was going to be slow, but it would at least have good quality output. And 11 revisions later, I had one that felt decent enough to release as v1.b to the world.

I still had not spent a lot of time with nanoFlare yet at this point, so I began testing it in preparation for this write up. It was terribly slow, like 1-3 tokens/sec because of the cache being bottlenecked by my ancient PCI bus. I also was still not quite happy with the quality of output the more time I spent prompting it.

I was upset to be honest. I felt like I had failed to achieve what I set out to make. And so I stubbornly continued my quixotic journey in pursuit of making this 27B parameter model work despite everyone online saying a working build under 10GB is not possible. I was so close, I knew it. Then something surprising happened. Unsloth announced version 3.0 of their Ultra-Dynamic build system and now they had not only one that was in the same ~7GB range as mine, but an even tinier IQ1_M build that's closer to 6! Now the major players were getting into the same place I was and reminding me that I'm just one guy who's only been doing this for less than 2 months. Luckily the 6GB build was just as bad as my worst builds and definitely worse than nanoFlare. But their 7GB model performed pretty well in the small amount of test prompts I gave it. I felt defeated. But then I tested its perplexity drift. It was scoring better (lower) than NanoFlare on the 2k context test, but then on the 8k version NanoFlare scored better than Unsloth's! There was still hope.

The next series of builds would all follow the opposite strategy I had employed before, I would do extensive manual testing on each one and not worry about automated tests till later. At first I wasn't making much progress, but then I took a peak at Unsloth's configuration and I realized they were prioritizing the QKV weights in the opposite order I was. AI assisted hallucination had struck again. I switched from prioritizing Q to V instead and suddenly I saw a noticeable gain in quality. This was it, I was getting closer! I got one build that was feeling pretty good but then it ran out of memory. I could no longer fit my target 10k context into VRAM, not even 8k. I had hit the wall. So I scaled it back and determined ~7130MB was about my maximum file size with my particular PC configuration if I wanted at least an 8k context on GPU. I shuffled the bits around a few times but was still not happy.

At this point I had been quantizing Qwen models so much that I had finally figured out what everything meant and was no longer relying on my AI assistant to help me figure things out. The next two builds I would make completely on my own and not even tell my electronic partner about it till after the fact. I was so close, but I was also exhausted. I had been working all day every day on this project for over 3 weeks. I decided to scale everything back to as small as possible, then ticked up the various tensors in quality following my renewed priority list until I hit my size target. I had learned in the prior couple builds that heavily compressing the token_embd weight has little effect on actual output quality, although it can diminish your perplexity score. So I crushed it as small it possibly could go so those bits could be spent elsewhere. I started up my quantizer and went to bed.

The next day I had done it. I had a model that although, still imperfect felt MUCH better in terms of real quality from hands on subjective testing. And I could fit my 8k context, and the file size was right on target, and thinking loops were mostly gone now. I finally had a version of NanoFlare that I felt like I could be proud of. And that version is try 42 AKA v1.c, the one I'm here to announce today.

Now let's look at some numbers. Despite being completely on GPU, NanoFlare is still a bit slow for me on my machine once the context starts to grow and so big automated tests like the MMLU take hours to run, possibly days. So I'm just going to give you the partial results of what data I have so far.

First, let's look at perplexity, which scores how far the quantized model has drifted from its original uncompressed version in terms of predictive output. Lower is better and zero would be a perfect score.

I'm very happy I was able to beat Unsloth here, at least with a longer 8k context. And I was surprised v1c beat out v1b despite being smaller. This is pretty good for such a heavily quantized model. However, for reference microFlare was able to score 6.2761 with a 2k context. So the tiny size still comes at a cost in quality.

Here are my incomplete, partial MMLU Pro scores (only first 20 questions per subject) for NanoFlare v1.c:

I'll continue to do more tests and will post more data in a follow-up sometime.

Some tips for when you're running it. I often see best output with top-k set to 20 and temperature set to 0.95. Qwen officially recommends a 1.0 temp, but I haven't had good results with that setting for NanoFlare. I'd advise turning on the DRY-multiplier and repeat penalities, I've been setting both of them to 1.05 lately. I also sometimes like to set its reasoning levels down to medium instead of the default xhigh, because it's a very thinky model.

Here is an example command that I have used to run it via Llama.cpp

llama-server -m /mnt/data/ai-models/nanoflare/nanoFlare-v1.c-Q2_0.gguf -t 2 -fa on --mlock -b 2048 -ub 512 -ngl all --no-mmap --host 127.0.0.1 --port 8080 -cram 7779 -ctk q5_0 -ctv iq4_nl -c 8192 -np 1 --no-cache-idle-slots --repeat-penalty 1.05 --dry-multiplier 1.05 --top-k 20 --temp 0.95 --chat-template-kwargs '{"reasoning_effort": "medium"}'

I hope NanoFlare can be useful to you. My goal with this project is to make open weight LLMs more accessible to everyone. Local AI will become common one day I believe, as most tasks don't require a top-tier "frontier" grade model, and so why pay a subscription fee for it and risk your privacy? If NanoFlare or MicroFlare have been useful to you please let me know. I'd love to hear more data on how well it's running on people's machines other than my own.

Also note if you'd like to support my work I now have a tip jar on Ko-fi. I'd love to expand the project further if I had the funding for more hardware and resources.

Next up I think will be a new revision of MicroFlare with everything I've learned creating NanoFlare employed. Then after that, I'm considering making a PicoFlare that will target a 4GB GPU next.