NanoFlare, microFlare, and This Project's Future
Last week I published my first highly compressed AI model, microFlare. It allows a 35B parameter model to be ran on a 16GB consumer grade GPU, or as low as a 4GB card when experts are offloaded, as it is a Mixture-of-Experts LLM. All while maintaining a relatively high quality by leveraging asymmetric quantization levels per layer.
This past weekend I released the beta for a new smaller model I'm calling NanoFlare. It's based off the newly released Qwen 3.8 27B. And I'm trying to get it squeezed down to fit entirely on an 8GB GPU this time. This is a dense model so you can't leverage the same CPU offloading techniques as a MoE model, but technically you can still do some offloading, however it will be incredibly slow when ran that way. Also note the beta version gets caught in thinking loops sometimes so you'll want to leverage repeat-penalty and dry flags. If it does mess up try again, it often succeeds on reprompt.
I've already made 11 different builds since the raw weights for Qwen 3.8 27B were first made public at the end of last week. I will be publishing a final version soon with a detailed write up. But from the downloads of the beta I've seen on HF so far since Friday, there definitely seems to be interest in what I'm doing here.
If you have found use out of microFlare or NanoFlare and would like to support me in my efforts, I have setup a Tip Jar on Ko-fi. I am doing all this work by myself with a fairly old PC right now. I am an unemployed software developer who was directly impacted by the AI hype in my field and do not have the funds to build a new machine at this time. If you'd like to donate a little cash so I can expand my endeavors to bigger platforms like 16GB and 24GB cards then please consider leaving me a tip at my Ko-fi. Even a dollar would help, it all adds up. Thanks