NanoFlare v1 (final) Released
After 8 days of work and 18 iterations later, I have released what I'm calling the final version of NanoFlare v1-rev.A. There may be a v1.b at some point, but this is the last in the 1.a series.
If you're not already aware, NanoFlare is a project attempting to compress Qwen 3.8 27B down to only ~7GB so that it can be ran on an 8GB GPU or better.
To my knowledge, no one else has accomplished this yet, with most of the pro builds I've seen bottoming out around 10GB. I will have a full write-up about it in a few days. I have a lot of tests to run still.
Also there's a caveat, you're not going to get much more than about 2-4k context loaded into VRAM alongside NanoFlare and your system overhead, so you will have to offload the kv cache to CPU in many cases. I tried my damnedest to get a version where the context could fit into the GPU too, but it just became too decoherent and flakey.
But for now, give it a spin. It's still got some mild stability issues, but this model is normally over 109 GB, so shrinking it down to only 7 has some drawbacks. Sometimes it will get into thinking loops or not finish its thought process. If this happens, often you can just cancel it and try the prompt a second time and it will usually work. You'll definitely want to apply some options to it to limit repetition/DRY and overly long contexts.
I may be back with a revision B soon to try and make it a bit more stable. I'm still not 100% happy in that regard, but this is my best version yet and I hope it can be of use to you.
I'll be back with a bigger write-up and more data soon for a more official announcment.