NanoFlare v1 RC1 Released
Work on compressing Qwen 3.8 27B down to a file that can be ran completely on an 8GB GPU has been a challenge. But I'm getting close. This morning I have released NanoFlare v1.a RC1.
It is more stable and smaller than beta 1 was. However, it's still a bit bigger than I would like. You can run RC1 with a 10k context from llama-cli or equivalent. But when loaded into llama-server it doesn't really have room for anything but an incredibly small kv cache. You need to offload the KV cache to CPU to make it really useful, but that slows it down considerably.
Perplexity scores are available on my Hugging Face ReadMe. But no test suites have been ran against any of these builds yet. Please use at your own risk, these preview versions are intended for testing only.
Another problem both the beta and RC1 have is getting into thinking loops. Sometimes these can be prevented with repeat and dry penalty settings, but often times if you see one repeating its thinking with the same paragraph over and over again, try canceling it out and running the prompt a second time. In my limited testing these problem prompts will usually finish correctly on the second try.
Note that the final version will probably have a worse perplexity score than the preview builds, as they are too large and I am now working toward a sub-7GB target. Tradeoffs must be made.
If anyone out there has tried these models out, please let me know how they are performing for you. I'd love to hear how it runs on a larger 12GB+ card. Also if anyone is interested in a version that includes the MTP let me know, it has been stripped out of the default version.