Some of us have big machines with lots of memory. WiCi Infer now runs a 177B parameter model on my Studio with 90+ tokens/second. It has a Local API hook at 127.0.0.1:8080/v1. Can we get this linked up to run in the Elements API window?
After a bit of memory cleanup and use, I’d say that the 177B/4-bit Qwen is running about one-tenth the speed of Claude on my Studio. Considering the huge model size, I find that respectable. It’s definitely workable. Most of the slowdown is inference (when it infers, it’s using the SSD).
