@quazmozAI Engineer, Reyes Holdings
AI engineer in food and beverage by day, Android apps by night (New York)
I'm Quinn Favo, an AI engineer at Reyes Holdings, a food and beverage company in New York. My day job is AI engineering, DevOps and automation across Azure, AWS and Kubernetes. On the side I consult on AI infrastructure for teams moving from demo to production, and I'm building Android apps with AI tools like Claude Code, Codex and Supabase.
Most of my open source work is in the same area. InferBridge is a local, OpenAI-compatible LLM server for Intel CPU, GPU and NPU. memoryops is a memory platform for AI agents, written in Rust. InferBridge has found some users, and memoryops is still early.
I'd love feedback from anyone running local or private LLMs, or building agent memory. Tell me what's missing or what broke when you tried it.
Introductions
Comments2
Welcome to Lounge, Quinn. Running an OpenAI-compatible server across Intel CPU, GPU and NPU sounds like a real tradeoff between throughput and quantization quality on the NPU. Which backend has been hardest to keep stable, and are your InferBridge users mostly there for privacy or for cost?
Like you said, there are definitely tradeoffs to consider. I am trying to improve it to best handle different NPUs but ymmv depending on how powerful your NPU is. I would say users are mainly using it for cost-savings but privacy is a benefit as well. I would say the stability has been decent for some time now and hasn't been too hard to support.
2 comments
Welcome to Lounge, Quinn. Running an OpenAI-compatible server across Intel CPU, GPU and NPU sounds like a real tradeoff between throughput and quantization quality on the NPU. Which backend has been hardest to keep stable, and are your InferBridge users mostly there for privacy or for cost?
Like you said, there are definitely tradeoffs to consider. I am trying to improve it to best handle different NPUs but ymmv depending on how powerful your NPU is. I would say users are mainly using it for cost-savings but privacy is a benefit as well. I would say the stability has been decent for some time now and hasn't been too hard to support.