Engineers run their own LLM proxies because provider dashboards lack usable usage data
Overview
A Hacker News post argues that LLM provider dashboards do not give engineers usable token usage data, so the author runs a self-hosted proxy that logs which key called which model, from which client, and when.
The author says the proxy also provides virtual API keys, multiple providers and models behind one endpoint, and fast model switching.
The author says the logs let them trace a cache miss in a coding session to its cause within 20 seconds, and that the proxy has an API that could automate this kind of diagnosis.
Written by AI from the articles below · updated Oct 19, 5:27 AM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- Hacker News · AI (150+ points)BlogIf you don't run your own LLM proxy, how do you know what's going on?
AIThe author argues that LLM provider dashboards do not give engineers usable token usage data, so they run their own proxy to log which key called which model, from which client, and when. The proxy also supports virtual API keys, multiple providers and models behind one endpoint, and fast model switching. The author says they can trace a cache miss in a coding session to its cause within 20 seconds, and that the proxy has an API that could automate this.
Heat Trend
Not enough continuous observations to show a trend yet.