How to Finally Track Your LLM Costs (Before They Spiral Out of Control)

So your company's monthly OpenAI bill just landed and it's... higher than you thought. Way higher.

You ask the finance team where the money went. They shrug and point to a single line item: "OpenAI API Usage". You ask the platform team. They point to a dozen different services all hitting the same API key. Nobody knows which team, which feature, or which user is actually burning through the budget.

Been there. It's a uniquely frustrating kind of hell for a dev team, and it's becoming the default state for companies building with AI.

When we first started letting multiple teams use our central GPT-4o key, everything was fine. Then the bill tripled in one month. After a week of frantic log-diving, we found the culprits: the marketing team was running a massive, unannounced content generation script for SEO, and the product team had a bug in a new summarization feature that was re-processing the same 100 documents every hour.

We were flying completely blind and burning cash. The problem isn't just the total cost, it's not knowing where it's going. Without granular cost attribution, you can't make smart decisions. You can't tell a customer their usage is too high, you can't tell the Head of Product that the shiny new AI feature is unprofitable, and you can't tell which models give you the best bang for your buck.

This isn't just a finance problem; it's a product strategy problem. And solving it requires more than just a bigger budget. It requires visibility.

The Common Ways to Track Costs (And Why They Mostly Suck)

Most teams, when faced with this chaos, end up trying one of three approaches. They range from "simple but useless" to "powerful but a pain to build."

Method 1: The Spreadsheet Method (One API Key Per Team)

The first-pass, "obvious" solution. Finance asks for a breakdown, so you give each team (or service) its own API key. On the surface, it makes sense. Each team gets its own bill. Problem solved, right?

Pros:

It's simple to explain to non-technical stakeholders.
It provides a very coarse level of team-based accountability.

Cons:

Operational Nightmare: Now you have to manage key rotation, permissions, and security for dozens of keys instead of one. When a key inevitably leaks (and it will), good luck figuring out which one it was and which services to rotate. This doesn't just add complexity; it adds significant security surface area.
No Real Insight: You still have to manually stitch together all the billing data in a spreadsheet every month to see the big picture. You know the team cost, but not the customer or feature cost, which is where the real insights are. What if one feature used by Team A is responsible for 90% of their bill? You'd never know. This approach doesn't scale past two or three teams and provides zero actionable intelligence.
Inflexible: What happens when two teams collaborate on a single feature? Do they share a key? Does one team "own" the cost? It creates organizational friction where there should be technical clarity.

Method 2: The DIY Logging Method (Custom Headers & Metadata)

This is the "we're smart engineers, we'll build it ourselves" approach. You keep a central API key but mandate that every API call must include custom headers like x-user-id or x-team-name. All this data gets piped into your logging system of choice (Elasticsearch, Datadog, etc.) for later analysis.

This is a huge step up from the spreadsheet method. If implemented perfectly, it can give you the granularity you need. But the devil is in the details, and those details are a full-time job.

Why it's harder than it looks:

Human Error: A developer, under pressure to ship a feature, forgets to add the x-customer-id header to a new API call. Boom. That cost data is now permanently misattributed or, worse, lost in the void. You have to enforce this with CI checks and constant code reviews, which adds friction to every single deployment.
Maintenance Burden: You've just signed up to build and maintain a whole new distributed system. You need a logging pipeline to slurp up all these headers, a database (like Elasticsearch or a data warehouse) to store it, and a dashboard (like Grafana or Looker) to query it. Who owns this system? Who pays for its infrastructure? Who's on call when it breaks? It's a significant engineering investment.

If you go down this path, a solid metadata strategy is non-negotiable.

A Deep Dive on Metadata Best Practices

Simply adding a user_id is a start, but a truly robust system gives you multi-dimensional visibility. Here's a more comprehensive list of tags you should be tracking:

user_id / customer_id: The absolute must-have. You need to know which of your customers is costing you the most to inform pricing, fair-use policies, and to identify potential abuse.
environment: (e.g., production, staging, development). You don't want your CI tests or a dev's experimental script running up your production bill. This is the first filter you should apply in any cost analysis.
feature_name: (e.g., doc_summarizer, chat_bot, content_generator). This tells you which parts of your product are the most expensive to run and helps you make product roadmap decisions. Maybe that "experimental" feature is secretly your most expensive one.
team_name / cost_center: Keeps the finance department happy and enables team-level accountability without the nightmare of multiple API keys.
model_name: While the API response gives you this, logging it explicitly upfront makes querying and filtering much simpler, especially if you're using multiple providers.
request_id / trace_id: For tying LLM costs to specific user actions or system processes. This is invaluable for debugging and performance analysis.

Getting this right requires discipline across the entire engineering organization.

Method 3: The Observability Proxy (The "Right" Way)

This approach involves routing all your LLM traffic through a single, intelligent gateway or proxy. The proxy is responsible for enriching requests with all the critical metadata and logging the requests and responses before forwarding them to the LLM provider.

This is how mature engineering teams handle observability for any critical infrastructure, and LLMs are no different. It centralizes control, eliminates the "forgot the header" problem, and guarantees that no request goes untracked.

Advantages:

Guaranteed Compliance: Since all traffic goes through one point, you can enforce metadata rules centrally. No code can ship that bypasses your tracking.
Reduced Dev Friction: Developers don't have to remember to add a bunch of custom headers to every call. They just point their request to the proxy's URL.
Flexibility: Want to add a new piece of metadata? Update the proxy. Want to switch from OpenAI to Anthropic? Update the routing logic in the proxy. The logic is centralized, not scattered across a dozen microservices.

A Concrete Example: How a Single User Cost Us $40

Here's a real story. We offer a "free forever" tier for one of our products. For months, we just looked at the total OpenAI bill, saw it was reasonable, and moved on. Our thinking was, "it's a free tier, how much could it possibly cost?"

Once we finally had per-user cost tracking in place (using a proxy), we discovered something alarming. One single user on the free tier was costing us $40 per month in GPT-4 API calls. They had integrated our document analysis feature into their own custom script and were running it thousands of times a day, far more intensely than we ever anticipated for a free user.

Without per-user tracking, we would never have known. We would have just seen our overall costs slowly creeping up by a few hundred dollars a year and assumed it was our paying customers. With this data, we could make an informed decision: we implemented a fair-use rate limit on the free tier. This simple change saved us nearly $500 a year from that one user alone, and more importantly, it protected our business model.

Stop Flying Blind. You Can't Optimize What You Can't See.

Trying to manage your LLM spend without proper observability is like trying to find a gas leak with a match. You might find it, but it's probably going to be painful. A lack of visibility doesn't just cost you money; it costs you the ability to make strategic decisions for your product and your company.

Whether you build a DIY solution with headers, which requires massive internal investment and discipline, or use a dedicated tool, you have to get visibility. We got so frustrated with the DIY approach that we built LLMeter for ourselves. It's a lightweight, open-source proxy that automatically adds the metadata you need and gives you a simple dashboard to see exactly where your money is going. No need for separate keys, and no need to build and maintain a whole new logging system.

If your LLM bill is a black box, it's time to fix it. Your budget, and your product strategy, will thank you.

Ready to get started? Check out the project and docs at llmeter.org and get your costs under control in the next 10 minutes.