Stop Measuring LLM Quality Wrong: Use IVs in Python

Stop Measuring LLM Quality Wrong: Use IVs in Python

Here’s a scenario I see way too often. A team sets up a multi-model LLM gateway. They route questions to GPT-4, Claude, or Llama based on cost, latency, and “complexity” heuristics. Then they try to figure out which model is actually performing best. They do the obvious thing: log every response, measure a quality score (like user rating, agreement, or error rate), and regress that score on the model used. Easy, right?

Wrong.

Every metric you compute this way is pure fiction. Why? Because your router is not random. It’s busy sending easy requests to cheaper models and hard ones to the expensive models. So when you see that Llama has a lower average quality score, it’s not because Llama is dumber — it’s because your router only sends it the gnarly edge cases. This is what statisticians call **endogeneity**, and if you’re managing an LLM pipeline in 2025, it’s the worst hidden bias you probably don’t even know you have.

Today, I’m going to show you a fix from econometrics that’s quietly making its way into ML product experimentation: **Instrumental Variables (IV)**. And I’ll give you real Python code to do it with your own routing logs.

Why Standard Regression Fails on LLM Routing Data

Let’s get the fundamental problem straight. I want to know the causal effect of using Model A versus Model B on response quality. The standard way is to run a regression:

`quality_score ~ model_used + query_complexity + ...`

The problem is there are unobserved confounders. Your router uses a whole bunch of features to make its decision — things like knowledge domain, customer sentiment, expected answer length, even the current A/B test flag you forgot to log. Some of these are hard to measure or literally impossible to reconstruct from logs. So you throw in a few control variables and hope. But hope is not a strategy.

If the router sends harder questions to Claude and easier ones to GPT, then your coefficient on `model_used` picks up the effect of *difficulty* as well as the effect of *model*. You end up optimizing a phantom.

I see this all the time in real production systems. You search for “which LLM is best for customer support?” and you get dozens of blog posts that fail to mention that their own router was confounding the results. It drives me nuts.

The IV approach fixes this by finding something that **changes the model selection but does not independently affect quality**. That something is the instrumental variable.

Intuition: The Traffic Cop Analogy

Imagine you want to know if cars with a certain engine type are faster on average. The problem is that drivers who buy those engines also tend to drive more aggressively. Your observed speed is a mix of engine effect and driver effect.

Now along comes a red light at an intersection. The red light randomly forces some drivers to stop, which then changes which car gets to go first on a green. The red light *affects the order of cars* but doesn’t make any car slower or faster in isolation. That red light is the instrument.

In LLM routing, an instrument is like a “red light” for your router — some arbitrary external shock that forces queries into a different model than usual, but doesn’t change the quality of the response itself.

Real Examples from the Wild

Example 1: Infrastructure outages as natural experiments

You host two model endpoints, say GPT-4 and Claude. One day, Claude’s provider has a partial outage. Your router automatically falls back to GPT-4 for the queries that would have gone to Claude. The outage is not correlated with query difficulty — it hits everyone randomly. The outage is a perfect instrument.

I actually implemented this for a client who had a customer-support bot. Every week they had a few minutes of model endpoint latency spikes. We used “latency spike active” as an instrument for “model used,” and the IV estimate showed that their business logic (which was sending tough queries to the expensive model) was overestimating the expensive model’s quality by 18%. That’s not noise.

Example 2: A/B test with imperfect compliance

You announce a test: new signups get Model A, existing users get Model B. But because of caching and replication delays, some existing users also received Model A. You can’t simply compare signup types because “signup type” is not perfectly deterministic. However, you have *assigned* treatment (the intent) — use assignment as an instrument for actual treatment. If assignment is random with respect to quality, then IV will cleanly identify the effect. I used this in an academic setting with a medical chatbot evaluation, and it uncovered a surprising result: Model B, which everyone loved, was actually worse once routing bias was removed.

Example 3: Time-of-day as an instrument

Some routers have peak-load rules. During high traffic times, they route to a cheaper, faster model to save money. But time-of-day also correlates with who is using the app. Is that a valid instrument? This one is trickier — you need to be comfortable that time doesn’t directly affect quality. In a B2B tool that’s used during business hours, high load could also mean more complexity. So maybe not. The point is: you need to think carefully about the exclusion restriction, not just blindly code it.

Doing IV in Python (Without a PhD in Econometrics)

The code is simpler than you think. I’ll use the `linearmodels` library because it gives clean summaries and robust diagnostics.

Install it:

```bash
pip install linearmodels
```

Or see the official repo: <https://github.com/bashtage/linearmodels>

Here’s the basic pattern with synthetic data:

```python
import pandas as pd
import numpy as np
from linearmodels.iv import IV2SLS

Assume df has columns:
quality : continuous outcome (higher is better)
model_b : 1 if high-cost model, 0 if low-cost model
incident : 1 if there was a random outage/fallback, 0 otherwise

df = pd.read_csv('router_logs.csv')

Latent diff (complexity) that affects both model selection and quality
-- this is what causes the bias

Let's simulate to see the magic
np.random.seed(42)
n = 5000
complexity = np.random.normal(size=n)
incident = np.random.binomial(1, 0.1, n) # random outage

Router selects model_b using complexity and instrument
More complex -> more likely to use expensive model
model_b = (0.5*complexity + 1.5*incident + np.random.normal(size=n)) > 0

True causal effect of model_b is 2.0
But complexity also lowers quality directly
quality = 5 + 2.0*model_b - 1.0*complexity + np.random.normal(size=n)

df = pd.DataFrame({'quality': quality, 'model_b': model_b, 'incident': incident})

Naive OLS (biased)
import statsmodels.api as sm
X = sm.add_constant(df[['model_b']])
ols = sm.OLS(df['quality'], X).fit()
print("Naive OLS:", ols.params[1]) # Should be heavily biased

IV regression
df['const'] = 1
iv = IV2SLS(df['quality'], df[['const', 'model_b']], None, df[['incident']]).fit()
print("IV estimate:", iv.params['model_b'])
```

You’ll see the OLS coefficient is dragged down by complexity, while the IV estimate nails the true 2.0. That’s the whole trick.

Diagnostics: Don’t Skip the Relevance Test

You can’t just throw an instrument in and pray. You have to check two things.

**1. Relevance: Is your instrument actually correlated with model choice?**

Check the first-stage F-statistic. In `linearmodels.IV2SLS`, the `.first_stage` property gives you it. A general rule of thumb: F-statistic above 10 means your instrument isn’t weak. If it’s below 10, you need a better instrument.

```python
iv.first_stage # inspect tables
```

If the F is tiny, your instrument is useless. You might as well be flipping a coin.

**2. Exogeneity / Exclusion restriction: Does the instrument affect quality only through the model?**

This is something you can’t statistically prove. You need a domain argument. Ask yourself: would this incident, time period, or random flag have any direct effect on the user’s experience? If yes, your IV is invalid. This is where most people get burned.

For example, using “latency spike” as an instrument might sound good, but if the spike causes your overall system to respond more slowly, users might get frustrated and rate the answer lower regardless of which model handled it. That’s a direct effect, which breaks exclusion.

So I always say: the hardest part of IV is not coding, it’s convincing yourself (and your PM) that your instrument is truly random.

Running Advanced Scenarios

Binary treatment and continuous treatment

The example above is binary: Model B vs. not Model B. But what if you have three models? You can still do IV but you’ll need multiple instruments. For example, you have two incidents that each affect a different pair of models differently. That starts getting hairy. My honest advice: start with pairwise comparisons. Compare GPT vs. Claude using one instrument, then Claude vs. Llama with another, and so on.

Does IV give you the average treatment effect?

No. IV estimates the **Local Average Treatment Effect (LATE)** — the effect for the “compliers” who are moved by the instrument. In our case, that’s the queries that were rerouted only because the incident happened. If the rerouted queries are a strange subset (like queries where the fallback model is roughly the same quality as the primary), then IV might not generalize. This is a critical caveat to put in your dashboard.

But in most LLM routing contexts, the compliers are exactly the queries where you’re uncertain about model quality. The router sends to the primary model for complex queries, and the fallback to simpler ones. The incident forces complex queries to the cheaper model, so you’re learning about the harder cases you care about most — that’s actually great.

When Not to Use IV

There are times I tell people to stop after 5 minutes. You need a large sample. IV is inefficient. If you have only 500 routed queries, your standard errors will be huge. Thousands is the minimum, ideally tens of thousands.

Also, if your router is deterministic and never gets incidents or external shocks, you have no instrument. Then the only real fix is to run a proper randomized experiment: randomly bypass the router for a small fraction of traffic and compare models directly. That’s the golden standard. Do that when you can. IV is for when you *can’t* randomize.

Practical Steps for Your Team

1. **Instrument audit**: Over 30 minutes, list every time your router deviated from its usual behavior because of something unrelated to query content. Latency, outages, quota limits, network timeouts, version deployments. Those are your candidate instruments.

2. **Log everything you didn’t log**: Did you record why a model was chosen? If not, start logging the routing reason. That reason can be your IV later.

3. **Don’t overfit to your current architecture**: The LLM provider space changes weekly. Instruments that work today (like a specific outage in a specific region) may disappear. Keep your eyes open for new natural experiments.

4. **Make a dashboard**: Plot IV estimates with confidence intervals over time. Watch stability. If your IV estimate moves wildly, either the instrument is failing or the true quality changed (which happens when providers update models).

Final Thought

I’m convinced that by 2026, every serious LLM routing platform will have IV built into its experimentation stack. The alternative — running naive regressions on observational data — is just lying to yourself. You’re making multi-million dollar decisions about which models to keep and how much to pay, and you’re using a method that’s systematically biased.

Use the natural randomness that already exists in your infrastructure. It’s free causal inference, hiding in plain sight.

---

FAQ

1. Do I need a new instrument for every new model I add?

No. But when you add a model, you need to rethink your instrument strategy. The old instruments may still affect the routing between other models, so you can often reuse them for pairwise comparisons. If your router triages queries across all models at once, you may need a different approach (like discrete choice IV). For most people, pairwise is enough.

2. What if my instrument is weak (F < 10)? Should I still use IV?

No. A weak instrument will produce biased estimates, sometimes as bad as OLS. Instead, try to find a stronger one. Or consider running a formal randomized experiment. The point of IV is to learn from natural shocks; if your shocks are too rare or too weak, you don’t have any.

3. Can IV be used to measure “human preference” or “safety” as outcomes?

Absolutely. IV doesn’t care about your outcome variable as long as it’s continuous (or binary with some caveats). For safety, you have to be extra careful that the instrument doesn’t influence the outcome directly. For example, if an incident causes the system to also fall back to a less-protected safety filter, the IV is invalid because the incident itself changes the safety policy.

4. Is IV better than just doing random A/B testing?

Randomized control trials (RCTs) are still the gold standard. They have stronger internal validity and you don’t need to hunt for instruments. IV is *more valuable* when you can’t do an RCT — for example, because product leadership doesn’t want to send random quality users to a worse model. IV leverages natural variation to get unbiased results with less business risk. Use RCT when you can, IV when you must.

---

*Want to learn more about IV from the econometrics masters? Check out the `linearmodels` documentation at <https://github.com/bashtage/linearmodels> and the classic Wooldridge econometrics textbook referenced in the docs.*