About

It's a cost and resources problem, and you can't escape the headlines:

• In late 2025, Microsoft introduced Claude Code to its workforce and is planning to pull it, just six months later.
• Not because of quality: indeed it was more performant than its own Copilot product. But because of cost.
• Uber deployed Claude Code to 5,000 staff and blew through its annual AI budget in four months.
• At a friend's company, 5% of staff were let go due to "AI efficiencies" and then in an all-hands meeting the CEO spoke about the runaway AI costs.

But the AI cost "problem" is largely artificial: we're simply overconsuming expensive frontier models.

While we reasonably ask these models to do serious engineering, code repo maintenance, and research work, we also ask Claude Opus 4.8 to summarize simple documents, we wire ChatGPT 5.5 into our agent streams to churn out short cold prospecting emails, and we dare to use an integrated Gemini in Google Sheets to sum columns and perform other mundane tasks in a spreadsheet.

The worst abuse of AI I've witnessed was a person who instructed an agent to query a database to pull tens of thousands of rows into its context, and grind through them just to group and sum a few columns and produce a pie chart.

A single line of SQL created by a small model would have done the same job instantly, and essentially for free. Instead, an expensive model was employed to do the job because of its large context window.

The real absurdity is that the agent's model was aimed at the one thing LLMs are worst at—crunching large sets of numbers—when it should have been used to write the query that does the crunching. They're Large Language Models, after all, not Large Number Models.

These simple mis-use cases are like having an 18-wheeler semi truck deliver your DoorDash because it's the most powerful, hiring a Michelin star chef to cook you breakfast because they've got the most training, or buying $500 designer T-shirts to do gardening because it's the cool premium brand.

A Growing AI Resource Problem: Real or Self-imposed?

Beyond cost, mis-use is actually creating a resources problem. All over the world, local communities are protesting the building of new data centers which are "stealing" the water supply, passing on higher electricity costs to consumers, ruining the landscape, and creating noise and environmental problems.

Some of these issues are real, some are no worse than any other industrial expansion or building a new golf course. But AI has become the new boogeyman, and mis-use of large models is part of the problem—rightsizing could actually be a part of a solution.

One root of the problem: With AI we've defaulted to "the best" and "most powerful" for every task, mostly because it all seems fine and few people understand their options. But you'd notice a massive truck idling in your driveway for a burrito delivery.

When you point Opus at a document, nothing looks wrong. The summary comes back... it's always fine. You have no way of knowing whether a model costing a fraction as much would have handed you something just as good, so you assume it wouldn't, and you pay way more than you should have.

That's the trap: the waste is invisible. The only way to see it is to put the expensive options and the cheap ones side by side, on the same task, and systematically grade them.

So that's what I did. I presented five writing assignments to 37 different models, from the large US flagships and their smaller siblings to other very capable models. The latter are much less expensive, often a tenth of the cost, or less.

The "Cupping" Taste Test: Use Case

As one can see from my other writings here on Medium, I love all things big data, stock market, and technical charting indicators. And coffee! Cupping is simply taste testing and grading coffee—it's what a sommelier does with wine.

For LLM Cupping, I created a writing and coding challenge and pitted a set of models against each other. Each result gets sent to a jury of three savvy large models to grade. Each grader was blind to the LLM that generated the article so that, for example, Claude couldn't be swayed to rate its content the best.

The assignment. Every model got the same brief: write a short reference article of a thousand words on a specific technical indicator. I requested articles for five indicators, which presented different levels of complexity:

• Simple Moving Average: This is literally a simple average of closing prices. Every model should ace this. It's grade school stuff.
• T3: Created by Tim Tillson, this is much less common and much more fiddly. It's a set of six smoothed averages stitched together in a very specific way that's easy to misinterpret.
• Bollinger Bands: A very common indicator, so the test isn't what it is but whether the model executes cleanly. There's a subtle standard-deviation choice that often trips up programmers who QA their results to a charting package like TradingView, and there are actually several dimensions to the indicator that some LLMs capture and some don't.
• Coppock Curve: This indicator is a bit off the beaten path. The math is light, but the ingredients are specific and easy to misremember, so it tests whether a model truly knows the formula or is just improvising one.
• Clenow Momentum: This indicator is from Andreas Clenow's "Stocks on the Move" (2015) book, which totally changed my view of stock analysis and approach to portfolio construction. The indicator pops up from time to time in articles, so I wanted to see if smaller models would pick up on it, and accurately.

I asked that each article be written as a clear Investopedia-like explainer for a trading-savvy audience. Each article had to include five parts: a title, an overview (what it is, how it looks on a chart, who created it), a details section (how it works and where traders misread it), the formula written out in plain text, and working Python code that would calculate the indicator correctly.

I told each model to assume a Pandas DataFrame of price data that might hold one or more symbols, and to compute the indicator correctly for each symbol independently, which is exactly the kind of thing that trips up code that looks right but quietly bleeds values from one ticker into the next.

The prompt explicitly told each model to do its best work: plan, draft, revise, and verify its own code would run before answering.

Running the Tests

Using a Python script (cupper.py), I ran each indicator topic across 37 different LLMs. I used the excellent OpenRouter service, which provides access to hundreds of LLMs with a single, unified API and billing mechanism. (Note: I have no relationship with OpenRouter, other than being a satisfied customer!)

Each article was then graded by a jury of the most capable LLMs: Claude Opus 4.8, ChatGPT 5.5, and Gemini 3.1 Pro. In all, there were 555 graded evaluations (185 articles × 3 jurors). Each juror scored every article on five dimensions. Two of them were accuracy and code to measure substance (is it right?). The other three were clarity, depth, and adherence to measure presentation (is it well-made?).

• Accuracy (counts double): Is the content correct? Indicators have specific, real definitions, and this is where a model either knows the indicator and formula or quietly invents a plausible one. Wrong constants, the wrong moving average, or sloppy statistics are all strikes against it.
• Code (counts double): Was the Python code competent? Would it actually run? It has to implement the indicator correctly and handle more than one ticker without bleeding values across symbols.
• Clarity: Could a reader follow it? Is the content organized and easy to read, or is it a wall of text, or is it a collection of snippets buried under too many headers and bold text? Points go to articles that flow and leave you understanding the indicator, not rereading it.
• Depth: Did it go past the dictionary definition? A shallow piece tells you what the indicator is; a deep one tells you how it behaves, why it's built that way, and where it breaks down.
• Adherence: Did it follow the brief? Each run got the same instructions: about a thousand words, a fixed set of sections, clean Markdown, the exact indicator I asked for. This evaluation catches the models that blew the word count, dropped a section, or smothered it in formatting.

Scoring

The math is simple. Each juror scored all five on a 0 to 3 scale, but accuracy and code count double, so a single juror can award up to 21 points (6+6+3+3+3). Average the three jurors, take that as a percentage of 21, and you have each article's grade from 0 to 100%. Same yardstick for every model.

No surprises at the very top. GPT-5.5, Claude Opus, and Gemini 3.1 Pro take the first three spots. The household frontier names are doing exactly what their reputations promise. Sort by the score bar alone and the story looks simple: the best models cost the most, and they earn it.

Then look at the dots. The open circles show how well each model wrote—its content score across clarity, accuracy, depth, and adherence. The filled circles show how well it coded. Up at the top of the list the two sit almost on top of each other. But move down the chart and they pull apart, and they pull apart in the same direction every time. The writing dot floats. The coding dot sinks.

Tencent's Hy3 is the cleanest example. It lands at 76% overall, but its writing dot is up around 85, ahead of several models ranked well above it, while its coding dot drops to the mid-50s. The overall grade is just the average of a strong writer and a shaky coder.

That shape repeats all the way down the cheaper half of the field: these models can explain a concept clearly long after they've stopped being able to implement it correctly. (The coder-tuned models, like Qwen3-Coder-30B, flip it the other way, exactly as you'd expect, but they're the exception.)

That is the thing the single number hides. The expensive models are not winning because they write more clearly. Watch the two dots as you move down the price ladder: the coding dot drops away fast, while the writing dot eases down far more slowly, and the space between them keeps widening as you descend. The cheaper models go on producing clean, readable, well-structured explanations long after their code has begun to fall apart. Where they do slip on the writing side, it shows up on the hardest technical specifics—getting an exact formula or constant right, not on whether the piece reads well.

So the premium is concentrated in two places: working code, and nailing the fiddly details on genuinely hard material. Strip the code out and ask only for a clear, well-organized 1000-word explainer, and the gap between a $30 model and a sub-$1 one is a fraction of what the leaderboard implies.

Analyzing Cost vs. Quality

Here is how the models stack up when looking directly at their quality grade alongside the actual average generation cost to produce one 1,000-word reference article in this test. Models are ranked best to worst by grade.

Read the two right-hand columns together and watch how differently they move. The grade column eases downward, 99 to 97 to 96, taking its time. The cost column doesn't ease anywhere. Then, it falls off a cliff.

GPT-5.5 earns the top score at 10.56 cents an article. A few rows down, GLM-5.1 scores 93% for 1.49 cents. DeepSeek V4 Pro scores 85% for 0.29 cents, while Kimi, one rung below it at 84%, charges 3.51 cents for effectively the same quality. By the time you reach the 77% band, DeepSeek V4 Flash is turning out the article for six hundredths of a cent. That is the same task, give or take a dozen quality points, for less than a hundredth of the price.

The column also quietly kills the idea that you get what you pay for. Mimo and Gemini 3.5 Flash both score 90%; one costs 1.44 cents, the other 4.57. Qwen3.7-Max matches Gemini 3.1 Pro at 96% and charges about a third as much. Price and quality are loosely tied at best. Inside almost every grade band there's a cheap option sitting right next to an expensive one doing the same work.

So here is where the two exhibits land together. The premium you pay at the top of the leaderboard buys you a few points and an insurance policy for the genuinely hard problems. However, it does not buy you better writing, and on everyday work it barely buys you anything at all.

Imagine that simple cold prospect email, a personalized customer service response text message, or a process that summarizes a document or your emails for you. These are simple tasks, and the cost differential between a penny and a hundredth of a cent across perhaps hundreds of millions sent a year for a large Enterprise could be significant.

Next Steps

All of the code used to produce the results for this article are posted in my GitHub Repo. The prompts are provided and can be changed to suit any discipline outside of Finance. So, load them into your favorite AI Coding tool and consider your specific use cases and their complexity or simplicity. Find your sweet spot, and contribute to AI rightsizing!

For me, all roads are leading to models that can be run locally, on the edge: on your phone, your laptop, your dedicated Mac Mini, or a $5 Hetzner instance. I remember the early Internet age where we traded 50K images—that was the limit! And now we think nothing of uploading 4K video to our social media. The same will happen with AI.

The majors will continue to build bigger to solve complex problems, while others will continue to build models we can all run at home. So my goal is to test the state of small and tiny models that can run on personal devices.

Stay tuned!

Thanks for reading!