Staying on the cheap model

I run a small site called hnhiring that indexes the monthly “Ask HN: Who is Hiring?” thread on Hacker News and lets you filter it by location. The thread is great, but it’s very US-focused. I was job hunting in Europe at the time, so I decided to build it mostly to scratch my own itch.

The way it works is pretty simple. A Go program pulls every post from a month’s thread, and then a model reads each post and pulls out the useful bits: the tech stack, the location, the salary, visa sponsorship, and a link to apply. Those results get saved as JSON and served from a single HTML file. Usually a full month is around 300 posts.

Most of the interesting work lives in that extraction step, and from the start I wanted it to stay cheap. That pretty much shaped every decision I made after that.

Batching

The obvious way to speed up extraction is to send the posts to the model in batches instead of one at a time. That’s somewhere around 10 to 20 posts per call, instead of one call for every single post, which cuts down the round trips a lot, and on paper it works great.

The problem is that it has a failure mode that’s hard to notice. When I ran batches against my test posts, one of them, faqta, which has no apply link at all, came back with the apply_url that belonged to openrent, the post right next to it in the same batch.

This is what those two posts should look like (simplified):

{ "id": "faqta",    "apply_url": null }
{ "id": "openrent", "apply_url": "https://openrent.com/jobs" }

And this is what came back when they were sent together. faqta is wrong now:

{ "id": "faqta",    "apply_url": "https://openrent.com/jobs" }
{ "id": "openrent", "apply_url": "https://openrent.com/jobs" }

What makes this sneaky is that nothing breaks. The JSON comes back valid, every field is there with the right type, and the model has just quietly moved a value from one post into the next. It looks like a contamination issue, where a value from one post leaks into another.

The reason is pretty mundane once you think about it. To the model, a batch isn’t a list of separate questions. It’s one document with one prompt but several sections, so it doesn’t really respect the virtual boundaries I thought I was drawing between them. Everything sits in one shared context, so sometimes a value just slips across.

one call per post              one call for all posts

post1 -> call                  post1 -\
post2 -> call                  post2 --> one shared context
post3 -> call                  post3 -/

Why not just use a better model?

This is the part that actually matters. On the cheap model, with reasoning turned off, the slipping just happened. The model wasn’t good enough at keeping the posts apart to avoid it on its own.

There were two obvious ways to fix it.

The first is to turn reasoning on. And it’s true that with reasoning on, the contamination happened a lot less, because the model thought harder about each post and mostly stopped mixing them up. But reasoning on this model is about four times slower. On my test set it went from 32 seconds up to 128. And the worst part is that I throw the reasoning away. The answer I actually keep lives in message.content, and the reasoning itself, over in reasoning_content, never gets used. So turning reasoning on would mean paying four times the time for output I was throwing out anyway.

The second is to just use a bigger, more expensive model. That would probably keep the posts apart too. Throwing more money to solve a problem can work, but it’s not efficient.

Both of them work, but both of them mean paying more, whether in time or in money, to work around a problem that isn’t really the model’s fault. The model was never designed to keep unrelated things apart inside one big prompt. I was the one putting them together.

One post per call

So I stopped putting them together and went one post per call instead. Now there’s no shared context to worry about, because there’s only ever one post in a call. Each post is its own request, and the model has nothing else to look at, so the contamination can’t happen, on any model, cheap or not.

Initially I assumed this would be slower than batching. But it’s still cheap, because I get to stay on the small model with reasoning off. I’m not paying for a bigger model to fix a problem I can just design out of the system.

Here’s what the numbers looked like on my test set of 13 posts:

The two clean rows are the interesting ones. Reasoning works, but it costs four times the time. One post per call works too, on the same cheap model, with no reasoning and no bigger model. That’s the one I went with.

The eval set

None of this would really be a choice without an eval set.

The eval set is just 13 posts with various scenarios: normal listings, awkward edge cases, a few I made hard on purpose, and some that aren’t job posts at all. I run an eval against it before I ship any change. It’s small, but it’s the only reason I can say anything about correctness with any real confidence.

The evals are how I found out the batch path was mixing posts up in the first place. They’re how I knew reasoning wasn’t worth four times the time. And they’re how I could move to one post per call on the cheap model and still trust that the results were right, every post coming back correct, same as before.

To be honest, without evals, the safe thing to do would be to just pay for the bigger model, or turn reasoning on, to be safe. Most people would, and it feels like the responsible choice. But really you’d just be paying for insurance against a problem you can measure, and that you can also just remove.

Final thoughts

The lesson is, don’t pay for a better model to fix a structural problem. When something is wrong because of how you built it, a more expensive model is the expensive answer, not the right one. Build it so the problem can’t happen, then measure so you know it doesn’t.

And that only really works if you measure. The eval set is the whole reason I could stay on the cheap model and still trust it. Without it, I’d just be guessing, and guessing gets expensive.