2 posts tagged with "evaluation"

I've spent the last six months writing skills for AI coding agents. Somewhere around month three, I started noticing something uncomfortable: some of our best-intentioned guidance was making things worse. Not on every task — just on enough tasks to make me question whether we were actually helping.

I couldn't prove it. I had gut feelings and a few suspicious eval runs, but nothing concrete enough to point at and say "this is why."

Then I read this paper.

Back in January, I started building skills for AI coding agents — domain-specific guidance that helps agents like GitHub Copilot, Claude, and others produce better code in areas they'd otherwise struggle with. My domain happened to be Azure Cosmos DB, but the lessons apply to anyone writing skills for any technology.

Six months later, what I learned about how to write for AI agents turned out to be far more interesting than the domain knowledge itself.