What the model actually sees

TL;DR: Tuning your agents is very important. Understanding what the model actually sees on each turn is the difference between tuning with a sledgehammer vs a scalpel.

What does the model actually see?

Before any of your tuned configs are seen there is a considerable amount of vendor-provided instruction that is out of your control. I am not saying this is bad…at least not most of the time. I am saying it is important to understand so that instructions don’t contradict, duplicate, cause unnecessary bloat and/or introduce model mis-interpretation. It typically looks like this:

Out of your control i.e. binary-injected:

#1 [main instructions]
#2 [text-output and session guidance]

*at this point you are looking at ~30k of vendor-provided instruction and the first thing the model sees. Let’s keep going.

In your control i.e. MD files, tools, hooks etc:

#3 [session start hook]
#4 [deferred MCP tool names]
#5 [agent types]
#6 [available skills]
#7 [claude.md + @imports + memory.md]
#8 [insert fancy auto-recall memory here]

then what you actually want to do

#9 [user prompt]

and finally one more vendor-provided instruction…

#10 [built-in tools]

So imagine the model sees “do this” then “don’t do this” or “memory says 1=green” then “memory says 1=blue”, dozens of times across many different categories of behavioral instruction.

You keep adding more rules and memories to prevent mistakes. Not really knowing that the model is just doing what it was told AND has to decide which contradictory instruction wins.

Not knowing it has dozens of instructions that all roughly say the same exact thing eating up your capacity for no value.

Last comment. This might be the craziest part. Think about what this means. A vendor can ship a new “feature” by adding a small block of text to the binary-injected instructions. That is essentially how Claude Code’s auto-memory feature became, well, a newsworthy feature. A few hundred lines of text.