Week notes: evaluation sets, and a useful config pattern

A shorter post, without an image, to show how quick notes look alongside longer pieces.

Write the tests first

Before touching a prompt this week, I wrote twenty example questions with the answers I’d accept. It took an hour and saved several. Every prompt change since has been checked against the same list, so “it feels better” has become “it gets 18 of 20 right, up from 14”.

One file for agent settings

Keeping model, tools and limits in a single file makes reviews much easier:

agent:
  name: supplier-assistant
  model: default
  max_steps: 8
  timeout_seconds: 60
tools:
  - search_suppliers
  - create_ticket
logging:
  level: info
  redact: [email, phone]

Reading

• A long thread on why tool descriptions matter more than the system prompt
• Notes on designing human approval steps that people don’t just click through

More next week.