One of my favorite books is Robert Pirsig’s Zen and the Art of Motorcycle Maintenance. Early on, he spins the question that haunts most of the product work I do:
Quality—you know what it is, yet you don’t know what it is. But that’s self-contradictory. But some things are better than others, that is, they have more quality. But when you try to say what the quality is, apart from the things that have it, it all goes poof! There’s nothing to talk about. But if you can’t say what Quality is, how do you know what it is, or how do you know that it even exists?
Pirsig never really defines Quality, and I think that’s the point. You recognize a good interface, a good product decision, a good paragraph, before you can fully explain why. Classical analysis can take the motorcycle apart. Romantic attention can feel the ride. Quality is what shows up before that split, at what he later calls the cutting edge of the present.
Not being able to easily define quality is an awkward fact for those of us designing with AI agents. Models are fantastic at producing statistically likely next tokens. But they usually suck at caring whether the result is good. This means a lot of the interesting work right now is not “better prompts.” It’s figuring out how to represent Quality well enough that an agent can aim at it, without pretending the representation itself is Quality.
Taste cannot be fully automated, but it can be captured, checked, and reused. That’s a lot of the goal right now. How can we capture our own sense of taste well enough that an AI can repeat it reliably?
Different ways people are capturing taste
I’ve been working to understand how people think about this fundamental question. So recently I’ve been looking at a cluster of tools that all try to put judgment into AI systems. They disagree on how and where that somewhere should be, which is interesting, but each approach help us understand the techniques that might work.
Judgment as a skill you run on outputs
One author I follow (nurijanian) has been writing PM skills for agents that look less like chat tricks and more like review muscles. /first-principles forces a product decision to reveal what has to be true to better understand it’s truth. /de-clever attacks the twin failure modes of AI fluff and human cleverness to root out common errors. He argues you should run those skills against the output, rather than against the chat that produced it, because it’s the artifact that matters — not merely the process.
To grade product outcomes, all that matters is the impact they create with users. We attempt to predict impact by judging outputs on a given set of scenarios. We also dogfood our own products, because getting more people to try their personal scenarios is helpful for assessing outcomes with real-ish users. Increasingly, though, PMs construct pools of representative examples called evals, then set criteria of judgment to test outputs. If the evals and criteria are representative of what happens in the world, we have a predictive process for estimating impact.
Craft as a workflow with built-in verification
ffmpeg-skill takes a different path. Instead of asking the model to “know FFmpeg,” it gives the agent a fixed editing workflow and twenty-one typed tools: probe the file, prefer lossless operations, plan with --dry-run, check the deliverable against the destination platform, then actually look at a contact sheet before declaring victory.
The skill’s own README is blunt about why this exists: an agent that merely “knows FFmpeg” still guesses frame rates, re-encodes files that only needed a stream copy, and reports done without opening the result. The package encodes craft as sequence and verification. Format checks (codec, size, true peak) can be fixed mechanically. Judgment checks (crop that loses the subject, loudness that boosts ambience into noise) still require a human call, and the skill is designed to surface those instead of papering over them.
Where nurijanian’s skills are portable taste for decisions, ffmpeg-skill is portable expertise encoded as taste within a domain. The agent doesn’t invent what good video delivery means every time. Someone already wrote down the order of operations and what “done” looks like.
Design systems as files agents can read
DESIGN.md, open-sourced by Google Labs (the world I work in), is the visual version of the same bet, and I think it’s one of the more important anti-slop tools shipping right now. The format is intentionally simple and extensible: machine-readable tokens in YAML paired with human-readable rationale written in markdown.
Tokens give exact values an AI can reuse. Prose tells the agent why those values exist and how to apply them when the tokens don’t cover everything. The CLI can lint broken references, check contrast, diff two versions for regressions, and export into preferred token formats, so the file can sit in a wider design pipeline rather than living as a one-off prompt. The reusability is the point.
The reason this matters for AI UI is that models have a strong prior toward a handful of generated-design tells: blue-green gradients on every hero, neon accents on dark mode dashboards, left-edge accent bars on too many cards, Inter everywhere, purple CTAs, identical rounded rectangles. Without a stronger identity to obey, agents reproduce the same crap all the time. And it drives me crazy! DESIGN.md is a powerful tool for overcoming this average slop with your you own sense of taste.
Guillermo Rauch put it nicely:
Your next design system is Markdown, and DESIGN.md is how you fight slop while scaling design taste across a large organization.
The ecosystem around the format is growing fast enough that you don’t have to start from a blank file. VoltAgent’s awesome-design-md is a curated archive of DESIGN.md files extracted from real product sites, each with light and dark previews you can drop into a project and point an agent at. Other catalogs are appearing too. The point isn’t to clone someone else’s brand. It’s that the representation is portable enough for people to share, fork, lint, and improve in public.
DESIGN.MD does not invent or even guarantee good design judgment, it only repeats it reliably. You still need to encode your sense of taste for it to do anything useful.
Critics help enforce Quality
You may have noticed that AI is lazy. If it can find a shortcut, it’s always going to exploit it and gaslight you into thinking it did the work thoroughly. So you need a system to overcome this and many other pedantic errors in the process. I wrote one approach to solving this conundrum in Check Your Work, using the math worksheet generator as the example.
In that system, I have one model that proposes problems, a separate SymPy verifier to check the math, and a wholly different critic system to grade whether the worksheet teaches the requested topic and gets harder across the page. Answer keys are meticulously audited and is transparent about what the AI system can’t honestly judge. These are all part of the rules I’ve built to enforce quality aligned with my judgement of what good looks like.
My Worksheet tool is one example of a critic-integrated loop. Critics are an essential part of how we express quality in a system we ship. They are not “quality” on their own, but critics play an important role in enforcing the rules that lead to my subjective point of view on quality.
Pirsig ran into a version in a scenario he plays out in his books. He talks about a teacher named Phaedrus teaching rhetoric. Phaedrus asked students to write about Quality without giving them a definition. The class could easily rank essays from better to worse with surprising agreement, but they struggled with a shared definition. That’s because people recognize quality but are broadly unable to define it. Grading outputs as a class only made sense because that recognition was shared enough to act on. An eval suite with a critic system is a modern version of the same bet:
Quality is real enough to show up as a pattern across many cases, and it’s possible to design a critic to recognize enough of that pattern to be useful.
When you build these systems it’s important that you carefully check what the grader is actually measuring. In the worksheet system, an over-strict numeric comparison once forced the authoring model to round good physics constants into tidier wrong ones. The path of least resistance became worse math in order to satisfy a checker that thought it was defending Quality. If your verification rejects correct answers, it will eventually manufacture wrong ones. It’s a common trap to watch out for.
Don’t confuse the scoreboard with the game. An eval suite and a grader are just the Quality judgments you chose to defend, written precisely enough that an agent can be held to them. When the suite goes green, you have evidence. When it goes red, you have a conversation about whether the product failed or the standard did. That is how dogfooding-plus-eval pools help PMs estimate user impact before a launch: the representation has to keep earning its keep against reality.
Quality as an ever moving target
In Lila, the follow-up to Zen, Pirsig splits Quality into Dynamic and static forms. I find this to be one of the most profound points he makes and it’s sad that it’s often lost on readers, because most people only remember the motorcycle book.
Dynamic Quality is the pre-intellectual cutting edge of reality, the source of all things, completely simple and always new… Static Quality emerges in the wake of Dynamic Quality. It is old and complex. It always contains a component of memory.
Dynamic Quality is the living edge of “this is better.” Static patterns are what remain after that judgment: the habits, standards, and files you can teach. Skills, DESIGN.md files, and graders are static patterns. They are incredibly useful. They are also lagging records of taste that already happened in a human mind. If you freeze them too hard, you get a museum. If you never write them down, every agent session starts from zero and drifts back toward the model’s prior.
We’re starting to see notions of Dynamic Quality in AI as well. Vercel recently wrote up how their agents use design.md in practice. They pair a public guidance file with a bounded stylesheet and an evaluation loop. The file doesn’t only describe brand; it names the recurring generated-design patterns they never want to see, so agents can recognize and avoid them.
They keep improving the system by only freezing the steps. When they generate anything new against the current guidance, they review the output, and encode accepted corrections back into the system or critics to have a dynamic quality system that evolves as tastes change. Guidance that helped one artifact and hurt another gets revised and standards that no longer prove fruitful are dropped. This notion of dynamic quality seems to be driving results for Vercel. In one matched test they shared, pages generated with their design.md showed 57% fewer failures than pages generated without it. I’ll be interested to see how this evolves over epochs of iteration and refinement. Will it truly be as dynamic as it seems?
Takeaways
The best AI systems today no longer try to encode taste into a single heroic prompt. They put clear indicators and instructions into reusable skills, implement critics to review artifacts, and build workflows that refuse to finish without verification. Design files carry instructions and reasoning, evals keep score, and results are incorporated into the system to keep them fresh.
None of that replaces the Pirsig problem. You still have to know Quality when you see it. The new craft is deciding which parts of that knowing can be written down, which parts can be checked by a machine, and which parts must only be judged by a person with a well calibrated sense of taste.
If you’ve found a representation of taste that still works after real users touched the product, I want to hear what form it took: skill, file, hook, eval, or something weirder. How are you building AI to deliver dynamic quality?