Note: This post was written by Vincent Seguin, 91’s Chief Technology Officer. This is the latest in a series about 91’s AI journey. You can read the previous post here.
Table of Contents
This post covers how we monitor what our AI does in production, and how our team has evolved in building software with agents. The common thread is ownership: in both cases, AI is blurring the lines between product and development, and we’re learning to work across them. As always, we don’t have all the answers, but here’s what we’ve learned so far.
Part 1: Watching how our agents behave
Shipping an agent is one thing; knowing whether it actually helped a real person with a real question on a random Tuesday afternoon is another.
We started where most teams start, with eval suites in CI. We run them against the SQL generation step behind Ask 91, and they’re great for catching regressions. What they can’t tell you is how a user felt when the answer came back wrong, or technically right but not useful. For that, you have to look at production.
Inspecting every conversation
We landed on as our eval and observability layer. Every Ask 91 conversation becomes a trace we can open, so we can see what the user asked, how the Receptionist routed it, which agent answered, the prompts that went in, and the answer that came out. We can follow a single thread turn by turn, or look at everything one user has asked.
A judge for frustration and confusion
The feature we’ve gotten the most value from is using an LLM as a “judge.” Each turn of a conversation gets scored by a model with one job: does this user look frustrated or confused?
The judge gets a prompt describing the signals we care about, like a user saying “that’s not what I asked,” asking for the same thing a third time, or giving up halfway through. It also gets the context a human reviewer would want, including the thread history and the user and account it belongs to. It returns a score and a short explanation of the score given.
This approach started as a hackathon project to score turns for user disappointment. It worked well enough to take seriously, and it was noisy enough to teach us something: our first definitions flagged far too much. The fix wasn’t a smarter model; it was tighter definitions, concrete examples in the prompt, and people reading what got flagged and telling us what was real.
Recently we made it official: the judge now posts what it finds straight into a Slack channel. Issues show up where the team already works instead of whenever someone remembers to go look.
Each flag arrives with the pertinent details: which agent answered, the question, the answer given, the message that got flagged, and the judge’s reasoning in plain words, with links to the thread and trace in Langfuse. The judge also tells us why it flagged the turn: whether the user rejected the answer, disputed the data, or complained that the answer missed the mark.
In its first week running in Slack, the judge flagged between 2% and 3% of customer conversations, roughly one in forty. That’s small enough that someone actually reads every flag, and big enough that something worth looking at shows up most days.
What it caught
A good example of why this matters: users kept asking Ask 91 how to do things, like adding a project or finding deactivated people. The support agent answered accurately, step by step, but the steps didn’t work because those users didn’t have permission to do them. The answer was correct, but the experience was still frustrating.
Once we caught this, the fix was simple: tell people up front when they’re missing a permission or a feature, then explain the steps. The judge catches the same pattern in the wild, like a user writing “I don’t see ‘Create Crew’” after finding the button missing.
Shared ownership
This example is also why I find behavioral monitoring so interesting. Traditional monitoring is clearly a dev responsibility. Is the service up? Is it fast? Are errors spiking? When something breaks, an engineer gets alerted and fixes it.
Behavioral monitoring doesn’t work that way. Everything can be technically healthy, and a user can still walk away without the help they need. When that happens, the cause could be a prompt, a missing permission message, a gap in the product, or a bit of each. There’s no clean line between “a product problem” and “a dev problem,” so there can’t be a clean handoff either.
That’s why the judge posts to a channel that both teams watch. Product looks at what users are actually experiencing and decides what good should look like. Dev digs into why the agent behaved the way it did and fixes the cause. Neither team owns it alone, and that shared ownership has quickly become one of the most valuable parts of the whole setup. Where we’d like to get to next is a ticket created and fixed automatically when users look frustrated about something. We’re not there yet, and it will take a lot of tuning, but that’s the direction.
That blurring of who owns what isn’t limited to AI monitoring. It’s starting to show up in how we build software overall, which brings me to the second half of this post.
Part 2: Where we’re headed as a team
A few weeks ago, I gave a talk called “One Layer Up” to our product and development teams. The core message sounds dramatic, but it’s sincere: the era of writing code is mostly over.
For most of software’s history, writing code was the longest part of building a product. Today, agents do much of that work in minutes. That’s an exciting shift, but it also raises an uncomfortable question for a lot of developers: if I’m not writing the code, what exactly is my job?
I don’t want to brush that question off. Many of us got into this field because we love the craft, and watching an agent produce in minutes what we’d have spent a day perfecting can feel like a real loss. But I don’t think the job is disappearing. I think it’s moving up a layer, the same way it has many times before. We went from assembly to C, and from C to managed languages where nobody reads the generated bytecode anymore. Each time, people worried, and each time the work became more ambitious rather than less. Describing intent and letting agents write the code is the next step on that path.
What the job becomes
So what does the job look like one layer up? For me, it comes down to judgment. Are we solving the right problem, the one customers actually have rather than the one we assumed? Does the solution fit our architecture and our product, and not just the ticket? Is the result something we’d be proud to own in two years? Unlike a compiler, an agent isn’t deterministic, so someone still has to stand behind what ships. The satisfaction shifts too: less of it comes from elegant code, and more from what we shipped and who it helped.
This is also where the lines between disciplines start to blur, much like they did with behavioral monitoring. When an agent can work across the whole stack, it makes more sense for one person to own a feature end to end than to pass it between “the backend person” and “the frontend person.” Most of our recent AI improvements were already built that way. Deep specialists matter more than ever in areas like infrastructure, data, performance and security, but the default is shifting toward broader ownership. And the blending doesn’t stop at engineering: product, design and development increasingly have to work as one team around the same problem. That’s why we’re expanding the internal AI toolkit we built for developers so it also covers design and writing product specs.
Staying in control
The natural concern is control. If agents write more of the code, how do we keep the codebase healthy? Our answer is what we call “the harness,” a safety net around our code that catches problems before they reach customers. It’s made up of linters and static analysis, tests at every level, deployment quality, and now the guidelines and skills our agents follow. None of it is new. When we scaled with people, the harness was what kept all that extra change safe. Now we’re scaling with agents, so the harness matters more than ever, and it’s never really done.
The habit we’re working hardest to build is simple: when an agent gets something wrong, don’t just fix the code. Ask why it happened. Was a guideline missing? Did it lack context? Would a test have caught it? Fixing the cause means the next run starts from a better place. It’s the same loop as the judge in Part 1: notice when something goes wrong, understand why, and improve the system rather than the single answer.
The new bottlenecks
One thing we’ve learned quickly is that when the middle of the process speeds up, the ends get squeezed. Specs, designs and technical plans still move at human speed, and without clear input, agents just build the wrong thing faster. On the other end, code review, QA and releases become the new queue. We’re already feeling it in code review, with more pull requests coming in and the same number of reviewers.
That’s where we’re focusing next. On the output side, we want to start automating part of code review, beginning with small, low-risk changes and expanding as we build trust. To prepare, we’re adding code review guidelines to each of our repositories, so agents review against the same standards our engineers do. On the input side, we’re using the expanded toolkit, with skills built alongside Product and Design that help draft specs and capture what good design looks like at 91. The clearer we can be about what “good” means, the better agents can follow it, and the more time our product and design partners get back to spend with customers.
Why go through all of this? Because we want to take on bigger, more ambitious problems for our customers, not just ship more of the same features faster. Whatever 91’s next chapter looks like, it will ask more of us than today, and changing how we work is how we make room for that ambition.
We’re early, and we’ll certainly get some things wrong along the way. But we’re moving up a layer on purpose, and I’d rather share what we’re learning than pretend we have it all figured out. If your team is going through something similar, I hope our experience offers some useful inspiration, and I’d love to hear what you’re learning too.
