KAIZEN Book a call
Open main menu
Book a call

How we run Kaizen

How we cut ten alerting agents down to one

Each of our AI agents sent useful alerts, but ten of them together were noise, and the urgent ones got lost among the rest. Now one agent decides what needs a person today, and everything else waits for the daily briefing.

By Ashish Tonse 5 min read

We run much of Kaizen on AI agents, and for a while we added one for each part of the business we wanted watched. We ended up with ten. Most of them could send an alert to a person on their own, and each alert made sense when you read it by itself.

Together, they were noise. One person had to read every alert, decide which ones mattered and set the rest aside, several times a day. Each agent also had its own idea of what counted as urgent, so a routine reminder could arrive looking just as pressing as a client waiting on a decision. The alerts that mattered were there, but they were hard to find among the rest.

We replaced the ten agents with one general agent and a set of scheduled jobs. The jobs still check their own parts of the business. Routine findings go into a queue for the daily briefing, and one set of written rules decides what is urgent enough to interrupt someone.

The lesson for anyone designing agents: noticing something and deciding to interrupt a person are two separate jobs. Give them to separate parts of the system.

Before and after: who decides to interrupt a person Before: many agents each checked one part of the business, and each could send an alert to a person using its own rule for what counts as urgent, so one person had to sort through all of them. After: scheduled jobs still check their own parts of the business, but one general agent with one set of written rules decides what happens to each finding. An urgent finding becomes a phone alert only when it is tied to a work item, and a repeat alert about the same item waits six hours by default. Everything else waits for the daily briefing, where each finding is checked again against the current record and expired items are dropped. BEFORE: EVERY AGENT COULD INTERRUPT EACH WITH ITS OWN RULE FOR URGENT Agent Agent Agent Agent More One person SORTS THROUGH EVERY ALERT AFTER: ONE AGENT DECIDES JOBS STILL CHECK THEIR OWN PARTS OF THE BUSINESS Scheduled job Scheduled job Scheduled job One general agent ONE SET OF WRITTEN RULES URGENT Alert to a phone ONLY WITH A WORK ITEM REPEATS WAIT SIX HOURS CAN WAIT Daily briefing EACH FINDING CHECKED AGAIN EXPIRED ITEMS DROPPED
Before, each agent decided for itself when to interrupt a person. Now the jobs report what they find, and one agent applies one set of written rules to decide what reaches a person today and what waits for the briefing.

Give one agent the decision about what is urgent

Our general agent holds routine findings for the briefing. It interrupts a person only for urgent deadlines, work that is blocked now, decisions needed the same day and urgent client support requests.

If you are designing a similar system, ask which part of it can weigh a finding against everything else happening that day. A specialist job can notice that a renewal is coming up. Deciding whether that renewal is worth an interruption needs a view of all the other work competing for attention.

What one job sees, and what the deciding agent sees On the left, a specialist job sees only its own part of the business: it notices that a renewal is coming up, and nothing else is in view. On the right, the general agent sees the whole day. The renewal sits alongside the kinds of finding that can interrupt a person under the written rules: an urgent deadline, work that is blocked now, a decision needed the same day and an urgent client support request. The job's part is noticing. The agent's part is deciding, by weighing the renewal against everything else competing for attention, using one set of written rules. A SPECIALIST JOB NOT IN VIEW NOT IN VIEW NOT IN VIEW NOT IN VIEW A renewal is coming up THE GENERAL AGENT: THE WHOLE DAY An urgent deadline Work that is blocked now A decision needed the same day An urgent client support request A renewal is coming up CAN INTERRUPT CAN INTERRUPT WEIGHED AGAINST THE REST CAN INTERRUPT CAN INTERRUPT NOTICING: ONE PART EACH DECIDING: ONE AGENT, ONE SET OF WRITTEN RULES
A job sees only its own part of the business. The general agent sees the renewal next to everything else that day, so it is the part that decides.

Write these rules before you add another way to send notifications. List the conditions that justify an interruption, and give routine findings a place to wait.

Check each finding again before the briefing

Our briefing instructions tell the agent to look up each record again before it repeats a finding. They also tell it to drop meeting acceptances and out-of-office replies that have expired, and to group receipts together.

So the briefing is a summary of where things stand now, built from the records themselves. It is more useful than a list of messages in the order they arrived.

When you design your own queue, decide what makes each kind of finding out of date, and make the briefing check for it. One good test: change the underlying record after a finding is queued and before the briefing runs. The briefing should report the current record, and drop the old text.

Every queued finding is checked again before the briefing On the left, six findings queued in the order they arrived: a meeting acceptance, a receipt, an out of office reply, a second receipt, a finding about a work item whose record has changed since, and a third receipt. Before the briefing, the agent looks up each record again. The expired meeting acceptance and out of office reply are dropped. The three receipts are grouped into one line. The work item is reported from its current record, not from the old text of the finding. The briefing on the right is a summary of where things stand now. The test: change a record after its finding is queued, and the briefing should show the current record. QUEUED, IN THE ORDER IT ARRIVED Meeting accepted Receipt Out of office reply Receipt Finding about a work item Receipt EXPIRED, DROPPED GROUPED EXPIRED, DROPPED GROUPED RECORD CHANGED GROUPED Look up each record again BEFORE THE BRIEFING THE BRIEFING: WHERE THINGS STAND The work item FROM ITS CURRENT RECORD, NOT THE OLD TEXT Receipts, grouped THREE RECEIPTS, ONE LINE DROPPED: TWO EXPIRED ITEMS THE TEST: CHANGE A RECORD AFTER ITS FINDING IS QUEUED. THE BRIEFING SHOWS THE CURRENT RECORD.
Findings wait in the order they arrived. Before the briefing, each one is looked up again: expired items drop out, receipts collapse into one line, and a changed record is reported as it stands now.

Our system cannot send an alert to a phone unless the alert is attached to a work item. The alert links to that record, so the person who gets it goes straight to the work it is about.

A repeat alert about the same work item waits six hours by default. The agent can override the wait when the situation has changed in a material way.

Some urgent events follow fixed rules and do not wait for the general agent, including client requests to schedule a meeting and failed jobs. These alerts still attach to a work item in the same way.

When an alert about the same work item may repeat A timeline for one work item. The first alert goes to a phone only because it is tied to a work item, and it opens that item. A repeat alert about the same work item waits six hours by default. If nothing material has changed, a repeat inside the six hours is held and can go out after the wait ends. If the situation has changed in a material way, the agent can override the wait and send the repeat sooner. Separately, some urgent events follow fixed rules, such as a client request to schedule a meeting or a failed job. They do not wait for the general agent, and they still attach to a work item. DEFAULT WAIT BEFORE A REPEAT: SIX HOURS FIRST ALERT SIX HOURS LATER First alert TIED TO A WORK ITEM SENT, AND IT OPENS THE WORK ITEM Repeat, nothing changed SAME WORK ITEM HELD CAN REPEAT Repeat, situation changed IN A MATERIAL WAY AGENT OVERRIDES THE WAIT FIXED-RULE EVENTS, SUCH AS A CLIENT REQUEST TO SCHEDULE A MEETING OR A FAILED JOB, DO NOT WAIT FOR THE GENERAL AGENT. THEY STILL ATTACH TO A WORK ITEM.
A repeat about the same work item waits six hours by default. A material change lets the agent send it sooner.

If you run your own agents, settle these questions in writing: what record must exist before an alert goes out, how often an alert can repeat, and what kind of change allows an override. You can inspect and test these rules alongside the model's instructions.

List every way an agent can interrupt a person

The specialist jobs are still part of our system. The general agent gives them one place to send routine findings and one rule for what is urgent.

If your team runs several AI workflows, start by listing every path that can interrupt a person. For each one, record:

  • Who decides whether it is urgent.
  • What evidence supports the finding.
  • When that evidence must be checked again.
  • What makes the finding expire.
  • Which work record the alert opens.
  • What stops repeat alerts about the same work.

This list is a good starting point for a conversation about AI implementation. It connects each agent's instructions to how people receive and act on what the agents produce.

For the same approach applied to deadlines, read How to automate follow-up when deadlines slip. For everything the agent does today, and the rules it follows, see How we run Kaizen on AI.

Want the same for your company?

Bring us two or three workflows that take up your team's week. We will tell you which one to automate first.

How we implement AI

Book a 30-minute call