The fastest way to fail with complaint-handling automation is to switch it on for every product on day one. The second fastest is a pilot so cautious that it never produces a number anyone trusts. This is a ninety-day plan between the two: one product line, real complaints, the firm’s own policy, people deciding every case, and a measured answer at the end about whether to go further.

In short

  • Days 1–30: connect and configure. Read-only access to contacts, customers, accounts, fees and prior complaints; your deadlines, redress rate and thresholds; your case controls and letter wording; roles; and a baseline measured before anything changes.
  • Days 31–60: one product line, live. Agents prepare every complaint on one line; handlers decide every one; a sample is worked both ways; the classifier and knowledge base are tuned as gaps appear.
  • Days 61–90: widen and decide. Turn on root-cause analysis and QA sampling across the line, add a second line, put validation evidence in front of risk, and take a measured recommendation to the committee.
  • Autonomy does not rise in ninety days. Agents start at L1 and are capped at L2 (Ask first). What the pilot earns is evidence, not permission.

Before day one: choose the scope

Pick one product line where complaints are frequent enough to measure in weeks and the process is well understood. Current accounts are a common first choice: fee complaints exercise classification, research, redress and letters together. A claims line is a good second, because it exercises the clock and vulnerability more than redress.

Decide up frontWhy it matters
The product lineEverything else is sized from it
Who decides during the pilotNamed handlers and leads, the same people throughout, so agreement rates mean something
The sample worked both waysA fixed share of complaints also prepared by hand, to compare outcomes
What success meansThresholds for agreement, deadlines kept and redress corrected, agreed before the pilot, not after
Your policy valuesDeadlines, redress rate, approval threshold, sampling rate, systemic threshold: replace every illustrative default
The exitWhat happens to complaints in flight if the pilot stops

The plan

Day 0 Day 15 Day 30 Day 45 Day 60 Day 75 Day 90Source systems, read-only Baseline measured Roles and sign-in Policy values and controls Letter wording and knowledge baseAgents prepare, people decide Sample worked both ways Classifier and KB tuning Root causes and QA sampling Second product line Validation and recommendation ConnectConfigureLive on one lineWiden and decideThe first ninety days

Ninety days in four overlapping tracks.

Days 1–30: connect and configure

Connect the sources, read-only

InsightService reads the firm’s systems from a source schema (SERVICE_SOURCE_SCHEMA) and keeps its own work in its own schema. For a first line it needs inbound contacts (emails, web forms, chat transcripts, call notes), customers and their accounts or policies, fees and transactions, prior complaints, and the knowledge-base articles replies are drafted from. Views over existing systems are usually enough; nothing is written back.

Set the policy

Every deadline, rate and threshold in the product is configuration, and every one ships as an illustrative placeholder. Replace them with the firm’s own values before a single real complaint is handled:

SettingWhat it controlsDemo value (illustrative)
SERVICE_ACK_DAYSDays to acknowledge a complaint5
SERVICE_FINAL_WEEKSWeeks to the final response8
SERVICE_DUE_SOON_DAYSWhen a deadline shows as due soon7
SERVICE_REMEDIATION_RATEAnnual simple interest added to refunds0.08
SERVICE_APPROVAL_THRESHOLDRedress above which a team lead approves250 (GBP)
SERVICE_QA_SAMPLE_RATEShare of closed cases sampled for QA0.2
SERVICE_SYSTEMIC_CUSTOMERSCustomers in a cluster before it can look systemic5

Then review the nine case controls in Rules Studio with compliance, and replace the template referral-rights wording in the final response letter with the firm’s approved text. The product flags that wording as a template on every draft until you do.

Rules Studio, Case controls: nine published rules that block or refer a close, such as No outcome recorded, Final response not sent, No root cause tagged, Remediation waiting for a decision, Complaint clock breached and Repeat complainant
Review the controls with compliance. Seven block a close; two refer it to a team lead.

Roles, and a baseline

Set up service agents, team leads and administrators through the identity service or your SSO. Measure the baseline before anything changes: preparation minutes per complaint, deadlines missed, redress corrected, and a sample of queries audited for hidden complaints. Without a baseline, day 90 has nothing to compare against.

Days 31–60: one product line, live

Switch on ingestion and work the chosen line's complaints on InsightService. The triage agent classifies each contact and quotes the sentence that makes it a complaint; handlers confirm or correct. On each case the agents run in order (triage, vulnerability spotter, case researcher, remediation calculator, reply drafter, supervisor) and every output waits for a person to accept, reject or ask.

What to watch each weekAgreement, corrections and the clock
1
Classification correctionshandlers

Every “not a complaint” or “actually a complaint” is a tuning signal for the lexicons.

2
Agreement with prepared decisionsActivity page

By agent and action. Low agreement on one action points at a fix, not a failure.

3
Redress changed before approvalteam leads

Any difference between the calculator and the approved sum needs a reason.

4
Deadlinescase queue

Acknowledgements and final responses due soon or overdue, compared with the baseline.

5
Letters rewrittenhandlers

Heavy rewrites usually mean the knowledge base needs an article, not that the drafter is wrong.

6
Weekly reviewpilot lead

Fix what the signals show; record every change to policy or lexicons.

Agent activity in InsightService: every agent step on every case and on the inbox, root causes and QA, with what was prepared and what is waiting on a person
Activity. Every agent step, what it prepared and what is waiting, which is where agreement is measured.

Days 61–90: widen and decide

With a month of cases on one line, turn on the oversight agents across it. The root-cause analyst groups complaints by cause and product across cases, the inbox and prior complaints, and flags clusters that look systemic for a person to own. The QA reviewer samples closed cases and checks each against the checklist, recalculating deadlines and redress; a person who did not close the case records the result.

Add a second product line, ideally one that exercises different parts of the process. Then assemble the evidence for the committee and for model risk or internal audit: agreement rates by agent and action, classification corrections, redress differences, deadlines against the baseline, QA results, and the decision records of a sample of closed cases.

yes

partly

no

Baseline, day 30

Compare at day 90

Pilot measures

Met the agreed thresholds?

Recommend the next lines

Fix and extend the pilot

Stop, with the evidence

The day-90 decision, made against thresholds agreed before the pilot.

Who should be in the room

RoleWhat they own in the pilot
Head of complaintsThe policy values, the case controls, the letter wording, and the day-90 recommendation
Pilot leadThe weekly review, the change log, and the baseline comparison
Handlers and team leadsAnswering every prepared decision honestly; approvals and referred closes
Data or platform engineerThe read-only source connections and the views behind them
Risk or internal auditReviewing the agents, the calculator and the controls before any widening
ComplianceApproving the referral-rights wording and any change to the controls

Keep the group small and the same for ninety days. Every change to a policy value, a lexicon, a knowledge-base article or a control should be logged with the date and the reason, so that the numbers at day 90 can be explained line by line.

About autonomy

Every InsightService agent starts at L1 (Draft) and is capped at L2 (Ask first) for every action. The platform recommends raising an action’s level only after people have agreed with what the agent prepared at least 90% of the time over at least 50 answers in 30 days, and an administrator makes the change. Even at L2, agents never send to a customer, approve or pay redress, decide whether a case is a complaint, or close a case. Ninety days is enough to measure agreement; it is not a reason to change who decides.

The Autonomy page in InsightService: the tenant ceiling, each agent's level per action, and its agreement and override rates over the last 30 days
Autonomy per agent and action. Levels are set by a person; agreement is what earns a recommendation.

Mistakes that cost a pilot

  • Leaving the illustrative values in place. The demo’s five days, eight weeks, 8% and £250 are placeholders. A pilot on placeholders measures the wrong deadlines and the wrong redress.
  • Sending the template wording. The final response letter carries referral-rights wording flagged as a template. Replace it with approved text before the first real letter; setting SERVICE_ALLOW_TEMPLATE_SEND to anything other than 1 makes the product refuse to send a final response that still contains the template marker.
  • Rubber-stamping. If handlers accept everything, agreement is meaningless. Make “reject” and “ask” normal, and look at a sample of accepted decisions.
  • Switching on live delivery too early. With SERVICE_COMMS_LIVE off, a sent reply or request is recorded but not delivered, and the sender sends it through the firm’s usual channel. Turn delivery on only once the letters and routing are approved.
  • Changing two things at once. A lexicon change and a knowledge-base change in the same week make it impossible to say which moved the numbers.

What to report at day 30, 60 and 90

Day 30

Connected and configured

  • Sources connected, read-only
  • Policy values replaced, controls reviewed, letter wording approved
  • Baseline measured
Day 60

One line live

  • Agreement by agent and action
  • Classification corrections and lexicon changes
  • Deadlines and redress against the baseline
Day 90

A measured recommendation

  • Root causes and QA results across the line
  • Second line started
  • Validation evidence and a go, fix or stop decision

For the head of complaints

Own the policy values and the controls from day one. The pilot is the time to make sure the product enforces your rules, with your wording, before the volume arrives.

For the service operations lead

Keep the same handlers and leads throughout, answer every prepared decision honestly, and log every correction. Those answers are the measurement.

For the board

Ask for the thresholds before the pilot starts and the comparison with the baseline at the end. A go, fix or stop decision against agreed numbers is the outcome to expect.

Ninety days buys evidence, not autonomy.

Start on one line with your own policy, let agents prepare and people decide every case, and measure against a baseline you took before anything changed.

Keep reading