The fastest way to fail with complaint-handling automation is to switch it on for every product on day one. The second fastest is a pilot so cautious that it never produces a number anyone trusts. This is a ninety-day plan between the two: one product line, real complaints, the firm’s own policy, people deciding every case, and a measured answer at the end about whether to go further.
In short
- Days 1–30: connect and configure. Read-only access to contacts, customers, accounts, fees and prior complaints; your deadlines, redress rate and thresholds; your case controls and letter wording; roles; and a baseline measured before anything changes.
- Days 31–60: one product line, live. Agents prepare every complaint on one line; handlers decide every one; a sample is worked both ways; the classifier and knowledge base are tuned as gaps appear.
- Days 61–90: widen and decide. Turn on root-cause analysis and QA sampling across the line, add a second line, put validation evidence in front of risk, and take a measured recommendation to the committee.
- Autonomy does not rise in ninety days. Agents start at L1 and are capped at L2 (Ask first). What the pilot earns is evidence, not permission.
Before day one: choose the scope
Pick one product line where complaints are frequent enough to measure in weeks and the process is well understood. Current accounts are a common first choice: fee complaints exercise classification, research, redress and letters together. A claims line is a good second, because it exercises the clock and vulnerability more than redress.
| Decide up front | Why it matters |
|---|---|
| The product line | Everything else is sized from it |
| Who decides during the pilot | Named handlers and leads, the same people throughout, so agreement rates mean something |
| The sample worked both ways | A fixed share of complaints also prepared by hand, to compare outcomes |
| What success means | Thresholds for agreement, deadlines kept and redress corrected, agreed before the pilot, not after |
| Your policy values | Deadlines, redress rate, approval threshold, sampling rate, systemic threshold: replace every illustrative default |
| The exit | What happens to complaints in flight if the pilot stops |
The plan
Ninety days in four overlapping tracks.
Days 1–30: connect and configure
Connect the sources, read-only
InsightService reads the firm’s systems from a source schema
(SERVICE_SOURCE_SCHEMA) and keeps its own work in its own schema. For a first line it
needs inbound contacts (emails, web forms, chat transcripts, call notes), customers and their
accounts or policies, fees and transactions, prior complaints, and the knowledge-base articles
replies are drafted from. Views over existing systems are usually enough; nothing is written back.
Set the policy
Every deadline, rate and threshold in the product is configuration, and every one ships as an illustrative placeholder. Replace them with the firm’s own values before a single real complaint is handled:
| Setting | What it controls | Demo value (illustrative) |
|---|---|---|
SERVICE_ACK_DAYS | Days to acknowledge a complaint | 5 |
SERVICE_FINAL_WEEKS | Weeks to the final response | 8 |
SERVICE_DUE_SOON_DAYS | When a deadline shows as due soon | 7 |
SERVICE_REMEDIATION_RATE | Annual simple interest added to refunds | 0.08 |
SERVICE_APPROVAL_THRESHOLD | Redress above which a team lead approves | 250 (GBP) |
SERVICE_QA_SAMPLE_RATE | Share of closed cases sampled for QA | 0.2 |
SERVICE_SYSTEMIC_CUSTOMERS | Customers in a cluster before it can look systemic | 5 |
Then review the nine case controls in Rules Studio with compliance, and replace the template referral-rights wording in the final response letter with the firm’s approved text. The product flags that wording as a template on every draft until you do.
Roles, and a baseline
Set up service agents, team leads and administrators through the identity service or your SSO. Measure the baseline before anything changes: preparation minutes per complaint, deadlines missed, redress corrected, and a sample of queries audited for hidden complaints. Without a baseline, day 90 has nothing to compare against.
Days 31–60: one product line, live
Switch on ingestion and work the chosen line's complaints on InsightService. The triage agent classifies each contact and quotes the sentence that makes it a complaint; handlers confirm or correct. On each case the agents run in order (triage, vulnerability spotter, case researcher, remediation calculator, reply drafter, supervisor) and every output waits for a person to accept, reject or ask.
Every “not a complaint” or “actually a complaint” is a tuning signal for the lexicons.
By agent and action. Low agreement on one action points at a fix, not a failure.
Any difference between the calculator and the approved sum needs a reason.
Acknowledgements and final responses due soon or overdue, compared with the baseline.
Heavy rewrites usually mean the knowledge base needs an article, not that the drafter is wrong.
Fix what the signals show; record every change to policy or lexicons.
Days 61–90: widen and decide
With a month of cases on one line, turn on the oversight agents across it. The root-cause analyst groups complaints by cause and product across cases, the inbox and prior complaints, and flags clusters that look systemic for a person to own. The QA reviewer samples closed cases and checks each against the checklist, recalculating deadlines and redress; a person who did not close the case records the result.
Add a second product line, ideally one that exercises different parts of the process. Then assemble the evidence for the committee and for model risk or internal audit: agreement rates by agent and action, classification corrections, redress differences, deadlines against the baseline, QA results, and the decision records of a sample of closed cases.
The day-90 decision, made against thresholds agreed before the pilot.
Who should be in the room
| Role | What they own in the pilot |
|---|---|
| Head of complaints | The policy values, the case controls, the letter wording, and the day-90 recommendation |
| Pilot lead | The weekly review, the change log, and the baseline comparison |
| Handlers and team leads | Answering every prepared decision honestly; approvals and referred closes |
| Data or platform engineer | The read-only source connections and the views behind them |
| Risk or internal audit | Reviewing the agents, the calculator and the controls before any widening |
| Compliance | Approving the referral-rights wording and any change to the controls |
Keep the group small and the same for ninety days. Every change to a policy value, a lexicon, a knowledge-base article or a control should be logged with the date and the reason, so that the numbers at day 90 can be explained line by line.
About autonomy
Every InsightService agent starts at L1 (Draft) and is capped at L2 (Ask first) for every action. The platform recommends raising an action’s level only after people have agreed with what the agent prepared at least 90% of the time over at least 50 answers in 30 days, and an administrator makes the change. Even at L2, agents never send to a customer, approve or pay redress, decide whether a case is a complaint, or close a case. Ninety days is enough to measure agreement; it is not a reason to change who decides.
Mistakes that cost a pilot
- Leaving the illustrative values in place. The demo’s five days, eight weeks, 8% and £250 are placeholders. A pilot on placeholders measures the wrong deadlines and the wrong redress.
- Sending the template wording. The final response letter carries referral-rights
wording flagged as a template. Replace it with approved text before the first real letter; setting
SERVICE_ALLOW_TEMPLATE_SENDto anything other than 1 makes the product refuse to send a final response that still contains the template marker. - Rubber-stamping. If handlers accept everything, agreement is meaningless. Make “reject” and “ask” normal, and look at a sample of accepted decisions.
- Switching on live delivery too early. With
SERVICE_COMMS_LIVEoff, a sent reply or request is recorded but not delivered, and the sender sends it through the firm’s usual channel. Turn delivery on only once the letters and routing are approved. - Changing two things at once. A lexicon change and a knowledge-base change in the same week make it impossible to say which moved the numbers.
What to report at day 30, 60 and 90
Connected and configured
- Sources connected, read-only
- Policy values replaced, controls reviewed, letter wording approved
- Baseline measured
One line live
- Agreement by agent and action
- Classification corrections and lexicon changes
- Deadlines and redress against the baseline
A measured recommendation
- Root causes and QA results across the line
- Second line started
- Validation evidence and a go, fix or stop decision
For the head of complaints
Own the policy values and the controls from day one. The pilot is the time to make sure the product enforces your rules, with your wording, before the volume arrives.
For the service operations lead
Keep the same handlers and leads throughout, answer every prepared decision honestly, and log every correction. Those answers are the measurement.
For the board
Ask for the thresholds before the pilot starts and the comparison with the baseline at the end. A go, fix or stop decision against agreed numbers is the outcome to expect.
Ninety days buys evidence, not autonomy.
Start on one line with your own policy, let agents prepare and people decide every case, and measure against a baseline you took before anything changed.