What stood out about Thushara was her ability to bring structure and clarity to ambiguous product problems. She did not just focus on making screens look good; she consistently tried to understand the user, the business context, and the edge cases before arriving at a solution. That mindset made her design work practical, thoughtful, and much easier for engineering to execute. She was especially strong at reducing friction in workflows that could otherwise feel overwhelming to users.
Overview
For two-plus years I have been Gistly's only designer, shaping most of the product with the founders and the engineering team (dashboards and reporting, the call-review workspace, objects and records, onboarding, and the scoring system) across 20+ end-to-end flows. This case study zooms in on the single hardest one: making the AI's quality scores something non-engineers can actually trust, verifying them when they're right and tuning the rubric when they're not.
Context
The product uses AI to score recorded sales and support calls against quality rubrics. A rubric, a template, is a set of categories and parameters, where each parameter is one question the AI answers about a call: "Did the agent greet the customer professionally?" The people who build and own these rubrics are QA analysts and team leads, not engineers.
For the AI's scores to be trusted, those users need to do two things: verify the AI is grading correctly, and tune the rubric when it is not. Designing that testing and tuning experience was my brief.
The problem
Authoring a rubric and testing it had become two disconnected surfaces. In the gap between them, three things were broken.
- 01Forms that driftA parameter was edited in the builder and again on a separate testing page, two forms of the same thing, quietly diverging.
- 02Weights you can't compareScoring weights lived inside a single parameter's editor, invisible to the siblings they're weighed against.
- 03No ground truthThere was no honest, affordable way to fix the correct answer every test has to measure against.
And underneath all three sat a number. To verify the AI at any real scale, a human has to confirm the correct answer for each question on each call. With a 100-call sample and 30–50 parameters, that is thousands of judgments before a single meaningful test can run. If the design did not make that cheap, no one would do it — and the feature would be theatre.
Constraints
How I approached it
I studied how developer eval tools structure the tweak, run, grade loop, then reframed it for a non-technical audience. Earlier user interviews had shown that users carry a high cognitive load and vary in how technical they are. AI-based design reviews and audits landed with their own recommendations; I treated them as input, not instruction, adopting what held and reconciling what did not against the product's reality. Their draft versus published model, for instance, did not match how calls are actually ingested. Throughout, I surfaced my assumptions and the genuinely open questions rather than designing around them.
The key decisions
Five decisions. Three close the breakdowns above; the others shape the loop itself.
Test and Tune repeat until the scores hold. Save publishes a new version, live going forward.
01
Author first, test later
Users author a rubric, run it on real calls, and only then tune when results look wrong. So a new template opens in a fast side panel for quick authoring; once it has run, each parameter gets its own page where testing lives. Testing actions do not appear until there is something to test, which keeps a first-timer's path simple. The tradeoff: a quick edit on a live parameter now costs a navigation. I judged the consistency worth it.
Figure: Template library, start from scratch, or from a prebuilt Support, Sales, or Collections rubric.
02
Closes:No ground truth
Use existing workflows instead of creating new ones
The user can correct the answers from the review flow itself, and it gets seeded to the verified set automatically. Instead of labelling from a blank slate, the system auto-seeds a small set of recently evaluated calls, pre-fills the AI's answers, and asks the reviewer only to confirm or correct them. Verification becomes a quick pass, not thousands of cold judgments: the difference between the feature being usable and abandoned.
Figure: Review tab, the AI's answer pre-filled per call, with one-click Confirm or Edit against real transcripts and timestamps.
03
Closes:Forms that drift
One editor, two contexts
Builder and testing use the same parameter editor in two frames: a panel while authoring, a page while tuning, backed by a single working draft. This kills the two-forms-that-drift problem and removes the risk of one surface overwriting the other. What is shared is the form and its state; the testing apparatus wraps around it where relevant.
04
Closes:Weights you can't compare
Set weights in one view
Category and parameter weights are relative across siblings, so they belong in a whole-template view, not inside one parameter. Pulling them out of the editor also made the editor identical in both contexts, which is what allowed it to be shared at all.
Figure: The whole-template view, category and parameter weights read as shares of the score, set where they can be compared.
05
Extends: Reusable for any check
A capability, not a place
Rather than a separate section, each entity hosts testing on its own page under its own URL. That kept the information architecture honest, fixed the breadcrumb, and let the same loop generalise to other check types without reinventing it.
Figure: Test tab, the tuned parameter re-run against the verified calls, reporting pass/fail against ground truth.
Handling ambiguity
A few decisions hinged on engineering realities I could not assume: whether scoring is versioned, whether test calls exist before a template goes live. Rather than guess, I documented each as an explicit open question with my recommendation and the dependency owner, so the team could resolve them deliberately instead of discovering them in build.
Outcome
The redesigned Review, Test and Versions flow lowers the smallest testable unit from an external, undefined QA process requiring thousands of judgments to an in-product loop where one verified call or record can create ground truth and run an evaluation.
The flow was aligned with the founders and the engineering team and adopted as the build direction: a single coherent loop, author, run, verify, tune, save, that a non-engineer can move through, with the ground-truth burden designed down to confirmations rather than cold labelling.
What I took from it
The hardest part was not the screens. It was finding the one number, thousands of judgments, that decided whether the feature could work at all, and designing the whole flow around defeating it. In B2B, the best decisions usually come from the second-order consequence of a workflow, not the surface. It also sharpened how I take feedback: keep what holds, reconcile what does not against real constraints, and surface the open questions instead of papering over them.