What makes an AI assistant useful
Research behind version 1 of LLM Suite at JPMorgan Chase: an assistant for developers, and a sales-enablement tool for the firm's salespeople. Prototype tests, a blue-sky focus group and feasibility triage produced a realistic roadmap and a shared bar for "useful."
At a glance
- Role
- Vice President, User Researcher, JPMorgan Chase; led the developer-side research for version 1
- Organization
- JPMorgan Chase, 2023–2024. LLM Suite, version 1: the firm's generative-AI platform, prototyped first for the internal engineering platform's developers and for sales enablement.
- Methods
- Prototype usability testing, blue-sky focus group, affinity diagramming, feasibility triage with engineering
- Outcome
- A prioritized, feasibility-triaged feature framework and a shared definition of a useful assistant
Summary
LLM Suite, the firm's generative-AI platform, began as an assistant with two audiences: developers losing time across documentation wikis, ticketing, Q&A forums and the service desk, and salespeople who needed pitch decks and market background on clients faster than they could assemble them. The open question for version 1 was scope: what should it do, what could it realistically do, and how would the team tell the difference? I tested the prototype's conversation tree, ran a blue-sky focus group with an AI/ML specialist in the room, and put every idea through feasibility triage on security, technical feasibility and effort. The team came away with a prioritized roadmap across management, code and documentation, and a definition of "useful" to hold the product to. The research I led was with developers; the bar was written to hold for both audiences.
What the team got
- A feature framework, triaged
- Candidate features across three areas (management, code, documentation), each scored on security, feasibility and effort, so aspiration and what could be built were visibly separated.
- Prototype findings
- From usability tests of the conversation tree: the points where developers expected context the assistant didn't have, where they gave up, and what an answer had to show before they would trust it.
- A definition of "useful"
- Conversational, aware of context, able to resolve the problem, and citing its sources when it can't. A bar the whole team could hold the product to.
The work
The challenge
For developers, access requests and build or pipeline errors were the worst of the time lost across fragmented channels. At the same time, the firm's salespeople were assembling pitch decks and client market background by hand from scattered sources. Version 1 of LLM Suite was being prototyped to serve both, and it needed a scope grounded in research and in what the technology could deliver.
Test the conversation, not just the screens
Prototype usability testing of the assistant's conversation tree showed where developers expected it to understand context, where they abandoned it, and what they needed to see before trusting an answer.
Imagine widely, then triage hard
A blue-sky focus group brought together developers, engineers and an AI/ML specialist. Ideation fed affinity diagramming, and every cluster then went through feasibility triage: security, technical feasibility and effort.
Ground ideas in what models can do
Feasibility was tied to real model capabilities and to real limits: data and training needs, hallucination, and the privacy considerations of running models in-house at a bank.
What came out of it
A prioritized feature framework across three areas, each item triaged by security, feasibility and effort:
- Management: access requests, environment and resource questions.
- Code: build and pipeline errors, and help in the context of the developer's own work.
- Documentation: finding the right answer across scattered sources.
And a definition of a useful assistant the whole team could hold the product to:
Conversational, aware of context, able to resolve the problem, and citing its sources when it can't.
Impact
Product and engineering got an evidence-based roadmap and a shared definition of "useful." The research separated aspiration from feasibility, so the team could commit to what the assistant could do well instead of promising everything. LLM Suite went on to become the firm's generative-AI platform; in later work I contributed research on how employees adopt and trust it.
What carries over
The same bar applies to any AI that answers questions or drafts replies for a team: does it resolve the task, does it know the context, and does it show where its answer came from? It is the bar I now use when evaluating an AI feature or scoring a candidate use for automation.
And the same method applies: let people imagine the assistant they want, then put someone in the room who knows what the model can do, and triage with both in view.