Thirteen years ago, 14 months after Geoffrey Hinton’s seminal deep learning tech talk, Google Chief Economist Hal Varian said that statisticians would be the sexiest job in the next 10 years. He was right, although we call them data scientists rather than statisticians. A new foundational machine learning model — deep learning — created demand for a new role to train those models.
It’s happening again. Five months ago, like Hinton’s 2007 talk, ChatGPT popularized a new foundational model: Large language models. Maybe we need a new role to train them. Perhaps the sexist job of the next 10 years will be AI bot experience engineer.
Sounds silly, I know. But when desktop computing took off 40 years ago, it created a new type of workforce — knowledge workers with their own computers and their own apps. They couldn’t write code or use mainframes, so a new “user experience” role emerged, led by pioneers such as Don Norman and Ben Shneiderman.
Similarly, bots are a new kind of productivity tool coming to businesses, and they too need a new experience design to improve productivity. I’ll illustrate this with a use case from Tom Davenport’s recent book “Working With AI” about computer-aided language translation of text.
Training a Translation Bot
Google, Amazon, Microsoft, and iFlytek have low-cost or free translators, but there is still demand for human translation for important documents like contracts and books. Today, human translators are often assisted by bot translators.
Early bots were fed large collections of translated documents, learned all at once, and then translated new documents in one pass. Their translations were the same for every user and every document style. Users had to wait for the next release of the bot to get improvements, which wasn’t an ideal experience.
Eventually, AI developers changed how bots are trained to include user feedback per document. A bot translated, a user made corrections or edits to the translated document, and the bot learned from the corrections. But there was still room for improvement.
Today, bots translate and learn from corrections sentence by sentence. The bot and user work through a document together, side by side, in real time. Bots learn individual users’ preferences and style. The bot learning experience went from once a year to once a document to every few seconds, and user interaction frequency increased accordingly:
Many AI/Machine Learning Use Cases and Many Bot Experiences
The same approach could be applied to many AI and machine learning use cases. Today, retail sales associates from home improvement stores to luxury apparel boutiques carry tablets that serve AI bot recommendations for products and advice. Many, like early document translators, learn in batch and aren’t personalized to individual sales associates or customers.
A bot experience engineer could change that, enabling them to learn from the day of the week, time of the day (for example, midday and mid-week home improvement customers are likely to be contractors), physical location within the store (kitchen vs. bathroom, handbags vs. dresses), sales quota of the associate, percentage of quota achieved to date, price sensitivity of the customer, and preferred color palette.
For next best action messages sent by financial advisors to their customers, bots that assist with what to say and how to say it could observe when an advisor overrides the bots suggestions, what message is actually sent, and learn individual advisor preferences. They could watch individual customer behavior to learn, for example, do they prefer stocks, mutual funds, or ETFs? Is a CNBC message tone more effective for Northern California tech workers, or a tone like The Motley Fool?
Predictive and prescriptive maintenance bots could observe which maintenance tasks are actually performed in response to its recommendations. Additionally, they could be fed the situational context such as factory productivity, order backlog, monthly maintenance budget, and number of repair technicians on duty. Based on all that, they could learn risk tolerances for different product lines, factories, and maintenance managers.
Evolution of Bot Experience
In the near-term, most AI use cases will involve user and bot interaction that evolves through phases. Consider the financial advisor example:
1. Assisted autonomy: Useful macros are provided to users, such as a button in their email program to pop up a customer’s profile and generate a draft message.
2. Partially automated: The bot learns from the user’s edits.
3. Highly automated: Users give the bot 50 people to contact today. The bot sends emails to those it’s confident that it has good messages for, writes drafts for the rest, and learns from the user’s edits.
4. Full autonomy: Users give the bot 50 people to contact with the goal of maximizing the number of scheduled phone calls over the next two weeks. The bot sends all 50 emails, occasionally does some A/B testing, and monitors the user’s calendar to see how many meetings get scheduled.
*Adapted from a recent presentation by Dr. John R. Goulding from Raytheon Technologies.
Skills Needed for Bot Experience
Bot experience engineers draw skills from a number of other fields.
Interaction design, information architecture, and usability evaluation are from user experience: Interaction design so that users can easily send data to bots, information architecture so that they’re clear on what they’re sending, and usability evaluation to determine whether the extra data is worth it.
From data science, they take design of experiments and bias detection: Design of experiments to estimate the incremental value of user feedback and additional bot data, and bias detection to flag when bots become too personal and deviate from their core role.
Lastly, from MLOps they borrow continuous testing and continuous monitoring.
The bottom line: AI bots are a new tool coming to businesses, and there are big productivity gains to be made by improving how they learn from and interact with workers. But bots, like personal computer users a generation ago, might need their own experience engineers to realize and verify those gains.