Some of the most valuable applications of genAI are designed for the enterprise: companies such as ASAPP, Observe.ai, and Moveworks are supercharging customer service personnel with essential tools, meanwhile, companies like JasperAI, Notion, and Grammarly are boosting enterprise employees' productivity by leveraging LLM-powered natural language interfaces. Bain estimates an average of 20% of time saved for enterprise employees across industries leveraging genAI in the workplace.
Recognizing the untapped potential, by 2026 80% of organizations will experiment with application programming interfaces or deploy generative AI-enabled applications, with some reports estimating this market to reach the size of $110B by 2030.
Global organizations are struggling with new technology adoption, with many companies needing help to scale beyond the isolated pilot phases to deliver real value. Enterprises will increasingly look for the right set of tools to scale their genAI efforts to move from scattered experiments to fully integrated and functioning products.
To create amazing customer experiences, product teams must be equipped with building, compliance, security, monitoring, and optimization tools that are built genAI first.
Happy customers not only purchase more frequently but also spend more and demonstrate greater loyalty. This makes it crucial for successful companies to constantly monitor and improve their Net Promoter Score (NPS) (Bain/ROI Rocket General Retail NPS Study 2023).
As an increasing number of companies turn to large language model-powered products to enhance their customer experience with superior customer service (Bain survey of CX leaders in the NPS Loyalty Forum community, October 2023), the risks of hallucinations and subsequent brand damage increase.
The impact of these incidents on end users' NPS, and more broadly on a company’s reputation, can be potentially catastrophic. Moreover, these instances are more common and recent than one might think. Consider how Air Canada LLM-powered chatbot promised customers a non-existing discount or how delivery firm DPD’s chatbot called its parent company “useless”.
To monitor NPS and other user analytics, product managers have traditionally turned to tools like MixPanel or Pendo. However, the advent of GenAI features has drastically altered the type and volume of data available.
Simple analytics, such as button clicks, activations, and conversions, are no longer sufficient to capture the richness and complexity of natural language text interfaces. This leaves product teams in the dark when deploying GenAI features to end users.
Key questions like, "How does my product drive conversions, foster monetization, and increase NPS?" can't be answered by basic analytics as we know them today. The rise of natural language as a means for users to interact with a product shifts focus from simple buttons and mouse clicks towards a more complex analysis of user behaviors. Much like adding a dimension to a two-dimensional space, the types of data and insights have become harder to interpret but potentially much more insightful.
In completely transforming the user experience, large language models (LLMs) are redefining the rules of the game. At the same time, this poses challenges to product builders, who cannot access the same insights through standard two-dimensional dashboards.
Onyx empowers product leaders with the deeper and more reliable user experience data they need to accelerate the development of GenAI into their products and mitigate the risks that come with it.
There's lots of people developing generative AI applications, right now. There's a lot of tools for developers of generative AI applications but there aren't tools that help a product manager easily understand how users are interacting with these products, to be able to understand how best to prioritize features and design their product strategy to go from version 1.0 to version 2.0.
Arvind Kalidindi, Onyx founder
Onyx is built for product managers who are looking to go from an initial pilot to a more complete version that adds value to users at scale. Such change requires substantial investments that must be backed by purpose-built telemetry of the user.
Onyx is starting by helping product teams automate the collection of telemetry information. This includes monitoring topics and the frequency of their discussion by users, how the product engages with users, user NPS scores and satisfaction levels, as well as warnings about potential LLM hallucinations and unwanted outputs. This data provides product teams with much-needed analytics to inform their product strategy, continuously improve product performance, and maximize user NPS.
Onyx vision doesn’t stop at clickthrough analytics for LLM products:
Our ambition is to enable our customers to move from the simple automated collection of user data to autonomously improving generative AI applications that continuously upgrade themselves to maximize NPS.
Onyx telemetry will be built by leveraging Bain’s strong credentials and frameworks in understanding the drivers of customer advocacy developed through more than 1,600 global engagements focused on improving customer experiences.
As the inventor of NPS and champion of CX, with strong expertise in cost control and reduction problems, Bain is the perfect launchpad for the soon-to-be go-to tool of LLM monitoring.
We couldn’t be more excited to welcome Arvind and the Onyx team to the Founder’s Studio to build the next-gen toolkit for GenAI product managers.
If you want to learn more about Onyx reach out directly to Arvind Kalidindi and visit Onyxai.io.
Onyx provides clarity on customer experiences in any LLM-powered product
Associate Product Manager, Milan
