Customer Overview
Lovehoney Group (LHG), a global leader in sexual wellness, is renowned for its innovative products and commitment to customer satisfaction. To enhance customer experience, LHG set out to integrate Generative AI (GenAI) into their customer service operations.
The Challenge
LHG aimed to develop an AI-driven chatbot capable of providing useful, accurate and empathetic responses to customer inquiries. When needed, the chatbot would use Retrieval-Augmented Generation (RAG), with a vector database at its core, to pull relevant documents from their data sources and address customer questions effectively. The challenges included verifying that the chatbot fully understands the user desires, retrieves the relevant information, and follows LHG’s guidelines and brand values in its responses.
The Solution
LHG initiated the project using Deepchecks’ evaluation environment, allowing them to experiment, measure, and refine their AI models prior to production. They customized the auto-annotation YAML to adapt what constituted good or bad interactions to their needs, focusing on properties like grounded in context, retrieval-quality properties, fluency and coherence.
Both technical users (engineers, data scientists) and non-technical stakeholders (e.g., product managers) engaged with Deepchecks to monitor and improve the chatbot’s quality. For example, verifying that it doesn’t recommend specific products where it shouldn’t, or that it doesn’t add incorrect details in inquiries about products.
LHG utilized Deepchecks’ evaluation environment for continuous improvement, running multiple versions of their chatbot to test different prompts, models, and configurations. The multi-version comparison feature allowed them to analyze performance across versions, facilitating informed decisions on enhancements. They also leveraged Deepchecks’ root-cause analysis tools, including score breakdown, actionable insights, and topic modeling. The topics were extracted using Claude-Sonnet-3.5 via Amazon Bedrock, offering deeper visibility into user interactions. Additionally, the built-in properties they relied on—such as Instruction Fulfillment—were powered by models through Bedrock, ensuring high-fidelity evaluations.
In production, LHG used Deepchecks to monitor performance and detect degradations. Thus ensuring the chatbot maintained high performance standards over time, and successfully deal with drifts.
The Results
Within a day, LHG commenced the evaluation process, progressing swiftly to assess relevant pipeline components using Deepchecks’ built-in and LLM-based properties. Remarkably, they transitioned to production within weeks, a testament to the efficiency of the collaboration.
The outcomes were significant:
- Deployment to Production: LHG’s team was able to confidently deploy to production, thanks to the full pipeline visibility, proactive monitoring and swift identification of issues.
- Continuous Improvement: Utilizing the evaluation environment, LHG regularly refined their chatbot, ensuring it met evolving customer needs. Throughout the project, LHG was able to improve each pipeline version iteration time by approximately 5X, significantly accelerating their model development and deployment cycles.
- Visibility to User Interactions: Gaining comprehensive insights about the chatbot interactions, LHG were able to understand user needs, topics that they inquire about, and problems they encounter. These enabled prioritizing and navigating improvements and features in LHG’s products and chatbot.
Reflecting on the collaboration, an LHG team member noted, “Working with Deepchecks felt like an extension of our own team. The support and tools provided were instrumental in bringing our AI chatbot to life.”
Through this partnership, Lovehoney Group exemplifies a pioneering approach to integrating reliable AI solutions into customer service, reinforcing their commitment to innovation and customer satisfaction.
