Enhancing Moovit’s Internal GenAI Pipeline with Deepchecks

Moovit is one of the world’s leading forces in urban mobility. Serving over 1.5 billion users across more than 3,500 cities in 112 countries, the company has transformed how people navigate public transportation. By combining official transit data, real-time crowd insights, and intelligent algorithms, Moovit empowers commuters to make smarter, faster, and more reliable travel decisions every day.

Behind the familiar journey-planning app lies a sophisticated data ecosystem – one that continuously interprets massive streams of transportation information from thousands of sources. Maintaining this level of precision at global scale requires constant innovation. That’s why Moovit has been at the forefront of adopting Generative AI to ensure every piece of information – from a train delay to a bus route update – is accurate, timely, and trustworthy.

As part of its ongoing mission to make public transit effortless, Moovit has been building an AI-powered assistant that lets users ask natural-language questions about transportation – things like “What’s the fastest way to get to the airport?” or “Is the 18 bus delayed right now?”

But while the concept is simple, the execution is anything but. Each query carries layers of complexity: users often type incomplete or ambiguous questions, use slang, or mix multiple intentions in one message. The system must first understand the user’s true intent, then combine that with contextual data – such as the user’s location, time, and local transit conditions – to retrieve the most relevant information.

Finally, the app must present that information in a way that’s clear, accurate, and trustworthy – because even small mistakes can cause confusion, missed connections, or loss of user trust.

Underneath these user-facing challenges lie deep technical ones:

• How to reliably extract and classify user intent.
• How to design and evaluate prompts and models that interpret queries correctly.
• How to ensure the retrieval and response-generation stages maintain high factual accuracy and contextual completeness.

For Moovit, solving these challenges wasn’t just about making an AI chatbot – it was about ensuring that every AI decision directly impacts the real-world mobility of millions of users.

Recognizing the importance of a strong trust layer, Moovit partnered with Deepchecks to bring structure, visibility, and confidence to their AI development process. Together, we built a dedicated evaluation and monitoring application on the Deepchecks platform – a single environment where Moovit could both validate new versions of their AI pipeline and continuously monitor performance in production.

The collaboration began with seamless onboarding. With guidance from Deepchecks’ solution engineers, Moovit quickly uploaded their data – user queries as inputs, model-generated responses as outputs, along with prompts and additional context such as intent analysis results.

Once the data was in place, Moovit started defining and testing quality properties to assess every part of their pipeline. They leveraged Deepchecks’ research-backed built-in properties, such as Completeness, and combined them with custom prompt-based properties like Output Faithfulness, which verifies that each claim in the model output is supported by the provided context.

Moovit also validated the pipeline across multiple languages to match their global footprint, using user-provided metrics to group, filter, and analyze interactions by intent, language, and other dimensions. To accelerate iteration, they used small but representative datasets to test and compare many pipeline versions quickly – then promoted stable versions to production.

Finally, Moovit connected Deepchecks to their operational stack via our Datadog integration: evaluation metrics, pre-built dashboards, and alerting rules were routed in real time to Moovit’s Datadog workspace so production versions immediately had the monitors and alerts they needed. Weak or anomalous samples could be pulled back into the evaluation dataset with a single click, closing the loop for the next experiment.

The collaboration yielded significant results:​

• Enhanced visibility (multi-language): Deepchecks provided full transparency across the pipeline, including evaluations in multiple languages to reflect Moovit’s global markets – enabling targeted analysis by intent and locale.
• Faster experiment velocity: Moovit increased the number of pipeline versions validated per month by 5X, enabling quicker A/B-style experiments and faster convergence to better-performing pipelines.
• Measurable quality lift: Guided by Deepchecks’ properties and iterative testing, Moovit improved response accuracy for complex queries by ~25%, translating into fewer user corrections and higher end-user trust.
• Production reliability (real-time monitoring): Deepchecks’ Datadog integration delivered evaluation metrics, pre-built dashboards, and alerts to Moovit in real time – reducing time-to-detect and remediation to minutes and ensuring production issues are surfaced and acted upon immediately.
• Increased Confidence: With real-time monitoring and comprehensive evaluation, Moovit can deploy updates confidently. As Vadik Vasin, Moovit’s software architect, noted – “Deepchecks turned our GenAI development from guesswork into a data-driven process. We move faster, detect issues earlier, and trust every release”.

Through this partnership, Moovit solidifies its position as a global leader in urban mobility, setting the standard for leveraging advanced AI solutions to deliver accurate, efficient, and user-focused transportation experiences.