Transforming Data Discovery with a Scalable Solution
Year: 2024
Role: UX Design Lead
The Challenge
Imagine needing to find a needle in a haystack, only to realize the haystack is scattered across two different barns. That was the reality for the pharmaceutical R&D team relying on their current data catalog platform.
Over the years, it had become disconnected, with business and technical data siloed in separate systems. Searching for data felt like an endless scavenger hunt, and there was no standardized way to manage or govern the catalog. Frustrated data users were resorting to workarounds, while overburdened data stewards struggled to keep up. Progress slowed, efficiency plummeted, and the need for a revamp was impossible to ignore. The client needed a platform that could scale with their growth, simplify data discovery, and bridge the gap between their disconnected systems. It was time for a change.
The Journey
As the UX Design Lead, I took the reins to lead this transformation. My role wasn’t just about designing—it was about orchestrating the entire process. I managed stakeholders, guided research activities, and mentored a junior designer, ensuring every detail was aligned with our vision.
Digging Deep Into User Needs
To start, I worked closely with subject matter experts and proxy users to understand the pain points of three key user groups: Data Owners, Data Stewards, and Data Consumers. Each group had distinct needs that informed how I approached the redesign.
Data Owners wanted simplicity—they needed to search for datasets quickly and get clear, usable results without wading through irrelevant noise. Data Stewards, on the other hand, needed control. They wanted easy tools to curate and refine metadata over time to improve search accuracy, while also being able to track and approve or deny metadata update requests to ensure the catalog’s integrity. Lastly, Data Consumers had a unique role: they primarily searched across all datasets within their organization, assessed whether the data was useful for their specific use cases by viewing details and metadata about a single asset, and then submitted access requests to Data Owners.
Additionally, we uncovered that Data Owners required a way to efficiently manage and govern access. They needed tools to review and approve or deny access requests submitted by Data Consumers, ensuring proper permissions and compliance were maintained across the system.
This feedback shaped the redesign. For Data Users, I prioritized simplicity, ensuring the system delivered the most relevant results first, accompanied by clear summaries that helped users quickly assess the data's relevance. For Data Stewards, I focused on empowerment, equipping them with tools to refine the system and manage metadata updates seamlessly. For Data Consumers, the goal was clarity—providing intuitive search capabilities, clear dataset summaries, and a streamlined process for submitting access requests. By addressing these varied needs, the redesign created a system that allowed all three groups to achieve their goals efficiently without overlapping or conflicting workflows.
Armed with our research, I led stakeholder sessions to align on a shared vision for the platform. These discussions helped prioritize features for the minimum viable product (MVP). We focused on three key functionalities to solve the most pressing challenges:
AI-Assisted Data Enrichment: Generative AI would seamlessly integrate business and technical data into a unified catalog, eliminating the manual effort of curation.
Context-Aware Search: A natural language interface would make it easy for users to ask questions and get personalized, relevant results.
Expert Tuning Mechanisms: Data stewards would have tools to annotate and refine search results, ensuring accuracy and continuous improvement over time.
Reimagining Established Ideas
When I set out to start redesigning the data catalog, I didn’t want to simply tweak what existed—I wanted to reimagine it entirely. Traditional data catalogs rely on dropdowns, faceted filters, and static search bars, but I knew there had to be a better way. What if users could find and refine their datasets simply by having a conversation? Enter the idea of an AI-powered chatbot.
The vision was ambitious: a conversational interface where users could ask questions, follow up with clarifications, and refine results in real-time. Instead of overwhelming users with thousands of datasets at once, the system would surface the top results and let them expand or narrow their search as needed. Metadata summaries would make it easy to scan for relevance, while detailed pages offered deeper insights into access, ownership, and other key details.
I wanted to eliminate the frustration of rigid, outdated workflows. No clunky filters or disruptive navigation—just seamless, natural interactions. Whether users needed to refine a query, pivot to a new search, drill down into dataset specifics, or manage requests, the system would make it feel effortless.
As I sketched out user flows, the potential became clear. Users could start with a broad query, refine results conversationally, and dive into details—all within the same interface. This wasn’t just about modernizing the catalog; it was about creating a smarter, more intuitive way for people to find and interact with data. It was bold, but it was exactly the kind of shake-up that was needed.
We brought our big ideas to life in Mural, collaborating with stakeholders to sketch out how an AI-powered chatbot could refine search results in real time.
Early low-fidelity wireframes bring the concept to life, showing how users can effortlessly refine their search results through an ongoing conversation. It’s search, but smarter—and way more intuitive.
Big Ideas Meet Even Bigger Roadblocks, a Detour, and a Path Forward
When we first presented our bold, chatbot-powered concepts to the client, they were thrilled with the vision. It was exciting to see their enthusiasm for our groundbreaking ideas. But as the development team conducted a parallel tech assessment, the reality set in: their current technology stack couldn’t support the robust AI integration we had envisioned. It was a tough pill to swallow, but I knew we had to adapt, and fast.
We picked up the pieces and kept moving. I pivoted the design to reflect a more traditional data catalog, but I wasn’t ready to abandon the chatbot entirely. Instead, I found ways to integrate a lighter version of it into the system. While we relied on traditional search and refine features like faceted filters, I focused on making them as intuitive and user-friendly as possible. The chatbot, even in its scaled-back form, became a key feature—a resource that could summarize search results, answer questions about data assets, and help users request access to specific datasets without ever having to manually fill out a time-consuming form.
Beyond the chatbot, we retained backend AI integration that worked behind the scenes to generate intelligent summaries of search results. This integration added predictive capabilities to surface the most relevant results, corrected spelling mistakes, and guided users toward more effective queries. These enhancements provided a smoother, smarter search experience without overwhelming the existing tech infrastructure.
For data stewards, I introduced affordances directly within the context of the data catalog screens, enabling them to update metadata without leaving the page. This integration streamlined their workflows, making it easier for them to refine and improve the data catalog over time. Combined with the chatbot and backend AI intelligence, these features enhanced usability by allowing stewards to improve search results in real-time and ensure metadata accuracy.
We hit reset and went back to basics, using low-fidelity sketches to map out ideas and nail down requirements before diving into high-fidelity designs.
Drawing inspiration from modern knowledge management platforms like Atlan, I ensured the design reflected best practices in active data catalog capabilities. Each design decision was grounded in research, with a focus on usability and scalability. While it wasn’t the fully AI-driven solution we had initially envisioned, the pivot allowed us to create a practical and impactful system that worked within the client’s constraints.
Outcome
The pivot wasn’t just a compromise—it became an opportunity to rethink the entire experience and deliver a data catalog that truly worked for its users. While the fully AI-driven chatbot remained a long-term vision, the final solution struck a thoughtful balance between innovation and practicality.
Key outcomes included:
Simplified Data Discovery
The redesigned system made finding data easier and faster. With improved faceted filters and a lighter chatbot integration, users could quickly search, refine, and access critical business data without frustration.
Empowered Data Stewards and Owners
Data stewards and owners gained new tools to monitor and process approval requests for data access, asset usage, and metadata revisions—all directly within the platform. These features streamlined governance, minimized bottlenecks, and improved data quality over time.
Time and Effort Saved
AI-assisted data enrichment reduced manual workloads while maintaining accuracy. Metadata summaries, streamlined workflows, and seamless asset requests helped users and stewards work more efficiently and effectively.
A Scalable Foundation for the Future
By blending user-centered design with forward-thinking technology, the catalog became a scalable, future-ready platform. It bridged immediate needs with long-term growth, setting the client up for success in an evolving data landscape.
