Datafloq Sponsored September 24, 2024

Collected at: https://datafloq.com/read/accelerating-data-discovery-reuse-portals/

Today, organizations see the benefits of becoming data-driven. They, therefore, aim to harness data to digitalize, become more efficient, reduce costs, increase productivity and drive greater innovation. A key way to achieve this is by implementing a centralized, one-stop-shop data portal using software such as Opendatasoft’s solution. This breaks down silos and makes data assets (prepared information that provides business value) available to all.

Yet, with more and more data being created, collected and shared, it is a challenge for Chief Data Officers (CDOs) to ensure that everyone within the organization can quickly and confidently find and access the right data for their needs in the right formats, without requiring technical skills or support.

Effective data discovery within data portals is essential. It enables everyone to easily find and access the data assets they need within their working lives quickly and with confidence. Data discovery is, therefore, a vital part of data sharing, digitalization, and data democratization – it scales the use of data assets across the organization by making searching for data fast and straightforward.

How does the data discovery process work?

Data discovery is an end-to-end process that covers data flows from collection to making information available to users. First, you need to gather all your data. That means connecting to all storage applications (data warehouses, data lakes, cloud storage), business intelligence tools, business applications, and IoT sensors to create a holistic view of information. Once you have mapped your data, the collection process can be automated to reduce administrative time and maximize speed.

Often, raw datasets may not deliver value on their own. They need to be enriched with information from other sources, such as geographical or reference data. At the same time, data should be standardized to ensure consistency, such as by normalizing fields and formats (such as for dates), while anonymizing personal information. Describe what data assets contain by adding comprehensive metadata to aid discovery and meet data governance standards.

Organizations can then centralize and publish the resulting data assets through their self-service data portal in accessible, usable formats such as visualizations, tables, maps and via APIs. However, simply publishing data is not enough to guarantee usage the final stage of data discovery is ensuring data can be easily found through a powerful, intuitive search interface that understands the meaning and context of queries.

Data discovery best practices

Following these best practices helps improve data discovery and usage:

• Centralizing data assets within a data portal to deliver comprehensive access to data
• Making discovery as seamless and intuitive as possible, just like finding a product on an e-commerce site or via a web search engine
• Building confidence by ensuring every data asset has a full description, including information on the owner and suggestions for reuse
• Focusing on metadata to fully describe and give context to data assets, making them easier to find

The benefits of data discovery for companies

Data discovery increases the value of your data by making it easy to use and benefiting organizations in seven ways:

• Saving time as people can find the right data, the first time, without having to run multiple searches
• Improves productivity and encourages greater data use by all
• Improves the quality and speed of decision-making based on access to insights derived from complete data
• Saves time and resources for the IT/data team as they don’t have to support users or find data for them
• Builds a data culture where everyone uses data, whatever their role
• Turns data into a true business asset
• Delivers ROI on overall investment in data technology

How AI-powered search improves data discovery

Given the increasing volume, variety and velocity of data being created, data discovery has never been more important or more difficult to achieve. Traditional keyword-based search engines make discovering the right data asset difficult, often returning irrelevant or too many results. This puts searchers off and hampers data reuse.

Instead, adopting AI-powered search delivers faster, more accurate and more relevant search results by using vector-based semantic search. This goes beyond literal keyword search matches, providing results based on a deep understanding of the intent and contextual meaning of search terms. For example, a query on the word ‘gasoline’ may return data asset results with terms such as ‘fuel’.

It also enables real-time suggestions to be made as users type in their search query, further reducing the time required to connect users to relevant data. All of this improves user

productivity, reduces the number of queries required to discover relevant data, streamlines administration, accelerates metadata creation, and simplifies content discovery, regardless of the volume of data available on the portal or any technical terms used to describe it. AI search is therefore vital to intelligent data discovery, encouraging usage and unlocking the productivity benefits of data sharing at scale.