Data Mesh Demo Recap: Breaking Down Data Silos with Internal Marketplace

Data mesh has gained traction in recent years, driven by its promise of reducing the burden on centralized data teams by removing bottlenecks, scaling data management and accelerating time to value from data. For those who haven't yet jumped on the bandwagon, data mesh is a sociotechnical approach that emphasizes decentralized domain ownership of data and the treatment of data as a product, enabled by a self-service data infrastructure. It aims to address the challenges of scaling data management, access and data sharing in the face of continuously increasing data sources and consumers. 

Data mesh expert Matthias Nicola has been guiding numerous Snowflake customers through the exploration, building and operation of data mesh architectures. These endeavors gave him a unique perspective on the compelling nature of data mesh as both a technical architecture and an organizational approach. In this blog post, we will recap highlights from our recent data mesh demo webinar, where Matthias discussed data mesh’s potential to transform businesses and dove into Snowflake solutions, including Internal Marketplace, to show how they can empower your organization to share documented and governed data products across business units. While data mesh is also subject to important organizational considerations, this webinar focused on data platform capabilities for the management of data products.

The data dilemma: Why traditional approaches fall short 

Traditional centralized data management teams often cannot keep up with the never-ending backlog of data requests. This leads to bottlenecks, hindering a company's ability to scale and adapt to the evolving data needs of its business. Additionally, central teams struggle with the lack of domain knowledge (in marketing, product, sales and other business units), causing delays in delivering the right data to the right data consumers. With each of these roadblocks slowing teams down, data collaboration is negatively impacted, resulting in longer time to value and reduced data agility. 

Snowflake solutions: Key capabilities needed for data mesh success 

To overcome these challenges, many forward-thinking companies are turning to a data mesh approach to reduce the organizational bottlenecks and limitations of traditional centralized data management. Data mesh architecture offers the potential for a more agile and scalable approach to data management, providing teams with greater autonomy and seamless data sharing. 

The four key principles of a data mesh are federated governance, domain-centric ownership and architecture, data as a product and self-service infrastructure. 

The Snowflake platform offers solutions and capabilities that align with data mesh principles, including:   

• Federated governance: Empower organizations with robust governance features, available through Snowflake Horizon Catalog, to control access, monitor usage and enforce security policies consistently across diverse domains and environments.  

• Domain-centric ownership: Snowflake's distributed, multi-cloud architecture supports domain-centric ownership by providing separate environments for different domains. This fosters independence while enabling seamless cross-domain data sharing and collaboration. 

• Data as a product: Enable domains to package data objects, metadata and governance policies as data products that can be easily published and discovered in Snowflake Internal Marketplace, helping ensure that trusted, high-quality data assets are readily accessible. Data products in Snowflake can also include data assets that are stored and managed outside of Snowflake.

• Self-service infrastructure: Snowflake's ease of use and interoperability with external sources simplify data access and empower teams to operate independently, reducing reliance on central teams and fostering a true self-service data infrastructure.

These capabilities provide the essential foundation for building and operating a successful data mesh architecture. 

By streamlining data sharing across diverse domains, many organizations are effectively breaking down traditional data silos and promoting a more integrated data ecosystem. Snowflake helps organizations such as global travel-tech leader FlixBus fully realize the transformative benefits of data mesh. FlixBus implemented a data mesh architecture and leveraged Snowflake Internal Marketplace and listings, which is a method to package up information with metadata, description and more as a data product. From there, FlixBus was able to share and mobilize its data to improve fleet operations, forecast supply and demand, measure business performance and provide self-service analytics. 

Unlocking data mesh success with Snowflake: Your top questions answered 

Data mesh can seem complex at first, but it doesn't have to be. During this live webinar, attendees expressed a strong interest in diving deeper into specific aspects of data mesh implementations with Snowflake. To provide clarity and guidance, we've compiled the questions that generated the most interest: 

Q: How do I choose between single versus multiple Snowflake accounts in the data mesh architecture?

The decision to use a single account or multiple Snowflake accounts really depends on your organization's specific needs, challenges or legal requirements. Some considerations to help you determine a single- or multiple-account approach include: 

• Whether or not you require maximum isolation between business domains, for legal or security reasons, with separate security configurations and user management per domain.

• Regions your organization operates in: Are you internationally operating with multiple businesses (teams residing across regions such as Asia, U.S. or Europe)? Or does your organization operate in a single geographic region?  

• Business units working on different cloud platforms (AWS, Google Cloud, Azure) in different regions and considering any localization requirements. Additionally, do you have security or legal requirements that span clouds and regions? 

Things such as measuring and controlling costs and allowing teams to configure security should also be considered. Some organizations choose to use a single account because it's simpler and more straightforward to manage. However, some choose multiple accounts to provide stronger privacy, accommodation for a broader set of regions and clouds and distinct security needs. 

Whatever your choice, one account or multiple, know that you are not locked in, because the Snowflake Internal Marketplace works seamlessly across one or more accounts and evolving account topologies, as shown in the image below.

Check out this article to dive deeper!   

Q: How can I set up domain profiles?

As an organization admin, you can configure domain profiles using Organization Profiles in the Internal Marketplace. Organization profiles are regular Snowflake objects that you can create, alter, drop, show and use to list information about the origin and ownership of data. Creating or altering a profile takes a YAML file as input that contains the domain name, a description and various domain attributes such as contact information or a default approver for requests to data products. This helps provide clear data ownership and data provenance to drive trust, data quality and structure, while strengthening governance across an organization. 

Q: What object types can be in a data product in Snowflake?

A data product can contain one or more objects of various kinds including: 

• Databases

• Schemas

• Tables

• Dynamic tables

• External tables

• Semi-structured data such as JSON and XML

• Files and unstructured documents, even if stored in storage buckets outside of Snowflake

• Apache Iceberg™ tables, even if stored outside of Snowflake and managed by a catalog such as AWS Glue or OneLake

• Views

• Materialized views

• Secure user-defined functions (UDFs)

• Snowflake Native Apps

• ML models

• Semantic views (private preview)

• On-premises data, if it is stored as files in an S3-compatible storage system

Q: How can I create data products based on Iceberg and Delta Lake tables?

If you are managing data outside of Snowflake, in Apache Iceberg, for example, you can still make that Iceberg table visible in Snowflake and include it as part of a data product in the Internal Marketplace. This allows that same data product to be interoperable with other compute engines that can access the same data. 

Q: “Chat with your data product” is a capability currently in private preview. What is it, and how does it work?

Snowflake introduces an innovative feature that allows data consumers to use natural language to interact with a data product. This requires the data product owner to include a semantic view in the listing, which describes the data’s content and structure, such as relationships, metrics, dimensions, and optional comments and synonyms. 

The semantic view is then used by Snowflake Cortex Analyst, a fully-managed LLM-powered feature that helps answer business questions based on your structured data in Snowflake. A data consumer can then interrogate the data product in powerful ways. Queries in natural language are automatically converted to SQL and executed to generate responses. The generated SQL can be reviewed and manipulated if desired. The data consumer can also ask open-ended questions such as “What can this data product help me with?” or “What are the limitations of this data product?”

Data mesh webinar: Five takeaways 

Here are the five top takeaways and key insights from our data mesh webinar: 

• Data mesh aims to address the limitations of traditional centralized data management by decentralizing data ownership and empowering business domains to manage their data. The evaluation of data mesh requires careful consideration to determine whether the sociotechnical implementation and capabilities align with your business goals. 

• Snowflake provides a platform that supports data mesh principles by enabling scalable and governed data ownership, facilitating easy sharing of data products and reducing complexity and cost.   

• Internal Marketplace is a key feature within Snowflake that allows domains to publish, discover, share and use governed data products within their company.   

• Snowflake enables autonomy for domain teams by providing separate environments with their own storage and compute resources, while also facilitating collaboration and data sharing across domains.   

• Snowflake helps reduce the complexity and cost of building and operating a data product platform by offering a pre-integrated, fully managed service with a wide range of features.

• Snowflake integrates with a broad ecosystem of tools and services, which enables companies to create custom architectures for their needs.    

By empowering business domains, facilitating data sharing and reducing complexity, a data mesh architecture can help organizations unlock the full potential of their data and drive better business outcomes. However, building and operating a data mesh requires a combination of vision, alignment and technical capabilities. You will need to tap into a variety of stakeholders, tools and solutions to be successful. The Snowflake platform provides a powerful and comprehensive foundation of solutions for organizations that are on this data mesh journey.