Snowflake: A Next-Gen Managed Service
Posted by PDI Marketing Team
Posted by PDI Marketing Team

Published: July 27, 2021

After a decade of increasing comfort using Integration Platform as a Service (iPaaS), it makes sense that a Data Warehouse Platform as a Service (dwPaaS) such as Snowflake should come along. Next, perhaps we'll see Business Intelligence Platform as a Service (BIPaaS)?

Snowflake, as a platform, is a purpose-built data warehouse-in-the-cloud architecture with the scalability, security, performance, concurrency, and affordability required for large-scale systems. It is similar to Amazon Redshift and Google BigQuery as it is elastic in its expandability, and expense-conscious in that it is a pay-for-what-you-use model. Where Snowflake goes further is that it is more of a data warehouse, purpose-built Managed Service.

Perhaps I am overstating slightly, but it depends on your definition of the service.

Think of the impact that came with Integration Platform as a Service. Compared to on-premise enterprise software data integration platforms, iPaaS was engineered to require no training for the user. Ux engineers had to radically rethink what data integration was about. Are there common patterns and thought processes for data integration architects and developers? How can we automate those common streams so people can achieve their results more quickly? iPaaS companies used the 80/20 rule to highlight the 20% of a data integration tool that you use 80% of the time. (Or is it more like 10% that you use 90% of the time?)

This simplification results in superior power for the company utilizing the product. So much more can be done, by more people, faster, yet with BETTER OVERSIGHT!  The better oversight piece comes because improved governance visualizations have been built into the different iPaaS tools to allow a view of what’s going on.

Now Snowflake has applied this same re-factoring to the Data Warehouse space. Will this re-thinking result in a similar win/win for the data warehouse storage/compute/administration platform? What is the 20% that is used 80% of the time? How can these actions be simplified and automated, giving the designers, administrators, and operators more power, with innovation through simplification?

iPaaS resulted in the democratization of integration services, allowing citizen integrators a tool to use for themselves. Snowflake/dwPaaS might result in a similar benefit, or at least the ability for smaller organizations to solve some data problems for themselves, as well as begin to address the originating cause for the data lake movement–agility.

As has been said many times before, the issue with data warehousing is the time it takes to incorporate new data into the data warehouse. For many reasons, "gates" have been created in front of the data warehouse to ensure that designs and data conform and that the system grows in a managed fashion. Data lakes are cropping up to allow a fast-path to make data available to data scientists and analysts more quickly.

Snowflake/dwPaaS could result in a corresponding democratization of data warehouse capabilities, giving teams a cost-effective way to experiment and grow the data warehouse by removing some of the "gates" that prevent the current agility. Obviously, chaos will result if corresponding energy isn’t applied to the "system" in order to reduce entropy, as they say in the thermodynamic world. But theoretically, teams could benefit from this new approach.

Remember, a Data Lake doesn’t have a hard and fast definition. It is a construct that is morphing as fast as technology changes, but foundationally Data Lakes exist to make more data available to more people faster. HOW a “data lake” gets constructed is an exercise left to each organization according to their situation, budget, skills, etc.

Currently, it seems the weak link with Snowflake, BigQuery, and Redshift is data integration. They all rely heavily on scripting and file-based staging (usually), two old-school approaches that add complexity and brittleness along with too many moving parts or things that can fail.

My suggested approach is combining iPaaS with dwPaaS to take advantage of the best of both worlds. Currently, Snaplogic and Informatica have certified cloud adapters that provide that next-generation data integration assist to go along with this new data warehouse platform managed service from Snowflake.

Now, all we need is Business Intelligence Platform as a Service (BIPaaS)!



Image of Posted by PDI Marketing Team

Posted by PDI Marketing Team

Pacific Data Integrators Offers Unique Data Solutions Leveraging AI/ML, Large Language Models (Open AI: GPT-4, Meta: Llama2, Databricks: Dolly), Cloud, Data Management and Analytics Technologies, Helping Leading Organizations Solve Their Critical Business Challenges, Drive Data Driven Insights, Improve Decision-Making, and Achieve Business Objectives.



Share
Share
Share

Join the discussion