Snowpark : Data engineering with Snowflake.

• Connect Snowpark API with snowflake.
• Basic read and write operations using Snowpark.
• Do's and Don'ts while using Snowpark.
• Building data components and basic data pipeline.

Latest Update : Snowpark procedure section added(sep 17 2023 )

What is Snowpark ?

With Snowpark, Snowflake allows developers to write code in their preferred language.

Along with Snowflake’s original SQL interface now snowflake allows you to write code in,

1. Python

2. Scala

3. Java

Some of the key features of Snowpark are,

1. Your code will be pushed to snowflake leveraging the compute power of snowflake warehouses.

2. You will not end up exporting data to different environment but rather your code is shipped to the data.

3. You can build complex data pipelines or data products using Snowpark

   
   

Snowpark also address below overheads in conventional data pipelines,

1. Long startup time of node clusters: Systems like Hadoop and Spark requires cluster of nodes to process data. Most of the time it takes 5-10 minutes to just start the cluster. In case of Snowpark we will be using snowflake warehouse to process our data.

2. Problem of small files , Problem of using right joins to shuffle data across nodes, Problem of garbage collection. uncertainty when the compute nodes goes down.

   
   

   All the above problems are well handled with Snowpark.

What you will learn ?

You will learn the basics of SNOWPARK API.

> Basic read and write operations.

> Read data from s3 and load that to snowflake table.

> We will do deep analysis of how SNOWPARKAPIworks.

> Do's and Don'ts of SNOWPARK.

> Build data components to process data.

> Build data pipeline to process data.