Real-Time Data Ingestion from Kafka to Snowflake

Within your AWS Console search for MSK in the search field at the top and select the matching result. Visit the Clusters screen, and then click Create Cluster.

The Quick Create option provides a good set of defaults for creating a Kafka cluster, so unless you've previous knowledge or experience to know you might want something different I'd suggest just confirming the details and then clicking Create cluster at the bottom of the screen.

Once you've started the cluster creation it may take about 15 minutes for provisioning to complete and for your broker to be available.

The Snowflake Pull from Kafka Connector by Ockam is invite only, to get an invite please drop us an email at hello@ockam.io. Once invited you'll receive an email with a link to get the app.

The first screen you're presented with will ask you to select the warehouse to utilize to activate the app.

Click the Grant button to the right of this screen. The app will then be automatically granted permissions to create a warehouse and create a compute pool.

Once the permissions grants complete, an Activate button will appear. Click it and the activation process will begin.

After the app activates you'll see a page that summarizes the privileges that the application now has. There's nothing we need to review or update on these screens yet, so proceed by clicking the Launch app button.

Run the following command on your local workstation:

This will install the Ockam Command and source in the required environment settings. If you've installed Ockam Command before you can skip this step.

Once Ockam Command installation is complete you can run:

Wrapped up in this single ockam enroll command are several steps that will bootstrap your first project and get you ready to go. It will:

• Generate an Ockam Identity and store its secret keys in a file system based Ockam Vault.
• Create an account with Ockam Orchestrator.
• Provision a trial Space and Project in the Orchestrator.
• Make your Ockam Identity the administrator of your new Project.

In this section we're going to provision an Ockam node that will run alongside our Kafka broker, and provide one of the ends of our point-to-point connection.

We need to generate an enrollment ticket to allow a new Ockam Node to join the project that was just created. This node will run alongside the Kafka broker, in the network where the Kafka is running:

The Ockam Node for Amazon MSK is a streamlined way to provision a managed Ockam Node within your private AWS VPC.

To deploy the node that will allow Snowflake to reach your Kafka broker visit the Ockam Node for Amazon MSK listing in the AWS Marketplace, and click the Continue to Subscribe button, and then Continue to Configuration.

On the configuration page choose the region that your Amazon MSK cluster is running in, and then click Continue to Launch followed by Launch.

The initial Create Stack screen pre-fills the fields with the correct information for your node, so you can press Next to proceed.

This screen has important details to you need to fill in:

• Stack name: Give this stack a recognisable name, you'll see this in various locations in the AWS Console. It'll also make it easier to clean these resources up later if you wish to remove them.
• VPC ID: The ID of the Virtual Private Cloud network to deploy the node in. Make sure it's the same VPC where you've deployed your Amazon MSK cluster.
• Subnet ID: Choose one of the subnets within your VPC, ensure MSK has a broker address available in that subnet.
• Enrollment ticket: Copy the contents of the kafka.ticket file we created earlier and paste it in here.
• Amazon MSK Bootstrap Server with Port: In the client configuration details for your Amazon MSK cluster you will have a list of bootstrap servers. Copy the hostname:port value for the Private endpoint that's in the same subnet you chose above.
• JSON Node Configuration: Copy the contents of the kafka.json file we created earlier and paste it in here.

We've now completed the highlighted part of the diagram below, and our Kafka broker node is waiting for another node to connect.

One end of our connection is now setup, it's time to connect the Snowflake end. We need to generate an enrollment ticket to allow another Ockam Node to join our project. This node will run in our Snowflake warehouse:

Take the contents of the file snowflake.ticket that we just created and paste it into "Provide the above Enrollment Ticket" form field in the "Configure app" setup screen in Snowflake.

To be able to authenticate with Ockam Orchestrator and then discover the route to our outlet, the Snowflake app needs to allow outbound connections to your Ockam project. Toggle the Grant access to egress and reach your Project and approve the connection by pressing Connect.

Toggle the Grant access to the tables where Kafka messages must be inserted and select the CUSTOMERS table.

Enter the name of the Kafka topic you want to map to the Snowflake table. In this example, the topic is customers.

Check "Messages are encoded with a schema" if you have a schema registry and the messages are encoded with a schema. The configuration to use to Launch Ockam node for Amazon MSK will need to be updated to include the schema registry details. Update $SCHEMA_REGISTRY_ADDRESS with the address of the schema registry. Make sure the ockam node has access to the schema registry.

Update other options from default values if needed.