Guide: VPC Peering

Peering your VPC with Cloudkarafka allows you to access the Kafka cluster using plaintext while still maintaining the security of your data in transit.
This also gives you access to zookeeper which is necessary for some of the kafka CLI scripts and other tools around Kafka.

Create a Kafka cluster

Create the Kafka cluster at cloudkarafka.com, make sure to select a subnet that doesn’t conflict with the subnet that your machines (in you account) is using.

Set up a VPC Peering Connection

Once the Kafka cluster is up and running you will find a menu option named VPC in the Console, here you will find account id, vpc id and region, this information is later needed when creating the peering connection.
Further, open a new tab in your browser and log in to your AWS account, go to VPC and Peering Connections and click Create Peering Connections.

Start by giving the connection a name.
For VPC (Requester) select the VPC of which your own machine is running. In the next section, you will enter the account id, vpc id and region information that you found on the VPC page for Cloudkarafka. After that, click Create Peering Connection. When the connection is created you need to edit the Route table for your VPC so the data that should go to CloudKarafka gets routed correctly. Select your VPC under “Your VPCs”. To open the route table for your VPC, hit the Description tag and select Route Table. Select the Route Tab and click Edit. A similar view should be shown

You now need to add the subnet for the kafka cluster (this information is also available under the VPC page at Cloudkarafka). Target should be set to the peering connection that you just created, prefixed pcx-

Now, all traffic from your VPC, via your entered subnet, will pass over the peering connection to CloudKarafka, and you will be able to connect to Kafka using the internal IP of the brokers and port 9092.
This also enables you to connect to Zookeeper using internal ip and port 2181.