guide to creating a tibco messaging - apache kafka ... · the following aws services, docker...

of14

Guide to creating a TIBCO Messaging - Apache Kafka Distribution (AKD)

Configuration in an AWS Kubernetes Environment

December 2018

TIBCOAKDinanAWSKubernetesEnvironment December2018

Page2of14

TableofContentsOverview................................................................................................................................................3

ApacheKafkaonKubernetesandAWS.....................................................................................................3Architecture...........................................................................................................................................4Prerequisites..........................................................................................................................................6

SpecializedKnowledge..............................................................................................................................6TechnicalRequirements............................................................................................................................6

DeploymentSteps..................................................................................................................................7Step1.PrepareYourAWSAccountandKubernetesTooling....................................................................7Step2.BuildtheAKD–DockerImages.....................................................................................................7Step3.UploadDockerImagestoAWSECR...............................................................................................8Step3.CreateKubernetesCluster.............................................................................................................9Step4.AccessKubernetesDashboard(Optional)...................................................................................11Step5.DeployZooKeeperandKafka.......................................................................................................12Step6.TestKafkaEnvironment...............................................................................................................13Step7.RemoveKubernetesCluster........................................................................................................14


Page3of14

Overview This guide was created to provide an example of creating and provisioning Apache Kafka on Amazon Web Services (AWS) running inside Docker containers and managed by the Kubernetes orchestrator. To successfully install the TIBCO Messaging - Apache Kafka Distribution (AKD) on Kubernetes in AWS, the following file is required:

• TIB_msg-akd-core-ce_2.0.1_docker-k8-aws.tar.gz

Apache Kafka on Kubernetes and AWS Apache Kafka is an open-source, distributed stream processing platform that has rapidly grown in popularity over past few years. Provisioning and configuring a distributed system can be time consuming activity. Failures are also inevitable for a service composed from multiple components running on different hardware connected by disruptive network. Setting up Apache Kafka on AWS with support of Kubernetes can ease cluster provisioning, adopt configuration best practices, and allow scaling of hardware resources according to evolving requirements. This may lead to reducing overall cost of ownership.


Page4of14

Architecture This guide builds the following TIBCO – Messaging Apache Kafka Distribution (AKD) environment in AWS for a new Kubernetes cluster with default parameters.

Figure 1: Architecture for TIBCO AKD on AWS


Page5of14

Using this guide, along with the accompanying software, sets up the following:

• Kubernetes cluster that spans three availability zones.

• Three ZooKeeper instances (machine type t2.medium) that guarantee high availability of cluster coordinator, each deployed in a different availability zone.

• Six Kafka brokers (machine type m4.2xlarge), two in each availability zone.

• Kafka cluster has been configured with replication factor of three and rack-awareness, which guarantees that each record will be replicated across all three availability zones. Outage of single zone will not interrupt the Kafka service, nor cause data unavailability.

• External access to the Kafka brokers is made available through AWS Elastic Load Balancer.

• At the time of the writing of this document, Kubernetes on AWS does not provide local pod storage. Kafka and ZooKeeper are configured to use fast GP2 SSD drives managed by Amazon Elastic Block Store. EBS improves node failover time, because storage from decommissioned node is reattached to new instance. The Kafka broker does not need to copy data from all assigned partitions, because it is already locally available.

Note - Users may choose smaller AWS EC2 machine types, but doing so requires changes in resources section and container environment variables (JVM heap size limit) of Kubernetes deployment.


Page6of14

Prerequisites Specialized Knowledge Before you deploy the defined architecture, we recommend that you become familiar with the following AWS services, Docker containers and Kubernetes orchestrator. (If you are new to AWS, see the Getting Started Resource Center.)

• Amazon EC2 • Amazon EBS • Amazon ECS • Identity and Access Management • Kubernetes • Docker

Technical Requirements This guide defines how to create and use Docker images of the TIBCO Apache Kafka Distribution version 2.0.1. Deployment manifests in TIB_msg-akd-core-ce_2.0.1_docker-k8-aws.tar.gz can be obtained from TIBCO Community - TIBCO AKD.


Page7of14

Deployment Steps Step 1. Prepare Your AWS Account and Kubernetes Tooling 1. If you do not already have an AWS account, create one at https://aws.amazon.com and follow

the on-screen instructions.

2. Use the region selector in the navigation bar to choose the AWS Region where you want to deploy TIBCO AKD on AWS.

Note –This guide uses KOPS to manage Kubernetes cluster running on AWS. Users may choose to leverage Amazon EKS service, but it is not available in all regions. Currently, only regions in the U.S. support EKS.

3. Install and configure Amazon AWS CLI on your workstation.

Note – Ensure the AWS credentials and config files are created, and contain the appropriate AWS key, secret key, profile, and region with the appropriate IAM Role.

4. Install Docker on your workstation to build the TIBCO AKD images.

5. Install kubectl command-line tool do manage and deploy applications to Kubernetes on your workstation.

6. Download TIB_msg-akd-core-ce_2.0.1_docker-k8-aws.tar.gz from TIBCO Community - TIBCO AKD.

7. Download the TIBCO Messaging – Apache Kafka Distribution (Community Edition) from TIBCO Community - Messaging

8. Install TIBCO AKD on your workstation. This is needed to to access Kafka in AWS from your workstation.

9. Install KOPS for quick provisioning of Kubernetes cluster on your workstation.

Note – With Kops, if the non-default AIM Role is being used, ensure the AWS_PROFILE variable is set to an AWS profile that has the appropriate role and permissions.

Export the following environment variable: export AWS_SDK_LOAD_CONFIG=1, when the default AIM Role is not used.

Step 2. Build the AKD – Docker Images 1. Ensure Docker is running on your workstation. Use docker images to verify Docker is available.


Page8of14

2. Open a terminal shell and navigate to the directory where the TIBCO Docker/AKD files are located. Unpack the TIB_msg-akd-core-ce_2.0.1_docker-k8-aws.tar.gz archive. The docker and kubernetes directories will be created with several subdirectories.

3. Copy TIB_msg-akd-core-ce_2.0.1_linux_x86_64.zip to the docker/kafka/bin and docker/zookeeper/bin directories. Unzip the file in both locations. The rpm and deb directories can be discarded.

4. Navigate back to the docker/zookeeper directory.

5. Execute the make build command to build the Zookeeper - Docker image. Alternatively, if you do not have make utility installed, use the run_build.sh script. The Docker image, tibco/zookeeper with the 2.0.1 tag will be created.

6. Navigate to the docker/kafka directory.

7. Execute make build command to build the Kafka – Docker image. Alternatively, if you do not have make utility installed, use the run_build.sh script. The Docker image, tibco/kafka with the 2.0.1 tag will be created.

8. To test the Docker images, open a second terminal shell, and navigate the docker/zookeeper directory. Execute make run command to start the Zookeeper – Docker image. In the first terminal shell, and navigate the docker/kafka directory. Execute make run command to start the Kafka – Docker image. Both should start successfully, and the Kafka broker should connect to Zookeeper.

Note – Resolve any issues before continuing.

Step 3. Upload Docker Images to AWS ECR 1. Create two new ECR repositories named tibco/kafka and tibco/zookeeper in AWS. The

repositories can be created via the AWS CLI. Please note the URL of your ECR repository (e.g. 123456789012.dkr.ecr.us-west-2.amazonaws.com).

$ aws ecr create-repository --repository-name tibco/kafka --region us-west-2 $ aws ecr create-repository --repository-name tibco/zookeeper --region us-west-2

2. Return to your terminal shell and retrieve the login command to use to authenticate your Docker client with AWS registry. Adjust AWS region and access keys created in step 1.

$ aws ecr get-login --no-include-email --region us-west-2

3. Execute Docker login command returned in the previous step.


Page9of14

4. Tag Kafka and ZooKeeper images and push them to the ECR repository using the URL of your ECR repository instead of 123456789012.

$ docker tag tibco/zookeeper:2.0.1 \ 123456789012.dkr.ecr.us-west-2.amazonaws.com/tibco/zookeeper:2.0.1 $ docker push \ 123456789012.dkr.ecr.us-west-2.amazonaws.com/tibco/zookeeper:2.0.1 $ docker tag tibco/kafka:2.0.1 \ 123456789012.dkr.ecr.us-west-2.amazonaws.com/tibco/kafka:2.0.1 $ docker push \ 123456789012.dkr.ecr.us-west-2.amazonaws.com/tibco/kafka:2.0.1

5. Navigate to the kubernetes directory. Edit the zookeeper/zookeeper.yaml and kafka/kafka.yaml manifests and update the Docker image URL with your ECR repository.

Step 3. Create Kubernetes Cluster 1. Create S3 bucket to store Kubernetes cluster configuration. Adjust bucket name as it must be

unique.

$ aws s3api create-bucket --bucket tibco-kops-state-store --region us-west-2

2. By default, KOPS uses public key ~/.ssh/id_rsa.pub to allow logging into the nodes of Kubernetes cluster. If the file does not exist on your workstation, create it by executing ssh-keygen command.

3. Create Kubernetes cluster without any nodes, as we will add them in the next steps.

$ kops create cluster \ --node-count=0 \ --node-size=t2.micro \ --zones=us-west-2a,us-west-2b,us-west-2c \ --name=kafka.k8s.local \ --state=s3://tibco-kops-state-store

4. Create the Zookeeper and Kafka instance groups. Once text editor pops up, replace Kafka machine type with m4.2xlarge. Update also number of machines, so that ZooKeeper has minimum three and Kafka six VMs.

$ kops create ig zookeepernodes --subnet us-west-2a,us-west-2b,us-west-2c --state=s3://tibco-kops-state-store --name=kafka.k8s.local $ kops create ig kafkanodes --subnet us-west-2a,us-west-2b,us-west-2c --state=s3://tibco-kops-state-store --name=kafka.k8s.local


Page10of14

Figure 2: ZooKeeper instance group

Figure 3: Kafka instance group

5. Update Kubernetes configuration. Wait for several minutes and check whether all nodes have been provisioned and joined the cluster.

$ kops update cluster --name kafka.k8s.local --state=s3://tibco-kops-state-store --yes $ kops validate cluster --state=s3://tibco-kops-state-store


Page11of14

Note – The AWS nodes, masters, and api-elb security groups are all created with a CIDR of 0.0.0.0/0 on the ssh and https ports. These can create a security vulnerability, and should be modified as required.

Final result should be similar to below listing.

Figure 4: Kubernetes cluster status

Step 4. Access Kubernetes Dashboard (Optional) 1. Issue below command from shell to deploy Kubernetes dashboard.

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

2. You can access Dashboard using the kubectl command-line tool by running the following command:

$ kubectl proxy & - to run the proxy in the background, or open a second terminal shell, and run kubectl proxy

3. Open web browser and access: http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/.

4. Select token access and retrieve the token with below command.


Page12of14

$ kops get secrets admin --type secret --state s3://tibco-kops-state-store -oplaintext

5. Login into the Dashboard using the token.

Figure 5: Kubernetes dashboard

Step 5. Deploy ZooKeeper and Kafka 1. Open command shell and navigate to the Kubernetes directory that was unpacked from the

archive TIB_msg-akd-core-ce_2.0.1_docker-k8-aws.tar.gz from step 2.

2. Execute make build command to deploy ZooKeeper and Kafka. Alternatively, if you do not have make utility installed, issue below set of commands manually. Scripts will create new Kubernetes namespace, services and dedicated storage class for ZooKeeper and Kafka pods.

$ kubectl apply -f ./namespace.json $ kubectl apply -f ./zookeeper/zookeeper-storage.yaml,./zookeeper/zookeeper-service.yaml,./zookeeper/zookeeper.yaml $ kubectl apply -f ./kafka/kafka-storage.yaml,./kafka/kafka-service.yaml,./kafka/kafka.yaml Note – It will take several minutes for this step to complete!


Page13of14

3. Execute the command below to review state of the cluster. Wait until all ZooKeeper and Kafka instances are up and running. Again, this will take some time to complete. It can take ~45 minutes to complete.

$ kubectl get pods -n kafka

4. If you are using Kubernetes dashboard, please remember to switch namespace to kafka while viewing pods.

Note – The deployment step for Zookeeper and Kafka create several new elb security groups. These AWS elb* security groups are all created with a CIDR of 0.0.0.0/0 on all ports. These can create a security vulnerability, and should be modified as required.

Step 6. Test Kafka Environment In the previous step we have deployed ZooKeeper and Kafka to a Kubernetes cluster running in the AWS cloud environment. ZooKeeper provides only internal access limited to Kubernetes cluster, so we will have to connect to one of the Kafka pods to create a new topic to test with. On the other hand, Kafka brokers expose local, as well as the external service via Amazon ELB.

1. Connect to one of Kafka pods and create a new sample topic.

$ kubectl exec -it kafka-0 -n kafka -- /bin/bash # kafka-topics.sh --create --zookeeper ${_KAFKA_ZOOKEEPER_CONNECT} --replication-factor 3 --partitions 100 --topic my-topic --config min.insync.replicas=2

2. While still connected to Kafka pod, try to send and receive messages. Please note that we use broker Kubernetes service, which is available only internally inside the cluster.

# kafka-console-producer.sh --broker-list kafka-0.broker:9092,kafka-1.broker:9092,kafka-2.broker:9092 --topic my-topic --request-required-acks all # kafka-console-consumer.sh --bootstrap-server kafka-0.broker:9092 --from-beginning --topic my-topic

3. To access Kafka directly from your local workstation, we have to discover Amazon ELB endpoint of at least one Kafka broker. Issue the command below to review the external IP and port number assigned to each Kafka broker. Note the external IP and port.

$ kubectl get services -n kafka -o wide


Page14of14

4. Send and receive messages from your local workstation. Please note that you do not need to provide the endpoint of every broker, the Kafka client will discover topology of the cluster once it establishes connectivity with any of the nodes. In the following example, af176121ab0ed11e88cfe060e158e737-819917813.us-west-2.elb.amazonaws.com:32400 is the external IP and port.

$ kafka-console-producer.sh --broker-list af176121ab0ed11e88cfe060e158e737-819917813.us-west-2.elb.amazonaws.com:32400 --topic my-topic --request-required-acks all $ kafka-console-consumer.sh --bootstrap-server af176121ab0ed11e88cfe060e158e737-819917813.us-west-2.elb.amazonaws.com:32400 --from-beginning --topic my-topic

5. Optionally download kafkacat utility to review Kafka cluster metadata – how nodes present themselves to external clients.

$ kafkacat -b af176121ab0ed11e88cfe060e158e737-819917813.us-west-2.elb.amazonaws.com:32400 –L

Step 7. Remove Kubernetes Cluster Once finished using Kafka, execute below command to remove Kubernetes cluster, if desired. This step can take several minutes to complete. ~10 minutes.

$ kops delete cluster --name kafka.k8s.local --state s3://tibco-kops-state-store --yes

guide to creating a tibco messaging - apache kafka ... · the following aws services, docker...

Documents