In the first blog post, we discussed Next-Gen Corda’s architecture. In the second blog post, we saw how to start a local all-in-one combined cluster with a static network. This article will show how to deploy a local Corda worker cluster using Kubernetes with a static network.
Pre-requisites
In this blog post, I will show you how to use Helm. Helm is a Kubernetes deployment tool that lets you automate the creation, packaging, configuration, and deployment of applications and services to Kubernetes clusters.
We will first start a local Kubernetes cluster using Minikube. After that will install Kafka and Postgres using Helm. We will then install Corda workers to the Kubernetes cluster using Helm.
Please note that Kafka and Database are prerequisites for a Corda cluster deployment. This blog will show how these prerequisites can be installed using Helm charts.
Note: There could be many ways to do the deployment mentioned below, depending on your use case, your needs, and your organisation’s policies. For example, for setting up Kubernetes locally, you could use either Docker Desktop, Minikube, Kind, MicroK8s or Rancher Desktop. For remote setup, there are many options like EKS, AKS and GKE. This blog shows you one way to deploy Kubernetes locally, and it should not be treated as a setting stone standard. Similarly, we currently don’t have any specific guidance on the configuration of PostgreSQL and Kafka. Helm charts to set up Kafka and Postgres provided by R3 are only for a development setup.
Step 1: Start a Kubernetes cluster using Minikube
For this blog, I will be executing the below commands on a Mac M1 computer. There are multiple ways to create a Kubernetes cluster. For this blog, I will use Minikube to start a local kubernetes cluster. Run the below command and this will start a Kubernetes cluster running on your system.
minikube start --cpus=6 --memory=8G
Step 2: Install Postgres, Kafka from a packaged Helm chart
Corda Helm charts are pushed as an OCI bundle to a docker registry. To install Kafka and Postgres is to install a packaged Helm chart as an OCI bundle pushed onto a docker registry, as shown below.
helm install prereqs -n corda --create-namespace oci://registry-1.docker.io/corda/corda-dev-prereqs --timeout 10m --wait
Pulled: registry-1.docker.io/corda/corda-dev-prereqs:0.2.0
Digest: sha256:577cbc404efa224e9cc8b5c4b6ca09574b515639d36a08a39f30290f5f888ba6
NAME: prereqs
LAST DEPLOYED: Thu Feb 16 14:19:05 2023
NAMESPACE: corda
STATUS: deployed
REVISION: 1
NOTES:
Kafka and PostgreSQL have been installed.
**WARNING** These are not intended for production use.
The following overrides should be used when installing a Corda Helm Chart into the same namespace:
bootstrap:
kafka:
replicas: 1
db:
cluster:
host: "prereqs-postgres"
username:
value: "corda"
password:
valueFrom:
secretKeyRef:
name: "prereqs-postgres"
key: "corda-password"
kafka:
bootstrapServers: "prereqs-kafka:9092"
sasl:
enabled: true
mechanism: "PLAIN"
username:
value: "admin"
password:
valueFrom:
secretKeyRef:
name: "prereqs-kafka"
key: "admin-password"
tls:
enabled: true
truststore:
valueFrom:
secretKeyRef:
name: "prereqs-kafka"
key: "ca.crt"
To access PostgreSQL from outside the Kubernetes cluster:
kubectl port-forward -n corda svc/prereqs-postgres 5432 &
export PGUSER=corda
export PGPASSWORD=$(kubectl get secret -n corda prereqs-postgres -o go-template='{{ index .data "corda-password" | base64decode }}')
psql -h localhost -p 5432 -d cordacluster
To access Kafka from outside the Kubernetes cluster:
export KAFKA_PASSWORD=$(kubectl get secret -n corda prereqs-kafka -o go-template='{{ index .data "admin-password" | base64decode }}')
echo "security.protocol=SASL_PLAINTEXT" > client.properties
echo "sasl.mechanism=PLAIN" >> client.properties
echo "sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=\"admin\" password=\"$KAFKA_PASSWORD\";" >> client.properties
kubectl port-forward -n corda $(kubectl get pods -n corda --selector=app.kubernetes.io/component=kafka,app.kubernetes.io/instance=prereqs -o=name) 9094 &
kafka-topics --list --bootstrap-server localhost:9094 --command-config client.properties
This will install Kafka and Postgres into the Corda Kubernetes namespace. The --wait option ensures all the pods are ready before returning.The timeout is set to 10 minutes to allow time to pull the images.
Note: Here Kafka is a single broker and should be used only for development purposes.
To know more about the details, Helm charts are pushed at the corda-helm-prereq repo.
Step 3: Port Forward for Postgres
Use the output from step 2 and expose the cluster’s Postgres to the outside world by executing the below command.
kubectl port-forward -n corda svc/prereqs-postgres 5432 &
You should also set the default namespace to Corda by executing below command.
kubectl config set-context --current --namespace=corda
Step 4: Install Corda using packaged Helm chart and pre-built docker images
It is possible to deploy a pre-built version of the Helm chart and the associated Docker images from the Artifactory.
helm install corda oci://registry-1.docker.io/corda/corda --version 5.0.0-Gecko1.0 --namespace corda --values https://gist.githubusercontent.com/snedamle/3fbadeb35f11ad13fecb204b099655cd/raw/94fc09db43a9048497157c3778866fd488090a3f/corda-dev-prereqs.yaml --wait
Pulled: registry-1.docker.io/corda/corda:5.0.0-Gecko1.0
Digest: sha256:c8e5aa99c47fa3aab94e93bb46e77d8c4d72b67bd9b15432cc423673c61b13f2
NAME: corda
LAST DEPLOYED: Fri Mar 24 13:31:53 2023
NAMESPACE: corda
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Extract the credentials for the initial admin user:
kubectl get secret corda-initial-admin-user -o go-template='{{ .data.username | base64decode }}'
kubectl get secret corda-initial-admin-user -o go-template='{{ .data.password | base64decode }}'
2. Expose the API endpoint on localhost by running this command:
kubectl port-forward --namespace corda deployment/corda-rest-worker 8888 &
3. The API endpoint definition can then be accessed via: https://localhost:8888/api/v1/swagger
The corda-dev-prereqs-yaml file lets you override the default Corda cluster configuration, e.g. the number of replicas. The above command will install Corda worker jars from the docker images and tells the chart where to find the Kafka and PostgreSQL instances created in the previous step.
Note: I have set up the database using the Helm chart. You could manually set up the database as well. You can manually generate the SQL scripts using the Corda-Cli tool.
The instructions on the GitHub wiki talk about how to build the Corda Docker images and then install Corda using the Helm chart from the source.
Step 5: Kubernetes Cluster Port forward and verify if the cluster is up
Follow the instructions, which are the output of the previous step, to verify if the cluster is running. Forward the port by executing the below command and hit https://localhost:8888/api/v1/swagger#/ to confirm if the cluster is up.
kubectl port-forward --namespace corda deployment/corda-rest-worker 8888 &
Once the cluster is up, we can use CSDE’s quickDeploy task to deploy our CorDapp. If you want to read more about what the quickDeploy task does, read the previous blog. Here, we will not run the startCorda task, as we already have a cluster up and running.
Step 6: Retrieve the Corda cluster RPC username and password
Get the Corda Rpc username and password from the output of the previous step. Update the cordaRpcUser and cordaRpcPasswd in CSDE’s gradle.properties to the values retrieved by running the below commands.
kubectl get secret corda-initial-admin-user -o go-template='{{ .data.username | base64decode }}' --namespace corda
kubectl get secret corda-initial-admin-user -o go-template='{{ .data.password | base64decode }}' --namespace corda
Update the RPC username and password in gradle.properties
Step 7: Run CSDE’s quickDeploy gradle task
quickDeploy task will internally create a .CPI file, and upload the .CPI file to the cluster, create and register virtual nodes.
Step 8: View worker logs
You can retrieve the list of worker pods by running the below command.
kubectl get pods -n corda
To retrieve logs for specific worker pods, retrieve the name of the worker from output of the above list and hit the below command.
kubectl logs -f -n corda corda-rest-worker-68f4db8b4-lbfdv
Step 9: Use the HTTP RPC API to hit a flow
You can look at these steps, which discuss triggering a flow.
Step 10: Cleanup
You can use the below commands to clean up the Corda cluster.
kubectl delete namespace corda
In the next blog post, we will talk about dynamic network deployments.
