Setting up Oracle Container Engine for Kubernetes (OKE) with three worker nodes

In this tutorial, I will explain how you set up a Kubernetes Cluster consisting of the Kubernetes Control Plane and the Data Plane (Node Pool) using the Oracle Container Engine (OKE) for Kubernetes. I will also deploy and delete two sample applications on the Kubernetes platform to prove it works. This tutorial will set the stage for future tutorials that will dive into networking services offered inside Kubernetes for container-hosted applications.

The Steps

• STEP 01: Create a new Oracle Kubernetes Cluster (OKE) and verify the components
• STEP 02: Verify the deployed Oracle Kubernetes Cluster (OKE) cluster components in the OCI console
• STEP 03: Verify if the Oracle Kubernetes Cluster (OKE) cluster is running (using the CLI)
• STEP 04: Deploy a sample NGINX application (using kubectl)
• STEP 05: Deploy a sample MySQL application (using helm chart)
• STEP 06: Clean up PODS and Namespaces

STEP 01 - Create a new Oracle Kubernetes Cluster and verify the components

• Click the hamburger menu.

1. Click on Developer Services.
2. Click Kubernetes Clusters (OKE).

• Click on Create Cluster.

1. Select quick create.
2. Click on Submit.

1. Enter a Cluster Name.
2. Select a Compartment.
3. Select the Kubernetes version.
4. Select the Kubernetes API endpoint to be a Public endpoint.
5. Select the Node Type to be Managed.
6. Scroll down.

1. Select the Kubernetes Worker nodes to be a Private workers.
2. Scroll down.

1. Leave the Node count (Worker nodes) default (3).
2. Click on Next.

1. Review the Cluster parameters.
2. Scroll down.

1. Review the Node Pools parameters.
2. Scroll down.

1. Do NOT select to create a Basic Cluster.
2. Click on Create cluster.

- Review the status of the different components that are created.

1. Review the status of the different components that are created.
   1. Make sure everything has a green check.
2. Click on Close.

1. Review that the status is CREATING.
2. Scroll down.

- Review the CLUSTER and NODEPOOL creation status. - The Kubernetes control CLUSTER is being created and the worker NODEPOOL will be created later.

• After a few minutes the Kubernetes control CLUSTER is successfully created.

• The worker NODEPOOL will now be created.

• After a few minutes the worker NODEPOOL is successfully created.

1. Click on Node Pools.
2. Notice that the (Worker) Nodes in the pool are still being created.
3. Click on the number 3 of (Worker Nodes).

• Notice that not all nodes and Not ready (yet).

• After a few minutes they will be Ready.

At this point, the Kubernetes Control cluster and worker nodes are fully deployed and configured inside Oracle Cloud Infrastructure (OCI). This is what we call the Oracle Cloud Kubernetes Engine (OKE).

STEP 02 - Verify the deployed Oracle Kubernetes Cluster cluster components in the OCI console

When you use the Oracle Cloud Kubernetes Engine (OKE) to create a Kubernetes cluster some resources will be created inside OCI to support this deployment.

The first and most important resource is the Virtual Cloud Network (VCN). Because I have chosen the “Quick Create” option a new VCN dedicated to OKE was created.

If you navigate through the OCI Console using the hamburger menu and then Networking > Virtual Cloud Networks (VCN) you will see the new VCN that was created.

Click on the VCN.

Inside the VCN you will see three Subnets, one Private and two Public Subnets to support the OKE deployment.

1. Click on CIDR Blocks/Prefixes to review the CIDR of the VCN.
2. Notice that 10.0.0.0/16 was assigned by OCI.

1. Click on the Route Tables to review the routing tables.
2. Notice that there are two routing tables created, one to route to private subnets, and one to route to public subnets.

1. Click on the Internet Gateways to review the Internet Gateway that will provide internet connectivity using the public subnets to and from the internet.
2. Notice there is only one Internet Gateway.

1. Click on the Security Lists to review the Security Lists that can be either Ingress or Egress Rules to protect connectivity between the subnets.
2. Note that there are three Security Lists, one used for Kubernetes (Worker) Node Connectivity protection, one used for Kubernetes API Endpoint protection, and one used for Kubernetes Services protection.

1. Click on the NAT Gateways to review the NAT Gateway that will provide internet connectivity using the private subnets to the internet.
2. Notice there is only one NAT Gateway.

1. Click on the Service Gateways to review the Service Gateway that will provide private access to specific Oracle services, without exposing the data to an internet gateway or NAT gateway.
2. Notice there is only one Service Gateway.

1. Navigate through the OCI Console using the hamburger menu and then Compute > Instances.
2. Notice three Instances that are created that will be used as the three Kubernetes Worker Nodes we specified during the deployment.

1. Navigate through the OCI Console using the hamburger menu and then IP Management > Reserved public IPs.
2. Notice there is one public IP address (ending with .166) reserved for the Kubernetes Public API endpoint.

If we place every piece of information we have just collected and place that in a diagram the diagram will look something like this:

This will help to put everything more into context (or perspective).

Tables with configuration details to deploy OKE

VCN

Resource	Name
VCN	• Name: oke-vcn-quick-IH-OKE-CLUSTER-af593850a • CIDR Block: 10.0.0.0/16 • DNS Resolution: Selected
Internet Gateway	• Name: oke-igw-quick-IH-OKE-CLUSTER-af593850a
NAT Gateway	• Name: oke-ngw-quick-IH-OKE-CLUSTER-af593850a
Service Gateway	• Name: oke-sgw-quick-IH-OKE-CLUSTER-af593850a • Services: All region Services in Oracle Services Network
DHCP Options	• DNS Type set to Internet and VCN Resolver

Subnets

Resource	Example
Public Subnet for Kubernetes API Endpoint	Purpose: Kubernetes API endpoint with the following properties: • Type: Regional • CIDR Block: 10.0.0.0/28 • Route Table: oke-public-routetable-IH-OKE-CLUSTER-af593850a • Subnet Access: Public • DNS Resolution: Selected • DHCP Options: Default • Security List: oke-k8sApiEndpoint-quick-IH-OKE-CLUSTER-af593850a
Private Subnet for Worker Nodes	Purpose: workernodes with the following properties: • Type: Regional • CIDR Block: 10.0.10.0/24 • Route Table: N/A • Subnet Access: Private • DNS Resolution: Selected • DHCP Options: Default • Security List: oke-nodeseclist-quick-IH-OKE-CLUSTER-af593850a
Private Subnet for Pods	Purpose: pods with the following properties: • Type: Regional • CIDR Block: 10.96.0.0/16 • Route Table: oke-private-routetable-IH-OKE-CLUSTER-af593850a • Subnet Access: Private • DNS Resolution: Selected • DHCP Options: Default • Security List: N/A
Public Subnet for Service Load Balancers	Purpose: Load balancers with the following properties: • Type: Regional • CIDR Block: 10.0.20.0/24 • Route Table: oke-private-routetable-IH-OKE-CLUSTER-af593850a • Subnet Access: Public • DNS Resolution: Selected • DHCP Options: Default • Security List: oke-svclbseclist-quick-IH-OKE-CLUSTER-af593850a

Route Tables

Resource	Example
Public Subnet for Kubernetes API Endpoint	Purpose: Kubernetes API endpoint with the following properties: • Type: Regional • CIDR Block: 10.0.0.0/28 • Route Table: oke-public-routetable-IH-OKE-CLUSTER-af593850a • Subnet Access: Public • DNS Resolution: Selected • DHCP Options: Default • Security List: oke-k8sApiEndpoint-quick-IH-OKE-CLUSTER-af593850a
Private Subnet for Worker Nodes	Purpose: workernodes with the following properties: • Type: Regional • CIDR Block: 10.0.10.0/24 • Route Table: N/A • Subnet Access: Private • DNS Resolution: Selected • DHCP Options: Default • Security List: oke-nodeseclist-quick-IH-OKE-CLUSTER-af593850a
Private Subnet for Pods	Purpose: pods with the following properties: • Type: Regional • CIDR Block: 10.96.0.0/16 • Route Table: oke-private-routetable-IH-OKE-CLUSTER-af593850a • Subnet Access: Private • DNS Resolution: Selected • DHCP Options: Default • Security List: N/A
Public Subnet for Service Load Balancers	Purpose: Load balancers with the following properties: • Type: Regional • CIDR Block: 10.0.20.0/24 • Route Table: oke-private-routetable-IH-OKE-CLUSTER-af593850a • Subnet Access: Public • DNS Resolution: Selected • DHCP Options: Default • Security List: oke-svclbseclist-quick-IH-OKE-CLUSTER-af593850a

Security List Rules for Public Kubernetes API Endpoint Subnet

The oke-k8sApiEndpoint-quick-IH-OKE-CLUSTER-af593850a security list has the ingress and egress rules shown below:

Ingress Rules:

Stateless	Source	IP Protocol	Source Port Range	Destination Port Range	Type and Code	Allows	Description
No	0.0.0.0/0	TCP	All	6443		TCP traffic for ports: 6443	External access to Kubernetes API endpoint
No	10.0.10.0/24	TCP	All	6443		TCP traffic for ports: 6443	Kubernetes worker to Kubernetes API endpoint communication
No	10.0.10.0/24	TCP	All	12250		TCP traffic for ports: 12250	Kubernetes worker to control plane communication
No	10.0.10.0/24	ICMP			3, 4	ICMP traffic for: 3, 4 Destination Unreachable: Fragmentation Needed and Don't Fragment was Set	Path discovery

Egress Rules:

Stateless	Destination	IP Protocol	Source Port Range	Destination Port Range	Type and Code	Allows	Description
No	All AMS Services In Oracle Services Network	TCP	All	443		TCP traffic for ports: 443 HTTPS	Allow Kubernetes Control Plane to communicate with OKE
No	10.0.10.0/24	TCP	All	All		TCP traffic for ports: All	All traffic to worker nodes
No	10.0.10.0/24	ICMP			3, 4	ICMP traffic for: 3, 4 Destination Unreachable: Fragmentation Needed and Don't Fragment was Set	Path discovery

Security List Rules for Private Worker Nodes Subnet

The oke-nodeseclist-quick-IH-OKE-CLUSTER-af593850a security list has the ingress and egress rules shown below:

Ingress Rules:

Stateless	Source	IP Protocol	Source Port Range	Destination Port Range	Type and Code	Allows	Description
No	10.0.10.0/24	All Protocols				All traffic for all ports	Allow pods on one worker node to communicate with pods on other worker nodes
No	10.0.0.0/28	ICMP			3, 4	ICMP traffic for: 3, 4 Destination Unreachable: Fragmentation Needed and Don't Fragment was Set	Path discovery
No	10.0.0.0/28	TCP	All	All		TCP traffic for ports: All	TCP access from Kubernetes Control Plane
No	0.0.0.0/0	TCP	All	22		TCP traffic for ports: 22 SSH Remote Login Protocol	Inbound SSH traffic to worker nodes
No	10.0.20.0/24	TCP	All	32291		TCP traffic for ports: 32291
No	10.0.20.0/24	TCP	All	10256		TCP traffic for ports: 10256
No	10.0.20.0/24	TCP	All	31265		TCP traffic for ports: 31265

Egress Rules:

No	10.0.10.0/24	All Protocols				All traffic for all ports	Allow pods on one worker node to communicate with pods on other worker nodes
No	10.0.0.0/28	TCP	All	6443		TCP traffic for ports: 6443	Access to Kubernetes API Endpoint
No	10.0.0.0/28	TCP	All	12250		TCP traffic for ports: 12250	Kubernetes worker to control plane communication
No	10.0.0.0/28	ICMP			3, 4	ICMP traffic for: 3, 4 Destination Unreachable: Fragmentation Needed and Don't Fragment was Set	Path discovery
No	All AMS Services In Oracle Services Network	TCP	All	443		TCP traffic for ports: 443 HTTPS	Allow nodes to communicate with OKE to ensure correct start-up and continued functioning
No	0.0.0.0/0	ICMP			3, 4	ICMP traffic for: 3, 4 Destination Unreachable: Fragmentation Needed and Don't Fragment was Set	ICMP Access from Kubernetes Control Plane
No	0.0.0.0/0	All Protocols				All traffic for all ports	Worker Nodes access to Internet

Security List Rules for Public Load Balancer Subnet

The oke-svclbseclist-quick-IH-OKE-CLUSTER-af593850a security list has the ingress and egress rules shown below:

Ingress Rules:

Stateless	Source	IP Protocol	Source Port Range	Destination Port Range	Type and Code	Allows	Description
No	0.0.0.0/0	TCP	All	80		TCP traffic for ports: 80

Egress Rules:

Stateless	Destination	IP Protocol	Source Port Range	Destination Port Range	Type and Code	Allows	Description
No	10.0.10.0/24	TCP	All	32291		TCP traffic for ports: 32291
No	10.0.10.0/24	TCP	All	10256		TCP traffic for ports: 10256
No	10.0.10.0/24	TCP	All	31265		TCP traffic for ports: 31265

STEP 03 - Verify if the Oracle Kubernetes Cluster cluster is running using the CLI

• Go back to the overview page of the Kubernetes Clusters (OKE).
• Click the hamburger menu > Developer Services > Kubernetes Clusters (OKE).
• Click on the Kubernetes Cluster that was just created.

1. Scroll down.
2. Click on Quick Start.

• Click on Access Cluster.

1. Select Cloud Shell Access.
2. Click on Copy to copy the command to allow access to the Kubernetes Cluster.
3. Click on Launch Cloud Shell.

• This is a diagram of how the connection will be made to perform management on the OKE cluster using Cloud Shell.

• The Cloud Shell will now start.

• Some informational messages will be shown on what is happening in the background.

• In this case, it is possible to let Cloud Shell run on different CPU architectures.
• Click on Close to close this informational message.

• Now we are ready to use the Cloud Shell to access to the Kubernetes Cluster.

• Paste in the command that was copied before.

• Issue the following command to gather information about the Kubernetes Cluster:

kubectl cluster-info

• Issue the following command to gather information about the Worker Nodes:

kubectl get nodes

• Issue another command to gather more information about the Worker Nodes:

kubectl get nodes -o wide

• Issue the following command to clear the screen and start with a fresh new screen.

clear

1. Notice that the previous output has been cleared (but is still accessible when you scroll up)
2. Click on minimize to minimize the Cloud Shell window.

• Click on Close to close the Access Cluster information.

STEP 04 - Deploy a sample NGINX application using kubectl

1. Make a note of the command to gather the Kubernetes Version.
2. Make a note of the command to gather the Deploy a sample application.
3. Click on Restore to restore the Cloud Shell window (to issue the commands).

• Issue this command to gather the Kubernetes Version:

kubectl version

• Issue the following commands to verify the current pods (applications) that are deployed:

kubectl get pods

• Notice that there are no resources found.

• Issue the following command to deploy a new (sample) application:

kubectl create -f https://k8s.io/examples/application/deployment.yaml

• Issue the following commands to verify the current pods (applications) that are deployed:

kubectl get pods

• Notice that there are PODS in the RUNNING state.
• This means that the application we just deployed is running.

• Issue the following command to gather the IP addresses to access the application:

kubectl get deploy,svc

• Notice that the newly deployed application does not have any IP addresses assigned and that only the Kubernetes Cluster has a ClusterIP service attached to it with an internal IP address.

• Issue the following command to look at the attached (network) services for the newly deployed application specifically:

kubectl get svc ngnix

• Notice that there are no (network) services deployed (or attached) for the deployed (NGINX) application.
• For this reason, we will not be able to access the application from another application or use the web browser to access the webpage in the NGINX web server.
• This will be addressed in a future tutorial.

STEP 05 - Deploy a sample MySQL application using helm chart

• A Helm chart is a package that contains all the necessary resources to deploy an application to a Kubernetes cluster.

1. Issue the following command to add the Bitnami Repository for the MySQL database Helm chart is a package:
2. Issue the following command to deploy a MySQL database on the Kubernetes Worker Nodes and also create a new namespace (mysql):

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install mysql bitnami/mysql -–namespace mysql --create-namespace

• Issue the following command to gather the deployed applications:
• This command will only display the deployed applications in the current (default) namespace):

kubectl get pods

• Notice that only the NGINX application is showing up in the default (current) namespace.
• This command will now display the deployed applications cluster-wide (all namespaces):

kubectl get pods -A -w

1. Notice that the NGINX application is running in the default namespace
2. Notice that the MySQL application is running in the (new) mysql namespace.

STEP 06 - Clean up PODS and Namespaces

• Now that we have deployed an application in the default namespace (NGINX) and another application in a new namespace (MySQL) using Helm charts, let's clean up the environment so we can start fresh whenever we need to.

1. Use this command to gather all the worker nodes (cluster-wide).
2. Use this command to gather all the running pods in the current (default) namespace
3. Use this command to gather all the namespaces.
4. Use this command to gather all the running pods in the current (default) namespace specifically.
5. Use this command to gather all the running pods in the mysql namespace specifically.

1.
kubectl get nodes -o wide
2.
kubectl get pods -o wide
3.
kubectl get namespaces
4.
kubectl get pods --namespace=default
5.
kubectl get pods --namespace=mysql

1. Issue this command to delete all deployments/pods in the default namespace.
2. Issue this command to verify if the deployments/pods are deleted.
3. Use this command to gather all the running pods in the mysql namespace specifically. Just Verify if this still exists.

1.
kubectl delete --all deployments --namespace=default
2.
kubectl get pods --namespace=default
3.
kubectl get pods --namespace=mysql

1. Issue this command to delete all deployments/pods AND the complete mysql namespace.
2. Use this command to gather all the namespaces, and verify if the mysql namespace is deleted.

1.
kubectl delete namespace mysql
2.
kubectl get namespaces

Conclusion

In this tutorial we have deployed a Kubernetes Control cluster and worker nodes are fully deployed and configured inside Oracle Cloud Infrastructure (OCI). This is what we call the Oracle Cloud Kubernetes Engine (OKE). I have also deployed two sample applications in two different namespaces where one application was deployed using a Helm Chart in a new namespace. In the end, we cleaned up the applications/pods. We have not deployed any network services for Kubernetes-operated applications/pods in this tutorial yet.