Create a quick Distributed multi nodes Cluster with Jenkins using a Master node in Rhel8 Os and 3 slaves nodes (one Ec2 instance of AWS, one Guest Os of Windows and my local ubuntu machine)

10 min readMar 24, 2021

Introduction

Before start, let’s be familiar with some technical terms of our demo.

So What is Jenkins? Jenkins is an open-source server that is written entirely in Java. It lets you execute a series of actions to achieve the continuous integration process, that too in an automated fashion.

Why we need distributed or Multi nodes Cluster of Jenkins?

In nowadays automation is very crucial in production world to achieve better in quality and quantity a task. Therefore come Jenkins as a continuous integration tool to set a single or multi nodes depending of our needs. So a single node cluster will only allow us to setup a Master and a worker/slave node working together to achieve a task but what if we want many tasks with different purposes to be done? That is where come a multi node cluster of Jenkins to manage all workers or slave nodes from a single Master node of Jenkins and now all different worker node can run different version of software to achieve the task.

What is my plan?

My plan today is to show you quickly how you can set a multi node cluster with Jenkins and perform various different tasks in different workers nodes.

Plan of work

So to achieve our plan,
1) i will setup First a Master Node on top of a Guest Rhel8 OS running in VirtualBox
2) Setup my local machine Ubuntu as a Worker node1
3) Launch a second Worker node2 on windows OS running on top of VirtualBox
4) And finnally a third Worker node3 of Ec2 instance from AWS
5) Launch some jobs to test our node
6) Conclusion

Setup Master Node

Jenkins installed on an Os by default become a single node cluster and works therefore as Master or slave node. But here in this demo we will set a Master node in a Rhel8 Os running on Virtual Box. So let’s proceed with below phases :

Step1: Download the Jenkins package on your machine according to the Os you are using . Reference https://www.jenkins.io/doc/book/installing/linux/ In my case i am using rhel8 OS, so

i) sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
  
ii) sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key       "If you’ve previously imported the key from Jenkins, the rpm --import will fail because   you already have a key. Please ignore that and move on."
iii) yum install jenkins

You will need to explicitly install a Java runtime environment, because Oracle’s Java RPMs are incorrect and fail to register as providing a java dependency. Thus, adding an explicit dependency requirement on Java would force installation of the OpenJDK JVM.

2.164 (2019–02) and newer: Java 8 or Java 11

2.54 (2017–04) and newer: Java 8

1.612 (2015–05) and newer: Java 7

Step2: Download the Java package

iv) sudo yum install jenkins java-1.8.0-openjdk-devel

Ref=> https://www.jenkins.io/doc/book/installing/linux/

Step3: Start the service of Jenkins

sudo systemctl daemon-reload 
sudo systemctl restart jenkins

Now typing ps -aux | grep jenkins => to check the jenkins service has started

jenkins successfully started and running on port 8080

Step4: Access the jenkins webui on your local machine:

For that your have to type in your browser the ip of your guest machine along the port 8080 and don’t forget to disable the firewall o your guest machine like:

sudo systemctl stop firewalld
ipconfig => to know the ip of your guest machine

Accessing jenkins webUi on my local machine browser.

Note: As for the 1st time you will be asked for admin password so look at: sudo cat /var/lib/jenkins/secrets/initialAdminPassword will print the password at console terminal. After login now, you can change the password of admin user: More Ref use=> https://www.jenkins.io/doc/book/installing/linux/

As I have already told you that that by default jenkins configure the node on which it is installed as a Master node, so to confirm that click on >Dashboard> then Manage Jenkins> and Under System Configuration> click on Manage Nodes & Clouds > and you can see that your node is configured as master .

Setup my local ubuntu machine as worker node1 :

I will set my local ubuntu machine as worker node1 and for that i need :

1) As my worker node1 will be a linux machine i need an ssh plugin on the Master node which will contact the local machine node1 for connection
2) An agent program java in my local machine which will be contacted by the Master node to establish a connection.

So first install the “ssh build agent” in the Master node by clicking Dashboard > Manage Jenkins > under System Configuration > select Manage Plugin > and under tab available > search ‘‘ssh’’ in the filter box and after click download to install it.

After install the plugin you will have to restart the jenkins service .

Now to set a worker node we need first to click under Dashboard>Nodes> click New Node like:

Now after you will be present to a page to set name for your worker node and select Permanent Agent button and then click ok:

So at next page, you will be prompt like this:

Note: At the 1) From the above screenshot, under “#of executors” you can set any number you want and this define the no of jobs that can be set in parallel.

And under “Remote root Directory”, you have to put the path to directory of worker node which will be used my the Master node to install software.

At 2) Under Launch method, you have to choose ssh agent as method of connection because the worker node1 is a linux machine on which the Master will instantiate a connection.

At 3) Under Host, just type the ip_address of your worker node1 there

At 4) Under Credentials, click add to enter the username and password of the worker node on which you want Master will connect.

At 5) Under Host Key Verification Strategy, select “Non verifying Verification Strategy” => this will avoid the issue of prompting to enter password when Master will try ssh connection to worker node.

And click Save it will setup the worker node ready as :

Now let’s launch a job to test our configuration are working

You can see that a job has been launched in our worker node1.

Note: To create permanent jobs in the slave node so you can reconfigure the worker node by choosing under “Usage” options “only build jobs with labels” so every jobs launched will be set in this worker node

And also activate the button “Restrict where the project can be run”, will allow you to type the node in which you want the job to run.

As Result now under the slave node1, you can see that second project or jobs have been tied successfully:

Setup a second worker node with Windows guest Os from VB:

Same thing as the first worker , we have to install first java runtime environment then associate the ip of this machine to Master node and know that the protocol used by windows to access master Jenkins node is jnpl .

Step1) Install java jdk on the windows machine and check if it exist like:

Step2) Go to Master Node and and a second node :

Just click under Dashboard>Nodes> click New Node then you will be presented with the page like:

Setting like

**Under Launch method**, you have to choose “Launch agent by connecting to master” as method of connection because the node is a windows machine and it will contact the Master node to instantiate a connection.

Now The second node come but not connected so we have to disable some security control:

Click on “Go to security configuration screen and change it” to modify security control. And after , under “Agents” choose “Random” point and Save.

After go and click again on the “Windows node2”, you will see:

Step 3) Now go login into your windows Machine and type in the windows console the following cmd: javaws http://192.168.43.85:8080/computer/windo_node2/jenkins-agent.jnlp or You can use the second option if you want to make a background connection.

After typing in the console the cmd you will be presented with above output

Master connected successfully:

2 Nodes connected successfully to Master Node

Setup a third node with AWS ec2 instances:

First login to your ec2 instance and install Java jdk because Jenkins agent works on java:

After login, install with sudo yum install java-1.8.0-openjdk:

First launch one Ec2 instance in AWS and if you need help take this Ref:https://docs.aws.amazon.com/quickstarts/latest/vmlaunch/step-1-launch-instance.html

Now go to Master Node in jenkins to add one more node like:

i) under “#of executors” you can set any number you want and this define the no of jobs that can be set in parallel.

And under “Remote root Directory”, you have to put the path to a directory/workspace of worker node which will be used my the Master node to manage it’s staff. if not set then go to login to ec2 instance and create thios folder like:

ii) Under Launch method, you have to choose ssh agent as method of connection because the worker node here is a linux machine on which the Master will instantiate a connection.

iii) Under Host, just type the public ip_address of your ec2 instance created from AWS :

iv) Under Credentials, click add to enter the username “ec2-user” and private-key which you gain while launching this instance and click add to save it.

v) Under Host Key Verification Strategy, select “Non verifying Verification Strategy” => this will avoid the issue of prompting to enter password when Master will try ssh connection to worker node.

And finally you should have this output and click save:

And awsLinus node joined successfully Master Node:

Now let’s test our ec2 instance node by launching a jobs in Master which will install docker and launch one container inside of our slave node:

See we don’t have docker installed inside

Launch a job now in Master node

And at the next page

Note: Activate the button “Restrict where the project can be run”, will allow you to type the node of awsLinux where the job will be run. And “Build” > select “Execute shell” > then these commands to install docker in this slave node and launch a container inside it and click save :

sudo yum install docker -y
sudo systemctl enable — now docker
sudo docker ps
sudo docker run -dit — name cont1 httpd

After click “build now” button to start the job

From console output of Master you can see

docker installed and container launched successfully from a job of Master node

Clicking on aws ec2Linux node, you can see the recent job tied successfully

CONCLUSION

Congratulation we made it, now with one Master node we can execute different tasks for different nodes and all this independently of worker nodes we are using. Automation is taking a boom growth in big Industries and by this small demo i have demonstrate one of the Use case of Continuous Integration and Deployment tool ex of Jenkins in our case and it’s power.

Thank You !!! and See you in another story ⌛…….