Setting Up a CI/CD Pipeline with Jenkins

Table of Contents

Jenkins, an open-source automation server, is one of the most widely used tools for setting up CI/CD pipelines. Its flexibility, extensive plugin ecosystem, and strong community support make it a popular choice for organizations of all sizes. Whether you’re managing a small project or orchestrating complex enterprise-level workflows, Jenkins provides the tools you need to automate and streamline your software delivery process.

Note: If don’t really understand what Continuous Intergration and Continuous Deployment is, explore our article for a quick, 5-minute breakdown of Continuous Integration and Continuous Deployment.

This guide will demonstrate how to install, configure, and create a simple CI/CD pipeline using Jenkins.

Prerequisite

Before you begin:

Deploy a server with your preferred operating system.
Install Java (JDK 11 or later) on the server.
Install Git on the server to manage code repositories.
(Optional) Install Docker if you plan to build and deploy containerized applications.
Access the server using SSH as a user with administrator privileges.

Installing Jenkins

Now that your server is ready, it’s time to install Jenkins. Follow the steps below to get Jenkins up and running on your system.

Add Jenkins GPG Key and Jenkins repository

sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
https://pkg.jenkins.io/debian/jenkins.io-2023.key
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \
https://pkg.jenkins.io/debian binary/ | sudo tee \
/etc/apt/sources.list.d/jenkins.list > /dev/null

Update the Package List

sudo apt update

Install Jenkins

sudo apt install jenkins

After installing Jenkins, start the Jenkins service:

sudo systemctl start jenkins
sudo systemctl enable jenkins

Access Jenkins UI

With Jenkins running, you can now access the Jenkins UI to complete the initial setup:

Open a web browser and navigate to http://<your-server-ip>:8080.

You will be prompted to unlock Jenkins. Find the initial admin password by running the following command on your server:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Enter the password in the Jenkins UI and proceed with the setup wizard.
You will be prompted to install suggested plugins or select specific plugins. It’s recommended to install the suggested plugins for a basic setup.

Create your first admin user and complete the setup.

Once the setup is complete, you’ll be redirected to the Jenkins dashboard, where you can start configuring your CI/CD pipeline.

Configuring Jenkins

With Jenkins installed and accessible, the next step is to configure it to suit your CI/CD pipeline needs. Follow the steps below to set up credentials for accessing external services, and configure global tools that Jenkins will use during the build and deployment processes.

Install Plugins

Jenkins’ true power lies in its extensive plugin ecosystem, which allows you to tailor the system to your specific needs. To get started with a CI/CD pipeline, you’ll need to install several essential plugins:

Git Plugin: Enables Jenkins to interact with Git repositories, allowing you to clone, build, and push code.
- Go to Jenkins Dashboard > Manage Jenkins > Manage Plugins > Available.
- Search for Git Plugin and install it.
Pipeline Plugin: This plugin is crucial for defining and running Jenkins pipelines using a Jenkinsfile. It allows you to automate your build, test, and deployment processes in a scripted manner.
- Search for Pipeline and install it.
Docker Pipeline Plugin (Optional): If you plan to use Docker within your CI/CD pipeline, this plugin integrates Docker commands into your Jenkinsfile, enabling you to build and manage Docker containers as part of your pipeline.
- Search for Docker Pipeline and install it.
Credentials Binding Plugin: This plugin simplifies the process of securely managing and using credentials (like API tokens or passwords) in your pipelines.
- Search for Credentials Binding Plugin and install it.

Once installed, these plugins will expand Jenkins’ capabilities, allowing you to build more sophisticated and automated pipelines.

Set Up Credentials

To interact with external services like GitHub, Docker Hub, or any other third-party systems, Jenkins needs to be able to authenticate securely. You can set up and manage credentials in Jenkins as follows:

Navigate to Credentials:
- Go to Jenkins Dashboard > Manage Jenkins > Manage Credentials.
Add Credentials:
- Select the appropriate domain (usually Global unless you’re segmenting credentials by project).
- Click (global) under Stores scoped to Jenkins, then Add Credentials.
Select the Credential Type:
- For GitHub or GitLab, choose Username with password or Secret text (for API tokens).
- For Docker Hub, select Username with password.
- Enter the required information, such as the username, password, or API token, and provide a meaningful ID for reference.
Save Credentials:
- Once you’ve added your credentials, they will be securely stored within Jenkins and can be referenced in your pipelines.

Setting up credentials ensures that Jenkins can seamlessly interact with other services, such as pulling code from a Git repository or pushing Docker images to a registry, without exposing sensitive information.

Configure Global Tools

For Jenkins to build, test, and deploy your applications, it needs access to various development tools. Configuring these global tools ensures that Jenkins knows where to find them on your system.

Navigate to Global Tool Configuration:
- Go to Jenkins Dashboard > Manage Jenkins > Global Tool Configuration.
Set Up Git:
- Under Git, click Add Git.
- If Git is installed in a standard location, Jenkins should auto-detect it. If not, specify the path to the Git executable.
Set up Maven (If applicable):
- If your project uses Maven for building, scroll to Maven, click Add Maven, and either let Jenkins install it automatically or specify the installation path.
Set Up Gradle (If applicable):
- Similar to Maven, under Gradle, click Add Gradle, and choose to install automatically or specify the path.

Once these tools are configured, Jenkins will be able to invoke them during pipeline execution, ensuring that your builds run smoothly regardless of the technologies involved.

Creating a Simple Pipeline

With Jenkins configured and ready to go, the next step is to create a pipeline that automates your build, test, and deployment processes. Follow the steps below to set up a basic pipeline, set up a new job to write a Jenkinsfile, configure build triggers, and run the pipeline.

Create a New Job

Jenkins allows you to create different types of jobs, each suited to specific tasks. For a CI/CD pipeline, the Pipeline job type is typically the best choice as it offers the most flexibility and is designed to work with Jenkinsfiles.

Navigate to New Item:
- From the Jenkins Dashboard, click New Item.
Choose Job Type:
- Enter a name for your job (e.g., MyFirstPipeline).
- Select Pipeline from the list of job types and click OK.
Configure the Job:
- After selecting the job type, you’ll be taken to the job configuration page where you can define how the pipeline should behave.
The Pipeline job type is ideal for complex CI/CD processes because it allows you to script the entire pipeline in a Jenkinsfile, making it easy to version control and maintain.

Write a Jenkinsfile

The Jenkinsfile is a text file that defines your CI/CD pipeline as code. It’s stored in your project’s source control repository and describes the steps Jenkins should take to build, test, and deploy your application.

Understand the Jenkinsfile Structure:
- A basic Jenkinsfile typically consists of stages like Build, Test, and Deploy.
- Each stage contains steps that define the actions Jenkins will perform.

Create a Simple Jenkinsfile:

pipeline {
    agent any

    stages {
        stage('Clone Repository') {
            steps {
                git 'https://github.com/your-repo/your-project.git'
            }
        }
        stage('Build') {
            steps {
                // Example build step
                sh 'make build'
            }
        }
        stage('Test') {
            steps {
                // Example test step
                sh 'make test'
            }
        }
    }
}

pipeline: The top-level block that defines the entire pipeline.
agent: Specifies where the pipeline should run. any means it can run on any available agent.
stages: A block that contains all the individual stages of the pipeline.
stage: Defines a single stage in the pipeline, like Clone Repository, Build, or Test.
steps: The commands or actions Jenkins should execute within a stage.

Add the Jenkinsfile to Your Repository:
- Save the Jenkinsfile in the root directory of your project’s repository.

Configure Build Triggers

Build triggers determine when and how your pipeline runs. Jenkins supports various triggers, from manual starts to automated triggers based on external events.

Navigate to Build Triggers:
- In your job’s configuration page, scroll down to the Build Triggers section.
Choose a Trigger Method:
- Poll SCM: Jenkins will periodically check your repository for changes. Specify the polling frequency (e.g., H/5 * * * * for every 5 minutes).
- GitHub Hook Trigger: If you’re using GitHub, configure a webhook to trigger the pipeline whenever there’s a new push or pull request.
- Build Periodically: Set up a cron-like schedule to run the pipeline at specific intervals.
- Manual Trigger: You can also trigger builds manually from the Jenkins dashboard.
Configure the Trigger:
- For webhooks, ensure that your repository is configured to send POST requests to the Jenkins server when changes occur. Jenkins will listen for these requests and trigger the pipeline accordingly.

Execute the Pipeline

Once everything is set up, it’s time to run the pipeline and see it in action.

Run the Pipeline:
- From your job’s page, click Build Now to manually trigger the pipeline.
- If you’ve configured automatic triggers, the pipeline will start whenever the specified event occurs (e.g., a code push).
Monitor the Build Process:
- Jenkins will display the progress of each stage in real-time. You can view logs, check the status of each stage, and troubleshoot any issues that arise.
- If the pipeline completes successfully, all stages will show a green checkmark.
Review the Results:
- Once the pipeline finishes, review the build logs and results. Jenkins will provide detailed information on what was executed, any errors that occurred, and the overall success or failure of the pipeline.

Integrating with Version Control

Integrating Jenkins with version control systems like GitHub or GitLab is essential for automating your CI/CD pipeline. Follow the steps below to set Jenkins to automatically trigger builds when changes are pushed to the repository, ensuring continuous integration and delivery.

Connecting Jenkins to GitHub

To get started, you need to connect Jenkins to your GitHub or GitLab repository. This connection allows Jenkins to receive notifications (webhooks) when changes occur in your repository and to authenticate securely when accessing your codebase.

Configure Authentication:
- Generate a Personal Access Token:
  - For GitHub: Go to GitHub > Settings > Developer settings > Personal access tokens > Generate new token. Select the necessary scopes, such as repo and admin:repo_hook.
- Add the Token to Jenkins:
  - In Jenkins, go to Manage Jenkins > Manage Credentials > (global) > Add Credentials.
  - Choose Secret text for the token, enter it, and give it a meaningful ID (e.g., github-token).
Set Up Webhooks:
- Go to your repository on GitHub, then Settings > Webhooks > Add webhook. Set the Payload URL to http://your-jenkins-url/github-webhook/ and select application/json as the content type. Choose which events will trigger the webhook, usually Just the push event and Pull requests.
Configure the Jenkins Job:
- Link to Your Repository:
  - In your Jenkins job configuration, under the Pipeline section, specify the URL of your GitHub. Use the format https://github.com/your-repo/your-project.git.
- Use Credentials:
  - Ensure that Jenkins uses the credentials (personal access token) you added earlier. Select them from the Credentials dropdown in the SCM section of your job configuration.

With authentication and webhooks in place, Jenkins is now connected to your GitHub or GitLab repository. This setup ensures that Jenkins can pull your code and trigger builds automatically.

Branch and Pull Request Builds

To fully leverage Jenkins in a collaborative development environment, it’s important to configure it to build feature branches and pull requests. This setup allows for early detection of integration issues, improving the overall quality of your code.

Set Up Multibranch Pipeline:

A Multibranch Pipeline job in Jenkins can automatically create pipelines for all branches in a repository, building each one based on its own Jenkinsfile.
- Create a Multibranch Pipeline Job:
  - From the Jenkins Dashboard, click New Item and select Multibranch Pipeline. Name your job (e.g., MyProject-Multibranch).
- Configure Branch Sources:
  - In the job configuration, under Branch Sources, add your GitHub. Use the credentials you configured earlier.
  - Set the Scan Multibranch Pipeline Triggers to check for new branches or updates periodically or in response to webhook events.
Branch Build Configuration:

Jenkins will automatically discover and build all branches in your repository that contain a Jenkinsfile. You can customize the behavior for different branches.
- Exclusions and Filters:
  - You can exclude certain branches from being built by configuring the Property Strategy within the branch sources. For example, you can set a regular expression to exclude branches with certain naming patterns.
- Specific Branch Configuration:
  - For more control, you can define specific behaviors for branches by using the when directive in your Jenkinsfile. For example, you can specify that certain stages only run for the main branch:
```
stage('Deploy') {
    when {
        branch 'main'
    }
    steps {
        // Deployment steps here
    }
}
```
Pull Request Builds:

Jenkins can automatically build pull requests, allowing you to test proposed changes before they are merged into the main branch.
- Enable Pull Request Builds:
  - In the Multibranch Pipeline job configuration, Jenkins automatically detects pull requests if you are using a supported source like GitHub or GitLab. These pull requests will appear as separate branches and can be built just like any other branch.
- Set Up Status Reporting:
  - Jenkins can report the status of the pull request build back to GitHub or GitLab. This integration allows you to see whether the build passed or failed directly within the pull request.
  - Ensure that your personal access token has the necessary permissions (repo:status for GitHub or api for GitLab).

Adding Notifications

To keep your team informed about the status of your builds and deployments, it’s important to set up notifications in Jenkins. Notifications can be sent via email or integrated with collaboration tools like Slack or Microsoft Teams, ensuring that everyone is updated on the success or failure of each pipeline.

Email Notifications

Email notifications are a basic yet effective way to alert your team about build results. Jenkins allows you to configure these notifications to be sent automatically at various stages of the pipeline.

Install Email Extension Plugin:
- Go to Jenkins Dashboard > Manage Jenkins > Manage Plugins > Available and search for Email Extension Plugin. Install it if it’s not already installed.
Configure SMTP Server:
- Go to Manage Jenkins > Configure System.

Scroll to the Extended E-mail Notification section and enter the SMTP server details:
- SMTP server: The address of your mail server (e.g., smtp.yourdomain.com).
- SMTP port: Typically 587 for TLS or 465 for SSL.
- Use SMTP Authentication: Check this box if your server requires authentication, and enter the username and password.
- Use SSL/TLS: Check this if your SMTP server requires SSL or TLS.

Configure Email Notifications in the Pipeline:

You can configure email notifications directly within your Jenkinsfile to send alerts based on the build status.


pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                sh 'make build'
            }
        }
    }

    post {
        success {
            emailext subject: "Build Success: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
                     body: "Good news! The build succeeded.",
                     to: 'team@example.com'
        }
        failure {
            emailext subject: "Build Failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
                     body: "Oops! The build failed. Please check the logs.",
                     to: 'team@example.com'
        }
    }
}

post: The post block defines actions that run at the end of the pipeline, such as sending emails.
success and failure: These sections define what happens when the build succeeds or fails. The emailext step is used to send the email.

Slack or Teams Integration

For teams using collaboration tools like Slack or Microsoft Teams, integrating Jenkins with these platforms provides a seamless way to receive build notifications directly in your team channels.

Slack Integration:
1. Install the Slack Notification Plugin:
  Go to Jenkins Dashboard > Manage Jenkins > Manage Plugins > Available and search for Slack Notification Plugin. Install it and restart Jenkins if necessary.
2. Configure Slack Integration:
  - Go to Manage Jenkins > Configure System.
  - Scroll down to the Slack section and configure it:
    - Workspace: Enter your Slack workspace.
    - Credential: Add your Slack integration token (this can be obtained from Slack by creating an incoming webhook or using an app like Jenkins CI).
    - Default Channel: Specify the default Slack channel where notifications should be sent.
3. Add Slack Notifications to Jenkinsfile:
```
pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                sh 'make build'
            }
        }
    }

    post {
        success {
            slackSend channel: '#build-notifications',
                      color: 'good',
                      message: "Build Success: ${env.JOB_NAME} #${env.BUILD_NUMBER}"
        }
        failure {
            slackSend channel: '#build-notifications',
                      color: 'danger',
                      message: "Build Failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}"
        }
    }
}
                
```

Microsoft Teams Integration:

Create an Incoming Webhook in Microsoft Teams:
In your Microsoft Teams, create an incoming webhook for the channel where you want to receive notifications.

Go to Teams > More options (⋯) > Connectors > Incoming Webhook. Name the webhook and copy the URL provided.
Install Microsoft Teams Notification Plugin:
Install the Office 365 Connector Plugin by going to Jenkins Dashboard > Manage Jenkins > Manage Plugins > Available and searching for Office 365 Connector Plugin. Install it and restart Jenkins if necessary.
Configure Teams Integration:
Go to your Jenkins job’s configuration page and scroll down to the Office 365 Connector section. Add the webhook URL obtained from Teams.

Add Teams Notifications to Jenkinsfile:


pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                sh 'make build'
            }
        }
    }

    post {
        success {
            office365ConnectorSend webhookUrl: 'https://outlook.office.com/webhook/your-webhook-url',
                                   status: 'Build Success: ${env.JOB_NAME} #${env.BUILD_NUMBER}',
                                   color: '00FF00'
        }
        failure {
            office365ConnectorSend webhookUrl: 'https://outlook.office.com/webhook/your-webhook-url',
                                   status: 'Build Failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}',
                                   color: 'FF0000'
        }
    }
}