Jenkins in practice – combinations of parallel and generic stages
2021-10-11
Jenkins is a powerful tool that allows you to automate processes that would otherwise have to be done manually. So it’s good to use it and use its capabilities to make your work easier. In this article we will show you how to automate repetitive tasks in various ways. Some seemingly simple Jenkins solutions and […]
Jenkins is a powerful tool that allows you to automate processes that would otherwise have to be done manually. So it’s good to use it and use its capabilities to make your work easier. In this article we will show you how to automate repetitive tasks in various ways.
Some seemingly simple Jenkins solutions and functions may cause some difficulties when used simultaneously. Dynamically generated stages (stage) and parallel stages may be problematic. That is why today we will show how to create several types of streams (pipeline) using different combinations of these functionalities, present their pros and cons and answer the question whether it is worth using them at all.
Parallel stages
Let’s start with the easier topic, i.e. the parallelization of our stages. As the name suggests, this allows us to carry out many stages simultaneously:
pipeline {
agent any
stages {
stage("tests") {
parallel{
stage("test1"){
steps{
sh "echo test1"
}
}
stage("test2"){
steps{
sh "echo test2"
}
}
}
}
}
}In the Blue Ocean view it looks like this:
Parallel stages in this way can operate on the same agent and in the same work area (as above). We can also define a separate agent for each of them:
pipeline {
agent none
stages {
stage("tests") {
parallel{
stage("test1"){
agent {
label "test-agent1"
}
steps{
sh "echo test1"
}
}
stage("test2"){
agent {
label "test-agent1"
}
steps{
sh "echo test2"
}
}
}
}
}
}We also have the option of setting our stages so that some of them are still sequential or put inside one of the parallel stages the next ones run one by one:
Examples of using parallel stages:
- performing many unrelated tests
- performing the same test scenarios on many systems simultaneously.
Disadvantages of parallel stages:
– slightly reduces the transparency of streams
+ significantly reduces the time it takes to carry out various processes (such as tests, for example) if they consist of independent stages.
Matrix
The Matrix allows us to define a matrix of stages run in parallel. It is an ideal tool when we need to perform the same operations on many systems and with different configurations.
When defining the matrix, we start by defining the axis:
matrix {
agent {label "${DISTRO}"}
axes {
axis {
name 'DISTRO'
values 'eurolinux', 'centos', 'scientific'
}
axis {
name 'BROWSER'
values 'firefox', 'chrome', 'opera'
}
}
stages {
stage("Test") {
steps {
echo "Do Build for ${DISTRO} - ${BROWSER}"
}
}
}
}We can define any number of axes. As a result, we get every possible combination of the values they contain:
As you can see, we can easily get functionality that would otherwise require multiple text swearing. This solution allows you to control the agent at which the stage is to be started, depending on the values selected on the axis. It also allows you to exclude selected combinations of values using excludes and much more. However, there are many possibilities. To read them, we encourage you to use documentation ;-)
Matrix pros and cons:
– due to the lack of an easy way to dynamically change the names of stages, the matrix becomes not very transparent, and to find the right variant, we need to hover over it with the pointer (mouse) to display its name.
+ the code is legible and clean
+ allows you to easily add successive stages.
Generic stages
Sometimes, however, we need more control over the course of stages or there are so many that the use of matrices becomes impractical. We can then generate such stages from the list. This requires the use of scripts and is as follows:
def OS_list = ["linux", "windows", "IOS", "android"]
pipeline {
agent eny
stages {
stage("Test all"){
steps{
script{
OS_list.each{
stage("$it"){
echo "$it"
}
}
}
}
}
}
}
Parallelization of generic stages
Creating such a stream is unfortunately not so easy. We must remember that parallel inside the buckle script behaves differently than the one that occurs in an ordinary declarative stream. In this case, he accepts the map. The keys will be displayed as the branch name, and the values should be the stages that need to be parallelized. At this point, we recommend experimenting as ways to create a map in groovy’m there is a lot and different combinations give different effects. Below is an example of generic parallel stages:
def OS_list = ["linux", "windows", "IOS", "android"]
pipeline {
agent none
stages {
stage("Test all"){
steps{
script{
parallel OS_list.collectEntries { OS -> [ "${OS}": {
stage("$OS") {
echo "$OS"
}
}]
}
}
}
}
}
}
This solution can be freely nested and parallelized:
def OS_list = ["linux", "windows", "IOS", "android"]
def BROWSERS = ["chrome", "firefox", "opera", "edge"]
pipeline {
agent none
stages {
stage("Test all"){
steps{
script{
parallel OS_list.collectEntries { OS -> [ "${OS}": {
stage("$OS") {
BROWSERS.each{
stage("$it"){
echo "$it"
}
}
}
}]}
}
}
}
}
}
Pros and cons of generic stages:
– requires the use of scripts inside the creek, which may reduce code transparency
– the use of scripts deprives us of the possibility of using some functionalities of the declarative stream, such as when
– no solution of the type excludes forces us to implement complicated mechanisms or the so-called ifologji
+ gives great freedom
+ we can freely nest stages
+ allows you to easily add successive stages.
Summary
Comparing the above solutions, we can see that the cleanest solution, due to the code, is to use a matrix. Unfortunately, this solution does not generate a stream that is transparent in browsing. The dynamically generated stages look much better, but require us to use scripts, minimal language skills groovy and deprives us of the possibility of using some functionalities. We hope that this material will help you in the future design and creation of jenkin streams.