CI (3)


Create a Jenkins Support Channel

When creating a large Jenkins setup, it is often a good idea to consider have a team governing, documenting, and managing the tools a little. These teams almost always require a clear support channel, otherwise everything becomes tribal knowledge, and that is not going to scale well.

Signature of a large Jenkins setup means it has

  • Multiple workers with varying operating systems
  • Multiple integrations to systems for static analysis, artifact repositories and code-review platform triggers such as pull-requests and patch sets
  • Support for an advanced authorization role strategy or single sign on from other platforms
  • Support for multiple build-systems such as MSBuild, Maven, Gradle, Ant, Make and so forth

When a large number of teams with a similarly large number of software development stacks starts to use it. Then it becomes relevant with being able to support the systems by providing a clear and obvious support-channel.

A common channel for issue-management is JIRA, which ironically started as an ITSM-platform. And another common place to store documentation is whatever source-control platform you have available, such as GitHub Pages, Wikimedia or Confluence.

So i would like to show how to include custom-integration in the Jenkins header directly into those tools.

Starting point

Configure JIRA Issue Collector

  • In JIRA, create a target project
  • Go to settings for the project
  • Go to Issue Collectors
  • Click + Add issue collector
  • Configure it as you may wish, make sure your select custom trigger-style
  • Submit the collector
  • Extract the link in the markup JIRA wants you to embed.

The collector-link will look something like this

https://YOUR-PROJECT.atlassian.net/s/aef99a9d9e96520927s6987-T/-p6br4e/b/5/a44af77267a987a660377e5c46e0fb64/_/download/batch/com.atlassian.jira.collector.plugin.jira-issue-collector-plugin:issuecollector/com.atlassian.jira.collector.plugin.jira-issue-collector-plugin:issuecollector.js?locale=en-US&collectorId=909eeccc

Configure Jenkins to embed collector and run some JavaScript

  • Go to Manage Jenkins
  • Go to System Configuration
  • Fill in Issue Collector URL: with the collector-link from the previous step.
  • Fill in the Theme Javascript part with an accessable location where you save this JavaScript. SonaType Nexus rawFiles-store works nicely for this purpose.
  • Get the following html
  • Put it into the system message
  • preview html should now put two buttons in the header
  • If you want you can include custom styling to the buttons in the simple theme css box. I used the following to achieve the looks below.

And voila! You have opened a support channel directly from your Jenkins header into your documentation and added support for adding JIRA issues directly into your JIRA project! Clicking the Report an issue! will open a JIRA Modal and allow your users to fill in a support issue as their own JIRA users.

Issue collectors can be customized to pre-populate fields quite extensively, read more on Atlassians own documentation.

#




Essential Jenkins CI

The great enablers of any successful DevOps is Continuous Integration (CI), Delivery and Deployment(CD). CI is a workflow at the core while delivery and deployment are extensions on top of that workflow. Described plainly it is the automation of manual work within software development. I have written about this before.

But in essence it is about letting something neutral build and compile all new and existing code, run unit and integration tests all the time as new changes are pushed. This can, and should, of course be run by the same developer who should also create more F.I.R.S.T class tests. It would be wrong to say that CI is about distrusting developers to be professionals. But it is about creating safeguards from accidents. By failing the build when something is wrong as early as possible, and also notifying stakeholders and developers that committed since the last build, we have a better change to fix it early.

This also has the side-effect that it encourages a workflow that makes developers to rely on a clean, compiling mainline with green tests. sometimes this also leads to a state where the mainline is always releasable or better yet, always deployed to production. It becomes a point of truth to the current state of the code. This also encourages creating more, and better, tests at all levels in the test-pyramid.

The way most teams accomplish this is by using a build or CI server. Now these days there are a few on the market, but the most popular one by far is Jenkins, likely because it is the only one licensed under the Creative Commons licensing, giving a’lot of flexibility to a very vibrant plugin community which likely contributes to its ever growing popularity. That is why this article will describe the essentials when working with Jenkins.

Meet Jenkins

At its core, Jenkins is not much more than a bunch of cron jobs with a web frontend. It is highly extendable with plugins that affect the system as a whole or that adds additional build-steps and triggers to a job.

Jobs

You can make a job run practically any build system for any language and collect the reports and results to the job dashboard. It can be integrated to other platforms such as Sonar to statically analyze code and look for obvious bugs. Jobs can also be made a part of the code-review process even before code is merged to a mainline. The flexibility is what makes it powerful.

In the dashboard you can also find the workspace link that takes you to the folder where the project has been built. This is good if you are debugging or setting your job up to understand what is happening behind the console log as you can inspect the files here.

General

Every job consists of a general section that allows you to describe, provide links to source control, and define any parameters that must be provided when triggering (starting) the job. Parametarized tests are a great way to provide a simple interface for less-technical users in Jenkins. Like running system tests towards different environments using a subset of all testcases. Where a choice parameter becomes a dropdown of different test servers to deploy it to and a CSV String parameter specifies the testcases to run.

SCM

This is the part where you tell Jenkins where to checkout code from. By default it supports GIT and SVN for source control. It can also support other SCMs through plugins such as Mercurial and CVS, why you would ever need to use them when GIT exists, but they exist.

Build Triggers

Jobs can be configured to trigger at specific times, after other jobs have executed, manually or trigger on SCM. SCM is a trigger on something new that has been pushed to a source code repository. But there can also be triggers that can come webhooks from pull-requests (GitHub, BitBucket), merge-requests (GitLab), Patch-sets(Gerrit) and so on.

Environment

Some plugins can inject variables that can be used throughout the job, use secrets and managed property files. These variables can usually be used in all fields that a configurable in the Jenkins job using the $VARIABLE notation. Jenkins also has a bunch of default environment variables that you can use at your leisure.

Build Steps

These steps are the bread and butter of a job, and you can do anything from write pure sh and bat scripts inline that Jenkins should execute to run build systems such as ANT, Maven, Gradle, MSBuild or dotnet to triggering SonarScanner in the OS. If you find yourself having more than five build steps then you should probably consider breaking the job up into several jobs instead of a super-job.

Post steps

These steps are used mostly to collect test and analysis reports from the workspace to show in the dashboard. It is also used to send notifications by email with the email notification step, slack or even call remote APIs to notify code-reviews that the build succeeded.

Standard Jobs in most projects

A common setup of Jenkins jobs that can usually be set up no matter which stack or systems you are building. Ranked in the order of priority.

  • CI Build: Compile, test and build artifacts that triggers on SCM against mainline, artifacts deployed to artifact storage to be possibly deployed later.
  • Sonar CI Build:: Static code analysis that triggers daily, weekly, monthly or after a release.
  • Pre-merge Build: Compile and test that triggers as part of code-review
  • Sonar pre-merge Build: Static code analysis that triggers as part of code-review and reviews the change
  • Deploy to test job: Deploy the latest artefact from CI Build or specific branch.
  • Release Job: Perform a release, deploy the release to artifact storage and deploy it to production.

Good luck with your Jenkins endeavours adopting CI/CD!

Comment on below if you found this useful or if i said something stupid.




Continuous Integration as a shared place of truth

  • 2018-04-17
  • CI
  • #Dev

I sometimes get to advocate the practice Continuous Integration and Delivery as part of a DevOps culture. Mostly for developers but sometimes managers and testers. If you ended up here then you most likely have some idea on what it is about, but to be sure even the business people are in on it let’s rehash the components that make up Continous Integration, Delivery and Deployment. As can be read from the words of everyones favourite self-declared loud-mouth on the design of enterprise software Martin Fowler. This is a process that can be split into a these key components.

Continuous Integration

  • Keep a single source code repository
  • Have one long-living mainline and keep the number of branches short
  • Automate a self-contained build towards the mainline
  • Automate tests as part of the build and thus make it self-testing, avoid end to end tests.
  • Everyone commits to mainline every day
  • Every commit should build the mainline on a machine separate from the team
  • Broken builds are fixed immediately
  • Keep the Build Fast
  • Test in a clone of the production environment
  • The latest executable or artifact is easy to find and start or deploy
  • Everyone can see what is happening

Continuous Delivery

Adds a final step to the continuous integration cycle that makes sure that the mainline is always deployed to a test-system whenever something is merged into mainline. Alternatively it is very easy to make an automated deployment from a particular feature branch or released version.

Continuous Deployment

Everything that is merged into mainline eventually goes into production if it passes all the tests, quality checks with all its bells and whizzles.

What’s the big deal?

CI/CD is intended not just to speed up development, but to help coordinate it. Having a shared mainline from which all new feature branches are created allows for an obvious point of integration and short-lived branches. If you always build what is going in to the mainline and always build what is on the TRUNK or HEAD then you have an automatic system that builds trust and supports the developers. This allows merging code into the mainline several times a day because you dare to do it. You dare to do it because there is a built trust that the system will be unforgiving if you do something stupid. I’d like tos uggest that it then acts as a shared truth between all developers that use it.

Shared truth

This shared truth is a powerful tool when it is stable and proven within a team. It allows the team to focus on what is important, features, because it can trust that the build will stop them from doing something stupid that breaks the tests or introduces unneccesary technical debt. But only as long as the build is fast, doesn’t produce false-positives and every commit is built. A flaky build, with tests that toggles or works half of the time undermines the shared truth, because if it can no longer be trusted then it is not really truth. Exactly like people, if they sometimes speak the truth and sometimes lie, then you cannot trust them. But if you have an impeachable person in your team who you can really trust to always rightfully stop you from doing something stupid, then that will truly empower a team.