Error script not yet approved for use

In part one of this three-part article series, I showed you how to install and configure Jenkins – and how to create build and release jobs for Maven projects via the Jenkins user interface. In part two, we programmatically created the same jobs with the Jenkins Job DSL and made the job list cleaner using views.

In this concluding third part, you will learn how to extract duplicated Groovy code into utility classes and how you can automatically create Jenkins jobs for new Java projects in the Git monorepo.

You can find the complete code created in this article series in my GitLab repository https://gitlab.com/SvenWoltmann/jenkins-tutorial-demo.

Jenkins security system «Script Security»

Before we proceed with the automation, it is essential to familiarize yourself with the Jenkins Script Security system. Previously, we were able to run all scripts without any problems because we created them as administrators. Jenkins maintains a whitelist of allowed scripts, to which it adds all scripts created by an admin. Regular users can later execute these scripts.

However, this only applies to scripts that an admin copies directly into a job. In the course of further automation, we want to configure Jenkins to download scripts from the Git repository. Downloaded scripts are not executable by regular users or the «SYSTEM» user. By default, a job is executed as the «SYSTEM» user – even if an admin starts it. Jenkins’ security system protects us from potentially dangerous groovy code that could be infiltrated by downloading from the git repository.

How to make downloaded scripts executable?

There are different ways to make downloaded Groovy scripts executable:

• Non-executable scripts automatically end up in a validation queue. An admin can manually approve them under «Manage Jenkins» → «In-process Script Approval». However, he would have to do this again each time we changed the Groovy code. This procedure could prove to be impracticable – depending on the composition of the team.
• Configure the security system so that the seed job always runs as an admin (by default, it runs as the SYSTEM user). This is insecure as it could introduce potentially dangerous code. Besides, this variant does not allow us to extract shared code into utility classes.
• Disable Script Security altogether («Manage Jenkins» → «Configure Global Security» → Disable checkbox «Enable script security for Job DSL scripts»). This is even more insecure, as any user could execute potentially dangerous scripts in any job. In the previous option, we reduced the risk at least to Groovy files downloaded by the seed job.
• Run the code in the «Groovy Sandbox». In the sandbox, the code is executable in principle but is limited to a fixed set of operations. This option only works together with the execution as an admin user. Otherwise, the scripts would have to be unlocked again via the validation queue. The sandbox prevents the execution of malicious code. This option is the only one that allows extracting code into utility classes.

In my opinion, the «Groovy Sandbox» in combination with the execution of the job as an admin user is the most useful option. In the following, I show you how to configure the security system accordingly.

Global configuration of Jenkins Script Security

First, we have to activate the possibility to configure the permissions for single jobs. To do this, we access the global security settings via «Manage Jenkins» → «Configure Global Security». There we have to add the «Per-project configurable Build Authorization» under «Access Control for Builds»:

In the checkboxes that appear, the strategy «Run as User who Triggered Build» should already be activated. We also have to tick the option «Run as a specific user», so that we can later configure a job always to run as an admin user.

We save the changes via the «Save» button. All further settings are done later in the Jenkins job configuration.

Jenkins Script Security error messages

In the following, I list the error messages that you might get from the security system. I also list the respective causes and possible solutions. You can skip this section for now. You can return to this section if you encounter any security-related error messages later in the tutorial.

ERROR: script not yet approved for use

Jenkins displays this error message when a job tries to execute a script that has not been whitelisted by an admin (either explicitly, or by the admin manually copying the script into a job).

Possible solutions:

• Jenkins automatically places the script into the validation queue. An admin can manually release it under «Manage Jenkins» → «In-process Script Approval».
• Execute the job as an admin user: Job page → «Authorization» → Activate «Configure Build Authorization» → Set «Authorize Strategy» to «Run as Specific User» or «Run as User who Triggered Build» and either select an admin user or start the job as an admin.

ERROR: startup failed: […] unable to resolve class […]

You get this error message when a Groovy script that does not run in the Groovy Sandbox tries to access another Groovy class on the file system.

Solution:

• The only way to access other classes is to execute the code in the Groovy Sandbox: Job configuration → «Process Job DSLs» → Tick «Use Groovy Sandbox».

ERROR: You must configure the DSL job to run as a specific user in order to use the Groovy sandbox

This error message appears if you have enabled the Groovy Sandbox for a job, but have not configured the security system to run the job as a specific user. By default, a job always runs as the «SYSTEM» user.

Solution:

• Run the job as a specific user: Job page → «Authorization» → Activate «Configure Build Authorization» → Set «Authorize Strategy» to «Run as Specific User» or «Run as User who Triggered Build». If you set a specific user, they should have admin rights again. Otherwise, you get the error message «script not yet approved for use» as long as an admin did not explicitly approve the script.

ERROR: Scripts not permitted to use method […] / staticMethod […]

This error message indicates that it is not allowed to call the particular method in the Groovy sandbox.

Solution:

• Jenkins automatically puts the method signature into the validation queue. An admin must manually approve it under «Manage Jenkins» → «In-process Script Approval».

ERROR: […] No signature of method: […] is applicable for argument types: […]

This error occurs when trying to configure the sparse checkout inside the Groovy sandbox using the configure() method shown above.

Solution:

• Instead of using the configure() method, the extensions must be configured using NodeBuilder (as shown above).

Job creation automation

In the previous article, we wrote Jenkins jobs as code. The code is located in the files BuildJobApplication1.groovy, BuildJobLibrary1.groovy, ReleaseJobApplication1.groovy, ReleaseJobLibrary1.groovy in the directory jenkins-jobs/src/jobs/ of our monorepo. (If you don’t work with a monorepo, you can also use a separate repository for the Jenkins jobs.)

To import the jobs into Jenkins, we manually created creator jobs, added one build step «Process Job DSLs» to each job, copied the Groovy code into it, and executed the job. If we want to change something in a job, we have to copy the code again into the respective creator job. With four jobs, this is still feasible – with a few hundred, it is no longer practicable. The previous procedure also had the disadvantage that we had to duplicate code that we needed several times instead of being able to extract it to helper classes.

Loading the Jenkins Job DSL code from the project repository

First of all, we don’t want to copy the code manually into the creator jobs, but let so-called loader jobs load the code from the Git repository. That’s pretty easy to set up.

I show you how to do this using the «Jenkins Tutorial Demo — Library 1» job as an example. You can apply the same procedure to the other three jobs.

On the Jenkins homepage, we create a new job via «New Item». We name it «Jenkins Tutorial Demo — Library 1 (DSL) Loader», as type, we choose «Freestyle project» once again. Under «Build», we add a «Process Job DSLs» step. So far, the process is identical to creating the creator jobs (therefore I refrain from showing screenshots until here).

Instead of selecting «Use the provided DSL script», we leave the default selection «Look on Filesystem». To make the job file available in the file system, we have to clone the GitLab repository first. So we scroll back up to «Source Code Management», select «Git» and enter the repository URL «[email protected]:SvenWoltmann/jenkins-tutorial-demo.git»:

Under «Additional Behaviours», we add «Sparse Checkout Paths» and enter jenkins-jobs/ so that only this directory is checked out. (If you have the Jenkins jobs in a separate repository, you can omit this step.)

Now we scroll down again to «Build» and enter the path of the Groovy file, jenkins-jobs/src/jobs/BuildJobLibrary1.groovy, in «DSL Scripts». We also activate the option «Use Groovy Sandbox» (as explained in the previous chapter):

We click on «Save». Before we can execute the job, we must specify that Jenkins has to run the job as an admin user (also explained in the previous chapter). To do this, we click on «Authorization» on the job page. (If you don’t see this point, please check if you have performed the steps in the Global configuration of Jenkins Script Security section correctly.)

On the «Authorization» page we activate «Configure Build Authorization» and select either «Run as User who Triggered Build» or «Run as Specific User» as the strategy. The difference is as follows:

• Run as User who Triggered Build: The job must ultimately be started by an admin user to execute the downloaded Groovy code (unless an admin has already whitelisted it).
• Run as Specific User: Use this option to specify a user with whose rights Jenkins executes the job – independent of the logged-in user. If you provide an admin user here, every logged-in user can start the job.

Since I am the only user on my Jenkins server, I choose the option «Run as User who Triggered Build»:

We save the changes and start the job with «Build Now». The job aborts with an error message despite all security-relevant settings we have made:

We cannot avoid this error message. It says that calling the specified method is not allowed within the Groovy sandbox. At the same time, Jenkins has written the method signature into the validation queue, where we can then approve it as follows. We reach the approval form via «Manage Jenkins» → «In-process Script Approval»:

We click on «Approve» and then find the method signature in the list of released signatures.

We have to repeat the steps «Run job» and «Approve method signature» three more times so that the list of approved signatures finally contains four entries:

We’ll start the job a fifth time. Now it runs successfully:

Even if we disregard approving method signatures (which we won’t have to repeat), it was still quite a manual effort. If we had to do that for the release jobs and the «application1» project, too, we wouldn’t have gained much over the previous fully manual approach. In the next section, I show you how to simplify this.

Creating multiple Jenkins jobs with a single seed job

To generate the three remaining jobs (the build job for the «application1» project and the two release jobs) and the view, we do not need to create any additional loader jobs. Instead, we can add the four additional Groovy files to the previously created loader job under «DSL Scripts»:

Since the loader job no longer only creates the build job for the «library1» project, we can rename it from «Jenkins Tutorial Demo — Library 1 (DSL) Loader» to «Jenkins Tutorial Demo — Seed Job». After rerunning it, we see the following status message:

We’ve successfully created four Jenkins jobs and one view with a single job. That was fast now!

What happens to jobs that already exist? If nothing has changed in the Groovy code, the job remains unchanged. If the Groovy code is changed, the job is adjusted. The workspace and the build history of the job remain unchanged.

In the last interim conclusion, I criticized that we duplicated some code (e.g., the code block for the sparse checkout). When we manually imported the code into Jenkins in the previous article, we had no way to extract shared functionality. Now that the code is in the file system, we can very well extract repetitive code by creating utility classes that are used by all jobs.

We create three classes ScmUtils.groovy, ReleaseUtils.groovy and ViewUtils.groovy in the subdirectory util/ and extract most of the duplicated code there. We leave the previous Groovy files unchanged and add the following four job files and one view file:

jenkins-jobs/src/jobs/BuildJobLibrary1WithUtils.groovy
jenkins-jobs/src/jobs/BuildJobApplication1WithUtils.groovy
jenkins-jobs/src/jobs/ReleaseJobLibrary1WithUtils.groovy
jenkins-jobs/src/jobs/ReleaseJobApplication1WithUtils.groovy
jenkins-jobs/src/jobs/ViewDSLJobsWithUtils.groovy

You can find the new job/view files and utility classes in the GitLab repository. Listing them here would unnecessarily blow up the article. I have specified the complete paths above so that we can copy this list directly into the seed job and then run it. The job finishes successfully and has created four new jobs and one new view:

We can now quickly create build and release jobs for a new Java project and import them into Jenkins via the seed job. Most of the functionality can now be adjusted centrally in the utility classes. After a change, all we have to do is to rerun the seed job.

Upgrading the seed job

However, when we add a new project, we still have to a) copy and paste two Groovy files, b) add them to the seed job, and c) run the job.

Items b) and c) can be changed quite easily: instead of listing concrete Groovy files in the seed job, we can also use wildcards; and to avoid having to start the seed job manually, we can define a time-based trigger, for example.

First, we replace the list of specific file names with the following three wildcard entries:

jenkins-jobs/src/jobs/BuildJob*.groovy
jenkins-jobs/src/jobs/ReleaseJob*.groovy
jenkins-jobs/src/jobs/View*.groovy 

We could also specify jenkins-jobs/src/jobs/*.groovy here, but the jobs/ directory also contains the sample file SimpleDSLJob.groovy, and I want to keep some control over the files to execute.

To run the seed job automatically, we add a trigger that polls the Git repository for changes every 15 minutes:

For the trigger to work, we need to change the «Authorize Strategy» for the seed job. It is currently set to «Run as User who Triggered Build». The build trigger executes the job as user «SYSTEM». This user does not have permission to execute code in the Groovy sandbox. Therefore we have to change the strategy to «Run as Specific User» and specify a user with admin rights. For the sake of simplicity, I’ll leave my name here, but you can also create a separate user with admin rights for this.

The only thing we need to do now to create a new job is to create the corresponding Groovy files in the Git repository. Everything else happens automatically. Even this final manual step can be automated. I show you how this works in the next – and at the same, time the last chapter of this article series.

Generating build and release jobs for new projects fully automatically

Nowhere does it say that Groovy files must be mapped one-to-one to Jenkins jobs. Eventually, a Groovy file contains an executable program that, in the previous examples, calls the mavenJob() method once. We can extend such a program – like any other – with loops and branches and thus implement arbitrarily complex logic.

In the following section, we create two generic job generators: one for build jobs and one for release jobs. The generators search for pom.xml files in the monorepo and create a Jenkins job for each one found (unless it is a module of a multi-module project). This possibility is a significant advantage of the monorepo. If we wanted to implement this logic for projects distributed across multiple repositories, we would have to use an API of the repository server to read the available repositories and check them out.

The generic build job in the BuildJobGeneric.groovy file:

import javaposse.jobdsl.dsl.jobs.MavenJob
import util.ProjectUtils
import util.ScmUtils

final String repoUrl = '[email protected]:SvenWoltmann/jenkins-tutorial-demo.git'

List<String> projectPaths = ProjectUtils.findProjectPaths(__FILE__)

projectPaths.each { projectPath ->
  MavenJob job = mavenJob('Jenkins Tutorial Demo - Build ' +
        ProjectUtils.removeTrailingSlash(projectPath)) {
    description "Automatically generated build job for Maven project " +
          "detected in folder $projectPath."

    logRotator {
      numToKeep 5
    }

    triggers {
      scm 'H/15 * * * *'
      snapshotDependencies true
    }

    rootPOM "${projectPath}pom.xml"
    goals 'clean install'
  }

  ScmUtils.setupGit(job, true, repoUrl, projectPath)
}
Code language: Groovy (groovy)

The code first calls the ProjectUtils.findProjectPaths() method and passes the filename of the currently executed script, provided by Jenkins in the __FILE__ variable. The method returns a list of the directories of all Maven projects in the repository. We now iterate over this list and create a Jenkins job for each project. The code inside the loop corresponds to the code we created in the Extract shared code section (BuildJobLibrary1WithUtils.groovy and BuildJobApplication1WithUtils.groovy), with the only difference that we generate the job name and description from the project path.

The generic release job, ReleaseJobGeneric.groovy looks like this. Again, the code within the loop corresponds to the previously created code in ReleaseJobLibrary1WithUtils.groovy and ReleaseJobApplication1WithUtils.groovy.

import javaposse.jobdsl.dsl.jobs.MavenJob
import util.ProjectUtils
import util.ReleaseUtils
import util.ScmUtils

final String repoUrl = '[email protected]:SvenWoltmann/jenkins-tutorial-demo.git'

List<String> projectPaths = ProjectUtils.findProjectPaths(__FILE__)

projectPaths.each { projectPath ->
  MavenJob job = mavenJob('Jenkins Tutorial Demo - Release ' +
        ProjectUtils.removeTrailingSlash(projectPath)) {
    description "Automatically generated release job for Maven project " +
          "detected in folder $projectPath."

    logRotator {
      numToKeep 5
    }

    rootPOM "${projectPath}pom.xml"
    goals 'clean install'
  }

  ScmUtils.setupGit(job, false, repoUrl, projectPath)

  ReleaseUtils.setupReleaseParams(job)
  ReleaseUtils.setupPreBuildSteps(job, projectPath)
  ReleaseUtils.setupPostBuildSteps(job, projectPath, repoUrl)
}
Code language: Groovy (groovy)

The ProjectUtils class with the findProjectPaths() and removeTrailingSlash() methods can be found in the GitLab repository, as can the other utility classes. I’ve documented those methods, so their functionality should be easy to understand.

Additionally, we create a file ViewGeneric.groovy, which creates a view for all generically created jobs:

import javaposse.jobdsl.dsl.View
import util.ViewUtils

View view = listView('Part III - Generic') {
  jobs {
    regex '(Jenkins Tutorial Demo - Build |Jenkins Tutorial Demo - Release ).*'
  }
}

ViewUtils.addDefaultColumns(view)
Code language: Groovy (groovy)

Before you can run the seed job, you need to disable the «Sparse Checkout» so that the findProjectPaths() method finds the library1/ and application1/ directories. (If you have the Jenkins jobs in a separate repository, you need to check out the repository(s) with the Java code instead.

When you run the seed job, you get error messages again because the code uses methods that are not yet available for the Groovy sandbox. You have to approve these methods again – one after the other – via «Manage Jenkins» → «In-process Script Approval». When you have finished, the whitelist should contain the following method signatures:

The seed job should now have created the view «Part III — Generic» with the following four jobs:

Our Groovy code has now automatically created build and release jobs for all existing Java projects in the monorepo. Finally, you can do the following experiment: create a new, simple Maven project in the monorepo. For example, copy the «library1» project into a «library2» project. As a result, build and release jobs should be appearing automatically in Jenkins.

Summary

In this series of articles I’ve shown you some of Jenkins’ most essential features such as build and release jobs for Maven projects, the Jenkins Job DSL and the so-called «seed jobs». There are countless extension possibilities, here a few examples:

• You can have the seed job automatically create build jobs for feature branches.
• You can have your release jobs build docker containers and push them into a docker registry (using the Fabric8 Maven plugin).
• You can deploy web applications via blue-green deployment.
• You can deploy Microservices into a Kubernetes cluster (also with the Fabric8 Maven plugin).

Ultimately, you can implement arbitrarily complex build and deployment logic through shell scripts in the various build steps and an extensive selection of plugins.

Do you have problems understanding the individual steps? Are you stuck somewhere? If so, feel free to write a comment under the article; I will do my best to help you.

If you like the article, feel free to share it using one of the share buttons at the end. Do you want to be informed when new articles are published on HappyCoders.eu? Then click here to sign up for HappyCoders.eu newsletter.