Turn on debug in Jenkins Operator deployment:
sed -i 's|\(args:\).*|\1\ ["--debug"\]|' deploy/operator.yaml -kubectl apply -f deploy/operator.yaml
Watch Kubernetes events:
kubectl get events --sort-by='{.lastTimestamp}'
Verify Jenkins master logs:
kubectl logs -f jenkins-<cr_name>
Verify the jenkins-operator logs:
jenkins-operator
kubectl logs deployment/jenkins-operator
This document answers the most frequently asked questions.
Delete the Jenkins master pod and wait for the new one to come up:
kubectl delete pod jenkins-<cr_name>
Jenkins can lose compatibility with its plugins or their dependencies. +If you want to reduce the probability of it happening, don’t use ‘latest’ Jenkins image tag. +Use set version of Jenkins image and declare plugins and all their dependencies in the Jenkins +Custom Resource under ‘plugins’. If you are not sure which plugins to pin, you can check the logs +from the ‘initial-config’ initcontainer or ‘jenkins-master’.
You can always add a custom Role for your Jenkins with the permissions you need and reference it in the +Jenkins Custom Resource under ‘spec.roles’. The Operator will create a RoleBinding for it. Be careful. +Operator may also not have these permissions. As a quick temporary workaround, you can manually bind this +role to the Operator service account.
In order to provide smooth extension, scalability and errorless backups, Jenkins needs to stay ephemeral. +There is no way to change volume for JENKINS_HOME. All the configurations should be volatile in Jenkins +and kept in a VCS.
Prepare your Kubernetes cluster and set up your kubectl access.
kubectl
Once you have a running Kubernetes cluster you can focus on installing Jenkins Operator according to the Installation guide.
Once you have a running Kubernetes cluster you can focus on installing Jenkins Operator according to the +Installation guide.
How to work with Jenkins Operator 0.1.x version. We recommend migrating to a newer version. -
How to work with Jenkins Operator 0.2.x version. We recommend migrating to a newer version. -
How to work with Jenkins Operator 0.3.x version. We recommend migrating to a newer version. -
How to work with Jenkins Operator 0.4.x version. We recommend migrating to a newer version. -
How to work with Jenkins Operator 0.4.x version. We recommend migrating to a newer version. +
How to work with Jenkins Operator 0.3.x version. We recommend migrating to a newer version. +
How to work with Jenkins Operator 0.2.x version. We recommend migrating to a newer version. +
How to work with Jenkins Operator 0.1.x version. We recommend migrating to a newer version. +
secrets
Once Jenkins Operator is up and running let’s deploy actual Jenkins instance. Create manifest e.g. jenkins_instance.yaml with following data and save it on drive.
jenkins_instance.yaml
apiVersion: jenkins.io/v1alpha2 @@ -887,7 +893,7 @@ kubectl get secret jenkins-operator-credentials-<cr_name> -o Last modified January 25, 2021 + Last modified August 19, 2021
Once you have running Kubernetes cluster you can focus on installing Jenkins Operator according to the Installation guide.
Deploy production ready Jenkins Operator manifest +
How to install Jenkins Operator
How to configure Jenkins with Operator +
Deploy production ready Jenkins manifest
How to customize Jenkins
Prevent loss of job history
Additional configuration for Azure Kubernetes Service -
Custom backup and restore provider
Jenkins security and hardening out of the box +
Additional configuration for Azure Kubernetes Service +
How to deal with Jenkins Operator problems -
Listed below are the two ways to deploy Jenkins Operator.
First, install Jenkins Custom Resource Definition:
kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/config/crd/bases/jenkins.io_jenkins.yaml
Then, apply the operator and other required resources:
Then, install the Operator and other required resources:
kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/all-in-one-v1alpha2.yaml
Watch Jenkins Operator instance being created:
kubectl get pods -w
$ helm repo add jenkins https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/chart $ helm install <name> jenkins/jenkins-operator -n <your-namespace>
In case you want to use released Chart v0.4.1, before installing/upgrading please install additional CRD into the cluster:
$ kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/chart/jenkins-operator/crds/jenkinsimage-crd.yaml
To add custom labels and annotations, you can use values.yaml file or pass them into helm install command, e.g.:
values.yaml
helm install
$ helm install <name> jenkins/jenkins-operator -n <your-namespace> --set jenkins.labels.LabelKey=LabelValue,jenkins.annotations.AnnotationKey=AnnotationValue
You can further customize Jenkins using values.yaml: @@ -1658,7 +1689,7 @@ Example:
If you wish to use the newest, not yet released version of the Operator, you can use one of nightly built snapshot images, however the maintainers of this project cannot guarantee their stability.
You can find nightly built images by heading to virtuslab/jenkins-operator Docker Hub repository and looking for images with tag in the form of “{git-hash}”, {git-hash} being the hash of master branch commit that you want to use snapshot of.
You can find nightly built images by heading to virtuslab/jenkins-operator Docker Hub repository and looking for images with tag in the form of {git-hash}, {git-hash} being the hash of master branch commit that you want to use snapshot of.
{git-hash}
fe81e5a
By default Jenkins Operator performs an initial security hardening of Jenkins instance +via groovy scripts to prevent any security gaps.
Currently Jenkins Operator generates a username and random password and stores them in a Kubernetes Secret. +However any other authorization mechanisms are possible and can be done via groovy scripts or configuration as code plugin. +For more information take a look at getting-started#jenkins-customization.
Any change to Security Realm or Authorization requires that user called jenkins-operator must have admin rights +because Jenkins Operator calls Jenkins API.
The list below describes all the default security setting configured by the Jenkins Operator:
Mode.EXCLUSIVE
JNLP3-connect
JNLP2-connect
JNLP-connect
/cli
If you would like to dig a little bit into the code, take a look here.
The Jenkins Operator generates and configures Basic Authentication token for Jenkins Go client +and stores it in a Kubernetes Secret.
Kubernetes API permissions are limited by the following roles:
Since Jenkins Operator must be able to grant permission for its deployed Jenkins masters +to spawn pods (the Jenkins Master role above), +the operator itself requires permission to create RBAC resources (the jenkins-operator role above).
Jenkins Master role
jenkins-operator role
Deployed this way, any subject which may create a Pod (including a Jenkins job) may +assume the jenkins-operator role by using its’ ServiceAccount, create RBAC rules, and thus escape its granted permissions. +Any namespace to which the jenkins-operator is deployed must be considered to implicitly grant all +possible permissions to any subject which can create a Pod in that namespace.
To mitigate this issue Jenkins Operator should be deployed in one namespace, and the Jenkins CR should be created in a separate namespace. +Section below contains instructions on how to do it.
You need to create two namespaces, for example we’ll call them jenkins for Jenkins and jenkins-operator for Jenkins Operator.
$ kubectl create ns jenkins-operator +$ kubectl create ns jenkins
Next, you need to install resources necessary for the Operator to work in the jenkins-operator namespace. To do that, +copy the manifest you see below to jenkins-operator-rbac.yamlfile.
jenkins-operator-rbac.yaml
--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: jenkins-operator +--- +# permissions to do leader election. +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: leader-election-role +rules: +- apiGroups: + - "" + - coordination.k8s.io + resources: + - configmaps + - leases + verbs: + - get + - list + - watch + - create + - update + - patch + - delete +- apiGroups: + - "" + resources: + - events + verbs: + - create + - patch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: leader-election-rolebinding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: leader-election-role +subjects: +- kind: ServiceAccount + name: jenkins-operator +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: jenkins-operator +rules: +- apiGroups: + - apps + resources: + - daemonsets + - deployments + - replicasets + - statefulsets + verbs: + - '*' +- apiGroups: + - apps + - jenkins-operator + resources: + - deployments/finalizers + verbs: + - update +- apiGroups: + - build.openshift.io + resources: + - buildconfigs + - builds + verbs: + - get + - list + - watch +- apiGroups: + - "" + resources: + - configmaps + - secrets + - services + verbs: + - create + - get + - list + - update + - watch +- apiGroups: + - "" + resources: + - events + verbs: + - create + - get + - list + - patch + - watch +- apiGroups: + - "" + resources: + - persistentvolumeclaims + verbs: + - get + - list + - watch +- apiGroups: + - "" + resources: + - pods + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - "" + resources: + - pods + - pods/exec + verbs: + - '*' +- apiGroups: + - "" + resources: + - pods/log + verbs: + - get + - list + - watch +- apiGroups: + - "" + resources: + - pods/portforward + verbs: + - create +- apiGroups: + - "" + resources: + - serviceaccounts + verbs: + - create + - get + - list + - update + - watch +- apiGroups: + - image.openshift.io + resources: + - imagestreams + verbs: + - get + - list + - watch +- apiGroups: + - jenkins.io + resources: + - '*' + verbs: + - '*' +- apiGroups: + - jenkins.io + resources: + - jenkins + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - jenkins.io + resources: + - jenkins/finalizers + verbs: + - update +- apiGroups: + - jenkins.io + resources: + - jenkins/status + verbs: + - get + - patch + - update +- apiGroups: + - rbac.authorization.k8s.io + resources: + - rolebindings + - roles + verbs: + - create + - get + - list + - update + - watch +- apiGroups: + - route.openshift.io + resources: + - routes + verbs: + - create + - get + - list + - update + - watch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: jenkins-operator +subjects: + - kind: ServiceAccount + name: jenkins-operator +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: jenkins-operator
Now install the required resources in jenkins-operator namespace with:
kubectl apply -n jenkins-operator -f jenkins-operator-rbac.yaml
There’s only one thing left to install in jenkins-operator namespace, and that is the Operator itself. The manifest +below contains the Operator as defined in all-in-one manifest found in Installating the Operator +page, the only difference is that the one here sets WATCH_NAMESPACE to the jenkins namespace we created.
WATCH_NAMESPACE
jenkins
Copy its content to jenkins-operator.yaml file.
jenkins-operator.yaml
apiVersion: apps/v1 +kind: Deployment +metadata: + name: jenkins-operator + labels: + control-plane: controller-manager +spec: + selector: + matchLabels: + control-plane: controller-manager + replicas: 1 + template: + metadata: + labels: + control-plane: controller-manager + spec: + serviceAccountName: jenkins-operator + securityContext: + runAsUser: 65532 + containers: + - command: + - /manager + args: + - --leader-elect + image: virtuslab/jenkins-operator:v0.6.0 + name: jenkins-operator + imagePullPolicy: IfNotPresent + securityContext: + allowPrivilegeEscalation: false + livenessProbe: + httpGet: + path: /healthz + port: 8081 + initialDelaySeconds: 15 + periodSeconds: 20 + readinessProbe: + httpGet: + path: /readyz + port: 8081 + initialDelaySeconds: 5 + periodSeconds: 10 + resources: + limits: + cpu: 100m + memory: 30Mi + requests: + cpu: 100m + memory: 20Mi + env: + - name: WATCH_NAMESPACE + value: jenkins + terminationGracePeriodSeconds: 10
Install the Operator in jenkins-operator namespace with:
kubectl apply -n jenkins-operator -f jenkins-operator.yaml
You have installed the Operator in jenkins-operator namespace, watching for Jenkins in jenkins namespace. Now +there are two things left to do: creating necessary Role and RoleBinding for the Operator in jenkins namespace, and +deploying actual Jenkins instance there.
Below you can find manifest with RBAC that need to be created in jenkins namespace. Copy its content to jenkins-ns-rbac.yaml file.
jenkins-ns-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: jenkins-operator +rules: + - apiGroups: + - apps + resources: + - daemonsets + - deployments + - replicasets + - statefulsets + verbs: + - '*' + - apiGroups: + - apps + - jenkins-operator + resources: + - deployments/finalizers + verbs: + - update + - apiGroups: + - build.openshift.io + resources: + - buildconfigs + - builds + verbs: + - get + - list + - watch + - apiGroups: + - "" + resources: + - configmaps + - secrets + - services + verbs: + - create + - get + - list + - update + - watch + - apiGroups: + - "" + resources: + - events + verbs: + - create + - get + - list + - patch + - watch + - apiGroups: + - "" + resources: + - persistentvolumeclaims + verbs: + - get + - list + - watch + - apiGroups: + - "" + resources: + - pods + verbs: + - create + - delete + - get + - list + - patch + - update + - watch + - apiGroups: + - "" + resources: + - pods + - pods/exec + verbs: + - '*' + - apiGroups: + - "" + resources: + - pods/log + verbs: + - get + - list + - watch + - apiGroups: + - "" + resources: + - pods/portforward + verbs: + - create + - apiGroups: + - "" + resources: + - serviceaccounts + verbs: + - create + - get + - list + - update + - watch + - apiGroups: + - image.openshift.io + resources: + - imagestreams + verbs: + - get + - list + - watch + - apiGroups: + - jenkins.io + resources: + - '*' + verbs: + - '*' + - apiGroups: + - jenkins.io + resources: + - jenkins + verbs: + - create + - delete + - get + - list + - patch + - update + - watch + - apiGroups: + - jenkins.io + resources: + - jenkins/finalizers + verbs: + - update + - apiGroups: + - jenkins.io + resources: + - jenkins/status + verbs: + - get + - patch + - update + - apiGroups: + - rbac.authorization.k8s.io + resources: + - rolebindings + - roles + verbs: + - create + - get + - list + - update + - watch + - apiGroups: + - route.openshift.io + resources: + - routes + verbs: + - create + - get + - list + - update + - watch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: jenkins-operator +subjects: + - kind: ServiceAccount + name: jenkins-operator + namespace: jenkins-operator +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: jenkins-operator
Now apply it with:
kubectl apply -n jenkins -f jenkins-ns-rbac.yaml
The last thing to do is to deploy Jenkins. Below you can find an example Jenkins resource manifest. +It’s the same as one used in Deploying Jenkins. +Copy it to jenkins-instance.yaml
jenkins-instance.yaml
apiVersion: jenkins.io/v1alpha2 +kind: Jenkins +metadata: + name: example +spec: + configurationAsCode: + configurations: [] + secret: + name: "" + groovyScripts: + configurations: [] + secret: + name: "" + jenkinsAPISettings: + authorizationStrategy: createUser + master: + disableCSRFProtection: false + containers: + - name: jenkins-master + image: jenkins/jenkins:2.277.4-lts-alpine + imagePullPolicy: Always + livenessProbe: + failureThreshold: 12 + httpGet: + path: /login + port: http + scheme: HTTP + initialDelaySeconds: 100 + periodSeconds: 10 + successThreshold: 1 + timeoutSeconds: 5 + readinessProbe: + failureThreshold: 10 + httpGet: + path: /login + port: http + scheme: HTTP + initialDelaySeconds: 80 + periodSeconds: 10 + successThreshold: 1 + timeoutSeconds: 1 + resources: + limits: + cpu: 1500m + memory: 3Gi + requests: + cpu: "1" + memory: 500Mi + seedJobs: + - id: jenkins-operator + targets: "cicd/jobs/*.jenkins" + description: "Jenkins Operator repository" + repositoryBranch: master + repositoryUrl: https://github.com/jenkinsci/kubernetes-operator.git
Now you can deploy it with:
kubectl apply -n jenkins -f jenkins-instance.yaml
With this, you have just set up Jenkins Operator and Jenkins in separate namespaces. Now the Operator will run in +its own namespace (jenkins-operator), watch for CRs in jenkins namespace, and deploy Jenkins there.
If you find a vulnerability or any misconfiguration in Jenkins, please report it in the issues.