mirror of
https://github.com/jenkinsci/kubernetes-operator.git
synced 2026-10-22 07:02:58 +02:00
Update docs, add assets directory
This commit is contained in:
antoniaklja
@@ -0,0 +1,83 @@
|
||||
# Developer guide
|
||||
|
||||
This document explains how to setup your development environment.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [operator_sdk][operator_sdk]
|
||||
- [dep][dep_tool] version v0.5.0+
|
||||
- [git][git_tool]
|
||||
- [go][go_tool] version v1.10+
|
||||
- [minikube][minikube] version v0.31.0+ (preferred Hypervisor - [virtualbox][virtualbox])
|
||||
- [docker][docker_tool] version 17.03+
|
||||
|
||||
## Clone repository and download dependencies
|
||||
|
||||
```bash
|
||||
mkdir -p $GOPATH/src/github.com/VirtusLab
|
||||
cd $GOPATH/src/github.com/VirtusLab/
|
||||
git clone git@github.com:VirtusLab/jenkins-operator.git
|
||||
cd jenkins-operator
|
||||
make go-dependencies
|
||||
```
|
||||
|
||||
## Build and run
|
||||
|
||||
Build and run **jenkins-operator**` locally:
|
||||
|
||||
```bash
|
||||
make build && make docker-build && make minikube-run EXTRA_ARGS='--minikube --local'
|
||||
```
|
||||
|
||||
Once minikube and **jenkins-operator** are up and running, apply Jenkins custom resource:
|
||||
|
||||
```bash
|
||||
kubectl apply -f deploy/crds/virtuslab_v1alpha1_jenkins_cr.yaml
|
||||
kubectl get jenkins -o yaml
|
||||
kubectl get po
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Run unit tests:
|
||||
|
||||
```bash
|
||||
make test
|
||||
```
|
||||
|
||||
Run e2e tests with minikube:
|
||||
|
||||
```bash
|
||||
make minikube-run
|
||||
make docker-build-e2e
|
||||
make e2e E2E_IMAGE=jenkins-operator
|
||||
```
|
||||
|
||||
## Hacks
|
||||
|
||||
### `pkg/apis/virtuslab/v1alpha1/jenkins_types` has changed
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
make deepcopy-gen
|
||||
```
|
||||
|
||||
### Getting Jenkins URL and basic credentials
|
||||
|
||||
```bash
|
||||
minikube service jenkins-operator-example --url
|
||||
kubectl get secret jenkins-operator-credentials-example -o 'jsonpath={.data.password}' | base64 -d
|
||||
```
|
||||
|
||||
|
||||
[dep_tool]:https://golang.github.io/dep/docs/installation.html
|
||||
[git_tool]:https://git-scm.com/downloads
|
||||
[go_tool]:https://golang.org/dl/
|
||||
[operator_sdk]:https://github.com/operator-framework/operator-sdk
|
||||
[fork_guide]:https://help.github.com/articles/fork-a-repo/
|
||||
[docker_tool]:https://docs.docker.com/install/
|
||||
[kubectl_tool]:https://kubernetes.io/docs/tasks/tools/install-kubectl/
|
||||
[minikube]:https://kubernetes.io/docs/tasks/tools/install-minikube/
|
||||
[virtualbox]:https://www.virtualbox.org/wiki/Downloads
|
||||
[jenkins-operator]:../README.md
|
||||
@@ -0,0 +1,339 @@
|
||||
# Getting Started
|
||||
|
||||
This document describes a getting started guide for **jenkins-operator** and an additional configuration.
|
||||
|
||||
1. [First Steps](#first-steps)
|
||||
2. [Deploy Jenkins](#deploy-jenkins)
|
||||
3. [Configure Seed Jobs and Pipelines](#configure-seed-jobs-and-pipelines)
|
||||
4. [Install Plugins](#install-plugins)
|
||||
5. [Configure Authorization](#configure-authorization)
|
||||
6. [Configure Backup & Restore](#configure-backup-&-restore)
|
||||
7. [Debugging](#debugging)
|
||||
|
||||
## First Steps
|
||||
|
||||
Prepare your Kubernetes cluster and set up access.
|
||||
Once you have running Kubernetes cluster you can focus on installing **jenkins-operator** according to the [Installation](installation.md) guide.
|
||||
|
||||
## Deploy Jenkins
|
||||
|
||||
Once jenkins-operator is up and running let's deploy actual Jenkins instance.
|
||||
Let's use example below:
|
||||
|
||||
```bash
|
||||
apiVersion: virtuslab.com/v1alpha1
|
||||
kind: Jenkins
|
||||
metadata:
|
||||
name: example
|
||||
spec:
|
||||
master:
|
||||
image: jenkins/jenkins
|
||||
seedJobs:
|
||||
- id: jenkins-operator-e2e
|
||||
targets: "cicd/jobs/*.jenkins"
|
||||
description: "Jenkins Operator e2e tests repository"
|
||||
repositoryBranch: master
|
||||
repositoryUrl: https://github.com/VirtusLab/jenkins-operator-e2e.git
|
||||
```
|
||||
|
||||
Watch Jenkins instance being created:
|
||||
|
||||
```bash
|
||||
kubectl get pods -w
|
||||
```
|
||||
|
||||
Get Jenkins credentials:
|
||||
|
||||
```bash
|
||||
kubectl get secret jenkins-operator-credentials-example -o 'jsonpath={.data.password}' | base64 -d
|
||||
```
|
||||
|
||||
Connect to Jenkins (minikube):
|
||||
|
||||
```bash
|
||||
minikube service jenkins-operator-example --url
|
||||
```
|
||||
|
||||
Connect to Jenkins (actual Kubernetes cluster):
|
||||
|
||||
```bash
|
||||
kubectl describe svc jenkins-operator-example
|
||||
kubectl jenkins-operator-example 8080:8080
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Configure Seed Jobs and Pipelines
|
||||
|
||||
Jenkins operator uses [job-dsl][job-dsl] and [ssh-credentials][ssh-credentials] plugins for configuring seed jobs
|
||||
and deploy keys.
|
||||
|
||||
## Prepare job definitions and pipelines
|
||||
|
||||
First you have to prepare pipelines and job definition in your GitHub repository using the following structure:
|
||||
|
||||
```
|
||||
cicd/
|
||||
├── jobs
|
||||
│ └── build.jenkins
|
||||
└── pipelines
|
||||
└── build.jenkins
|
||||
```
|
||||
|
||||
**cicd/jobs/build.jenkins** it's a job definition:
|
||||
|
||||
```
|
||||
#!/usr/bin/env groovy
|
||||
|
||||
pipelineJob('build-jenkins-operator') {
|
||||
displayName('Build jenkins-operator')
|
||||
|
||||
definition {
|
||||
cpsScm {
|
||||
scm {
|
||||
git {
|
||||
remote {
|
||||
url('https://github.com/VirtusLab/jenkins-operator.git')
|
||||
credentials('jenkins-operator')
|
||||
}
|
||||
branches('*/master')
|
||||
}
|
||||
}
|
||||
scriptPath('cicd/pipelines/build.jenkins')
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**cicd/jobs/build.jenkins** it's an actual Jenkins pipeline:
|
||||
|
||||
```
|
||||
#!/usr/bin/env groovy
|
||||
|
||||
def label = "build-jenkins-operator-${UUID.randomUUID().toString()}"
|
||||
def home = "/home/jenkins"
|
||||
def workspace = "${home}/workspace/build-jenkins-operator"
|
||||
def workdir = "${workspace}/src/github.com/VirtusLab/jenkins-operator/"
|
||||
|
||||
podTemplate(label: label,
|
||||
containers: [
|
||||
containerTemplate(name: 'jnlp', image: 'jenkins/jnlp-slave:alpine'),
|
||||
containerTemplate(name: 'go', image: 'golang:1-alpine', command: 'cat', ttyEnabled: true),
|
||||
]) {
|
||||
|
||||
node(label) {
|
||||
dir(workdir) {
|
||||
stage('Init') {
|
||||
timeout(time: 3, unit: 'MINUTES') {
|
||||
checkout scm
|
||||
}
|
||||
container('go') {
|
||||
sh 'apk --no-cache --update add make git gcc libc-dev'
|
||||
}
|
||||
}
|
||||
|
||||
stage('Build') {
|
||||
container('go') {
|
||||
sh 'make build'
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Configure Seed Jobs
|
||||
|
||||
Jenkins Seed Jobs are configured using `Jenkins.spec.seedJobs` section from your custom resource manifest:
|
||||
|
||||
```
|
||||
apiVersion: virtuslab.com/v1alpha1
|
||||
kind: Jenkins
|
||||
metadata:
|
||||
name: example
|
||||
spec:
|
||||
master:
|
||||
image: jenkins/jenkins:lts
|
||||
seedJobs:
|
||||
- id: jenkins-operator
|
||||
targets: "cicd/jobs/*.jenkins"
|
||||
description: "Jenkins Operator repository"
|
||||
repositoryBranch: master
|
||||
repositoryUrl: https://github.com/VirtusLab/jenkins-operator.git
|
||||
```
|
||||
|
||||
If your GitHub repository is **private** you have to configure corresponding **privateKey** and Kubernetes Secret:
|
||||
|
||||
```
|
||||
apiVersion: virtuslab.com/v1alpha1
|
||||
kind: Jenkins
|
||||
metadata:
|
||||
name: example
|
||||
spec:
|
||||
master:
|
||||
image: jenkins/jenkins:lts
|
||||
seedJobs:
|
||||
- id: jenkins-operator
|
||||
targets: "cicd/jobs/*.jenkins"
|
||||
description: "Jenkins Operator repository"
|
||||
repositoryBranch: master
|
||||
repositoryUrl: git@github.com:VirtusLab/jenkins-operator.git
|
||||
privateKey:
|
||||
secretKeyRef:
|
||||
name: deploy-keys
|
||||
key: jenkins-operator
|
||||
```
|
||||
|
||||
And Kubernetes Secret:
|
||||
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: deploy-keys
|
||||
data:
|
||||
jenkins-operator-e2e: |
|
||||
-----BEGIN RSA PRIVATE KEY-----
|
||||
MIIJKAIBAAKCAgEAxxDpleJjMCN5nusfW/AtBAZhx8UVVlhhhIKXvQ+dFODQIdzO
|
||||
oDXybs1zVHWOj31zqbbJnsfsVZ9Uf3p9k6xpJ3WFY9b85WasqTDN1xmSd6swD4N8
|
||||
...
|
||||
```
|
||||
|
||||
**jenkins-operator** will automatically discover and configure all seed jobs.
|
||||
|
||||
You can verify if deploy keys were successfully configured in Jenkins **Credentials** tab.
|
||||
|
||||
|
||||
|
||||
You can verify if your pipelines were successfully configured in Jenkins Seed Job console output.
|
||||
|
||||
|
||||
|
||||
## Jenkins Customisation
|
||||
|
||||
Jenkins can be customized using groovy scripts or configuration as code plugin. All custom configuration is stored in
|
||||
the **jenkins-operator-user-configuration-example** ConfigMap which is automatically created by **jenkins-operator**.
|
||||
|
||||
```
|
||||
kubectl get configmap jenkins-operator-user-configuration-example -o yaml
|
||||
|
||||
apiVersion: v1
|
||||
data:
|
||||
1-configure-theme.groovy: |2
|
||||
|
||||
import jenkins.*
|
||||
import jenkins.model.*
|
||||
import hudson.*
|
||||
import hudson.model.*
|
||||
import org.jenkinsci.plugins.simpletheme.ThemeElement
|
||||
import org.jenkinsci.plugins.simpletheme.CssTextThemeElement
|
||||
import org.jenkinsci.plugins.simpletheme.CssUrlThemeElement
|
||||
|
||||
Jenkins jenkins = Jenkins.getInstance()
|
||||
|
||||
def decorator = Jenkins.instance.getDescriptorByType(org.codefirst.SimpleThemeDecorator.class)
|
||||
|
||||
List<ThemeElement> configElements = new ArrayList<>();
|
||||
configElements.add(new CssTextThemeElement("DEFAULT"));
|
||||
configElements.add(new CssUrlThemeElement("https://cdn.rawgit.com/afonsof/jenkins-material-theme/gh-pages/dist/material-light-green.css"));
|
||||
decorator.setElements(configElements);
|
||||
decorator.save();
|
||||
|
||||
jenkins.save()
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
labels:
|
||||
app: jenkins-operator
|
||||
jenkins-cr: example
|
||||
watch: "true"
|
||||
name: jenkins-operator-user-configuration-example
|
||||
namespace: default
|
||||
```
|
||||
|
||||
When **jenkins-operator-user-configuration-example** ConfigMap is updated Jenkins automatically runs the **jenkins-operator-user-configuration** Jenkins Job which executes all scripts.
|
||||
|
||||
## Install Plugins
|
||||
|
||||
To install a plugin please add **2-install-slack-plugin.groovy** script to the **jenkins-operator-user-configuration-example** ConfigMap:
|
||||
|
||||
```
|
||||
apiVersion: v1
|
||||
data:
|
||||
1-configure-theme.groovy: |2
|
||||
|
||||
import jenkins.*
|
||||
import jenkins.model.*
|
||||
import hudson.*
|
||||
import hudson.model.*
|
||||
import org.jenkinsci.plugins.simpletheme.ThemeElement
|
||||
import org.jenkinsci.plugins.simpletheme.CssTextThemeElement
|
||||
import org.jenkinsci.plugins.simpletheme.CssUrlThemeElement
|
||||
|
||||
Jenkins jenkins = Jenkins.getInstance()
|
||||
|
||||
def decorator = Jenkins.instance.getDescriptorByType(org.codefirst.SimpleThemeDecorator.class)
|
||||
|
||||
List<ThemeElement> configElements = new ArrayList<>();
|
||||
configElements.add(new CssTextThemeElement("DEFAULT"));
|
||||
configElements.add(new CssUrlThemeElement("https://cdn.rawgit.com/afonsof/jenkins-material-theme/gh-pages/dist/material-light-green.css"));
|
||||
decorator.setElements(configElements);
|
||||
decorator.save();
|
||||
|
||||
jenkins.save()
|
||||
2-install-slack-plugin.groovy: |2
|
||||
|
||||
import jenkins.model.*
|
||||
import java.util.logging.Level
|
||||
import java.util.logging.Logger
|
||||
|
||||
def instance = Jenkins.getInstance()
|
||||
def plugins = instance.getPluginManager()
|
||||
def updateCenter = instance.getUpdateCenter()
|
||||
def hasInstalledPlugins = false
|
||||
|
||||
Logger logger = Logger.getLogger('jenkins.instance.restart')
|
||||
|
||||
if (!plugins.getPlugin("slack")) {
|
||||
logger.log(Level.INFO, "Installing plugin: slack")
|
||||
|
||||
updateCenter.updateAllSites()
|
||||
def plugin = updateCenter.getPlugin("slack")
|
||||
def installResult = plugin.deploy()
|
||||
while (!installResult.isDone()) sleep(10)
|
||||
hasInstalledPlugins = true
|
||||
instance.save()
|
||||
}
|
||||
|
||||
if (hasInstalledPlugins) {
|
||||
logger.log(Level.INFO, "Successfully installed slack plugin, restarting ...")
|
||||
// Queue a restart of the instance
|
||||
instance.save()
|
||||
instance.doSafeRestart(null)
|
||||
} else {
|
||||
logger.log(Level.INFO, "No plugins need installing.")
|
||||
}
|
||||
```
|
||||
|
||||
Then **jenkins-operator** will automatically trigger **jenkins-operator-user-configuration** Jenkins Job again.
|
||||
|
||||
## Configure Backup & Restore (work in progress)
|
||||
|
||||
Not implemented yet.
|
||||
|
||||
## Debugging
|
||||
|
||||
Watch Kubernetes events:
|
||||
|
||||
```bash
|
||||
kubectl get events --sort-by='{.lastTimestamp}'
|
||||
```
|
||||
|
||||
Verify Jenkins master logs:
|
||||
|
||||
```bash
|
||||
kubectl logs -f jenkins-master-example
|
||||
```
|
||||
|
||||
[job-dsl]:https://github.com/jenkinsci/job-dsl-plugin
|
||||
[ssh-credentials]:https://github.com/jenkinsci/ssh-credentials-plugin
|
||||
@@ -0,0 +1,52 @@
|
||||
# How it works
|
||||
|
||||
This document describes a high level overview how **jenkins-operator** works.
|
||||
|
||||
1. [Architecture and design](#architecture-and-design)
|
||||
2. [Operator State](#operator-state)
|
||||
3. [System Jenkins Jobs](#system-jenkins-jobs)
|
||||
3. [Jenkins Docker Images](#jenkins-docker-images)
|
||||
|
||||
## Architecture and design
|
||||
|
||||
The **jenkins-operator** design incorporates the following concepts:
|
||||
- watches any changes of manifests and maintain desired state according to deployed custom resource manifest
|
||||
- implements a main reconciliation loop which consists of two smaller reconciliation loops - base and user
|
||||
|
||||
|
||||
|
||||
**Base** reconciliation loop takes care of reconciling base Jenkins configuration, which consists of:
|
||||
- Ensure Manifests - monitors any changes in manifests
|
||||
- Ensure Jenkins Pod - creates and verifies status of Jenkins master Pod
|
||||
- Ensure Jenkins Configuration - configures Jenkins instance including hardening, initial configuration for plugins, etc.
|
||||
- Ensure Jenkins API token - generates Jenkins API token and initialized Jenkins client
|
||||
|
||||
**User** reconciliation loop takes care of reconciling user provided configuration, which consists of:
|
||||
- Ensure Restore Job - creates Restore job and ensures that restore has been successfully performed
|
||||
- Ensure Seed Jobs - creates Seed Jobs and ensures that all of them have been successfully executed
|
||||
- Ensure User Configuration - executed user provided configuration, like groovy scripts, configuration as code or plugins
|
||||
- Ensure Backup Job - creates Backup job and ensures that backup has been successfully performed
|
||||
|
||||
|
||||
|
||||
## Operator State
|
||||
|
||||
Operator state is kept in custom resource status section, which is used for storing any configuration events or job statuses managed by the operator.
|
||||
It helps to maintain or recover desired state even after operator or Jenkins restarts.
|
||||
|
||||
## System Jenkins Jobs
|
||||
|
||||
The operator or Jenkins instance can be restarted at any time and any operation should not block the reconciliation loop.
|
||||
Taking this into account we implemented custom jobs API for executing system jobs (seed jobs, groovy scripts, etc.) according to the operator lifecycle.
|
||||
|
||||
Main assumptions are:
|
||||
- do not block reconciliation loop
|
||||
- fire job, requeue reconciliation loop and verify job status next time
|
||||
- handle retries if case of failure
|
||||
- handle build expiration (deadline)
|
||||
- keep state in the custom resource status section
|
||||
|
||||
## Jenkins Docker Images
|
||||
|
||||
**jenkins-operator** is fully compatible with **jenkins:lts** docker image and does not introduce and hidden changes there.
|
||||
In case of need docker image can be easily changed in custom resource manifest as long as it supports standard Jenkins file system structure.
|
||||
@@ -0,0 +1,39 @@
|
||||
# Installation
|
||||
|
||||
This document describes installation procedure for **jenkins-operator**.
|
||||
|
||||
## Requirements
|
||||
|
||||
To run **jenkins-operator**, you will need:
|
||||
- running Kubernetes cluster
|
||||
- kubectl
|
||||
|
||||
## Configure Custom Resource Definition
|
||||
|
||||
Install Jenkins Custom Resource Definition:
|
||||
|
||||
```bash
|
||||
kubectl create -f deploy/crds/virtuslab_v1alpha1_jenkins_crd.yaml
|
||||
```
|
||||
|
||||
## Deploy jenkins-operator
|
||||
|
||||
Deploy **jenkins-operator** with RBAC (it may take some time):
|
||||
|
||||
```bash
|
||||
kubectl create -f deploy/service_account.yaml
|
||||
kubectl create -f deploy/role.yaml
|
||||
kubectl create -f deploy/role_binding.yaml
|
||||
kubectl create -f deploy/operator.yaml
|
||||
```
|
||||
|
||||
Watch **jenkins-operator** instance being created:
|
||||
|
||||
```bash
|
||||
kubectl get pods -w
|
||||
```
|
||||
|
||||
Now **jenkins-operator** should be up and running in `default` namespace.
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# Jenkins Security
|
||||
|
||||
By default **jenkins-operator** performs an initial security hardening of Jenkins instance via groovy scripts to prevent any security gaps.
|
||||
|
||||
## Jenkins Access Control
|
||||
|
||||
Currently **jenkins-operator** generates a username and random password and stores them in a Kubernetes Secret.
|
||||
However any other authorization mechanisms are possible an can be done via groovy scripts or configuration as code plugin.
|
||||
For more information take a look at [getting-started#jenkins-customization](getting-started.md#jenkins-customisation).
|
||||
|
||||
## Jenkins Hardening
|
||||
|
||||
The list below describes all the default security setting configured by the **jenkins-operator**:
|
||||
- basic settings - use `Mode.EXCLUSIVE` - Jobs must specify that they want to run on master node
|
||||
- enable CSRF - Cross Site Request Forgery Protection is enabled
|
||||
- disable usage stats - Jenkins usage stats submitting is disabled
|
||||
- enable master access control - Slave To Master Access Control is enabled
|
||||
- disable old JNLP protocols - `JNLP3-connect`, `JNLP2-connect` and `JNLP-connect` are disabled
|
||||
- disable CLI - CLI access of `/cli` URL is disabled
|
||||
- configure kubernetes-plugin - secure configuration for Kubernetes plugin
|
||||
|
||||
If you would like to dig a little bit into the code, take a look [here](../pkg/controller/jenkins/configuration/base/resources/base_configuration_configmap.go).
|
||||
|
||||
## Jenkins API access
|
||||
|
||||
**jenkins-operator** generates and configures Basic Authentication token for Jenkins go client stores it in a Kubernetes Secret.
|
||||
|
||||
## Report a Security Vulnerability
|
||||
|
||||
If you find a vulnerability or any misconfiguration in Jenkins, please report it in the [issues](https://github.com/VirtusLab/jenkins-operator/issues).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user