Added example ArgoCD + helmfile deployment (#1882)
This commit is contained in:
Luke Plausin
GitHub
parent
91aa0acddb
commit
4d01eddd3d
|
|
@ -0,0 +1,11 @@
|
|||
apiVersion: v2
|
||||
name: argo-cd
|
||||
description: Kubernetes continuous deployment system
|
||||
type: application
|
||||
version: 3.2.0
|
||||
appVersion: "3.2.0" # ArgoCD version
|
||||
|
||||
dependencies:
|
||||
- name: argo-cd
|
||||
version: 3.2.0
|
||||
repository: https://argoproj.github.io/argo-helm
|
||||
|
|
@ -0,0 +1,136 @@
|
|||
# ArgoCD Deployment
|
||||
|
||||
This example is intended to show how you can use Helmfile with ArgoCD in a private way. The example deployment demonstrates how to deploy ArgoCD with the helmfile plugin, and how to use helmfile within ArgoCD to connect to various private external systems.
|
||||
|
||||
If you intend to follow this example, please pay close attention to the caveats. This chart will not work if you deploy it as-is, you will need to customise many parts of it to match the infrastructure of your organisation.
|
||||
|
||||
## What is the point of this chart?
|
||||
|
||||
There are many organisations out there which would like to widen Kubernetes adoption but struggle with the requirement to keep all data confidential and follow best practices. For example, one best practice is to to implement a CI/CD process for all deployments including Helm charts. As part of this process we may want to keep secrets in private Vault secret storage, use Helm charts from a private chart museum, customised Helm values from a private Git repository and docker images from a private docker repository. ArgoCD alone cannot do all of these things at once, but we can achieve this if we configure it with Helmfile.
|
||||
|
||||
This example is intended to show how you COULD use ArgoCD together with Helmfile to implement a fully private CI/CD system. Your deployment may be a litte bit different, or very different to this example. It's intended only as an example.
|
||||
|
||||
The helm chart in this repo can be deployed with Helmfile, if you follow the prerequisites. In order for the deployment to work correctly, you will need to follow the prerequisites and update all of the `values.yaml` file entries with a `# TODO` comment.
|
||||
|
||||
# Deployment planning
|
||||
|
||||
* Getting started: https://argoproj.github.io/argo-cd/getting_started/
|
||||
* Helm Chart: https://github.com/argoproj/argo-helm/tree/master/charts/argo-cd
|
||||
* SSO Setup: https://argoproj.github.io/argo-cd/operator-manual/user-management/microsoft/#azure-ad-app-registration-auth-using-oidc
|
||||
* RBAC: https://medium.com/dzerolabs/configuring-sso-with-azure-active-directory-on-argocd-d20be4ba753b
|
||||
* Vault
|
||||
* Private Chartmuseum (Harbor)
|
||||
|
||||
What are the interactions between the different systems?
|
||||
|
||||
* ArgoCD will run on Kubernetes. We must use a custom build of ArgoCD as the official build doesn't ship with the runtime binaries which we will need to execute Helmfile and interact with Vault.
|
||||
* ArgoCD interacts with AzureAD for SSO. ArgoCD has to be configured as an enterprise app in Azure AD for authorization and authentication by an administrator. We put the client ID and client secrets into Vault for safe keeping.
|
||||
* ArgoCD interacts with Vault in order to pull secrets for customising Helm deployments. Sometimes we need to inject secrets into the values.yaml file of a Helm deployment. ArgoCD will use an AppRole for authentication into Vault, and this is done once during the deployment. The ArgoCD Vault credentials are saved in Vault for safe keeping.
|
||||
* ArgoCD may need to interact with a private Git server. This is so that ArgoCD can access the `helmfile.yaml` files before executing Helmfile. Additional configuration files (like `conf.d` and other `values.yaml`) files can be kept in here.
|
||||
* ArgoCD will authenticate to a private chart museum. This is so that we can keep the private helm charts private.
|
||||
|
||||
# Prerequisites
|
||||
|
||||
## Docker image
|
||||
|
||||
* Build the docker image from the `argocd.dockerfile` file in this folder.
|
||||
* Upload the image to your private docker registry and make a note of the registry URL and tag.
|
||||
* Customise the `values.yaml` file in this folder.
|
||||
Replace `argo-cd.global.image.repository` for the docker registry URL of your private registry.
|
||||
Replace `argo-cd.global.image.tag` for the tag which you used to upload your private docker image.
|
||||
|
||||
## Configure Single Sign-On for ArgoCD
|
||||
|
||||
* Log into AzureAD as an administrator.
|
||||
* Create a new App Registration in your tenant for ArgoCD.
|
||||
* Create a new Client ID and client secret.
|
||||
* Save the client ID and client secret into your vault server.
|
||||
- Save the client ID and client secret into Vault. Suggested keys: `internal/argocd/auth#azure_oidc_client_id` and `internal/argocd/auth#azure_oidc_client_secret`.
|
||||
* Update `values.yaml` for your deployment
|
||||
- Update `argo-cd.server.config.oidc.config` - replace `SOME_AZURE_AD_TENANT` for your actual plaintext AzureAD tenant ID.
|
||||
- Update `argo-cd.server.config.oidc.config` - replace `SOME_AZURE_AD_UUID` for your actual plaintext AzureAD client app ID.
|
||||
- Verify that the Vault path specified in `argo-cd.config.secret.extra.oidc.azure.clientSecret` is the correct path in Vault.
|
||||
* Customise the ingress objects in the source to match what your expected external URL will be. Update `values.yaml` to have the correct values for the hostname you plan to use.
|
||||
- `argo-cd.config.server.(url|hostname)`
|
||||
```yaml
|
||||
url: 'https://my.argocd.deployment.org' # TODO
|
||||
hostname: my.argocd.deployment.org # TODO
|
||||
```
|
||||
|
||||
## Configure authentication from ArgoCD into Vault (or read next section on how to disable it)
|
||||
|
||||
* Read the documentation from Hashicorp on AppRoles. Provision a new AppRole for ArgoCD with policies.
|
||||
- In the example the authentication path in Vault is assumed to be `auth/approle/login`. If that's not the case then update line 123 in `values.yaml`.
|
||||
* Store the AppRole credentials for ArgoCD in Vault. We will use Helmfile to pull them from here when we launch the chart.
|
||||
- Suggested paths: `internal/vault/argocd#role_id` and `internal/vault/argocd#secret_id`
|
||||
* Verify that the Vault paths in `values.yaml` are correct
|
||||
- `argo-cd.config.secret.extra.vault_role_id`
|
||||
- `argo-cd.config.secret.extra.vault_secret_id`
|
||||
* Replace `ROLE_ID` and `SECRET_ID` in `values.yaml` (line 124) with the actual plaintext values (values key `argocd.server.config.configManagementPlugins`). This is not 100% secure but was the only way that I could manage to get the rest of the deployment working. I advise you add `values.yaml` to `.gitignore` after doing this step.
|
||||
|
||||
## Disable Vault (optional)
|
||||
|
||||
If you are not going to use Vault, then you need to update the values to get your deployment to use Helmfile.
|
||||
* If you are not going to use Vault, then remove the line in `values.yaml` that starts with `export VAULT_TOKEN=...`.
|
||||
|
||||
## Configure authentication from ArgoCD into private Git repository
|
||||
|
||||
* Create a new API user in your Git system (username + password auth). ArgoCD will use these credentials to pull `values.yaml` and `helmfile.yaml` for your project.
|
||||
* Save the username and password into Vault
|
||||
- Recommended keys: `internal/git/users/argocd#username` and `internal/git/users/argocd#password`
|
||||
* Verify that the Vault paths in `values.yaml` are correct
|
||||
- `argo-cd.config.secret.extra.git_username`
|
||||
- `argo-cd.config.secret.extra.git_password`
|
||||
* Update the settings in `values.yaml` for repositories - remove the example Git URLS and replace with actual Git URLs for your project.
|
||||
- `https://my.git.server.org/my-team/my-repo-one.git`
|
||||
- `https://my.git.server.org/my-team/my-repo-two.git`
|
||||
|
||||
## Configure authentication from ArgoCD to your private Docker registry and Chartmuseum
|
||||
|
||||
* This guide is written with Harbor in mind, which is both a docker registry and chart museum. If you have separate systems in place you might need to configure these things separately.
|
||||
* Create a new set of credentials for Harbor and save them into Vault.
|
||||
- Suggested paths: `internal/harbor/users/argocd#username`, `internal/harbor/users/argocd#username`.
|
||||
* Verify that the Vault paths in `values.yaml` are correct
|
||||
- `argo-cd.config.secret.extra.harbor_username`
|
||||
- `argo-cd.config.secret.extra.harbor_password`
|
||||
* Change the Helm URL on line 95 in `values.yaml` for the actual URL of your Helm Chartmuseum deployment.
|
||||
- `https://my.harbor.deployment.org/chartrepo/my-project`
|
||||
|
||||
# Deploy ArgoCD for the first time
|
||||
|
||||
Once you've completed the prerequisites you are ready to deploy ArgoCD for the first time. There is a helmfile that you can use to do this.
|
||||
|
||||
Steps:
|
||||
|
||||
* Auth to Vault and Kubernetes
|
||||
* Add ArgoCD Helm repo
|
||||
- ```bash
|
||||
helm repo add argo https://argoproj.github.io/argo-helm
|
||||
helm install --name argocd --namespace argocd argo/argo-cd
|
||||
```
|
||||
* Download dependencies (`helmfile dep update`)
|
||||
* Deploy chart (`helmfile apply`)
|
||||
- If everything worked correctly, then ArgoCD should have deployed using your customised docker image into your cluster.
|
||||
|
||||
* If the ingress deployed correctly then we should be able to access the UI: https://my.argocd.deployment.org
|
||||
* Grab the admin password from `argocd-initial-admin-secret secret` and use it to log into the UI.
|
||||
```bash
|
||||
echo "<password>" | base64 -d
|
||||
```
|
||||
* Verify that SSO works by signing on with AzureAD.
|
||||
* Verify that Helmfile appears in the list of configured plugins in the UI.
|
||||
|
||||
# Configure ArgoCD apps for deployment with Helmfile
|
||||
|
||||
* Log into ArgoCD in the UI and configure a new app.
|
||||
* Make sure that it's using the Helmfile plugin.
|
||||
* For the source, select one of the Git repos which you configured previously.
|
||||
* Make sure that your Helm repo (including `helmfile.yaml` is present in the Git repository).
|
||||
* ArgoCD will pull the project from your private Git server and read the `helmfile.yaml` file.
|
||||
* ArgoCD will also execute helmfile directly in the container using the files checked out from your project.
|
||||
* If you configured ArgoCD with logins for Vault and private Chartmuseums, these can be used in ArgoCD by Helmfile.
|
||||
* You can use the diff feature and resource views in ArgoCD when using Helmfile. Unfortunately you can no longer use the `values.yaml` editor in the UI. You can change `values.yaml` for deployments by creating `values.yaml` files and committing them to the folders in your project Git repo. You can also customise them using `!vault` tags.
|
||||
|
||||
If you would like an example, this ArgoCD + helmfile deployment can itself be deployed by ArgoCD + helmfile (you may need to drop the pods after you do deploy this way). Any other projects - just structure them in the same way as this example.
|
||||
|
||||
Be careful when reconfiguring a project which was previously deploy by ArgoCD with Helmfile. There are some differences around the way in which annotations are used.
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
# Custom Dockerfile to install required helm plugins
|
||||
FROM argoproj/argocd:latest
|
||||
|
||||
USER root
|
||||
# Download OS dependencies
|
||||
RUN apt-get update && \
|
||||
apt-get install -y \
|
||||
curl git wget unzip && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
|
||||
# Download helmfile
|
||||
RUN wget https://github.com/roboll/helmfile/releases/download/v0.138.7/helmfile_linux_amd64 && \
|
||||
mv helmfile_linux_amd64 /usr/local/bin/helmfile && \
|
||||
chmod a+x /usr/local/bin/helmfile
|
||||
|
||||
# Download Vault
|
||||
RUN wget https://releases.hashicorp.com/vault/1.5.0/vault_1.5.0_linux_amd64.zip -O /tmp/vault.zip --quiet && \
|
||||
unzip -p /tmp/vault.zip vault > /usr/local/bin/vault && \
|
||||
chmod a+x /usr/local/bin/vault && \
|
||||
rm /tmp/vault.zip
|
||||
|
||||
USER argocd
|
||||
|
||||
# Install helm-secrets plugin (as argocd user)
|
||||
RUN helm plugin install https://github.com/jkroepke/helm-secrets --version v3.6.0
|
||||
|
||||
ENV HELM_PLUGINS="/home/argocd/.local/share/helm/plugins/"
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
# What is a helmfile? Read here...
|
||||
# https://github.com/roboll/helmfile
|
||||
|
||||
# Before deployment, export the required env vars
|
||||
# export HELM_SECRETS_DRIVER=vault
|
||||
|
||||
# Commands to deploy:
|
||||
# helmfile deps # < download dependencies
|
||||
# helmfile template # < template stuff. You might have to be signed into vault
|
||||
# helmfile sync # < deploy everything
|
||||
# helmfile apply # < deploy diff only
|
||||
|
||||
helmDefaults:
|
||||
wait: true
|
||||
timeout: 600
|
||||
recreatePods: true
|
||||
force: false
|
||||
|
||||
commonLabels:
|
||||
system: argocd
|
||||
|
||||
releases:
|
||||
- name: argocd # name of this release
|
||||
namespace: argocd # target namespace
|
||||
createNamespace: true # helm 3.2+ automatically create release namespace (default true)
|
||||
labels: # Arbitrary key value pairs for filtering releases
|
||||
env: prod
|
||||
chart: "." # the chart being installed to create this release, referenced by `repository/chart` syntax
|
||||
version: 3.2.2 # the semver of the chart. range constraint is supported
|
||||
missingFileHandler: Warn # set to either "Error" or "Warn". "Error" instructs helmfile to fail when unable to find a values or secrets file. When "Warn", it prints the file and continues.
|
||||
|
||||
# will attempt to decrypt secrets using helm-secrets plugin
|
||||
secrets:
|
||||
# {{ requiredEnv "HELM_SECRETS_DRIVER" }}
|
||||
- values.yaml
|
||||
verify: false
|
||||
wait: true
|
||||
timeout: 600
|
||||
recreatePods: true
|
||||
force: false
|
||||
installed: true
|
||||
atomic: true
|
||||
cleanupOnFail: false
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
kind: Ingress
|
||||
apiVersion: extensions/v1beta1
|
||||
metadata:
|
||||
name: argocd-ingress
|
||||
namespace: argocd
|
||||
labels:
|
||||
app: argocd
|
||||
app.kubernetes.io/managed-by: Helm
|
||||
app.kubernetes.io/component: ingress
|
||||
app.kubernetes.io/name: argocd-ingress
|
||||
app.kubernetes.io/part-of: argocd
|
||||
annotations:
|
||||
external-dns.alpha.kubernetes.io/hostname: {{ .Values.argo-cd.server.config.hostname }}
|
||||
spec:
|
||||
tls:
|
||||
- hosts:
|
||||
- {{ .Values.argo-cd.server.config.hostname }}
|
||||
secretName: argocd-secret
|
||||
rules:
|
||||
- host: {{ .Values.argo-cd.server.config.hostname }}
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
backend:
|
||||
serviceName: argocd-server
|
||||
servicePort: http
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
apiVersion: v1
|
||||
kind: Secret
|
||||
type: kubernetes.io/dockerconfigjson
|
||||
metadata:
|
||||
name: dockerpullsecrets
|
||||
stringData:
|
||||
.dockerconfigjson: |
|
||||
{
|
||||
"auths" : {
|
||||
"my.harbor.deployment.org" : {
|
||||
{{ $authstring := (printf "%v:%v" .Values.secrets.harbor.username .Values.secrets.harbor.password ) }}
|
||||
"auth": "{{ $authstring | b64enc }}"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
apiVersion: v1
|
||||
kind: Secret
|
||||
type: secret
|
||||
metadata:
|
||||
name: argocd-git
|
||||
stringData:
|
||||
username: "{{ .Values.secrets.git.username }}"
|
||||
password: "{{ .Values.secrets.git.password }}"
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
apiVersion: v1
|
||||
kind: Secret
|
||||
type: secret
|
||||
metadata:
|
||||
name: argocd-harbor
|
||||
stringData:
|
||||
username: "{{ .Values.secrets.harbor.username }}"
|
||||
password: "{{ .Values.secrets.harbor.password }}"
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
apiVersion: v1
|
||||
kind: Secret
|
||||
type: secret
|
||||
metadata:
|
||||
name: argocd-vault
|
||||
stringData:
|
||||
role_id: "{{ .Values.secrets.vault.role_id }}"
|
||||
secret_id: "{{ .Values.secrets.vault.secret_id }}"
|
||||
|
|
@ -0,0 +1,154 @@
|
|||
# TODO - Any of the fields with TODO must be changed for the deployment to work.
|
||||
secrets:
|
||||
harbor:
|
||||
username: !vault internal/harbor/users/argocd#username # TODO
|
||||
password: !vault internal/harbor/users/argocd#password # TODO
|
||||
git:
|
||||
username: !vault internal/git/users/argocd#username # TODO
|
||||
password: !vault internal/git/users/argocd#password # TODO
|
||||
vault:
|
||||
role_id: !vault internal/vault/argocd#role_id # TODO
|
||||
secret_id: !vault internal/vault/argocd#secret_id # TODO
|
||||
|
||||
argo-cd:
|
||||
global:
|
||||
image:
|
||||
repository: my.private.docker.registry.local/internal/argocd # TODO
|
||||
tag: 1.2.3 # TODO
|
||||
imagePullPolicy: IfNotPresent
|
||||
securityContext: {}
|
||||
# runAsUser: 999
|
||||
# runAsGroup: 999
|
||||
# fsGroup: 999
|
||||
imagePullSecrets:
|
||||
- name: dockerpullsecrets
|
||||
hostAliases: []
|
||||
# - ip: 10.20.30.40
|
||||
# hostnames:
|
||||
# - git.myhostname
|
||||
# set harbor creds in environment, set HELM_SECRET_DRIVER as well
|
||||
server:
|
||||
## Argo server log format: text|json
|
||||
logFormat: text
|
||||
## Argo server log level
|
||||
logLevel: debug
|
||||
env:
|
||||
- name: "HELM_SECRETS_DRIVER"
|
||||
value: "vault"
|
||||
- name: "HARBOR_USERNAME"
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: argocd-harbor
|
||||
key: username
|
||||
- name: "HARBOR_PASSWORD"
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: argocd-harbor
|
||||
key: password
|
||||
# Unfortunately these envvars don't seem to be working correctly.
|
||||
- name: "VAULT_APPROLE_ID"
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: argocd-vault
|
||||
key: role_id
|
||||
- name: "VAULT_APPROLE_SECRET"
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: argocd-vault
|
||||
key: secret_id
|
||||
config:
|
||||
application.instanceLabelKey: argocd.argoproj.io/instance
|
||||
# TODO - customise oidc.config - replace SOME_AZURE_AD_TENANT and SOME_AZURE_AD_UUID with the plaintext values
|
||||
oidc.config: |
|
||||
name: Azure
|
||||
issuer: SOME_AZURE_AD_TENANT
|
||||
clientID: SOME_AZURE_AD_UUID
|
||||
clientSecret: $oidc.azure.clientSecret
|
||||
requestedIDTokenClaims:
|
||||
groups:
|
||||
essential: true
|
||||
requestedScopes:
|
||||
- openid
|
||||
- profile
|
||||
- email
|
||||
# TODO - customise repositories to include correct git URLs
|
||||
repositories: |
|
||||
- type: git
|
||||
url: https://my.git.server.org/my-team/my-repo-one.git
|
||||
usernameSecret:
|
||||
key: username
|
||||
name: argocd-git
|
||||
passwordSecret:
|
||||
key: password
|
||||
name: argocd-git
|
||||
- type: git
|
||||
url: https://my.git.server.org/my-team/my-repo-two.git
|
||||
usernameSecret:
|
||||
name: argocd-git
|
||||
key: username
|
||||
passwordSecret:
|
||||
name: argocd-git
|
||||
key: password
|
||||
|
||||
- type: helm
|
||||
url: https://my.harbor.deployment.org/chartrepo/my-project
|
||||
usernameSecret:
|
||||
name: argocd-harbor
|
||||
key: username
|
||||
passwordSecret:
|
||||
name: argocd-harbor
|
||||
key: password
|
||||
|
||||
url: 'https://my.argocd.deployment.org' # TODO
|
||||
hostname: my.argocd.deployment.org # TODO
|
||||
configManagementPlugins: |
|
||||
# If you just want to use helm-secrets this will work.
|
||||
- name: helm-secrets
|
||||
generate: # Command to generate manifests YAML
|
||||
command: ["/bin/bash", "-c"]
|
||||
args: ["echo \"$HELM_VALUES\" > ./values-local.yaml && helm secrets -d vault template $HELM_OPTS -n $ARGOCD_APP_NAMESPACE -f ./values-local.yaml $ARGOCD_APP_NAME . && rm ./values-local.yaml"]
|
||||
|
||||
# You can use helm-secrets via helmfile also.
|
||||
- name: helmfile
|
||||
init:
|
||||
command: ["/bin/bash", "-c"]
|
||||
args:
|
||||
- >
|
||||
helmfile repos ;
|
||||
helm dependency update ;
|
||||
true
|
||||
generate:
|
||||
command: ["/bin/bash", "-c"]
|
||||
args:
|
||||
- >
|
||||
export VAULT_TOKEN=$(vault write auth/approle/login role_id=ROLE_ID secret_id=SECRET_ID | grep token | head -n 1 | tr -s ' ' | cut -d ' ' -f 2) &&
|
||||
helmfile template --skip-deps
|
||||
# ^^^ I am not sure why but I can't seem to get it working without hardcoding the secrets here :(
|
||||
# TODO - replace ROLE_ID and SECRET_ID with the actual plaintext values
|
||||
|
||||
## ArgoCD rbac config
|
||||
## reference https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/rbac.md
|
||||
rbacConfig:
|
||||
policy.csv: |
|
||||
p, role:org-admin, applications, *, */*, allow
|
||||
p, role:org-admin, clusters, get, *, allow
|
||||
p, role:org-admin, repositories, get, *, allow
|
||||
p, role:org-admin, repositories, create, *, allow
|
||||
p, role:org-admin, repositories, update, *, allow
|
||||
p, role:org-admin, repositories, delete, *, allow
|
||||
g, "OrgAdmin", role:org-admin
|
||||
g, "ReadOnly", role:readonly
|
||||
policy.default: 'role:readonly'
|
||||
scopes: '[roles, email]'
|
||||
|
||||
## Argo Configs
|
||||
configs:
|
||||
secret:
|
||||
extra:
|
||||
oidc.azure.clientSecret: !vault internal/argocd/auth#azure_oidc_client_secret # TODO
|
||||
harbor_username: !vault internal/harbor/users/argocd#username # TODO
|
||||
harbor_password: !vault internal/harbor/users/argocd#password # TODO
|
||||
git_username: !vault internal/git/users/argocd#username # TODO
|
||||
git_password: !vault internal/git/users/argocd#password # TODO
|
||||
vault_role_id: !vault internal/vault/argocd#role_id # TODO
|
||||
vault_secret_id: !vault internal/vault/argocd#secret_id # TODO
|
||||
Loading…
Reference in New Issue