* Add flag to remap registries for any registry mirror
The purpose of this PR is to add an option to remap registries, a kind of generalized `--registry-mirror`.
This is helpful for air-gapped environments and/or when local registry mirrors are available (not limited to docker.io).
This allows user to reference any images without having to change their location.
It also permit to separate infra related configuration (the mirrors) from CI/CD pipeline definition by using an environment variable for example (the reason behind the early return if flag provided but empty).
Therefore you can have a pipeline calling kaniko with `--registry-map=$REGISTRY_MAP` and have the `REGISTRY_MAP` populated via the runner's env by another team, and the absence of env wouldn't trigger a failure, it makes the pipeline env independent.
I've also considered the option of environment variables directly but it doesn't seems to be in kaniko's philosophy.
This makes quite some duplicated code :/ One option to keep the mirror flag and behavior would be to use only one codebase and convert `--registry-mirror=VALUE` to `--registry-map=index.docker.io=VALUE` internally. Suggestions welcome!
* Configure logging config sooner to be able to use it in flag parsing
* Replace registry mirrors by maps logic and use env var
* Add env vars to README.md
* Fix test
* Add an integration test to reproduce #2892
* Fix go compilation
* Fix docker run cmd
* Fixing entrypoint
* Test warmer with cache in a volume.
* Add missing comma
* Fix imports
* Fix dir
* Add logs
* fix
* Use test framework to log
* Fix warmer failing if image already in cache.
* Fix format.
* Allow to disable the fallback to the default registry on image pull
When one or more registry mirror(s) are deffined with the 'registry-mirror' argument, if none of those mirrors include the image,
the current behavior is to fallback to the default registry.
If a whitelist (or some image restriction) is applied at the mirror side, fallbacking to the default registry makes that restriction useless.
This new argument allows to skip the fallback and abort the build if the mirror rejects an image.
If it is not set, is completelly transparent.
* fix typo on command help
If a non-empty directory gets replaced with something other than a
directory (e.g. file or symlink), the files in that directory also get
deleted. However, there should not be any whiteout files for them in the
layer. If there were, the resulting tar file would not be extractable.
Fixes#2308
As all Kaniko variants should be upgraded at once, all following the same specifications, it is easier for maintainers to bump Go or other image dependencies in a single place.
On top of that, there were an issue building `ppc64le` images because of a semantic error on the if condition that checks the arch used.
Signed-off-by: Ferran Vidal <ferran.vidal.p@gmail.com>
* fix(ci): Bump golangci-lint to 1.51.1
* chore(lint): fix gofmt and goimport issues
* chore(lint): fix linter issues
- Adapted error comparison according to linter recommendation
- Disabled noctx linting for http request where canceling makes no sense
- Disabled nilerror linting where nil error is returned on purpose
- Disabled makezero linter where slice is explicitly deepcopied
* chore(ci): Update go version in tests workflows
* fix(ci): Allow boilerplate years from 2000-2099
Previously the regex only allowed the copyright notice to contain the
years 2018,2019,2020,2021, or 2022. This commit widens to regex to
20\d\d allowing any year in the range [2000-2099]
* feat(ci): Replace minikube with k3s for intregration tests
The existing setup for minikube is very complicated, replicating most of
the setup steps for a full kubernetes cluster in an only partially
supported minikube configuration (driver=none). Furthermore the existing
setup has been broken for sometime, likely, at least in part due to the
changes to CNI and CRI in recent kubernetes versions.
Since what we actually need is only a running Kubernetes cluster on the
node and access to a registry on localhost:5000, we can switch the
extremely complicated minikube setup for a lightweight cluster using
k3s. Minikube came with a default addon for running a registry on every
node, but the same is not the case for k3s, instead we make use of the
package helm controller and its HelmChart CR to deploy twuni/docker-registry.helm
and expose it on localhost using the integrated LoadBalancer controller.
* fix(test-684): pin base container version
The dockerfile for the regression test connected to issue 684 used a
rolling tag as base image, making it flaky and fail since it was
introduced.
This commit pins the base image to the digest of bionic-20200219, which,
based on the date of the commit that introduced to the dockerfile would
be the most newest ubuntu build and likely what the "rolling" tag
resolved to back then. Since this also an image from the pre-oci days of
ubuntu, this circumvents a bug in container-diff as well
(https://github.com/GoogleContainerTools/container-diff/issues/389)
* Adds the ability to provide `--cache-repo` as an OCI image layout path
- Adds cache.LayoutCache to implement cache.LayerCache interface
- When opts.CacheRepo has "oci:" prefix, instantiates a LayoutCache
Signed-off-by: Natalie Arellano <narellano@vmware.com>
* Add integration test for layout cache
Signed-off-by: Natalie Arellano <narellano@vmware.com>
* Updates from PR review
Signed-off-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Natalie Arellano <narellano@vmware.com>
* fix: Correct flatten function in layers
- Added a test.
- Cache current image, track deletes in `whiteouts` as well as normal adds in `layers`.
- Fix ugly delete behavior of `layerHashCache`.
Delete it when crerating a new snapshot.
- Slight cleanup in `snapshot.go`.
- Format ugly `WalkFS` function.
* fix: Add symbolic link changes to Hasher and CacheHasher
* fix: Better log messages
* fix(ci): Integration tests
* fix(ci): Add `--no-cache` to docker builds
* fix(ci): Pass credentials for error integration test
* np: Missing .gitignore in `hack`
* np: Capitalize every log message
- Correct some linting.
* fix: Key function
- Merge only last layer onto `currentImage`.
* fix: Remove old obsolete `cacheHasher`
* remove configurables from constants
* add configurables to config init
* add kaniko dir flag
* cleanup pkg constants
* replace buildcontext constants pkg occurrences with config
* add KanikoDir to KanikoOptions
* replace executor constants pkg occurrences with config
* remove redundant KanikoDir assignment
* replace constants to config for IntermediateStagesDir
* fix imports
* add default kaniko directory into constants
* add check for kanikoDir on use
* update init to use default path constant
* update executor kanikoDir check
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* alter checkKanikoDir parameter
* add TestKanikoDir func
* update error handling style
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* remove shorthand flag usage from test
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* add docstring to integration test
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* remove shorthand flag from kaniko-dir
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* Bump github.com/docker/docker to latest release
* Pin deps to docker/docker's requirements
fix build breakages
fix unit test
Trying to see whether am-i-in-a-container checks are broken for integration tests...
* Run GitHub Actions on pushes and PRs to main, not master
* fix go install ecr-login, bump to latest v0.6.0
* master -> main in some integration tests
* skip no-ref integration test
* Collapse integration test workflows into one config
Remove last traces of Travis, including env vars and `make travis-setup`
and badges in README
* fix lint
* make integration tests pass on PRs
On my machine this makes image rebuilds go from ~5m to 1.5s.
This also required setting DOCKER_BUILDKIT=1 in integration test image
builds.
I also took the opportunity to bump the Go version used in those tests
to Go 1.17 to match the image build processes, and tidied up the
workflow files a bit too (renaming for consistency, typos, whitespace,
etc.)
* Debugging the integration testing.
It seems to be consistently failing (also fix a typo in the file/leg name).
I'm disabling `-v` for `go test` because it interleaves the test output in ways that make it painful to read a single failing test's output when `t.Parallel` is involved.
* Try swapping registry:2 for the registry add-on
* Drop logf and make Logger private
* git: accept explicit commit hash for git context
When checking out code from non-github repositories, the typical
assumptions may not be valid, e.g. that the only interesting
non-branch commits have ref names starting with refs/pull. A specific
example is fetching an un-merged commit from a gerrit repository by
commit hash.
This change just looks at the second part of the git context path and
checks if it's a SHA commit hash, and if so, will fetch and check out
this commit after cloning the repository.
Sample context argument:
https://github.repo/project#e1772f228e06d15facdf175e5385e265b57068c0
* ci: fix test script to recognize any non-zero exit as an error
hack/linter.sh didn't properly install golangci-lint in hack/bin as I
already have another version of golangci-lint on my PATH, but then it
failed to execute because it was looking for it specifically in
hack/bin.
When the executable is not found, the exit code is 127 instead of 1,
and so test.sh ignored the error.
Two fixes:
1. `test.sh`:
- Use `if (script) ...` instead of assigning / checking a result
variable to determine if each validation script passed or failed.
2. `hack/linter.sh`:
- Instead of checking for golangci-lint on the path, just
specifically check for an executable file (`test -x`) in the
expected location.
Co-authored-by: Wade Carpenter <wwade@users.noreply.github.com>
* chore: add workflows for pr tests
* fix unit tests
* fix formatting
* chore: fix gobuild
* change minikube script
* chore: fix lint install script
* chore: ignore and fix tests
* fix lint and run gofmt
* lint fixes
* k8s executor image only
* fix Makefile
* fix travis env variables
* more info on k8s tests
* fix travis run
* fix
* fix
* fix
* fix log
* some more changes
* increase timeout
* delete travis.yml and fix multiple copy tests
* fix registry mirror
* fix lint
* add concurency
* last attemot to fix k8 integrations
* diff id for diff workflows
* Fix composite cache key for multi-stage copy command (#1706)
PR #1518 reintroduced COPY layers caching using the `--cache-copy-layers`
flag. Unfortunately, this PR also introduced a bug by not including the
stage digest into the caching key of the COPY command when the
`--cache-copy-layers` flag was not set. As a result, kaniko would use
any previous (possibly stalled) layer from the cache because the digest
of the "COPY --from" command would never change.
PR author probably expected Go to fallthrough in the switch just like C
does. However, this is not the case. Go does not fallthrough in
switch-statements by default and requires the fallthrough keyword to be
used. Note that this keyword is not available in type-switches though,
because it wouldn't work properly with typings.
* refactor: add an abstract copy command interface to avoid code duplication
* fix typo in error message
Co-authored-by: Tejal Desai <tejal29@gmail.com>
* exit codes of failed RUN commands may now be propagated to the caller by using the --propagate-exit-code flag
* removed exit code propagation flag; the exit function now unwraps the error and looks for an exit code
* added integration test for exit code propagation
* added trailing new line in test Dockerfile_exit_code_propagation; tidying up README.md removing all traces of error propagation flag
* moved exit code propagation test to integration_test.go; moved exit code propagation test files to 'testdata' to avoid interference with other integration tests
Fixes#1473
The initial implementation of the registry mirror only allowed a single mirror, and if pulling from the mirror failed, the build would fail.
This change introduces:
- multiple registry mirrors instead of a single one
- fallback if an image can't be pulled from a registry
This is the same behavior as the docker daemon and will allow using a registry mirror such as `mirror.gcr.io` which is incomplete and doesn't have all the content that the default registry on docker.io has.
Note that there are no changes in the CLI flags, the `--registry-mirror` flag is still valid. But now it can be used multiple times to set up more than one registry mirror.
Co-authored-by: Tejal Desai <tejaldesai@google.com>
Previously there was a need to import public GCR images into the local
docker registry and replace the base images to use the local docker
registry. This is no longer needed since Kaniko now works with public
GCR images also for unauthenticated users.
Matthias Schneider
Damien Degois
Aaron Prindle
Maxime BOSSARD
Fernando Giannetti
Andreas Fleig
Ferran Vidal
Joël Pepper
Natalie Arellano
Lukas
Gabriel Nützi
Jack
Jason Hall
Matt Moore
Tejal Desai
wwade
Gilbert Gilb's
Benjamin Krenn
Vincent Behar
Yoan Blanc
Tejal Desai
Ben Einaudi
Carlos Alexandro Becker
Sam Stoelinga
Jordan Goasdoué
cvgw