* chore: go get -u ./...
* chore: go mod tidy
* chore: go mod vendor
* chore: fix compilation for buildkit >= 0.15.0
* chore: upgrade to Go 1.24
* chore: upgrade the Debian container used in an integration test
* Add flag to remap registries for any registry mirror
The purpose of this PR is to add an option to remap registries, a kind of generalized `--registry-mirror`.
This is helpful for air-gapped environments and/or when local registry mirrors are available (not limited to docker.io).
This allows user to reference any images without having to change their location.
It also permit to separate infra related configuration (the mirrors) from CI/CD pipeline definition by using an environment variable for example (the reason behind the early return if flag provided but empty).
Therefore you can have a pipeline calling kaniko with `--registry-map=$REGISTRY_MAP` and have the `REGISTRY_MAP` populated via the runner's env by another team, and the absence of env wouldn't trigger a failure, it makes the pipeline env independent.
I've also considered the option of environment variables directly but it doesn't seems to be in kaniko's philosophy.
This makes quite some duplicated code :/ One option to keep the mirror flag and behavior would be to use only one codebase and convert `--registry-mirror=VALUE` to `--registry-map=index.docker.io=VALUE` internally. Suggestions welcome!
* Configure logging config sooner to be able to use it in flag parsing
* Replace registry mirrors by maps logic and use env var
* Add env vars to README.md
* Fix test
* Add an integration test to reproduce #2892
* Fix go compilation
* Fix docker run cmd
* Fixing entrypoint
* Test warmer with cache in a volume.
* Add missing comma
* Fix imports
* Fix dir
* Add logs
* fix
* Use test framework to log
* Fix warmer failing if image already in cache.
* Fix format.
This commit adds the skip option for otiai10.Copy to skip the /kaniko
directory when the root is being copied. The files under /kaniko dir
should be ignored and thus this shall not cause any loss of information.
fixes: GoogleContainerTools#2033
* Create directories with the right UID/GID
* Forgot to create the actual directory
* Integration test creation of intermediate files with correct ownership
* ADD version of the test
* Allow to disable the fallback to the default registry on image pull
When one or more registry mirror(s) are deffined with the 'registry-mirror' argument, if none of those mirrors include the image,
the current behavior is to fallback to the default registry.
If a whitelist (or some image restriction) is applied at the mirror side, fallbacking to the default registry makes that restriction useless.
This new argument allows to skip the fallback and abort the build if the mirror rejects an image.
If it is not set, is completelly transparent.
* fix typo on command help
If a non-empty directory gets replaced with something other than a
directory (e.g. file or symlink), the files in that directory also get
deleted. However, there should not be any whiteout files for them in the
layer. If there were, the resulting tar file would not be extractable.
Fixes#2308
As all Kaniko variants should be upgraded at once, all following the same specifications, it is easier for maintainers to bump Go or other image dependencies in a single place.
On top of that, there were an issue building `ppc64le` images because of a semantic error on the if condition that checks the arch used.
Signed-off-by: Ferran Vidal <ferran.vidal.p@gmail.com>
* fix(ci): Bump golangci-lint to 1.51.1
* chore(lint): fix gofmt and goimport issues
* chore(lint): fix linter issues
- Adapted error comparison according to linter recommendation
- Disabled noctx linting for http request where canceling makes no sense
- Disabled nilerror linting where nil error is returned on purpose
- Disabled makezero linter where slice is explicitly deepcopied
* chore(ci): Update go version in tests workflows
* fix(ci): Allow boilerplate years from 2000-2099
Previously the regex only allowed the copyright notice to contain the
years 2018,2019,2020,2021, or 2022. This commit widens to regex to
20\d\d allowing any year in the range [2000-2099]
* feat(ci): Replace minikube with k3s for intregration tests
The existing setup for minikube is very complicated, replicating most of
the setup steps for a full kubernetes cluster in an only partially
supported minikube configuration (driver=none). Furthermore the existing
setup has been broken for sometime, likely, at least in part due to the
changes to CNI and CRI in recent kubernetes versions.
Since what we actually need is only a running Kubernetes cluster on the
node and access to a registry on localhost:5000, we can switch the
extremely complicated minikube setup for a lightweight cluster using
k3s. Minikube came with a default addon for running a registry on every
node, but the same is not the case for k3s, instead we make use of the
package helm controller and its HelmChart CR to deploy twuni/docker-registry.helm
and expose it on localhost using the integrated LoadBalancer controller.
* fix(test-684): pin base container version
The dockerfile for the regression test connected to issue 684 used a
rolling tag as base image, making it flaky and fail since it was
introduced.
This commit pins the base image to the digest of bionic-20200219, which,
based on the date of the commit that introduced to the dockerfile would
be the most newest ubuntu build and likely what the "rolling" tag
resolved to back then. Since this also an image from the pre-oci days of
ubuntu, this circumvents a bug in container-diff as well
(https://github.com/GoogleContainerTools/container-diff/issues/389)
WORKDIR ignores the currently set USER and creates the new directories
with the root user ownership.
This changes that, by executing a chown after the mkdir if needed, and
also handle the case where the provided USER is an uid and the passwd
file is not available to resolve to the username.
Fixes#2259
Signed-off-by: Aris Buzachis <buzachis.aris@gmail.com>
Signed-off-by: Aris Buzachis <buzachis.aris@gmail.com>
* Adds the ability to provide `--cache-repo` as an OCI image layout path
- Adds cache.LayoutCache to implement cache.LayerCache interface
- When opts.CacheRepo has "oci:" prefix, instantiates a LayoutCache
Signed-off-by: Natalie Arellano <narellano@vmware.com>
* Add integration test for layout cache
Signed-off-by: Natalie Arellano <narellano@vmware.com>
* Updates from PR review
Signed-off-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Natalie Arellano <narellano@vmware.com>
* fix: getUIDandGID is able to resolve non-existing users and groups
A common pattern in dockerfiles is to provide a plain uid and gid number, which doesn't neccesarily exist inside the os.
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* test: add chown dockerfile
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore: format
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore: add comment
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* tests: fix chown dockerfile
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* refactor: split up getIdsFromUsernameAndGroup func
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* fix: implement raw uid logic for LookupUser
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* test: add dockerfiles for integration test
* fix: lookup user error message
* test: add dockerfiles for non-existing user testcase
* fix: forgot error check
* tests: fix syscall credentials test
* chore: add debug output for copy command
* tests: set specific gid for integration dockerfile
* tests: fix syscall credentials test
github runner had the exact uid that i was testing on, so the groups were not empty
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* tests: fix test script
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore: apply golangci lint checks
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* fix: reset file ownership in createFile if not root owned
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore: logrus.Debugf missed format variable
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore(test-script): remove go html coverage
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* test(k8s): increase wait timeout
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* Fix missing file permissions on multi-stage build
Fixes#2075
When a file with the setuid bit is copied from one stage
to another, the permissions were not copied over properly after
setting ownership on directory and the file itself.
* Update pkg/util/fs_util.go
Co-authored-by: Jason Hall <jason@chainguard.dev>
* Adding boilerplate to dockerfile
* Add bash check to bail with exit code 1 if setuid not present
Co-authored-by: Jason Hall <jason@chainguard.dev>
* fix: Correct flatten function in layers
- Added a test.
- Cache current image, track deletes in `whiteouts` as well as normal adds in `layers`.
- Fix ugly delete behavior of `layerHashCache`.
Delete it when crerating a new snapshot.
- Slight cleanup in `snapshot.go`.
- Format ugly `WalkFS` function.
* fix: Add symbolic link changes to Hasher and CacheHasher
* fix: Better log messages
* fix(ci): Integration tests
* fix(ci): Add `--no-cache` to docker builds
* fix(ci): Pass credentials for error integration test
* np: Missing .gitignore in `hack`
* np: Capitalize every log message
- Correct some linting.
* fix: Key function
- Merge only last layer onto `currentImage`.
* fix: Remove old obsolete `cacheHasher`
* remove configurables from constants
* add configurables to config init
* add kaniko dir flag
* cleanup pkg constants
* replace buildcontext constants pkg occurrences with config
* add KanikoDir to KanikoOptions
* replace executor constants pkg occurrences with config
* remove redundant KanikoDir assignment
* replace constants to config for IntermediateStagesDir
* fix imports
* add default kaniko directory into constants
* add check for kanikoDir on use
* update init to use default path constant
* update executor kanikoDir check
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* alter checkKanikoDir parameter
* add TestKanikoDir func
* update error handling style
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* remove shorthand flag usage from test
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* add docstring to integration test
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* remove shorthand flag from kaniko-dir
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* Fix resolving argument over mulit-stage build
- Building multi-stage Dockerfiles did not correctly resolve
ARG commands over different stages. Now, each stage depends on the
build arguments set by the stage before.
Closes: #1911
* Fix resolving empty arg commands
* Bump github.com/docker/docker to latest release
* Pin deps to docker/docker's requirements
fix build breakages
fix unit test
Trying to see whether am-i-in-a-container checks are broken for integration tests...
* Run GitHub Actions on pushes and PRs to main, not master
* fix go install ecr-login, bump to latest v0.6.0
* master -> main in some integration tests
* skip no-ref integration test
* Collapse integration test workflows into one config
Remove last traces of Travis, including env vars and `make travis-setup`
and badges in README
* fix lint
* make integration tests pass on PRs
On my machine this makes image rebuilds go from ~5m to 1.5s.
This also required setting DOCKER_BUILDKIT=1 in integration test image
builds.
I also took the opportunity to bump the Go version used in those tests
to Go 1.17 to match the image build processes, and tidied up the
workflow files a bit too (renaming for consistency, typos, whitespace,
etc.)
* Debugging the integration testing.
It seems to be consistently failing (also fix a typo in the file/leg name).
I'm disabling `-v` for `go test` because it interleaves the test output in ways that make it painful to read a single failing test's output when `t.Parallel` is involved.
* Try swapping registry:2 for the registry add-on
* Drop logf and make Logger private
* git: accept explicit commit hash for git context
When checking out code from non-github repositories, the typical
assumptions may not be valid, e.g. that the only interesting
non-branch commits have ref names starting with refs/pull. A specific
example is fetching an un-merged commit from a gerrit repository by
commit hash.
This change just looks at the second part of the git context path and
checks if it's a SHA commit hash, and if so, will fetch and check out
this commit after cloning the repository.
Sample context argument:
https://github.repo/project#e1772f228e06d15facdf175e5385e265b57068c0
* ci: fix test script to recognize any non-zero exit as an error
hack/linter.sh didn't properly install golangci-lint in hack/bin as I
already have another version of golangci-lint on my PATH, but then it
failed to execute because it was looking for it specifically in
hack/bin.
When the executable is not found, the exit code is 127 instead of 1,
and so test.sh ignored the error.
Two fixes:
1. `test.sh`:
- Use `if (script) ...` instead of assigning / checking a result
variable to determine if each validation script passed or failed.
2. `hack/linter.sh`:
- Instead of checking for golangci-lint on the path, just
specifically check for an executable file (`test -x`) in the
expected location.
Co-authored-by: Wade Carpenter <wwade@users.noreply.github.com>
* chore: add workflows for pr tests
* fix unit tests
* fix formatting
* chore: fix gobuild
* change minikube script
* chore: fix lint install script
* chore: ignore and fix tests
* fix lint and run gofmt
* lint fixes
* k8s executor image only
* fix Makefile
* fix travis env variables
* more info on k8s tests
* fix travis run
* fix
* fix
* fix
* fix log
* some more changes
* increase timeout
* delete travis.yml and fix multiple copy tests
* fix registry mirror
* fix lint
* add concurency
* last attemot to fix k8 integrations
* diff id for diff workflows
* Fix composite cache key for multi-stage copy command (#1706)
PR #1518 reintroduced COPY layers caching using the `--cache-copy-layers`
flag. Unfortunately, this PR also introduced a bug by not including the
stage digest into the caching key of the COPY command when the
`--cache-copy-layers` flag was not set. As a result, kaniko would use
any previous (possibly stalled) layer from the cache because the digest
of the "COPY --from" command would never change.
PR author probably expected Go to fallthrough in the switch just like C
does. However, this is not the case. Go does not fallthrough in
switch-statements by default and requires the fallthrough keyword to be
used. Note that this keyword is not available in type-switches though,
because it wouldn't work properly with typings.
* refactor: add an abstract copy command interface to avoid code duplication
* fix typo in error message
Co-authored-by: Tejal Desai <tejal29@gmail.com>
* avoid deleting whiteouts if they are in ignore list
* fix snapshotter ignore list
* include filesystem mounts in ignorelist of snapshotter
* clean up ignore list logic
* add unit and integration tests for #1652
* fix tests and ignore list updates
* exit codes of failed RUN commands may now be propagated to the caller by using the --propagate-exit-code flag
* removed exit code propagation flag; the exit function now unwraps the error and looks for an exit code
* added integration test for exit code propagation
* added trailing new line in test Dockerfile_exit_code_propagation; tidying up README.md removing all traces of error propagation flag
* moved exit code propagation test to integration_test.go; moved exit code propagation test files to 'testdata' to avoid interference with other integration tests
Fixes#1473
The initial implementation of the registry mirror only allowed a single mirror, and if pulling from the mirror failed, the build would fail.
This change introduces:
- multiple registry mirrors instead of a single one
- fallback if an image can't be pulled from a registry
This is the same behavior as the docker daemon and will allow using a registry mirror such as `mirror.gcr.io` which is incomplete and doesn't have all the content that the default registry on docker.io has.
Note that there are no changes in the CLI flags, the `--registry-mirror` flag is still valid. But now it can be used multiple times to set up more than one registry mirror.
Co-authored-by: Tejal Desai <tejaldesai@google.com>
* Extend .dockerignore integration test with copies in later stages
.dockerignore should continue to apply when copying from the build context in later stages, but it currently doesn't
* Replace excluded global with passed along FileContext struct
This new FileContext struct allows much cleaner handling of context specific file exclusions.
The global excluded file state is no longer needed.
Additionally this also fixes the issue where excluded files aren't being applied for build context copies in later build stages.
Michael Plump
Matthias Schneider
Aaron Prindle
Damien Degois
Maxime BOSSARD
JeromeJu
Manish Giri
Adrià Garriga-Alonso
Fernando Giannetti
Andreas Fleig
Ferran Vidal
Joël Pepper
Aris Buzachis
Natalie Arellano
Gabriel Nützi
Lukas
Tony De La Nuez
Jack
Jason Hall
Andrei Kvapil
François JACQUES
Matt Moore
Tejal Desai
wwade
Gilbert Gilb's
Kamal Nasser
Benjamin Krenn
Vincent Behar
Tinjo Schöni
Yoan Blanc