If a non-empty directory gets replaced with something other than a
directory (e.g. file or symlink), the files in that directory also get
deleted. However, there should not be any whiteout files for them in the
layer. If there were, the resulting tar file would not be extractable.
Fixes#2308
* feat: cache dockerfile images through warmer
* Fix logical error in conditional statement
* Addressed review feedback
1. Updated help text for the --build-arg flag to indicate it should be used with the dockerfile flag.
2. Updated the documentation to include the optional --build-arg flag.
3. Added unit tests for `ParseDockerfile`, covering scenarios for missing Dockerfile, invalid Dockerfile, single stage Dockerfile, multi-stage Dockerfile and Args Dockerfile
---------
Co-authored-by: 连奔驰 <benchi.lian@thoughtworks.com>
* Rename IgnoreListPath to MountInfoPath in config & constants
The string points to /proc/self/mountinfo
* fs_util_test.go: fix tests failing when /tmp mountpoint present
The tests
* Test_GetFSFromLayers_ignorelist
* Test_GetFSFromLayers_with_whiteouts_include_whiteout_disabled
* Test_GetFSFromLayers_with_whiteouts_include_whiteout_enabled
were failing on systems with a /tmp mountpoint:
fs_util.InitIgnoreList() adds all mountpoints to the ignore list,
but the tests were expecting file operations in a /tmp subdirectory.
This change provides an empty mountinfo list for the affected tests.
Fixes#1779
* Removed block on use --cache-copy-layers with multistage builds
* Removed using digest in composite key with command COPY --from
* COPY --from command uses src as file context (only changed files will be reason for change hash)
* ARG and ENV changed before COPY dont change composite key
* Add and fix some tests
* Caching work same as caching in docker buildx
Co-authored-by: Sergei Kraev <skraev@tradingview.com>
`IsSrcRemoteFileURL` was doing a `http.Get` call to make sure the URL was valid, but not surfacing any errors.
Because the error from the http.Get call is not handled, some useful information can be buried.
It also means kaniko will download the file twice during a build, once to validate, and once to actually add the file
to the image.
Removing the http.Get call and validating the URL is valid, and has the correct schema and hostname will stop
the double handling, and allow any errors to be surfaced through the error handing in the file download function.
Fixes#1590
Signed-off-by: Angus Williams <anguswilliams@gmail.com>
* Add mTLS (client cert) support
Add support for Mutual TLS (mTLS) client certificate authentication.
The expected format of the new --registry-client-cert flag is the same
as the existing --registry-certificate flag, which will allow
different client certificates for different registries:
--registry-client-cert my.registry.url=/path/to/cert.crt,/path/to/key.key
* tidy: Rename mTLS (Client Cert) flag to be in line with others
This flag didn't describe that it was for the client certs uses with
the registry. Although this should be reasonably obvious, I like the
consistency with the other registry flag.
* test: Added unit tests for mTLS (Client Cert) loading
* test: Add 2 more tests for comma split formatting
since the comma splitting is a new portion of code let's make sure
that that format works well too in other cases
* tidy: Fix formatting of flag help text
* tidy: Made invalid cert format error consistent
I was running the tests and saw the message:
Failed to load client certificate/key '/path/to/client/certificate.cert' for my.registry.name, format is my.registry.name=/path/to/cert,/path/to/key
I then realized that it'd be a lot nicer if this showed the user what
they input, and how they should change it (rather than decomposing it:
Failed to load client certificate/key 'my.registry.name=/path/to/client/certificate.cert', expected format: my.registry.name=/path/to/cert,/path/to/key
* test: Fixed incorrect test argument
This didn't fail the test before because it's only attempting to show
that certs only get loaded and used for their associated registry but
it's important to keep this correct.
This case is covered by the test below, "RegistriesClientCertificates
incorrect cert format"
* doc: Add new flag to README.md
* mod: Fail to push if there was a problem loading client certs
Rather than warning that there was an issue, we should fail if the
requested client certificates were not found or failed to load.
This feels a lot better than waiting for the build to finish then
failing later.
* mod: Return an error if the certificate authority fails to load, just like client certs
The MakeTransport function was changed in the previous commit to
allow returning errors if there was a problem loading certificates,
rather than just print warnings.
This feels a lot better as you get the error immediately that there's
a problem to fix, rather than getting a warning, then later an error
that the server's certificate could not be verified.
* tidy: fix golint issues
* Add support for configurable compression algorithm (gzip, zstd) and compression level
We want to make the layer compression in kaniko configurable, so we have added two optional command line arguments “--compression” and “--compression-level”. The former allows the user to specify a compression algorithm (zstd, gzip) and the latter can be used to specify the compression level.
Depending on the selected compression algorithm and level we modify the set of layerOptions that are used to create tarball layers in `push.go` and `build.go`.
The actual implementation of the zstd support can be found in our fork of the go-containerregistry package for which we have filed this PR: google/go-containerregistry#1487
The changes should be fully backwards compatible.
* Restrict inputs for compression flag to gzip and zstd
This change will ensure that users can only specify supported compression algorithms (`zstd`, `gzip`) to the `--compression` flag.
* Fix incorrect type for switch statements on config.Compression
* somehow now the only thing that doesnt work is devices.Device
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* this gets rid of all the compiler errors in the vendored code
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* fixed some things but a bunch of tests and maybe some compiler steps are still failing
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* all the things i figured out how to fix so far
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* guess i had to redo go mods after rebasing again
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* update docker constants to be SHOUTY CASE now
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* include DestPath in resolveEnv
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* fix one mistake in Docker lib upgrade and some typos/deprecations in the file
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* last changes (hopefully) to update to new docker libs
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
---------
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* fix(ci): Bump golangci-lint to 1.51.1
* chore(lint): fix gofmt and goimport issues
* chore(lint): fix linter issues
- Adapted error comparison according to linter recommendation
- Disabled noctx linting for http request where canceling makes no sense
- Disabled nilerror linting where nil error is returned on purpose
- Disabled makezero linter where slice is explicitly deepcopied
* chore(ci): Update go version in tests workflows
* fix(ci): Allow boilerplate years from 2000-2099
Previously the regex only allowed the copyright notice to contain the
years 2018,2019,2020,2021, or 2022. This commit widens to regex to
20\d\d allowing any year in the range [2000-2099]
* feat(ci): Replace minikube with k3s for intregration tests
The existing setup for minikube is very complicated, replicating most of
the setup steps for a full kubernetes cluster in an only partially
supported minikube configuration (driver=none). Furthermore the existing
setup has been broken for sometime, likely, at least in part due to the
changes to CNI and CRI in recent kubernetes versions.
Since what we actually need is only a running Kubernetes cluster on the
node and access to a registry on localhost:5000, we can switch the
extremely complicated minikube setup for a lightweight cluster using
k3s. Minikube came with a default addon for running a registry on every
node, but the same is not the case for k3s, instead we make use of the
package helm controller and its HelmChart CR to deploy twuni/docker-registry.helm
and expose it on localhost using the integrated LoadBalancer controller.
* fix(test-684): pin base container version
The dockerfile for the regression test connected to issue 684 used a
rolling tag as base image, making it flaky and fail since it was
introduced.
This commit pins the base image to the digest of bionic-20200219, which,
based on the date of the commit that introduced to the dockerfile would
be the most newest ubuntu build and likely what the "rolling" tag
resolved to back then. Since this also an image from the pre-oci days of
ubuntu, this circumvents a bug in container-diff as well
(https://github.com/GoogleContainerTools/container-diff/issues/389)
WORKDIR ignores the currently set USER and creates the new directories
with the root user ownership.
This changes that, by executing a chown after the mkdir if needed, and
also handle the case where the provided USER is an uid and the passwd
file is not available to resolve to the username.
Fixes#2259
Signed-off-by: Aris Buzachis <buzachis.aris@gmail.com>
Signed-off-by: Aris Buzachis <buzachis.aris@gmail.com>
* Adds the ability to provide `--cache-repo` as an OCI image layout path
- Adds cache.LayoutCache to implement cache.LayerCache interface
- When opts.CacheRepo has "oci:" prefix, instantiates a LayoutCache
Signed-off-by: Natalie Arellano <narellano@vmware.com>
* Add integration test for layout cache
Signed-off-by: Natalie Arellano <narellano@vmware.com>
* Updates from PR review
Signed-off-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Natalie Arellano <narellano@vmware.com>
On version 1.9.0 context fetch from S3 is broken since `url.Parse` requires full path: 90e426ba3f/pkg/util/bucket/bucket_util.go (L77)
Currently on a --context=s3://my-bucket/my-path/context.tar.gz we would pass only my-bucket/my-path/context.tar.gz which breaks context fetch
Closes#2200
- Adds a new option, InitialFSUnpacked
- When opts.InitialFSUnpacked is true, the first stage builder will
skip unpacking the file system; later stages are unaffected
Signed-off-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Natalie Arellano <narellano@vmware.com>
* fix: getUIDandGID is able to resolve non-existing users and groups
A common pattern in dockerfiles is to provide a plain uid and gid number, which doesn't neccesarily exist inside the os.
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* test: add chown dockerfile
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore: format
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore: add comment
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* tests: fix chown dockerfile
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* refactor: split up getIdsFromUsernameAndGroup func
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* fix: implement raw uid logic for LookupUser
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* test: add dockerfiles for integration test
* fix: lookup user error message
* test: add dockerfiles for non-existing user testcase
* fix: forgot error check
* tests: fix syscall credentials test
* chore: add debug output for copy command
* tests: set specific gid for integration dockerfile
* tests: fix syscall credentials test
github runner had the exact uid that i was testing on, so the groups were not empty
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* tests: fix test script
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore: apply golangci lint checks
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* fix: reset file ownership in createFile if not root owned
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore: logrus.Debugf missed format variable
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore(test-script): remove go html coverage
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* test(k8s): increase wait timeout
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* Write parent directories to tar before whiteout files
Fixes#1149
The OCI image spec does not specify this order but it's a good idea and Docker
does the same.
When manually comparing layers created by Docker and Kaniko there are still
some differences (that container-diff does not show):
* Kaniko adds / to layers
* For `mkdir /test`, docker adds `/test` and an opaque whiteout file
`/test/.wh..wh..opq`. Kaniko only adds `/test/` (and /).
* snapshot_test: cleanup
Fix typos and use listFilesInTar() where possible
* Fix missing file permissions on multi-stage build
Fixes#2075
When a file with the setuid bit is copied from one stage
to another, the permissions were not copied over properly after
setting ownership on directory and the file itself.
* Update pkg/util/fs_util.go
Co-authored-by: Jason Hall <jason@chainguard.dev>
* Adding boilerplate to dockerfile
* Add bash check to bail with exit code 1 if setuid not present
Co-authored-by: Jason Hall <jason@chainguard.dev>
* fix: Correct flatten function in layers
- Added a test.
- Cache current image, track deletes in `whiteouts` as well as normal adds in `layers`.
- Fix ugly delete behavior of `layerHashCache`.
Delete it when crerating a new snapshot.
- Slight cleanup in `snapshot.go`.
- Format ugly `WalkFS` function.
* fix: Add symbolic link changes to Hasher and CacheHasher
* fix: Better log messages
* fix(ci): Integration tests
* fix(ci): Add `--no-cache` to docker builds
* fix(ci): Pass credentials for error integration test
* np: Missing .gitignore in `hack`
* np: Capitalize every log message
- Correct some linting.
* fix: Key function
- Merge only last layer onto `currentImage`.
* fix: Remove old obsolete `cacheHasher`
In Dockerfile, if there is something like:
```
RUN setcap cap_net_raw=+ep /path/to/binary
```
kaniko won't detect that there is a change on file `/path/to/binary` and
thus discards this layer. This patch allows the hasher function to
actually look at `security.capability` extended attributes.
Fix#1936
Kubernetes was not being detected by files not by /proc/?/cgroup
contents. Now it detects the kubernetes runtime if any of those
conditions are met:
* /var/run/secrets/kubernetes.io/serviceaccount exists
* /proc/mounts has the mount for "/" with fs type "overlay"
This simplifies usage of Kaniko in GitLab CI environments and
means that it's not longer necessary to manually cobble together
the config with the right values from the environment in
error-prone shell commands.
* Use canonical image architectures. Closes#1995.
Prior to this change, Kaniko was not using the platform's canonical value.
* Update platform unit test.
* remove configurables from constants
* add configurables to config init
* add kaniko dir flag
* cleanup pkg constants
* replace buildcontext constants pkg occurrences with config
* add KanikoDir to KanikoOptions
* replace executor constants pkg occurrences with config
* remove redundant KanikoDir assignment
* replace constants to config for IntermediateStagesDir
* fix imports
* add default kaniko directory into constants
* add check for kanikoDir on use
* update init to use default path constant
* update executor kanikoDir check
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* alter checkKanikoDir parameter
* add TestKanikoDir func
* update error handling style
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* remove shorthand flag usage from test
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* add docstring to integration test
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* remove shorthand flag from kaniko-dir
Co-authored-by: Jason Hall <jasonhall@redhat.com>
Fixed the codeql issue
```
Sanitizing untrusted URLs is an important technique for preventing attacks such as request forgeries and malicious redirections. Often, this is done by checking that the host of a URL is in a set of allowed hosts.
If a regular expression implements such a check, it is easy to accidentally make the check too permissive by not escaping regular-expression meta-characters such as ..
Even if the check is not used in a security-critical context, the incomplete check may still cause undesirable behavior when it accidentally succeeds.
```
Prior to this change, Kaniko disabled logging for its ecr-login
dependency by redirecting the logger's output to ioutil.Discard.
However, since that used the logrus standard logger, this had the effect
of completely disabling (redirecting to Discard) _all_ logging done by
Kaniko.
This caused integratiion test failures, and left any user of Kaniko
without any logging.
This fix updates the dependency on ecr-login to include a fix for this
issue, and updates to the new API introduced in that fix.
Previously the ecr-login cred helper we compile into the executor/warmer
binaries would log a warning when they were asked for credentials for a
non-ECR image. This takes advantage of recent work in ecr-login to let
consumers like kaniko disable that logging.
* Fix resolving argument over mulit-stage build
- Building multi-stage Dockerfiles did not correctly resolve
ARG commands over different stages. Now, each stage depends on the
build arguments set by the stage before.
Closes: #1911
* Fix resolving empty arg commands
* Correctly handle platforms that include CPU variants
Prior to this change, Kaniko would only select the platform-specific
image from a multi-platform image using the OS and architecture to
select the image. This leads to problems when there are two platforms
that are indistinguishable using only this information (e.g.,
linux/arm/v5 and linux/arm/v7).
This change more explicitly selects the right image from a
multi-platform image, taking CPU variant into account (v5 vs v7), using
containerd's CPU variant detection logic.
This also moves platform defaulting up as early as possible in execution
as it can go, so that malformed platform values are detected as soon as
possible.
* set platform in unit test
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Bump github.com/docker/docker to latest release
* Pin deps to docker/docker's requirements
fix build breakages
fix unit test
Trying to see whether am-i-in-a-container checks are broken for integration tests...
Andreas Fleig
alexezio
Kraev Sergei
Angus Williams
Bob Du
Fedor V
Eric
Aaron Prindle
Lavrenti Frobeen
Lavrenti Frobeen
Joe Kimmel
Joël Pepper
Aris Buzachis
Natalie Arellano
almg80
Abirdcfly
dmr
Gabriel Nützi
Ramy
Lukas
Hingbong Lo
Tony De La Nuez
Jason D'Amour
zhouhaibing089
Víctor
François JACQUES
ePirat
Diego Gonzalez
Jack
Igor Scheller
Naveen
Jason Hall
Florian Apolloner
Eng Zer Jun
Dawei Ma