* Create directories with the right UID/GID
* Forgot to create the actual directory
* Integration test creation of intermediate files with correct ownership
* ADD version of the test
filepath.Clean shows up in profiles as a hot spot, and there seem to be
many redundant calls, particularly in ignorelist handling. We can avoid
these redundant calls by pre-cleaning entries in the ignore list, and
providing fast paths when we know we're already dealing with a cleaned
candidate path.
Before:
580ms 3.03% 72.35% 590ms 3.08% path/filepath.(*lazybuf).append (inline)
390ms 2.03% 74.39% 990ms 5.16% path/filepath.Clean
After:
0.13s 0.69% 84.01% 0.17s 0.91% path/filepath.(*lazybuf).append (inline)
0.13s 0.69% 84.70% 0.31s 1.65% path/filepath.Clean
* Rename IgnoreListPath to MountInfoPath in config & constants
The string points to /proc/self/mountinfo
* fs_util_test.go: fix tests failing when /tmp mountpoint present
The tests
* Test_GetFSFromLayers_ignorelist
* Test_GetFSFromLayers_with_whiteouts_include_whiteout_disabled
* Test_GetFSFromLayers_with_whiteouts_include_whiteout_enabled
were failing on systems with a /tmp mountpoint:
fs_util.InitIgnoreList() adds all mountpoints to the ignore list,
but the tests were expecting file operations in a /tmp subdirectory.
This change provides an empty mountinfo list for the affected tests.
Fixes#1779
`IsSrcRemoteFileURL` was doing a `http.Get` call to make sure the URL was valid, but not surfacing any errors.
Because the error from the http.Get call is not handled, some useful information can be buried.
It also means kaniko will download the file twice during a build, once to validate, and once to actually add the file
to the image.
Removing the http.Get call and validating the URL is valid, and has the correct schema and hostname will stop
the double handling, and allow any errors to be surfaced through the error handing in the file download function.
Fixes#1590
Signed-off-by: Angus Williams <anguswilliams@gmail.com>
* Add mTLS (client cert) support
Add support for Mutual TLS (mTLS) client certificate authentication.
The expected format of the new --registry-client-cert flag is the same
as the existing --registry-certificate flag, which will allow
different client certificates for different registries:
--registry-client-cert my.registry.url=/path/to/cert.crt,/path/to/key.key
* tidy: Rename mTLS (Client Cert) flag to be in line with others
This flag didn't describe that it was for the client certs uses with
the registry. Although this should be reasonably obvious, I like the
consistency with the other registry flag.
* test: Added unit tests for mTLS (Client Cert) loading
* test: Add 2 more tests for comma split formatting
since the comma splitting is a new portion of code let's make sure
that that format works well too in other cases
* tidy: Fix formatting of flag help text
* tidy: Made invalid cert format error consistent
I was running the tests and saw the message:
Failed to load client certificate/key '/path/to/client/certificate.cert' for my.registry.name, format is my.registry.name=/path/to/cert,/path/to/key
I then realized that it'd be a lot nicer if this showed the user what
they input, and how they should change it (rather than decomposing it:
Failed to load client certificate/key 'my.registry.name=/path/to/client/certificate.cert', expected format: my.registry.name=/path/to/cert,/path/to/key
* test: Fixed incorrect test argument
This didn't fail the test before because it's only attempting to show
that certs only get loaded and used for their associated registry but
it's important to keep this correct.
This case is covered by the test below, "RegistriesClientCertificates
incorrect cert format"
* doc: Add new flag to README.md
* mod: Fail to push if there was a problem loading client certs
Rather than warning that there was an issue, we should fail if the
requested client certificates were not found or failed to load.
This feels a lot better than waiting for the build to finish then
failing later.
* mod: Return an error if the certificate authority fails to load, just like client certs
The MakeTransport function was changed in the previous commit to
allow returning errors if there was a problem loading certificates,
rather than just print warnings.
This feels a lot better as you get the error immediately that there's
a problem to fix, rather than getting a warning, then later an error
that the server's certificate could not be verified.
* tidy: fix golint issues
* somehow now the only thing that doesnt work is devices.Device
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* this gets rid of all the compiler errors in the vendored code
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* fixed some things but a bunch of tests and maybe some compiler steps are still failing
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* all the things i figured out how to fix so far
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* guess i had to redo go mods after rebasing again
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* update docker constants to be SHOUTY CASE now
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* include DestPath in resolveEnv
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* fix one mistake in Docker lib upgrade and some typos/deprecations in the file
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* last changes (hopefully) to update to new docker libs
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
---------
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* fix(ci): Bump golangci-lint to 1.51.1
* chore(lint): fix gofmt and goimport issues
* chore(lint): fix linter issues
- Adapted error comparison according to linter recommendation
- Disabled noctx linting for http request where canceling makes no sense
- Disabled nilerror linting where nil error is returned on purpose
- Disabled makezero linter where slice is explicitly deepcopied
* chore(ci): Update go version in tests workflows
* fix(ci): Allow boilerplate years from 2000-2099
Previously the regex only allowed the copyright notice to contain the
years 2018,2019,2020,2021, or 2022. This commit widens to regex to
20\d\d allowing any year in the range [2000-2099]
* feat(ci): Replace minikube with k3s for intregration tests
The existing setup for minikube is very complicated, replicating most of
the setup steps for a full kubernetes cluster in an only partially
supported minikube configuration (driver=none). Furthermore the existing
setup has been broken for sometime, likely, at least in part due to the
changes to CNI and CRI in recent kubernetes versions.
Since what we actually need is only a running Kubernetes cluster on the
node and access to a registry on localhost:5000, we can switch the
extremely complicated minikube setup for a lightweight cluster using
k3s. Minikube came with a default addon for running a registry on every
node, but the same is not the case for k3s, instead we make use of the
package helm controller and its HelmChart CR to deploy twuni/docker-registry.helm
and expose it on localhost using the integrated LoadBalancer controller.
* fix(test-684): pin base container version
The dockerfile for the regression test connected to issue 684 used a
rolling tag as base image, making it flaky and fail since it was
introduced.
This commit pins the base image to the digest of bionic-20200219, which,
based on the date of the commit that introduced to the dockerfile would
be the most newest ubuntu build and likely what the "rolling" tag
resolved to back then. Since this also an image from the pre-oci days of
ubuntu, this circumvents a bug in container-diff as well
(https://github.com/GoogleContainerTools/container-diff/issues/389)
WORKDIR ignores the currently set USER and creates the new directories
with the root user ownership.
This changes that, by executing a chown after the mkdir if needed, and
also handle the case where the provided USER is an uid and the passwd
file is not available to resolve to the username.
Fixes#2259
Signed-off-by: Aris Buzachis <buzachis.aris@gmail.com>
Signed-off-by: Aris Buzachis <buzachis.aris@gmail.com>
* fix: getUIDandGID is able to resolve non-existing users and groups
A common pattern in dockerfiles is to provide a plain uid and gid number, which doesn't neccesarily exist inside the os.
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* test: add chown dockerfile
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore: format
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore: add comment
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* tests: fix chown dockerfile
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* refactor: split up getIdsFromUsernameAndGroup func
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* fix: implement raw uid logic for LookupUser
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* test: add dockerfiles for integration test
* fix: lookup user error message
* test: add dockerfiles for non-existing user testcase
* fix: forgot error check
* tests: fix syscall credentials test
* chore: add debug output for copy command
* tests: set specific gid for integration dockerfile
* tests: fix syscall credentials test
github runner had the exact uid that i was testing on, so the groups were not empty
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* tests: fix test script
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore: apply golangci lint checks
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* fix: reset file ownership in createFile if not root owned
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore: logrus.Debugf missed format variable
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* chore(test-script): remove go html coverage
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* test(k8s): increase wait timeout
Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
* Fix missing file permissions on multi-stage build
Fixes#2075
When a file with the setuid bit is copied from one stage
to another, the permissions were not copied over properly after
setting ownership on directory and the file itself.
* Update pkg/util/fs_util.go
Co-authored-by: Jason Hall <jason@chainguard.dev>
* Adding boilerplate to dockerfile
* Add bash check to bail with exit code 1 if setuid not present
Co-authored-by: Jason Hall <jason@chainguard.dev>
* fix: Correct flatten function in layers
- Added a test.
- Cache current image, track deletes in `whiteouts` as well as normal adds in `layers`.
- Fix ugly delete behavior of `layerHashCache`.
Delete it when crerating a new snapshot.
- Slight cleanup in `snapshot.go`.
- Format ugly `WalkFS` function.
* fix: Add symbolic link changes to Hasher and CacheHasher
* fix: Better log messages
* fix(ci): Integration tests
* fix(ci): Add `--no-cache` to docker builds
* fix(ci): Pass credentials for error integration test
* np: Missing .gitignore in `hack`
* np: Capitalize every log message
- Correct some linting.
* fix: Key function
- Merge only last layer onto `currentImage`.
* fix: Remove old obsolete `cacheHasher`
In Dockerfile, if there is something like:
```
RUN setcap cap_net_raw=+ep /path/to/binary
```
kaniko won't detect that there is a change on file `/path/to/binary` and
thus discards this layer. This patch allows the hasher function to
actually look at `security.capability` extended attributes.
Fix#1936
Kubernetes was not being detected by files not by /proc/?/cgroup
contents. Now it detects the kubernetes runtime if any of those
conditions are met:
* /var/run/secrets/kubernetes.io/serviceaccount exists
* /proc/mounts has the mount for "/" with fs type "overlay"
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
When os.Stat returns an error different from ErrNotExist,
mkdirAllWithPermissions may panic with a nil pointer
derefence due to insufficient error checking.
Avoid the panic by bailing out, returning the error to the
caller.
* avoid deleting whiteouts if they are in ignore list
* fix snapshotter ignore list
* include filesystem mounts in ignorelist of snapshotter
* clean up ignore list logic
* add unit and integration tests for #1652
* fix tests and ignore list updates
which means we can now:
- set up one or more mirrors
- set up registries certificates
- skip TLS verify
- use plain HTTP
using the same set of flags that are defined for the executor
* Extend .dockerignore integration test with copies in later stages
.dockerignore should continue to apply when copying from the build context in later stages, but it currently doesn't
* Replace excluded global with passed along FileContext struct
This new FileContext struct allows much cleaner handling of context specific file exclusions.
The global excluded file state is no longer needed.
Additionally this also fixes the issue where excluded files aren't being applied for build context copies in later build stages.
This allows those using Kaniko as a library to add ignored dirs and
files without needs to modify /proc/self/mountinfo or doing other
strange things with mount -t tmpfs
Manish Giri
Adrià Garriga-Alonso
tal66
Anna Levenberg
Quan Zhang
Vishal Khot
Aaron Prindle
geekvest
Diego Gonzalez
Aaron Lehmann
Andreas Fleig
Angus Williams
Eric
Joe Kimmel
Joël Pepper
Aris Buzachis
Abirdcfly
Lukas
Tony De La Nuez
Gabriel Nützi
zhouhaibing089
Víctor
Igor Scheller
Eng Zer Jun
Jason Hall
Andrei Kvapil
Lars Seipel
Travis DePrato
François JACQUES
Tejal Desai
nihilo
Kamal Nasser
Jon Friesen
Sascha Schwarze
Vincent Behar
Sladyn
Shashank
Tinjo Schöni
Joe Kutner
Yoan Blanc