* Fix#3032: Remove query parameters in ADD command when the destination is a directory
* fixing linter URL sorry forget to lint
* add error in extractFilename and realize that ResolveEnvironmentReplacement better execute before getting the filename
* add /path possible in registry maps and/or mirror
* Fixing Unit test Test_ExtractPathFromRegistryURL
* fix typo library
* fix unit test on ExtractPathFromRegistryURL
* fixing go lint url
* fix typo s/ectract/extract
If git clone context is a private self-signed repository, we allow user
to add --git insecure-skip-tls=true flag in the option. The value is
default to false, this behavior is in accordance with the go-git
package.
* Add flag to remap registries for any registry mirror
The purpose of this PR is to add an option to remap registries, a kind of generalized `--registry-mirror`.
This is helpful for air-gapped environments and/or when local registry mirrors are available (not limited to docker.io).
This allows user to reference any images without having to change their location.
It also permit to separate infra related configuration (the mirrors) from CI/CD pipeline definition by using an environment variable for example (the reason behind the early return if flag provided but empty).
Therefore you can have a pipeline calling kaniko with `--registry-map=$REGISTRY_MAP` and have the `REGISTRY_MAP` populated via the runner's env by another team, and the absence of env wouldn't trigger a failure, it makes the pipeline env independent.
I've also considered the option of environment variables directly but it doesn't seems to be in kaniko's philosophy.
This makes quite some duplicated code :/ One option to keep the mirror flag and behavior would be to use only one codebase and convert `--registry-mirror=VALUE` to `--registry-map=index.docker.io=VALUE` internally. Suggestions welcome!
* Configure logging config sooner to be able to use it in flag parsing
* Replace registry mirrors by maps logic and use env var
* Add env vars to README.md
* Fix test
* Prevent extra snapshot when using new run
* Add unit tests for initializing snapshotter
There should be no snapshot for RunV2. Added a test for SingleSnapshot
as well to prove that the test actually works (rather than `initialized`
just not being read or set properly).
* Add an integration test to reproduce #2892
* Fix go compilation
* Fix docker run cmd
* Fixing entrypoint
* Test warmer with cache in a volume.
* Add missing comma
* Fix imports
* Fix dir
* Add logs
* fix
* Use test framework to log
* Fix warmer failing if image already in cache.
* Fix format.
This commit adds the skip option for otiai10.Copy to skip the /kaniko
directory when the root is being copied. The files under /kaniko dir
should be ignored and thus this shall not cause any loss of information.
fixes: GoogleContainerTools#2033
* Fix warmer memory leak. Write down images directly into a temp file. Add a script to test warmer in boxed memory conditions. Fixes: #2754
* Document usage of boxed_warm_in_docker.sh integration test.
* Create directories with the right UID/GID
* Forgot to create the actual directory
* Integration test creation of intermediate files with correct ownership
* ADD version of the test
* feat: add a retry with result function enabled by --image-download-retry (#2853)
* impl: add a retry with result function
* fix ci errs
* test: add unit tests
* gofmt
* make debian a const
* update param description
This feature allows one to specify an https URL for any of the
digest-file options, resulting in an HTTP PUT to the provided
URL. This could for example be a (pre-signed) URL to S3 or GCS.
Currently the final digest is only written to the local filesystem,
which disappears and is not accessible when Kaniko is run in a
managed container service like AWS ECS.
By supporting https a single implementation supports all storage
services, without the need for special code for S3, GCS, etc..
`sync` system call triggers a full page cache sync which may not always
work, especially in kubernetes environment where it is easy to be
interfered by others. I have seen several cases where a broken nfs mount
is blocking kaniko from doing its job.
With `syncfs`, it only writes cache back to disk for the current
filesystem that is used by kaniko which is supposed to be more reliable.
Ensure zstd compression only gets applied to oci images.
When adding a layer to an image ensure that they are compatable if not convert them.
Create function to convert mediatypes between oci and docker types.
This commit changes the condition check for the behavior when no-push is
set to true while destinations are needed. Prior this change, users would
have to set destinations even when noPush option is set to true. More
specifically, a workaround for tar files to be generated when --no-push is
true and destinations is empty is provided where a dummy destination would be
set.
filepath.Clean shows up in profiles as a hot spot, and there seem to be
many redundant calls, particularly in ignorelist handling. We can avoid
these redundant calls by pre-cleaning entries in the ignore list, and
providing fast paths when we know we're already dealing with a cleaned
candidate path.
Before:
580ms 3.03% 72.35% 590ms 3.08% path/filepath.(*lazybuf).append (inline)
390ms 2.03% 74.39% 990ms 5.16% path/filepath.Clean
After:
0.13s 0.69% 84.01% 0.17s 0.91% path/filepath.(*lazybuf).append (inline)
0.13s 0.69% 84.70% 0.31s 1.65% path/filepath.Clean
* Allow to disable the fallback to the default registry on image pull
When one or more registry mirror(s) are deffined with the 'registry-mirror' argument, if none of those mirrors include the image,
the current behavior is to fallback to the default registry.
If a whitelist (or some image restriction) is applied at the mirror side, fallbacking to the default registry makes that restriction useless.
This new argument allows to skip the fallback and abort the build if the mirror rejects an image.
If it is not set, is completelly transparent.
* fix typo on command help
If a non-empty directory gets replaced with something other than a
directory (e.g. file or symlink), the files in that directory also get
deleted. However, there should not be any whiteout files for them in the
layer. If there were, the resulting tar file would not be extractable.
Fixes#2308
* feat: cache dockerfile images through warmer
* Fix logical error in conditional statement
* Addressed review feedback
1. Updated help text for the --build-arg flag to indicate it should be used with the dockerfile flag.
2. Updated the documentation to include the optional --build-arg flag.
3. Added unit tests for `ParseDockerfile`, covering scenarios for missing Dockerfile, invalid Dockerfile, single stage Dockerfile, multi-stage Dockerfile and Args Dockerfile
---------
Co-authored-by: 连奔驰 <benchi.lian@thoughtworks.com>
* Rename IgnoreListPath to MountInfoPath in config & constants
The string points to /proc/self/mountinfo
* fs_util_test.go: fix tests failing when /tmp mountpoint present
The tests
* Test_GetFSFromLayers_ignorelist
* Test_GetFSFromLayers_with_whiteouts_include_whiteout_disabled
* Test_GetFSFromLayers_with_whiteouts_include_whiteout_enabled
were failing on systems with a /tmp mountpoint:
fs_util.InitIgnoreList() adds all mountpoints to the ignore list,
but the tests were expecting file operations in a /tmp subdirectory.
This change provides an empty mountinfo list for the affected tests.
Fixes#1779
* Removed block on use --cache-copy-layers with multistage builds
* Removed using digest in composite key with command COPY --from
* COPY --from command uses src as file context (only changed files will be reason for change hash)
* ARG and ENV changed before COPY dont change composite key
* Add and fix some tests
* Caching work same as caching in docker buildx
Co-authored-by: Sergei Kraev <skraev@tradingview.com>
`IsSrcRemoteFileURL` was doing a `http.Get` call to make sure the URL was valid, but not surfacing any errors.
Because the error from the http.Get call is not handled, some useful information can be buried.
It also means kaniko will download the file twice during a build, once to validate, and once to actually add the file
to the image.
Removing the http.Get call and validating the URL is valid, and has the correct schema and hostname will stop
the double handling, and allow any errors to be surfaced through the error handing in the file download function.
Fixes#1590
Signed-off-by: Angus Williams <anguswilliams@gmail.com>
* Add mTLS (client cert) support
Add support for Mutual TLS (mTLS) client certificate authentication.
The expected format of the new --registry-client-cert flag is the same
as the existing --registry-certificate flag, which will allow
different client certificates for different registries:
--registry-client-cert my.registry.url=/path/to/cert.crt,/path/to/key.key
* tidy: Rename mTLS (Client Cert) flag to be in line with others
This flag didn't describe that it was for the client certs uses with
the registry. Although this should be reasonably obvious, I like the
consistency with the other registry flag.
* test: Added unit tests for mTLS (Client Cert) loading
* test: Add 2 more tests for comma split formatting
since the comma splitting is a new portion of code let's make sure
that that format works well too in other cases
* tidy: Fix formatting of flag help text
* tidy: Made invalid cert format error consistent
I was running the tests and saw the message:
Failed to load client certificate/key '/path/to/client/certificate.cert' for my.registry.name, format is my.registry.name=/path/to/cert,/path/to/key
I then realized that it'd be a lot nicer if this showed the user what
they input, and how they should change it (rather than decomposing it:
Failed to load client certificate/key 'my.registry.name=/path/to/client/certificate.cert', expected format: my.registry.name=/path/to/cert,/path/to/key
* test: Fixed incorrect test argument
This didn't fail the test before because it's only attempting to show
that certs only get loaded and used for their associated registry but
it's important to keep this correct.
This case is covered by the test below, "RegistriesClientCertificates
incorrect cert format"
* doc: Add new flag to README.md
* mod: Fail to push if there was a problem loading client certs
Rather than warning that there was an issue, we should fail if the
requested client certificates were not found or failed to load.
This feels a lot better than waiting for the build to finish then
failing later.
* mod: Return an error if the certificate authority fails to load, just like client certs
The MakeTransport function was changed in the previous commit to
allow returning errors if there was a problem loading certificates,
rather than just print warnings.
This feels a lot better as you get the error immediately that there's
a problem to fix, rather than getting a warning, then later an error
that the server's certificate could not be verified.
* tidy: fix golint issues
* Add support for configurable compression algorithm (gzip, zstd) and compression level
We want to make the layer compression in kaniko configurable, so we have added two optional command line arguments “--compression” and “--compression-level”. The former allows the user to specify a compression algorithm (zstd, gzip) and the latter can be used to specify the compression level.
Depending on the selected compression algorithm and level we modify the set of layerOptions that are used to create tarball layers in `push.go` and `build.go`.
The actual implementation of the zstd support can be found in our fork of the go-containerregistry package for which we have filed this PR: google/go-containerregistry#1487
The changes should be fully backwards compatible.
* Restrict inputs for compression flag to gzip and zstd
This change will ensure that users can only specify supported compression algorithms (`zstd`, `gzip`) to the `--compression` flag.
* Fix incorrect type for switch statements on config.Compression
Matthias Schneider
Prima Adi Pradana
Alessandro Bitocchi
Jérémie Augustin
Matheus Pimenta
schwannden
Damien Degois
Kraev Sergei
Aaron Prindle
Asher
Bob Du
Maxime BOSSARD
JeromeJu
Manish Giri
Adrià Garriga-Alonso
Anna Levenberg
tal66
Lio李歐
Anna Levenberg
Quan Zhang
zhouhaibing089
Vishal Khot
Logan Price
guangwu
geekvest
Diego Gonzalez
Julian
Aaron Lehmann
Fernando Giannetti
Andreas Fleig
alexezio
Kraev Sergei
Angus Williams
Fedor V
Eric
Lavrenti Frobeen
Lavrenti Frobeen