* Prevent extra snapshot when using new run
* Add unit tests for initializing snapshotter
There should be no snapshot for RunV2. Added a test for SingleSnapshot
as well to prove that the test actually works (rather than `initialized`
just not being read or set properly).
Ensure zstd compression only gets applied to oci images.
When adding a layer to an image ensure that they are compatable if not convert them.
Create function to convert mediatypes between oci and docker types.
* Removed block on use --cache-copy-layers with multistage builds
* Removed using digest in composite key with command COPY --from
* COPY --from command uses src as file context (only changed files will be reason for change hash)
* ARG and ENV changed before COPY dont change composite key
* Add and fix some tests
* Caching work same as caching in docker buildx
Co-authored-by: Sergei Kraev <skraev@tradingview.com>
* Add support for configurable compression algorithm (gzip, zstd) and compression level
We want to make the layer compression in kaniko configurable, so we have added two optional command line arguments “--compression” and “--compression-level”. The former allows the user to specify a compression algorithm (zstd, gzip) and the latter can be used to specify the compression level.
Depending on the selected compression algorithm and level we modify the set of layerOptions that are used to create tarball layers in `push.go` and `build.go`.
The actual implementation of the zstd support can be found in our fork of the go-containerregistry package for which we have filed this PR: google/go-containerregistry#1487
The changes should be fully backwards compatible.
* Restrict inputs for compression flag to gzip and zstd
This change will ensure that users can only specify supported compression algorithms (`zstd`, `gzip`) to the `--compression` flag.
* Fix incorrect type for switch statements on config.Compression
* somehow now the only thing that doesnt work is devices.Device
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* this gets rid of all the compiler errors in the vendored code
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* fixed some things but a bunch of tests and maybe some compiler steps are still failing
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* all the things i figured out how to fix so far
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* guess i had to redo go mods after rebasing again
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* update docker constants to be SHOUTY CASE now
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* include DestPath in resolveEnv
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* fix one mistake in Docker lib upgrade and some typos/deprecations in the file
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* last changes (hopefully) to update to new docker libs
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
---------
Signed-off-by: Joe Kimmel <jkimmel@vmware.com>
* Adds the ability to provide `--cache-repo` as an OCI image layout path
- Adds cache.LayoutCache to implement cache.LayerCache interface
- When opts.CacheRepo has "oci:" prefix, instantiates a LayoutCache
Signed-off-by: Natalie Arellano <narellano@vmware.com>
* Add integration test for layout cache
Signed-off-by: Natalie Arellano <narellano@vmware.com>
* Updates from PR review
Signed-off-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Natalie Arellano <narellano@vmware.com>
- Adds a new option, InitialFSUnpacked
- When opts.InitialFSUnpacked is true, the first stage builder will
skip unpacking the file system; later stages are unaffected
Signed-off-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Natalie Arellano <narellano@vmware.com>
* fix: Correct flatten function in layers
- Added a test.
- Cache current image, track deletes in `whiteouts` as well as normal adds in `layers`.
- Fix ugly delete behavior of `layerHashCache`.
Delete it when crerating a new snapshot.
- Slight cleanup in `snapshot.go`.
- Format ugly `WalkFS` function.
* fix: Add symbolic link changes to Hasher and CacheHasher
* fix: Better log messages
* fix(ci): Integration tests
* fix(ci): Add `--no-cache` to docker builds
* fix(ci): Pass credentials for error integration test
* np: Missing .gitignore in `hack`
* np: Capitalize every log message
- Correct some linting.
* fix: Key function
- Merge only last layer onto `currentImage`.
* fix: Remove old obsolete `cacheHasher`
* remove configurables from constants
* add configurables to config init
* add kaniko dir flag
* cleanup pkg constants
* replace buildcontext constants pkg occurrences with config
* add KanikoDir to KanikoOptions
* replace executor constants pkg occurrences with config
* remove redundant KanikoDir assignment
* replace constants to config for IntermediateStagesDir
* fix imports
* add default kaniko directory into constants
* add check for kanikoDir on use
* update init to use default path constant
* update executor kanikoDir check
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* alter checkKanikoDir parameter
* add TestKanikoDir func
* update error handling style
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* remove shorthand flag usage from test
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* add docstring to integration test
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* remove shorthand flag from kaniko-dir
Co-authored-by: Jason Hall <jasonhall@redhat.com>
* Fix resolving argument over mulit-stage build
- Building multi-stage Dockerfiles did not correctly resolve
ARG commands over different stages. Now, each stage depends on the
build arguments set by the stage before.
Closes: #1911
* Fix resolving empty arg commands
* Bump github.com/docker/docker to latest release
* Pin deps to docker/docker's requirements
fix build breakages
fix unit test
Trying to see whether am-i-in-a-container checks are broken for integration tests...
* Remove tarball.WithCompressedCaching flag to resolve OOM Killed error
Large images cannot be build as the kaniko container will be killed due to an OOM error. Removing the tarball compression drastically reduces the memory required to push large image layers. Fixes#1680
This change may increase the build time for smaller images. Therefore a command line option to trigger the compression or a more intelligent behaviour may be useful.
* Add new command line flag to toggle compressed caching
* Add unittest for build with --compressed-caching command line flag set to false
* chore: add workflows for pr tests
* fix unit tests
* fix formatting
* chore: fix gobuild
* change minikube script
* chore: fix lint install script
* chore: ignore and fix tests
* fix lint and run gofmt
* lint fixes
* k8s executor image only
* fix Makefile
* fix travis env variables
* more info on k8s tests
* fix travis run
* fix
* fix
* fix
* fix log
* some more changes
* increase timeout
* delete travis.yml and fix multiple copy tests
* fix registry mirror
* fix lint
* add concurency
* last attemot to fix k8 integrations
* diff id for diff workflows
* Fix composite cache key for multi-stage copy command (#1706)
PR #1518 reintroduced COPY layers caching using the `--cache-copy-layers`
flag. Unfortunately, this PR also introduced a bug by not including the
stage digest into the caching key of the COPY command when the
`--cache-copy-layers` flag was not set. As a result, kaniko would use
any previous (possibly stalled) layer from the cache because the digest
of the "COPY --from" command would never change.
PR author probably expected Go to fallthrough in the switch just like C
does. However, this is not the case. Go does not fallthrough in
switch-statements by default and requires the fallthrough keyword to be
used. Note that this keyword is not available in type-switches though,
because it wouldn't work properly with typings.
* refactor: add an abstract copy command interface to avoid code duplication
* fix typo in error message
Co-authored-by: Tejal Desai <tejal29@gmail.com>
* avoid deleting whiteouts if they are in ignore list
* fix snapshotter ignore list
* include filesystem mounts in ignorelist of snapshotter
* clean up ignore list logic
* add unit and integration tests for #1652
* fix tests and ignore list updates
which means we can now:
- set up one or more mirrors
- set up registries certificates
- skip TLS verify
- use plain HTTP
using the same set of flags that are defined for the executor
* Extend .dockerignore integration test with copies in later stages
.dockerignore should continue to apply when copying from the build context in later stages, but it currently doesn't
* Replace excluded global with passed along FileContext struct
This new FileContext struct allows much cleaner handling of context specific file exclusions.
The global excluded file state is no longer needed.
Additionally this also fixes the issue where excluded files aren't being applied for build context copies in later build stages.
Cached COPY layers are expensive in that they both need to be retrieved
over the network and occupy space in the layer cache.
They are unnecessary in that we already have all resources needed to
execute the COPY locally, and doing so is a trivial file-system
operation. This is in contrast to RUN layers, which can do
arbitrary and unbounded work.
The end result is that cached COPY commands were more expensive when
cached, not less. Remove them.
Resolves#1357
Nikita
Kraev Sergei
Asher
tal66
Logan Price
geekvest
Kraev Sergei
Fedor V
Lavrenti Frobeen
Lavrenti Frobeen
Joe Kimmel
Natalie Arellano
Ramy
Hingbong Lo
Gabriel Nützi
Jason D'Amour
Jack
Florian Apolloner
Jason Hall
Dawei Ma
Janosch Maier
Gilbert Gilb's
Tejal Desai
Rhianna
Jose Donizetti
Kamal Nasser
Matt Moore
Vincent Behar
Lars
mickkael
Tinjo Schöni
Ian Kerins
Tejal Desai
Ben Einaudi
Jordan Goasdoue