update stage code so that when comparing the BaseName of
a stage against the recorded, lowercase version of a Stage name
the BaseName is also lowercased.
- Change integration tests to use docker hub instead of GCR due to bug
in library that requires authentication with gcr.io even for public
images. See #966 for bug tracking this.
- Make uploading to GCS bucket configurable through a flag --uploadToGCS
- Utilize a locally deployed docker registry in travis CI to remove
dependency of needing to authenticate with GCP. This requires host
networking such that we can access the registry on localhost:5000
- Use the commit that's being tested for `TestGitBuildcontext`
- Remove duplicate GitBuildContext case that's now covered by default in
travis CI.
* add util.GetFSFromLayers
* GetFSFromImage delegates to GetFSFromLayers
* add FSOpts and FSConfig for GetFSFromLayers
* add tests for GetFSFromLayers
* add gomock for test support
* add mock_v1 for layers
ArgsEscaped according to Docker docs should only be set in Windows
environments: https://docs.docker.com/engine/api/v1.30/
It was causing integration test to fail with following message:
```
FAIL: TestRun/test_Dockerfile_test_metadata (8.48s)
"Diff": {
"Adds": [
"ArgsEscaped: true"
],
"Dels": [
"ArgsEscaped: false"
]
```
However docker 18.xx returns ArgsEscaped: true
whereas docker 19.xx returns ArgsEscaped: false
Hence this patch also adds the docker version check to the integration
to ignore ArgsEscaped being different when 18.xx is used.
Previously it would mount .config/gcloud directory which is not
recommended for systems such as CI that authenticate with Google Cloud.
This commit allows you to set the path to a service account.
By default previous behaviour will be as before so this shouldn't break
existing systems that run the integration test.
From Golang 1.13 release notes:
Testing flags are now registered in the new Init function, which is
invoked by the generated main function for the test. As a result,
testing flags are now only registered when running a test binary, and
packages that call flag.Parse during package initialization may cause
tests to fail.
The executor accepts a few arguments dockerfile, context, cache-dir and
digest-file that all represent paths. This commits allows those paths to
be relative to the working directory of the executor.
Fixes#732#731#675
* Revert "Change cache key calculation to be more reproducible. (#525)"
This reverts commit 1ffae47fdd.
* Add logging of composition key back
* Do not include build args in cache key
This should be save, given that the commands will have the args included
when the cache key gets built.
* Add parent directories of adding files
* Add integration Dockerfile to test parent directory permissions
* Remove unnecessary helper method
* Use a file on the internet for integration Dockerfile
- We were validating usernames/groupnames existed in etc/passwd. Docker does not do this
- We were incorrectly caching USER commands. This was fixed automatically by fixing the first part.
and our snapshot optimizations.
If a previous base image has a volume, the directory is added to the
list of files to snapshot. That directory may not actually exist in the image.
We previously had an optimization that would skip snapshotting mutli-stage images
when in an intermediate stage, until the very end.
This conflicted with another optimization to avoid snapshotting when no files had changed.
Before we were using the full image digest, but that contains a timestamp. Now
we only use the layers themselves and the image config (env vars, etc.).
Also fix a bug in unpacking the layers themselves. mtimes can change during unpacking,
so set them all once at the end.
This works around mtime-precision issues that cause us to differ from docker-built images
when the touch command executes too quickly to modify the mtime.
Also one fix in the .travis.yml beacuse something appears to have changed in their default
go installation.
This PR adds support for the dockerignore file. Previously when kaniko
had support for the dockerignore file, kaniko first went through the
build context and deleted files that were meant to be ignored. This
resulted in a really bad bug where files in user mounted volumes would
be deleted (my bad).
This time around, instead of modifying the build context at all, kaniko
will check if a file should be excluded when executing ADD/COPY
commands. If a file should be excluded (based on the .dockerignore) it
won't be copied over from the buildcontext and shouldn't end up in the
final image.
I also added a .dockerignore file and Dockerfile as an integration test,
which should fail if the dockerignore is not being processed correctly or if files aren't being excluded correctly.
Also, I removed all the integration testing from the previous version of the
dockerignore support.
Right now kaniko only supports COPY --from=<another stage>.
This commit adds support for the case where the referenced image is a remote image
in a registry that has not been used as a stage yet in the build.
* adding benchmarking code
* enable writing to file
* fix build
* time more stuff
* adding benchmarking to integration tests
* compare docker and kaniko times in integration tests
* Switch to setting benchmark file with an env var
* close file at the right time
* fix integration test with environment variables
* fix integration tests
* Adding benchmarking documentation to DEVELOPEMENT.md
* human readable benchmarking steps
From the docs on filepath.SkipDir:
> If the function returns SkipDir when invoked on a non-directory file, Walk skips the remaining files in the containing directory
This was causing the bug in #457. Since the file `/etc/hosts` was in the whitelist, when filepath.SkipDir was called the entire etc directory was skipped.
This change only returns filepath.SkipDir on directories.
When we execute multistage builds, we store the fs of each intermediate
stage at /kaniko/<stage number> if it's used later in the build. This
created a bug when extracting hardlinks, because we weren't appending
the new directory to the link path.
So, if `/tmp/file1` and `/tmp/file2` were hardlinked, kaniko was trying
to link `/kaniko/0/tmp/file1` to `/tmp/file2` instead of
`/kaniko/0/tmp/file2`. This change will append the correct directory to
the link, and fixes#437#362#352#342.
This change fixes that by properly "replaying" the Dockerfile and mutating the config when
calculating cache keys. Previously we were looking at the wrong cache key for each command
when there was more than one.
* parse arg commands at the top of dockerfiles
* fix pointer reference bug and remove debugging
* fixing tests
* account for meta args with no value
* don't take fs snapshot if / is the only changed path
* move metaArgs inside KanikoStage
* removing unused property
* check for any directory instead of just /
* remove unnecessary check
I improved handling of the .dockerignore file by:
1. Using docker's parser to parse the .dockerignore and using their
helper functions to determine if a file should be deleted
2. Copying the Dockerfile we are building to /kaniko/Dockerfile so that
if the Dockerfile is specified in .dockerignore it won't be deleted, and
if it is specified in the .dockerignore it won't end up in the final
image
3. I also improved the integration test to create a temp directory with
files to ignore, and updated the .dockerignore to include exclusions (!)
It seems like .dockerignore deletes files containeed within the file locally upon
docker build, so I created a temporary test dir to make sure the
--ignore flag works. We make sure it exists before building docker and
kaniko for Dockerfile_test_ignore, and then delete it after the builds
have completed.
Added a --ignore flag to ignore packages and files in the build context.
This should mimic the .dockerignore file. Before starting the build, we
go through and delete ignored files from the build context.
* comments
* initial commit for persisent volume caching
* cache warmer works
* general cleanup
* adding some debugging
* adding missing files
* Fixing up cache retrieval and cleanup
* fix tests
* removing auth since we only cache public images
* simplifying the caching logic
* fixing logic
* adding volume cache to integration tests. remove auth from cache warmer image.
* add building warmer to integration-test
* move sample yaml files to examples dir
* small test fix
I merged a contributor's PR which modifed the HasFilepathPrefix function
to take an additional argument, but the PR hadn't been rebased. One of the
liting tests in Travis caught this bug.
* set default HOME env properly
* set HOME to / if user is set by uid
* fix test
* continue to skip user_run test
* fix unit test to match new functionality
* Enable overwriting of links (solves #351)
* add integration test to check extraction of images with replaced hardlinks
* Prevent following symlinks during extracting normal files
This fixes#359, #361, #362.
When running container-diff in integration tests, both stdout and stderr were being returned
and unmarshalled into the container-diff json object. If container-diff
was giving any error messages (such as, if it didn't have permissions to
extract a file), this would fail even if ultimately no
differences between the filesystems existed.
I updated the RunCommands to only return stdout and print stderr if the
command fails.
To add layer caching to kaniko, I added two flags: --cache and
--use-cache.
If --use-cache is set, then the cache will be used, and if --cache is
specified then that repo will be used to store cached layers. If --cache
isn't set, a cache will be inferred from the destination provided.
Currently, caching only works for RUN commands. Before executing the
command, kaniko checks if the cached layer exists. If it does, it pulls
it and extracts it. It then adds those files to the snapshotter and
append a layer to the config history. If the cached layer does not exist, kaniko executes the command and
pushes the newly created layer to the cache.
All cached layers are tagged with a stable key, which is built based off
of:
1. The base image digest
2. The current state of the filesystem
3. The current command being run
4. The current config file (to account for metadata changes)
I also added two integration tests to make sure caching works
1. Dockerfile_test_cache runs 'date', which should be exactly the same
the second time the image is built
2. Dockerfile_test_cache_install makes sure apt-get install can be
reproduced
Yoan Blanc
Tejal Desai
Cole Wippern
cvgw
Ben Einaudi
Sam Stoelinga
Tejal Desai
Don McCasland
Victor Noel
Monard Vong
Hugues Alary
Carlos Alexandro Becker
Priya Wadhwa
Kartik Verma
Johannes 'fish' Ziemke
dlorenc
Daisuke Taniwaki
Sharif Elgamal
peter-evans
Cyrille Hemidy
Kartik Verma
xanonid