Commit Graph

338 Commits

Author SHA1 Message Date
Nao YONASHIRO 75fdad7319 chore: fix typo 2019-05-17 03:17:08 +09:00
priyawadhwa 38c1735d92
Merge pull request #655 from GijsWeterings/master
Add `--digest-file` flag to output built digest to file.
2019-05-14 12:37:14 -07:00
Gijs 3686b65426 Process feedback of priyawadhwa regarding naming/nilcheck.
Changes the argument flag from `--digestfile` to `--digest-file`.
Skips an unneeded nil check.
2019-05-11 15:14:25 +02:00
Johannes 'fish' Ziemke 8c732f6f52 Fix kaniko caching (#639)
* Revert "Change cache key calculation to be more reproducible. (#525)"

This reverts commit 1ffae47fdd.

* Add logging of composition key back

* Do not include build args in cache key

This should be save, given that the commands will have the args included
when the cache key gets built.
2019-05-10 09:57:03 -05:00
Gijs a6e3ddfc79 Add `--digestfile` flag to output built digest to file.
This flag, when set, takes a file in the container and writes the image digest to it. This can be used to extract the exact digest of the built image by surrounding tooling without having to parse the logs from Kaniko, for example by pointing the file to a mounted volume or to a file used durint exit status, such as with Kubernetes' [Termination message policy](https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/)]

When the flag is not set, the digest is not written to file and the executor behaves as before. The digest is also written to file in case of a tarball or a `--no-push`.

Closes #654
2019-05-02 14:57:44 +02:00
Dirk Gustke dd9d081447 this is quite spammy in my multistage build (#640)
.. and as i am surely not the only one, move it down to debug.
2019-04-15 13:22:46 -07:00
dlorenc c8fabdf6e4
Fix arg handling for multi-stage images in COPY instructions. (#621) 2019-03-22 12:24:43 -05:00
Daisuke Taniwaki 1bf4421047 Fix parent directory permissions (#619)
* Add parent directories of adding files

* Add integration Dockerfile to test parent directory permissions

* Remove unnecessary helper method

* Use a file on the internet for integration Dockerfile
2019-03-19 12:40:15 -05:00
Jason Hall 3fa411ceb9 Check push permissions before building images (#622)
* Check push permissions before building images

* Fix doc comment

* improve error messages
2019-03-19 12:39:59 -05:00
Daisuke Taniwaki 28bfb75a31 Fix file mode bug (#618)
* Fix file mode

* Add Dockerfile for special file permission test
2019-03-18 13:28:24 -05:00
dlorenc 246cc92a33
Optimize file copying and stage saving between stages. (#605)
This change calculates the exact files and directories needed between
stages used in the COPY command. Instead of saving the entire
stage as a tarball, we now save only the necessary files.
2019-03-13 07:47:28 -07:00
dlorenc 9912ccbf8d
Fix USER handling. There were two issues: (#600)
- We were validating usernames/groupnames existed in etc/passwd. Docker does not do this
- We were incorrectly caching USER commands. This was fixed automatically by fixing the first part.
2019-03-07 07:05:24 -08:00
dlorenc 2abe109eb2
Environment variables should be replaced in URLs in ADD commands. (#580)
We were previously explicitly skipping this for some reason, but Docker
seems to expand these in URLs so we should too.
2019-02-25 10:42:34 -06:00
dlorenc 4b7e2b3a06
Update the cache warmer to also save manifests. (#576) 2019-02-21 14:09:22 -06:00
dlorenc 378a3f9573
Look for manifests in the local cache next to the full images. (#570)
Calculating a manifest from a v1.tarball is very expensive. We can
store those locally as well, and use them if they exist.

This should eventually be replaced with oci layout support once that exists
in ggcr.
2019-02-19 13:54:41 -06:00
Jason Hall faadb2af86 Log "Skipping paths under..." to debug (#571)
This reduces noise in the log output, since it isn't terribly useful to most end users.
2019-02-19 13:54:26 -06:00
Fredrik Lönnegren 1b54bda32e Fix typo 2019-02-19 16:43:20 +01:00
Shuhei Kitagawa e8564f0d28 Add missing tests for bucket util (#565)
* Add CheckDeepEqual test util

* Fix for srcContext with slash suffix

* Add tests for bucket_util.go
2019-02-19 09:05:23 -06:00
dlorenc 4feed0ff35
Stop storing a separate cache hash. (#560)
This is unrequired, mtimes should be taken into account during caching.
2019-02-14 12:24:43 -06:00
dlorenc 8d78db4842
Refactor snapshotting (#561) 2019-02-14 12:14:28 -06:00
dlorenc 15e70d4142
Speed up workdir by always returning an empty filelist (rather than a nil one). (#557)
Nil indicates that a full snapshot is required. Empty indicates that nothing should be snapshotted.
2019-02-13 11:55:08 -06:00
dlorenc 877abd30ed
Refactor whitelist handling. (#559)
Also speed up stage deletion.
2019-02-13 11:17:56 -06:00
dlorenc 8179c47f0d
Refactor the build loop to fetch stagebuilders earlier. (#558)
This will help with optimizations.
2019-02-12 20:43:22 -06:00
James Rawlings af50bcec69 let's return errors when warming a cache (#553)
* let's return errors when warming a cache

* Fix lint error in warm.go
2019-02-08 15:14:53 -06:00
dlorenc 9047ccf7cc
This fixes a bug in the interaction between volumes in base images (#555)
and our snapshot optimizations.

If a previous base image has a volume, the directory is added to the
list of files to snapshot. That directory may not actually exist in the image.
2019-02-08 14:40:37 -06:00
dlorenc e14b660947
Fix a bug in volume handling for multi-stage images. (#554)
We were previously not resetting the whitelist between stages, this caused issues with volumes.
2019-02-07 09:01:49 -06:00
Anurag Goel 82fe355f14 Fix issues with layer caching, noPush and tarPath (#549)
* Set TarPath to empty when pushing a layer

* Fix issues with layer caching, noPush and tarPath.

- Layer caching should work even when tarPath is specified, so this
commit changes the value of tarPath to empty when caching layers.

- When an image is built with just the tarPath and noPush
is true, we should still create the tarBall (which wasn't happening
before this commit).

* Set no-push to false for cache layers

* Remove extra log

* go-imports fix
2019-02-06 10:16:09 -06:00
dlorenc c2514305ef
Fix a bug in snapshotting with multi-stage images. (#546)
We previously had an optimization that would skip snapshotting mutli-stage images
when in an intermediate stage, until the very end.

This conflicted with another optimization to avoid snapshotting when no files had changed.
2019-01-30 13:57:02 -06:00
Daisuke Taniwaki f8f59ea4c6 Add insecure-registry and tls-skip-verify-registry flags (#537) 2019-01-29 13:29:47 -06:00
dlorenc 1ffae47fdd
Change cache key calculation to be more reproducible. (#525)
Before we were using the full image digest, but that contains a timestamp. Now
we only use the layers themselves and the image config (env vars, etc.).

Also fix a bug in unpacking the layers themselves. mtimes can change during unpacking,
so set them all once at the end.
2019-01-23 13:46:12 -06:00
dlorenc fcd1976d3b
Make the Digest calculation faster for locally-cached images. (#534)
Right now when we find a v1.Tarball in the local disk cache, we
recompute the digest. This is very expensive and redundant, because
we store tarballs by their digest and use that as a key to look them up.
2019-01-22 13:28:21 -06:00
dlorenc 996bec885f
Refactor our credential fetching to only get k8s credentials if we are on k8s. (#522) 2019-01-11 11:36:43 -07:00
dlorenc 170e0a2d94
Add a lot more timing data. (#518) 2019-01-10 13:27:55 -07:00
dlorenc 9ab66560db
Simplify snapshotting. (#517) 2019-01-09 15:31:02 -07:00
dlorenc 2255837142
Tighten up the timing around Dockerfile commands. (#514)
Right now this timing also includes the snapshot time.
2019-01-09 10:34:23 -08:00
Myers Carpenter 1591f41f22 USER command needs /etc/passwd, thus require the fs to be unpacked (#508) 2019-01-08 17:26:54 -08:00
dlorenc 5f6fbfe74f
Add support for timing data in JSON format. (#510) 2019-01-08 17:24:47 -08:00
dlorenc a044e2b6e4
Even faster snapshotting with godirwalk. (#504)
This switches from filepath.Walk to godirwalk.Walk for even faster snapshotting.
A quick test shows a 40% improvement on the dockerfile_mv_add build.
2019-01-03 13:10:18 -06:00
linuxshokunin c3afcc0c7d Adding capability to get Dockerfile from URL. (#500)
* Changed to set http or https in Dockerfile path.

* Fixed return value when error.

* Fixed ineffectual error
2019-01-03 10:42:15 -06:00
MMeent e3bb8bc71a Adds COPY --from=previous stage name/number validation (#491)
* Adds COPY --from=previous stage name/number validation

This fixes an issue in which COPY --from=previous-stage-name would try to download docker image previous-stage-name instead of checking that previous-stage-name could be a named stage.

* Fix linting issues

goimports is implemented as 'gofmt + extras', so this should fix import warnings as well.

* Fix linting issues

Fixes linting issues introduced in the merge

* Fix linting issues.
2019-01-02 11:42:36 -06:00
dlorenc 8ced0930f4
Add more benchmarks (#487) 2018-12-17 10:23:40 -06:00
Sharif Elgamal 29b7c3e879
Adding TTL to caching (#488)
* Adding TTL to caching

* uncomment added lines

* cache TTL works for layers now

* remove debugging

* parse booleans correctly

* parse booleans correctly everywhere

* fix boolean parsing condition

* refactor benchmarking calls

* defer file properly
2018-12-13 14:20:25 -08:00
Priya Wadhwa a34ba5c233 Fixed merge conflict 2018-12-11 13:53:19 -08:00
Priya Wadhwa 7fd164deab Only parse .dockerignore once 2018-12-11 13:31:51 -08:00
Priya Wadhwa 9b01772cde Add logging 2018-12-11 12:44:23 -08:00
Priya Wadhwa b0b36ed85a Re-add support for .dockerignore file
This PR adds support for the dockerignore file. Previously when kaniko
had support for the dockerignore file, kaniko first went through the
build context and deleted files that were meant to be ignored. This
resulted in a really bad bug where files in user mounted volumes would
be deleted (my bad).

This time around, instead of modifying the build context at all, kaniko
will check if a file should be excluded when executing ADD/COPY
commands. If a file should be excluded (based on the .dockerignore) it
won't be copied over from the buildcontext and shouldn't end up in the
final image.

I also added a .dockerignore file and Dockerfile as an integration test,
which should fail if the dockerignore is not being processed correctly or if files aren't being excluded correctly.
Also, I removed all the integration testing from the previous version of the
dockerignore support.
2018-12-10 15:20:25 -08:00
Andrew Rynhard 01329d5ac1 Fix intermediate layer caching (#474)
* Fix intermediate layer caching

* Move the if statement into the ShouldTakeSnapshot function.

Also add some unit tests.
2018-12-10 11:34:06 -08:00
Sharif Elgamal 7f9ea39bf7
Avoid the cachedImage/remoteImage call loop (#483)
* Avoid the cachedImage/remoteImage call loop

* missed one function

* fix unit tests

* proper bool comparison
2018-12-10 10:11:05 -08:00
dlorenc 7611ea7a1d
Add support for COPY --from=<an unrelated image>. (#479)
Right now kaniko only supports COPY --from=<another stage>.
This commit adds support for the case where the referenced image is a remote image
in a registry that has not been used as a stage yet in the build.
2018-12-06 12:44:03 -06:00
priyawadhwa 0b7fa58ca2
Merge pull request #459 from aduong/symlinks
Overwrite existing dest when copying symlink and preserve link target
2018-11-30 09:28:26 -08:00