public_git/kaniko
1
0
Fork 0
mirror of https://github.com/GoogleContainerTools/kaniko synced 2026-10-04 05:58:59 +02:00
Code Issues Releases Wiki Activity

Merge branch 'master' of github.com:GoogleContainerTools/kaniko into cache-fix

Browse Source
This commit is contained in:
Sharif Elgamal
2019-08-02 13:21:12 -07:00
parent a74dc78c65 80421f2a73
commit 6ad4cdf314
464 changed files with 73028 additions and 1470 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.travis.yml
+1 -1
View File
@@ -2,7 +2,7 @@ language: go
os: linux
go:
- 1.10.x
- "1.10"
go_import_path: github.com/GoogleContainerTools/kaniko
script:
CHANGELOG.md
+114 -6
View File
@@ -1,4 +1,112 @@
# v0.6.0 Release - 11/06/2018
# v0.10.0 Release - 2019-06-19
## Bug Fixes
* Fix kaniko caching [#639](https://github.com/GoogleContainerTools/kaniko/pull/639)
* chore: fix typo [#665](https://github.com/GoogleContainerTools/kaniko/pull/665)
* Fix file mode bug [#618](https://github.com/GoogleContainerTools/kaniko/pull/618)
* Fix arg handling for multi-stage images in COPY instructions. [#621](https://github.com/GoogleContainerTools/kaniko/pull/621)
* Fix parent directory permissions [#619](https://github.com/GoogleContainerTools/kaniko/pull/619)
* Environment variables should be replaced in URLs in ADD commands. [#580](https://github.com/GoogleContainerTools/kaniko/pull/580)
* Update the cache warmer to also save manifests. [#576](https://github.com/GoogleContainerTools/kaniko/pull/576)
* Fix typo in error message [#569](https://github.com/GoogleContainerTools/kaniko/pull/569)
## New Features
* Add SkipVerify support to CheckPushPermissions. [#663](https://github.com/GoogleContainerTools/kaniko/pull/663)
* Creating github Build Context [#672](https://github.com/GoogleContainerTools/kaniko/pull/672)
* Add `--digest-file` flag to output built digest to file. [#655](https://github.com/GoogleContainerTools/kaniko/pull/655)
* README.md: update BuildKit/img comparison [#642](https://github.com/GoogleContainerTools/kaniko/pull/642)
* Add documentation for --verbosity flag [#634](https://github.com/GoogleContainerTools/kaniko/pull/634)
* Optimize file copying and stage saving between stages. [#605](https://github.com/GoogleContainerTools/kaniko/pull/605)
* Add an integration test for USER unpacking. [#600](https://github.com/GoogleContainerTools/kaniko/pull/600)
* Added missing documentation for --skip-tls-verify-pull arg [#593](https://github.com/GoogleContainerTools/kaniko/pull/593)
* README.me: update Buildah description [#586](https://github.com/GoogleContainerTools/kaniko/pull/586)
* Add missing tests for bucket util [#565](https://github.com/GoogleContainerTools/kaniko/pull/565)
* Look for manifests in the local cache next to the full images. [#570](https://github.com/GoogleContainerTools/kaniko/pull/570)
* Make the run_in_docker script support caching. [#564](https://github.com/GoogleContainerTools/kaniko/pull/564)
* Refactor snapshotting [#561](https://github.com/GoogleContainerTools/kaniko/pull/561)
* Stop storing a separate cache hash. [#560](https://github.com/GoogleContainerTools/kaniko/pull/560)
* Speed up workdir by always returning an empty filelist (rather than a… [#557](https://github.com/GoogleContainerTools/kaniko/pull/557)
* Refactor whitelist handling. [#559](https://github.com/GoogleContainerTools/kaniko/pull/559)
* Refactor the build loop to fetch stagebuilders earlier. [#558](https://github.com/GoogleContainerTools/kaniko/pull/558)
## Additonal PRs
* Improve changelog dates [#657](https://github.com/GoogleContainerTools/kaniko/pull/657)
* Change verbose output from info to debug [#640](https://github.com/GoogleContainerTools/kaniko/pull/640)
* Check push permissions before building images [#622](https://github.com/GoogleContainerTools/kaniko/pull/622)
* Bump go-containerregistry to 8c1640add99804503b4126abc718931a4d93c31a [#609](https://github.com/GoogleContainerTools/kaniko/pull/609)
* Update go-containerregistry [#599](https://github.com/GoogleContainerTools/kaniko/pull/599)
* Log "Skipping paths under..." to debug [#571](https://github.com/GoogleContainerTools/kaniko/pull/571)
Huge thank you for this release towards our contributors:
- Achilleas Pipinellis
- Adrian Duong
- Akihiro Suda
- Andreas Bergmeier
- Andrew Rynhard
- Anthony Weston
- Anurag Goel
- Balint Pato
- Christie Wilson
- Daisuke Taniwaki
- Dan Cecile
- Dirk Gustke
- dlorenc
- Fredrik Lönnegren
- Gijs
- Jake Shadle
- James Rawlings
- Jason Hall
- Johan Hernandez
- Johannes 'fish' Ziemke
- Kartik Verma
- linuxshokunin
- MMeent
- Myers Carpenter
- Nándor István Krácser
- Nao YONASHIRO
- Priya Wadhwa
- Sharif Elgamal
- Shuhei Kitagawa
- Valentin Rothberg
- Vincent Demeester
# v0.9.0 Release - 2019-02-08
## Bug Fixes
* Bug fix with volumes declared in base images during multi-stage builds
* Bug fix during snapshotting multi-stage builds.
* Bug fix for caching with tar output.
# v0.8.0 Release - 2019-01-29
## New Features
* Even faster snapshotting with godirwalk
* Added TTL for caching
## Updates
* Change cache key calculation to be more reproducible.
* Make the Digest calculation faster for locally-cached images.
* Simplify snapshotting.
## Bug Fixes
* Fix bug with USER command and unpacking base images.
* Added COPY --from=previous stage name/number validation
# v0.7.0 Release - 2018-12-10
## New Features
* Add support for COPY --from an unrelated image
## Updates
* Speed up snapshotting by using filepath.SkipDir
* Improve layer cache upload performance
* Skip unpacking the base image in certain cases
## Bug Fixes
* Fix bug with call loop
* Fix caching for multi-step builds
# v0.6.0 Release - 2018-11-06
## New Features
* parse arg commands at the top of dockerfiles [#404](https://github.com/GoogleContainerTools/kaniko/pull/404)
@@ -23,7 +131,7 @@
* fix releasing the cache warmer [#418](https://github.com/GoogleContainerTools/kaniko/pull/418)
# v0.5.0 Release - 10/16/2018
# v0.5.0 Release - 2018-10-16
## New Features
* Persistent volume caching for base images [#383](https://github.com/GoogleContainerTools/kaniko/pull/383)
@@ -42,7 +150,7 @@
* Don't cut everything after an equals sign [#381](https://github.com/GoogleContainerTools/kaniko/pull/381)
# v0.4.0 Release - 10/01/2018
# v0.4.0 Release - 2018-10-01
## New Features
* Add a benchmark package to store and monitor timings. [#367](https://github.com/GoogleContainerTools/kaniko/pull/367)
@@ -101,7 +209,7 @@
* Fix handling of the volume directive [#334](https://github.com/GoogleContainerTools/kaniko/pull/334)
# v0.3.0 Release - 7/31/2018
# v0.3.0 Release - 2018-07-31
New Features
* Local integration testing [#256](https://github.com/GoogleContainerTools/kaniko/pull/256)
* Add --target flag for multistage builds [#255](https://github.com/GoogleContainerTools/kaniko/pull/255)
@@ -113,7 +221,7 @@ Bug Fixes
* Multi-stage errors when referencing earlier stages [#233](https://github.com/GoogleContainerTools/kaniko/issues/233)
# v0.2.0 Release - 7/09/2018
# v0.2.0 Release - 2018-07-09
New Features
* Support for adding different source contexts, including Amazon S3 [#195](https://github.com/GoogleContainerTools/kaniko/issues/195)
@@ -122,7 +230,7 @@ New Features
* Update go-containerregistry so kaniko works better with Harbor and Gitlab[#227](https://github.com/GoogleContainerTools/kaniko/pull/227)
* Push image to multiple destinations [#184](https://github.com/GoogleContainerTools/kaniko/pull/184)
# v0.1.0 Release - 5/17/2018
# v0.1.0 Release - 2018-05-17
New Features
* The majority of Dockerfile commands are feature complete [#1](https://github.com/GoogleContainerTools/kaniko/issues/1)
CONTRIBUTING.md
+51 -4
View File
@@ -1,7 +1,15 @@
# How to Contribute
# Contributing to Kaniko
We'd love to accept your patches and contributions to this project. There are
just a few small guidelines you need to follow.
We'd love to accept your patches and contributions to this project!!
To get started developing, see our [DEVELOPMENT.md](./DEVELOPMENT.md).
In this file you'll find info on:
- [The CLA](#contributor-license-agreement)
- [The code review process](#code-reviews)
- [Standards](#standards) around [commit messages](#commit-messages) and [code](#coding-standards)
- [Finding something to work on](#finding-something-to-work-on)
## Contributor License Agreement
@@ -20,4 +28,43 @@ again.
All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose. Consult
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
information on using pull requests.
information on using pull requests.
## Standards
This section describes the standards we will try to maintain in this repo.
### Commit Messages
All commit messages should follow [these best practices](https://chris.beams.io/posts/git-commit/),
specifically:
- Start with a subject line
- Contain a body that explains _why_ you're making the change you're making
- Reference an issue number one exists, closing it if applicable (with text such as
["Fixes #245" or "Closes #111"](https://help.github.com/articles/closing-issues-using-keywords/))
Aim for [2 paragraphs in the body](https://www.youtube.com/watch?v=PJjmw9TRB7s).
Not sure what to put? Include:
- What is the problem being solved?
- Why is this the best approach?
- What other approaches did you consider?
- What side effects will this approach have?
- What future work remains to be done?
### Coding standards
The code in this repo should follow best practices, specifically:
- [Go code review comments](https://github.com/golang/go/wiki/CodeReviewComments)
## Finding something to work on
Thanks so much for considering contributing to our project!! We hope very much you can find something
interesting to work on:
- To find issues that we particularly would like contributors to tackle, look for
[issues with the "help wanted" label](https://github.com/GoogleContainerTools/kaniko/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22).
- Issues that are good for new folks will additionally be marked with
["good first issue"](https://github.com/GoogleContainerTools/kaniko/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
DEVELOPMENT.md
+21
View File
@@ -97,6 +97,27 @@ Requirements:
These tests will be kicked off by [reviewers](#reviews) for submitted PRs.
### Benchmarking
The goal is for Kaniko to be at least as fast at building Dockerfiles as Docker is, and to that end, we've built
in benchmarking to check the speed of not only each full run, but also how long each step of each run takes. To turn
on benchmarking, just set the `BENCHMARK_FILE` environment variable, and kaniko will output all the benchmark info
of each run to that file location.
```shell
docker run -v $(pwd):/workspace -v ~/.config:/root/.config \
-e BENCHMARK_FILE=/workspace/benchmark_file \
gcr.io/kaniko-project/executor:latest \
--dockerfile=<path to Dockerfile> --context=/workspace \
--destination=gcr.io/my-repo/my-image
```
Additionally, the integration tests can output benchmarking information to a `benchmarks` directory under the
`integration` directory if the `BENCHMARK` environment variable is set to `true.`
```shell
BENCHMARK=true go test -v --bucket $GCS_BUCKET --repo $IMAGE_REPO
```
## Creating a PR
When you have changes you would like to propose to kaniko, you will need to:
Gopkg.lock
Generated
+196 -6
View File
@@ -339,6 +339,21 @@
pruneopts = "NUT"
revision = "7567d47988d82a78b052c4b9ba1e61399e0e7897"
[[projects]]
digest = "1:de4a74b504df31145ffa8ca0c4edbffa2f3eb7f466753962184611b618fa5981"
name = "github.com/emirpasic/gods"
packages = [
"containers",
"lists",
"lists/arraylist",
"trees",
"trees/binaryheap",
"utils",
]
pruneopts = "NUT"
revision = "f6c17b524822278a87e3b3bd809fec33b51f5b46"
version = "v1.9.0"
[[projects]]
digest = "1:1b91ae0dc69a41d4c2ed23ea5cffb721ea63f5037ca4b81e6d6771fbb8f45129"
name = "github.com/fsnotify/fsnotify"
@@ -430,11 +445,13 @@
version = "v0.2.0"
[[projects]]
digest = "1:f1b23f53418c1b035a5965ac2600a28b16c08643683d5213fb581ecf4e79a02a"
digest = "1:16c8837e951303ef6388132bc875337660a48ea2dedf1c941ca118ea92d2a3d2"
name = "github.com/google/go-containerregistry"
packages = [
"pkg/authn",
"pkg/authn/k8schain",
"pkg/internal/retry",
"pkg/logs",
"pkg/name",
"pkg/v1",
"pkg/v1/daemon",
@@ -444,12 +461,13 @@
"pkg/v1/random",
"pkg/v1/remote",
"pkg/v1/remote/transport",
"pkg/v1/stream",
"pkg/v1/tarball",
"pkg/v1/types",
"pkg/v1/v1util",
]
pruneopts = "NUT"
revision = "88d8d18eb1bde1fcef23c745205c738074290515"
revision = "273af77a08b28b49cc2cff2dd8ae50a5094dac74"
[[projects]]
digest = "1:f4f203acd8b11b8747bdcd91696a01dbc95ccb9e2ca2db6abf81c3a4f5e950ce"
@@ -546,6 +564,14 @@
revision = "76626ae9c91c4f2a10f34cad8ce83ea42c93bb75"
version = "v1.0"
[[projects]]
branch = "master"
digest = "1:62fe3a7ea2050ecbd753a71889026f83d73329337ada66325cbafd5dea5f713d"
name = "github.com/jbenet/go-context"
packages = ["io"]
pruneopts = "NUT"
revision = "d14ea06fba99483203c19d92cfcd13ebe73135f4"
[[projects]]
digest = "1:ac6d01547ec4f7f673311b4663909269bfb8249952de3279799289467837c3cc"
name = "github.com/jmespath/go-jmespath"
@@ -561,6 +587,22 @@
revision = "ab8a2e0c74be9d3be70b3184d9acc634935ded82"
version = "1.1.4"
[[projects]]
digest = "1:75b5172f534d05a5abff749f1cf002351f834a2a5592812210aca5e139f9ddad"
name = "github.com/karrick/godirwalk"
packages = ["."]
pruneopts = "NUT"
revision = "cceff240ca8af695e41738831646717e80d2f846"
version = "v1.7.7"
[[projects]]
digest = "1:29e44e9481a689be0093a0033299b95741d394a97b28e0273c21afe697873a22"
name = "github.com/kevinburke/ssh_config"
packages = ["."]
pruneopts = "NUT"
revision = "81db2a75821ed34e682567d48be488a1c3121088"
version = "0.5"
[[projects]]
digest = "1:d0164259ed17929689df11205194d80288e8ae25351778f7a3421a24774c36f8"
name = "github.com/mattn/go-shellwords"
@@ -577,6 +619,14 @@
revision = "c12348ce28de40eed0136aa2b644d0ee0650e56c"
version = "v1.0.1"
[[projects]]
digest = "1:a4df73029d2c42fabcb6b41e327d2f87e685284ec03edf76921c267d9cfc9c23"
name = "github.com/mitchellh/go-homedir"
packages = ["."]
pruneopts = "NUT"
revision = "ae18d6b8b3205b561c79e8e5f69bff09736185f4"
version = "v1.0.0"
[[projects]]
branch = "master"
digest = "1:e92b50943cf297f740ce62ee527a1044215689f0daea2b51bb9be4265e2b2b36"
@@ -671,6 +721,22 @@
revision = "1949ddbfd147afd4d964a9f00b24eb291e0e7c38"
version = "v1.0.2"
[[projects]]
branch = "master"
digest = "1:15057fc7395024283a7d2639b8afc61c5b6df3fe260ce06ff5834c8464f16b5c"
name = "github.com/otiai10/copy"
packages = ["."]
pruneopts = "NUT"
revision = "7e9a647135a142c2669943d4a4d29be015ce9392"
[[projects]]
digest = "1:cf254277d898b713195cc6b4a3fac8bf738b9f1121625df27843b52b267eec6c"
name = "github.com/pelletier/go-buffruneio"
packages = ["."]
pruneopts = "NUT"
revision = "c37440a7cf42ac63b919c752ca73a85067e05992"
version = "v0.2.0"
[[projects]]
branch = "master"
digest = "1:3bf17a6e6eaa6ad24152148a631d18662f7212e21637c2699bff3369b7f00fa2"
@@ -738,6 +804,14 @@
pruneopts = "NUT"
revision = "05ee40e3a273f7245e8777337fc7b46e533a9a92"
[[projects]]
digest = "1:d917313f309bda80d27274d53985bc65651f81a5b66b820749ac7f8ef061fd04"
name = "github.com/sergi/go-diff"
packages = ["diffmatchpatch"]
pruneopts = "NUT"
revision = "1744e2970ca51c86172c8190fadad617561ed6e7"
version = "v1.0.0"
[[projects]]
digest = "1:b2339e83ce9b5c4f79405f949429a7f68a9a904fed903c672aac1e7ceb7f5f02"
name = "github.com/sirupsen/logrus"
@@ -762,6 +836,19 @@
revision = "583c0c0531f06d5278b7d917446061adc344b5cd"
version = "v1.0.1"
[[projects]]
digest = "1:ccca1dcd18bc54e23b517a3c5babeff2e3924a7d8fc1932162225876cfe4bfb0"
name = "github.com/src-d/gcfg"
packages = [
".",
"scanner",
"token",
"types",
]
pruneopts = "NUT"
revision = "f187355171c936ac84a82793659ebb4936bc1c23"
version = "v1.3.0"
[[projects]]
branch = "master"
digest = "1:e865a1cd94806d1bb0eaa0bffba2ccb5e25ac42e1f55328c83d5e3399c9961a4"
@@ -790,6 +877,14 @@
revision = "38ec4ddb06dedbea0a895c4848b248eb38af221b"
version = "v0.10.2"
[[projects]]
digest = "1:3148cb3478c26a92b4c1a18abb9428234b281e278af6267840721a24b6cbc6a3"
name = "github.com/xanzy/ssh-agent"
packages = ["."]
pruneopts = "NUT"
revision = "640f0ab560aeb89d523bb6ac322b1244d5c3796c"
version = "v0.2.0"
[[projects]]
digest = "1:51db8b83c5320ce0522433046efec5fda0d34a2308ff969627e1ca206de77f35"
name = "go.opencensus.io"
@@ -814,11 +909,27 @@
[[projects]]
branch = "master"
digest = "1:670ac353e434afad9b6d96ba9735f243c5808c58ed67fdce9768e160d84389ba"
digest = "1:32c6e2719119a24b32fd3f4e5680e68f6b29752616441f526d055987c2df9364"
name = "golang.org/x/crypto"
packages = [
"cast5",
"curve25519",
"ed25519",
"ed25519/internal/edwards25519",
"internal/chacha20",
"internal/subtle",
"openpgp",
"openpgp/armor",
"openpgp/elgamal",
"openpgp/errors",
"openpgp/packet",
"openpgp/s2k",
"pkcs12",
"pkcs12/internal/rc2",
"poly1305",
"ssh",
"ssh/agent",
"ssh/knownhosts",
"ssh/terminal",
]
pruneopts = "NUT"
@@ -1003,6 +1114,77 @@
revision = "d2d2541c53f18d2a059457998ce2876cc8e67cbf"
version = "v0.9.1"
[[projects]]
digest = "1:47a697b155f5214ff14e68e39ce9c2e8d93e1fb035ae5ba7e247d044e0ce64e3"
name = "gopkg.in/src-d/go-billy.v4"
packages = [
".",
"helper/chroot",
"helper/polyfill",
"osfs",
"util",
]
pruneopts = "NUT"
revision = "83cf655d40b15b427014d7875d10850f96edba14"
version = "v4.2.0"
[[projects]]
digest = "1:d1b3d8ed1789f92b3d8192f8822cf422f803cd690139a8537a61313145fcae44"
name = "gopkg.in/src-d/go-git.v4"
packages = [
".",
"config",
"internal/revision",
"plumbing",
"plumbing/cache",
"plumbing/filemode",
"plumbing/format/config",
"plumbing/format/diff",
"plumbing/format/gitignore",
"plumbing/format/idxfile",
"plumbing/format/index",
"plumbing/format/objfile",
"plumbing/format/packfile",
"plumbing/format/pktline",
"plumbing/object",
"plumbing/protocol/packp",
"plumbing/protocol/packp/capability",
"plumbing/protocol/packp/sideband",
"plumbing/revlist",
"plumbing/storer",
"plumbing/transport",
"plumbing/transport/client",
"plumbing/transport/file",
"plumbing/transport/git",
"plumbing/transport/http",
"plumbing/transport/internal/common",
"plumbing/transport/server",
"plumbing/transport/ssh",
"storage",
"storage/filesystem",
"storage/filesystem/dotgit",
"storage/memory",
"utils/binary",
"utils/diff",
"utils/ioutil",
"utils/merkletrie",
"utils/merkletrie/filesystem",
"utils/merkletrie/index",
"utils/merkletrie/internal/frame",
"utils/merkletrie/noder",
]
pruneopts = "NUT"
revision = "7b6c1266556f59ac436fada3fa6106d4a84f9b56"
version = "v4.6.0"
[[projects]]
digest = "1:b233ad4ec87ac916e7bf5e678e98a2cb9e8b52f6de6ad3e11834fc7a71b8e3bf"
name = "gopkg.in/warnings.v0"
packages = ["."]
pruneopts = "NUT"
revision = "ec4a0fea49c7b46c2aeb0b51aac55779c607e52b"
version = "v0.1.2"
[[projects]]
digest = "1:7c95b35057a0ff2e19f707173cc1a947fa43a6eb5c4d300d196ece0334046082"
name = "gopkg.in/yaml.v2"
@@ -1094,7 +1276,7 @@
version = "kubernetes-1.11.0"
[[projects]]
digest = "1:b960fc62d636ccdc3265dd1e190b7f5e7bf5f8d29bf4f02af7f1352768c58f3f"
digest = "1:2f523dd16b56091fab1f329f772c3540742920e270bf0f9b8451106b7f005a66"
name = "k8s.io/client-go"
packages = [
"discovery",
@@ -1146,8 +1328,8 @@
"util/integer",
]
pruneopts = "NUT"
revision = "7d04d0e2a0a1a4d4a1cd6baa432a2301492e4e65"
version = "kubernetes-1.11.0"
revision = "2cefa64ff137e128daeddbd1775cd775708a05bf"
version = "kubernetes-1.11.3"
[[projects]]
digest = "1:e345c95cf277bb7f650306556904df69e0904395c56959a56002d0140747eda0"
@@ -1174,7 +1356,9 @@
"github.com/aws/aws-sdk-go/service/s3",
"github.com/aws/aws-sdk-go/service/s3/s3manager",
"github.com/docker/docker/builder/dockerfile",
"github.com/docker/docker/builder/dockerignore",
"github.com/docker/docker/pkg/archive",
"github.com/docker/docker/pkg/fileutils",
"github.com/docker/docker/pkg/signal",
"github.com/genuinetools/amicontained/container",
"github.com/google/go-cmp/cmp",
@@ -1189,14 +1373,20 @@
"github.com/google/go-containerregistry/pkg/v1/remote",
"github.com/google/go-containerregistry/pkg/v1/tarball",
"github.com/google/go-github/github",
"github.com/karrick/godirwalk",
"github.com/moby/buildkit/frontend/dockerfile/instructions",
"github.com/moby/buildkit/frontend/dockerfile/parser",
"github.com/moby/buildkit/frontend/dockerfile/shell",
"github.com/otiai10/copy",
"github.com/pkg/errors",
"github.com/sirupsen/logrus",
"github.com/spf13/cobra",
"github.com/spf13/pflag",
"golang.org/x/net/context",
"golang.org/x/oauth2",
"golang.org/x/sync/errgroup",
"gopkg.in/src-d/go-git.v4",
"gopkg.in/src-d/go-git.v4/plumbing",
"k8s.io/client-go/discovery",
]
solver-name = "gps-cdcl"
Gopkg.toml
+6 -2
View File
@@ -33,12 +33,16 @@ required = [
[[constraint]]
name = "k8s.io/client-go"
version = "kubernetes-1.11.0"
version = "kubernetes-1.11.3"
[[constraint]]
name = "github.com/google/go-containerregistry"
revision = "88d8d18eb1bde1fcef23c745205c738074290515"
revision = "273af77a08b28b49cc2cff2dd8ae50a5094dac74"
[[override]]
name = "k8s.io/apimachinery"
version = "kubernetes-1.11.0"
[[constraint]]
name = "gopkg.in/src-d/go-git.v4"
version = "4.6.0"
Makefile
+2 -1
View File
@@ -14,12 +14,13 @@
# Bump these on release
VERSION_MAJOR ?= 0
VERSION_MINOR ?= 6
VERSION_MINOR ?= 10
VERSION_BUILD ?= 0
VERSION ?= v$(VERSION_MAJOR).$(VERSION_MINOR).$(VERSION_BUILD)
VERSION_PACKAGE = $(REPOPATH/pkg/version)
SHELL := /bin/bash
GOOS ?= $(shell go env GOOS)
GOARCH = amd64
ORG := github.com/GoogleContainerTools
README.md
+143 -74
View File
@@ -12,36 +12,64 @@ This enables building container images in environments that can't easily or secu
kaniko is meant to be run as an image, `gcr.io/kaniko-project/executor`.
We do **not** recommend running the kaniko executor binary in another image, as it might not work.
- [Kaniko](#kaniko)
- [How does kaniko work?](#how-does-kaniko-work)
- [Known Issues](#known-issues)
_If you are interested in contributing to kaniko, see [DEVELOPMENT.md](DEVELOPMENT.md) and [CONTRIBUTING.md](CONTRIBUTING.md)._
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
- [How does kaniko work?](#how-does-kaniko-work)
- [Known Issues](#known-issues)
- [Demo](#demo)
- [Using kaniko](#using-kaniko)
- [kaniko Build Contexts](#kaniko-build-contexts)
- [Running kaniko](#running-kaniko)
- [Running kaniko in a Kubernetes cluster](#running-kaniko-in-a-kubernetes-cluster)
- [Kubernetes secret](#kubernetes-secret)
- [Running kaniko in gVisor](#running-kaniko-in-gvisor)
- [Running kaniko in Google Cloud Build](#running-kaniko-in-google-cloud-build)
- [Running kaniko in Docker](#running-kaniko-in-Docker)
- [Running kaniko in Docker](#running-kaniko-in-docker)
- [Caching](#caching)
- [Caching Layers](#caching-layers)
- [Caching Base Images](#caching-base-images)
- [Pushing to Different Registries](#pushing-to-different-registries)
- [Pushing to Amazon ECR](#pushing-to-amazon-ecr)
- [Additional Flags](#additional-flags)
- [--build-arg](#--build-arg)
- [--cache](#--cache)
- [--cache-dir](#--cache-dir)
- [--cache-repo](#--cache-repo)
- [--cleanup](#--cleanup)
- [--digest-file](#--digest-file)
- [--insecure](#--insecure)
- [--insecure-pull](#--insecure-pull)
- [--no-push](#--no-push)
- [--reproducible](#--reproducible)
- [--single-snapshot](#--single-snapshot)
- [--snapshotMode](#--snapshotmode)
- [--skip-tls-verify](#--skip-tls-verify)
- [--skip-tls-verify-pull](#--skip-tls-verify-pull)
- [--target](#--target)
- [--tarPath](#--tarpath)
- [--verbosity](#--verbosity)
- [Debug Image](#debug-image)
- [Security](#security)
- [Comparison with Other Tools](#comparison-with-other-tools)
- [Community](#community)
- [Limitations](#limitations)
- [mtime and snapshotting](#mtime-and-snapshotting)
_If you are interested in contributing to kaniko, see [DEVELOPMENT.md](DEVELOPMENT.md) and [CONTRIBUTING.md](CONTRIBUTING.md)._
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
### How does kaniko work?
## How does kaniko work?
The kaniko executor image is responsible for building an image from a Dockerfile and pushing it to a registry.
Within the executor image, we extract the filesystem of the base image (the FROM image in the Dockerfile).
We then execute the commands in the Dockerfile, snapshotting the filesystem in userspace after each one.
After each command, we append a layer of changed files to the base image (if there are any) and update image metadata.
### Known Issues
## Known Issues
kaniko does not support building Windows containers.
## Demo
@@ -60,7 +88,7 @@ To use kaniko to build and push an image for you, you will need:
kaniko's build context is very similar to the build context you would send your Docker daemon for an image build; it represents a directory containing a Dockerfile which kaniko will use to build your image.
For example, a `COPY` command in your Dockerfile should refer to a file in the build context.
You will need to store your build context in a place that kaniko can access.
You will need to store your build context in a place that kaniko can access.
Right now, kaniko supports these storage solutions:
- GCS Bucket
- S3 Bucket
@@ -69,16 +97,18 @@ Right now, kaniko supports these storage solutions:
_Note: the local directory option refers to a directory within the kaniko container.
If you wish to use this option, you will need to mount in your build context into the container as a directory._
If using a GCS or S3 bucket, you will first need to create a compressed tar of your build context and upload it to your bucket.
If using a GCS or S3 bucket, you will first need to create a compressed tar of your build context and upload it to your bucket.
Once running, kaniko will then download and unpack the compressed tar of the build context before starting the image build.
To create a compressed tar, you can run:
```shell
tar -C <path to build context> -zcvf context.tar.gz .
```
Then, copy over the compressed tar into your bucket.
Then, copy over the compressed tar into your bucket.
For example, we can copy over the compressed tar to a GCS bucket with gsutil:
```
```shell
gsutil cp context.tar.gz gs://<bucket name>
```
@@ -87,11 +117,15 @@ When running kaniko, use the `--context` flag with the appropriate prefix to spe
| Source | Prefix |
|---------|---------|
| Local Directory | dir://[path to a directory in the kaniko container] |
| GCS Bucket | gs://[bucket name]/[path to .tar.gz] |
| GCS Bucket | gs://[bucket name]/[path to .tar.gz] |
| S3 Bucket | s3://[bucket name]/[path to .tar.gz] |
| Git Repository | git://[repository url] |
If you don't specify a prefix, kaniko will assume a local directory.
For example, to use a GCS bucket called `kaniko-bucket`, you would pass in `--context=gs://kaniko-bucket/path/to/context.tar.gz`.
For example, to use a GCS bucket called `kaniko-bucket`, you would pass in `--context=gs://kaniko-bucket/path/to/context.tar.gz`.
### Using Private Git Repository
You can use `Personal Access Tokens` for Build Contexts from Private Repositories from [GitHub](https://blog.github.com/2012-09-21-easier-builds-and-deployments-using-git-over-https-and-oauth/).
### Running kaniko
@@ -124,7 +158,7 @@ To create a secret to authenticate to Google Cloud Registry, follow these steps:
kubectl create secret generic kaniko-secret --from-file=<path to kaniko-secret.json>
```
_Note: If using a GCS bucket in the same GCP project as a build context, this service account should now also have permissions to read from that bucket._
_Note: If using a GCS bucket in the same GCP project as a build context, this service account should now also have permissions to read from that bucket._
The Kubernetes Pod spec should look similar to this, with the args parameters filled in:
@@ -212,28 +246,29 @@ We can run the kaniko executor image locally in a Docker daemon to build and pus
### Caching
#### Caching Layers
kaniko currently can cache layers created by `RUN` commands in a remote repository.
kaniko currently can cache layers created by `RUN` commands in a remote repository.
Before executing a command, kaniko checks the cache for the layer.
If it exists, kaniko will pull and extract the cached layer instead of executing the command.
If not, kaniko will execute the command and then push the newly created layer to the cache.
Users can opt in to caching by setting the `--cache=true` flag.
A remote repository for storing cached layers can be provided via the `--cache-repo` flag.
If this flag isn't provided, a cached repo will be inferred from the `--destination` provided.
If this flag isn't provided, a cached repo will be inferred from the `--destination` provided.
#### Caching Base Images
kaniko can cache images in a local directory that can be volume mounted into the kaniko image.
To do so, the cache must first be populated, as it is read-only. We provide a kaniko cache warming
kaniko can cache images in a local directory that can be volume mounted into the kaniko image.
To do so, the cache must first be populated, as it is read-only. We provide a kaniko cache warming
image at `gcr.io/kaniko-project/warmer`:
```shell
docker run -v $(pwd):/workspace gcr.io/kaniko-project/warmer:latest --cache-dir=/workspace/cache --image=<image to cache> --image=<another image to cache>
```
`--image` can be specified for any number of desired images.
`--image` can be specified for any number of desired images.
This command will cache those images by digest in a local directory named `cache`.
Once the cache is populated, caching is opted into with the same `--cache=true` flag as above.
The location of the local cache is provided via the `--cache-dir` flag, defaulting at `/cache` as with the cache warmer.
Once the cache is populated, caching is opted into with the same `--cache=true` flag as above.
The location of the local cache is provided via the `--cache-dir` flag, defaulting at `/cache` as with the cache warmer.
See the `examples` directory for how to use with kubernetes clusters and persistent cache volumes.
### Pushing to Different Registries
@@ -302,57 +337,21 @@ To configure credentials, you will need to do the following:
### Additional Flags
#### --snapshotMode
You can set the `--snapshotMode=<full (default), time>` flag to set how kaniko will snapshot the filesystem.
If `--snapshotMode=time` is set, only file mtime will be considered when snapshotting (see
[limitations related to mtime](#mtime-and-snapshotting)).
#### --build-arg
This flag allows you to pass in ARG values at build time, similarly to Docker.
You can set it multiple times for multiple arguments.
#### --single-snapshot
This flag takes a single snapshot of the filesystem at the end of the build, so only one layer will be appended to the base image.
#### --reproducible
Set this flag to strip timestamps out of the built image and make it reproducible.
#### --tarPath
Set this flag as `--tarPath=<path>` to save the image as a tarball at path instead of pushing the image.
#### --target
Set this flag to indicate which build stage is the target build stage.
#### --no-push
Set this flag if you only want to build the image, without pushing to a registry.
#### --insecure
Set this flag if you want to push images to a plain HTTP registry. It is supposed to be used for testing purposes only and should not be used in production!
#### --skip-tls-verify
Set this flag to skip TLS certificate validation when pushing images to a registry. It is supposed to be used for testing purposes only and should not be used in production!
#### --insecure-pull
Set this flag if you want to pull images from a plain HTTP registry. It is supposed to be used for testing purposes only and should not be used in production!
#### --skip-tls-verify-pull
Set this flag to skip TLS certificate validation when pulling images from a registry. It is supposed to be used for testing purposes only and should not be used in production!
#### --cache
Set this flag as `--cache=true` to opt in to caching with kaniko.
#### --cache-dir
Set this flag to specify a local directory cache for base images. Defaults to `/cache`.
_This flag must be used in conjunction with the `--cache=true` flag._
#### --cache-repo
Set this flag to specify a remote repository which will be used to store cached layers.
@@ -362,15 +361,77 @@ If `--destination=gcr.io/kaniko-project/test`, then cached layers will be stored
_This flag must be used in conjunction with the `--cache=true` flag._
#### --cache-dir
Set this flag to specify a local directory cache for base images. Defaults to `/cache`.
#### --digest-file
_This flag must be used in conjunction with the `--cache=true` flag._
Set this flag to specify a file in the container. This file will
receive the digest of a built image. This can be used to
automatically track the exact image built by Kaniko.
For example, setting the flag to `--digest-file=/dev/termination-log`
will write the digest to that file, which is picked up by
Kubernetes automatically as the `{{.state.terminated.message}}`
of the container.
#### --insecure-registry
Set this flag to use plain HTTP requests when accessing a registry. It is supposed to be used for testing purposes only and should not be used in production!
You can set it multiple times for multiple registries.
#### --skip-tls-verify-registry
Set this flag to skip TLS cerificate validation when accessing a registry. It is supposed to be used for testing purposes only and should not be used in production!
You can set it multiple times for multiple registries.
#### --cleanup
Set this flag to cleanup the filesystem at the end, leaving a clean kaniko container (if you want to build multiple images in the same container, using the debug kaniko image)
Set this flag to clean the filesystem at the end of the build.
#### --insecure
Set this flag if you want to push images to a plain HTTP registry. It is supposed to be used for testing purposes only and should not be used in production!
#### --insecure-pull
Set this flag if you want to pull images from a plain HTTP registry. It is supposed to be used for testing purposes only and should not be used in production!
#### --no-push
Set this flag if you only want to build the image, without pushing to a registry.
#### --reproducible
Set this flag to strip timestamps out of the built image and make it reproducible.
#### --single-snapshot
This flag takes a single snapshot of the filesystem at the end of the build, so only one layer will be appended to the base image.
#### --skip-tls-verify
Set this flag to skip TLS certificate validation when pushing to a registry. It is supposed to be used for testing purposes only and should not be used in production!
#### --skip-tls-verify-pull
Set this flag to skip TLS certificate validation when pulling from a registry. It is supposed to be used for testing purposes only and should not be used in production!
#### --snapshotMode
You can set the `--snapshotMode=<full (default), time>` flag to set how kaniko will snapshot the filesystem.
If `--snapshotMode=time` is set, only file mtime will be considered when snapshotting (see
[limitations related to mtime](#mtime-and-snapshotting)).
#### --target
Set this flag to indicate which build stage is the target build stage.
#### --tarPath
Set this flag as `--tarPath=<path>` to save the image as a tarball at path instead of pushing the image.
#### --verbosity
Set this flag as `--verbosity=<panic|fatal|error|warn|info|debug>` to set the logging level. Defaults to `info`.
### Debug Image
@@ -406,19 +467,20 @@ You may be able to achieve the same default seccomp profile that Docker uses in
Similar tools include:
- [BuildKit](https://github.com/moby/buildkit)
- [img](https://github.com/genuinetools/img)
- [orca-build](https://github.com/cyphar/orca-build)
- [umoci](https://github.com/openSUSE/umoci)
- [buildah](https://github.com/projectatomic/buildah)
- [buildah](https://github.com/containers/buildah)
- [FTL](https://github.com/GoogleCloudPlatform/runtimes-common/tree/master/ftl)
- [Bazel rules_docker](https://github.com/bazelbuild/rules_docker)
All of these tools build container images with different approaches.
`img` can perform as a non root user from within a container, but requires that
the `img` container has `RawProc` access to create nested containers. `kaniko`
does not actually create nested containers, so it does not require `RawProc`
access.
BuildKit (and `img`) can perform as a non root user from within a container, but requires
seccomp and AppArmor to be disabled to create nested containers. `kaniko`
does not actually create nested containers, so it does not require seccomp and AppArmor
to be disabled.
`orca-build` depends on `runc` to build images from Dockerfiles, which can not
run inside a container (for similar reasons to `img` above). `kaniko` doesn't
@@ -432,8 +494,15 @@ filesystem is sufficiently complicated). However it has no `Dockerfile`-like
build tooling (it's a slightly lower-level tool that can be used to build such
builders -- such as `orca-build`).
`buildah` requires the same privileges as a Docker daemon does to run, while
`kaniko` runs without any special privileges or permissions.
`Buildah` specializes in building OCI images. Buildah's commands replicate all
of the commands that are found in a Dockerfile. This allows building images
with and without Dockerfiles while not requiring any root privileges.
Buildah’s ultimate goal is to provide a lower-level coreutils interface to
build images. The flexibility of building images without Dockerfiles allows
for the integration of other scripting languages into the build process.
Buildah follows a simple fork-exec model and does not run as a daemon
but it is based on a comprehensive API in golang, which can be vendored
into other tools.
`FTL` and `Bazel` aim to achieve the fastest possible creation of Docker images
for a subset of images. These can be thought of as a special-case "fast path"
benchmark.sh
Executable
+21
View File
@@ -0,0 +1,21 @@
# Copyright 2018 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#!/bin/bash
set -ex
script_name=$0
script_full_path=$(dirname "$0")
export BENCHMARK=true
export IMAGE_REPO="gcr.io/kaniko-test/benchmarks"
./${script_full_path}/integration-test.sh
cmd/executor/cmd/root.go
+32 -28
View File
@@ -20,15 +20,16 @@ import (
"fmt"
"os"
"path/filepath"
"regexp"
"strings"
"time"
"github.com/GoogleContainerTools/kaniko/pkg/buildcontext"
"github.com/GoogleContainerTools/kaniko/pkg/config"
"github.com/GoogleContainerTools/kaniko/pkg/constants"
"github.com/GoogleContainerTools/kaniko/pkg/dockerfile"
"github.com/GoogleContainerTools/kaniko/pkg/executor"
"github.com/GoogleContainerTools/kaniko/pkg/timing"
"github.com/GoogleContainerTools/kaniko/pkg/util"
"github.com/docker/docker/pkg/fileutils"
"github.com/genuinetools/amicontained/container"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
@@ -49,6 +50,7 @@ func init() {
addHiddenFlags(RootCmd)
}
// RootCmd is the kaniko command that is run
var RootCmd = &cobra.Command{
Use: "executor",
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
@@ -67,7 +69,7 @@ var RootCmd = &cobra.Command{
if err := resolveDockerfilePath(); err != nil {
return errors.Wrap(err, "error resolving dockerfile path")
}
return removeIgnoredFiles()
return nil
},
Run: func(cmd *cobra.Command, args []string) {
if !checkContained() {
@@ -76,6 +78,9 @@ var RootCmd = &cobra.Command{
}
logrus.Warn("kaniko is being run outside of a container. This can have dangerous effects on your system")
}
if err := executor.CheckPushPermissions(opts); err != nil {
exit(errors.Wrap(err, "error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again"))
}
if err := os.Chdir("/"); err != nil {
exit(errors.Wrap(err, "error changing to root dir"))
}
@@ -86,6 +91,21 @@ var RootCmd = &cobra.Command{
if err := executor.DoPush(image, opts); err != nil {
exit(errors.Wrap(err, "error pushing image"))
}
benchmarkFile := os.Getenv("BENCHMARK_FILE")
// false is a keyword for integration tests to turn off benchmarking
if benchmarkFile != "" && benchmarkFile != "false" {
f, err := os.Create(benchmarkFile)
if err != nil {
logrus.Warnf("Unable to create benchmarking file %s: %s", benchmarkFile, err)
}
defer f.Close()
s, err := timing.JSON()
if err != nil {
logrus.Warnf("Unable to write benchmark file: %s", err)
}
f.WriteString(s)
}
},
}
@@ -108,8 +128,12 @@ func addKanikoOptionsFlags(cmd *cobra.Command) {
RootCmd.PersistentFlags().BoolVarP(&opts.NoPush, "no-push", "", false, "Do not push the image to the registry")
RootCmd.PersistentFlags().StringVarP(&opts.CacheRepo, "cache-repo", "", "", "Specify a repository to use as a cache, otherwise one will be inferred from the destination provided")
RootCmd.PersistentFlags().StringVarP(&opts.CacheDir, "cache-dir", "", "/cache", "Specify a local directory to use as a cache.")
RootCmd.PersistentFlags().StringVarP(&opts.DigestFile, "digest-file", "", "", "Specify a file to save the digest of the built image to.")
RootCmd.PersistentFlags().BoolVarP(&opts.Cache, "cache", "", false, "Use cache when building image")
RootCmd.PersistentFlags().BoolVarP(&opts.Cleanup, "cleanup", "", false, "Clean the filesystem at the end")
RootCmd.PersistentFlags().DurationVarP(&opts.CacheTTL, "cache-ttl", "", time.Hour*336, "Cache timeout in hours. Defaults to two weeks.")
RootCmd.PersistentFlags().VarP(&opts.InsecureRegistries, "insecure-registry", "", "Insecure registry using plain HTTP to push and pull. Set it repeatedly for multiple registries.")
RootCmd.PersistentFlags().VarP(&opts.SkipTLSVerifyRegistries, "skip-tls-verify-registry", "", "Insecure registry ignoring TLS verify to push and pull. Set it repeatedly for multiple registries.")
}
// addHiddenFlags marks certain flags as hidden from the executor help text
@@ -140,6 +164,9 @@ func cacheFlagsValid() error {
// resolveDockerfilePath resolves the Dockerfile path to an absolute path
func resolveDockerfilePath() error {
if match, _ := regexp.MatchString("^https?://", opts.DockerfilePath); match {
return nil
}
if util.FilepathExists(opts.DockerfilePath) {
abs, err := filepath.Abs(opts.DockerfilePath)
if err != nil {
@@ -163,7 +190,7 @@ func resolveDockerfilePath() error {
// copy Dockerfile to /kaniko/Dockerfile so that if it's specified in the .dockerignore
// it won't be copied into the image
func copyDockerfile() error {
if err := util.CopyFile(opts.DockerfilePath, constants.DockerfilePath); err != nil {
if _, err := util.CopyFile(opts.DockerfilePath, constants.DockerfilePath, ""); err != nil {
return errors.Wrap(err, "copying dockerfile")
}
opts.DockerfilePath = constants.DockerfilePath
@@ -181,12 +208,12 @@ func resolveSourceContext() error {
}
if opts.Bucket != "" {
if !strings.Contains(opts.Bucket, "://") {
// if no prefix use Google Cloud Storage as default for backwards compatibility
opts.SrcContext = constants.GCSBuildContextPrefix + opts.Bucket
} else {
opts.SrcContext = opts.Bucket
}
}
// if no prefix use Google Cloud Storage as default for backwards compatibility
contextExecutor, err := buildcontext.GetBuildContext(opts.SrcContext)
if err != nil {
return err
@@ -200,29 +227,6 @@ func resolveSourceContext() error {
return nil
}
func removeIgnoredFiles() error {
if !dockerfile.DockerignoreExists(opts) {
return nil
}
ignore, err := dockerfile.ParseDockerignore(opts)
if err != nil {
return err
}
logrus.Infof("Removing ignored files from build context: %s", ignore)
files, err := util.RelativeFiles("", opts.SrcContext)
if err != nil {
return errors.Wrap(err, "getting all files in src context")
}
for _, f := range files {
if rm, _ := fileutils.Matches(f, ignore); rm {
if err := os.RemoveAll(f); err != nil {
logrus.Errorf("Error removing %s from build context", f)
}
}
}
return nil
}
func exit(err error) {
fmt.Println(err)
os.Exit(1)
deploy/Dockerfile
+7 -5
View File
@@ -16,21 +16,23 @@
FROM golang:1.10
WORKDIR /go/src/github.com/GoogleContainerTools/kaniko
COPY . .
RUN make
# Get GCR credential helper
ADD https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.4.3-static/docker-credential-gcr_linux_amd64-1.4.3.tar.gz /usr/local/bin/
RUN tar -C /usr/local/bin/ -xvzf /usr/local/bin/docker-credential-gcr_linux_amd64-1.4.3.tar.gz
ADD https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.5.0/docker-credential-gcr_linux_amd64-1.5.0.tar.gz /usr/local/bin/
RUN tar -C /usr/local/bin/ -xvzf /usr/local/bin/docker-credential-gcr_linux_amd64-1.5.0.tar.gz
RUN docker-credential-gcr configure-docker
# Get Amazon ECR credential helper
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login
RUN make -C /go/src/github.com/awslabs/amazon-ecr-credential-helper linux-amd64
COPY . .
RUN make
FROM scratch
COPY --from=0 /go/src/github.com/GoogleContainerTools/kaniko/out/executor /kaniko/executor
COPY --from=0 /usr/local/bin/docker-credential-gcr /kaniko/docker-credential-gcr
COPY --from=0 /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/linux-amd64/docker-credential-ecr-login /kaniko/docker-credential-ecr-login
COPY files/ca-certificates.crt /kaniko/ssl/certs/
COPY files/config.json /kaniko/.docker/
COPY --from=0 /root/.docker/config.json /kaniko/.docker/config.json
ENV HOME /root
ENV USER /root
ENV PATH /usr/local/bin:/kaniko
deploy/Dockerfile_debug
+6 -5
View File
@@ -17,14 +17,15 @@
# Stage 0: Build the executor binary and get credential helpers
FROM golang:1.10
WORKDIR /go/src/github.com/GoogleContainerTools/kaniko
COPY . .
RUN make
# Get GCR credential helper
ADD https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.4.3-static/docker-credential-gcr_linux_amd64-1.4.3.tar.gz /usr/local/bin/
RUN tar -C /usr/local/bin/ -xvzf /usr/local/bin/docker-credential-gcr_linux_amd64-1.4.3.tar.gz
ADD https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.5.0/docker-credential-gcr_linux_amd64-1.5.0.tar.gz /usr/local/bin/
RUN tar -C /usr/local/bin/ -xvzf /usr/local/bin/docker-credential-gcr_linux_amd64-1.5.0.tar.gz
RUN docker-credential-gcr configure-docker
# Get Amazon ECR credential helper
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login
RUN make -C /go/src/github.com/awslabs/amazon-ecr-credential-helper linux-amd64
COPY . .
RUN make
# Stage 1: Get the busybox shell
FROM gcr.io/cloud-builders/bazel:latest
@@ -41,7 +42,7 @@ COPY --from=1 /distroless/bazel-genfiles/experimental/busybox/busybox/ /busybox/
# Declare /busybox as a volume to get it automatically whitelisted
VOLUME /busybox
COPY files/ca-certificates.crt /kaniko/ssl/certs/
COPY files/config.json /kaniko/.docker/
COPY --from=0 /root/.docker/config.json /kaniko/.docker/config.json
ENV HOME /root
ENV USER /root
ENV PATH /usr/local/bin:/kaniko:/busybox
deploy/Dockerfile_warmer
-1
View File
@@ -22,7 +22,6 @@ RUN make out/warmer
FROM scratch
COPY --from=0 /go/src/github.com/GoogleContainerTools/kaniko/out/warmer /kaniko/warmer
COPY files/ca-certificates.crt /kaniko/ssl/certs/
COPY files/config.json /kaniko/.docker/
ENV HOME /root
ENV USER /root
ENV PATH /usr/local/bin:/kaniko
deploy/cloudbuild.yaml
+1 -9
View File
@@ -3,9 +3,6 @@ steps:
- name: "gcr.io/cloud-builders/docker"
args: ["build", "-f", "deploy/Dockerfile",
"-t", "gcr.io/kaniko-project/executor:${COMMIT_SHA}", "."]
- name: "gcr.io/cloud-builders/docker"
args: ["build", "-f", "deploy/Dockerfile",
"-t", "gcr.io/kaniko-project/executor:latest", "."]
# Then, we want to build kaniko:debug
- name: "gcr.io/cloud-builders/docker"
args: ["build", "-f", "deploy/Dockerfile_debug",
@@ -17,12 +14,7 @@ steps:
- name: "gcr.io/cloud-builders/docker"
args: ["build", "-f", "deploy/Dockerfile_warmer",
"-t", "gcr.io/kaniko-project/warmer:${COMMIT_SHA}", "."]
- name: "gcr.io/cloud-builders/docker"
args: ["build", "-f", "deploy/Dockerfile_warmer",
"-t", "gcr.io/kaniko-project/warmer:latest", "."]
images: ["gcr.io/kaniko-project/executor:${COMMIT_SHA}",
"gcr.io/kaniko-project/executor:latest",
"gcr.io/kaniko-project/executor:debug-${COMMIT_SHA}",
"gcr.io/kaniko-project/executor:debug",
"gcr.io/kaniko-project/warmer:${COMMIT_SHA}",
"gcr.io/kaniko-project/warmer:latest"]
"gcr.io/kaniko-project/warmer:${COMMIT_SHA}"]
files/config.json
-12
View File
@@ -1,12 +0,0 @@
{
"auths": {
},
"credHelpers": {
"appengine.gcr.io": "gcr",
"asia.gcr.io": "gcr",
"eu.gcr.io": "gcr",
"gcr.io": "gcr",
"gcr.kubernetes.io": "gcr",
"us.gcr.io": "gcr"
}
}
integration/.dockerignore
+3
View File
@@ -0,0 +1,3 @@
# A .dockerignore file to make sure dockerignore support works
ignore/**
!ignore/foo
integration/cmd.go
+2 -9
View File
@@ -18,7 +18,6 @@ package integration
import (
"bytes"
"fmt"
"os/exec"
"testing"
)
@@ -26,14 +25,8 @@ import (
// RunCommandWithoutTest will run cmd and if it fails will output relevant info
// for debugging before returning an error. It can be run outside the context of a test.
func RunCommandWithoutTest(cmd *exec.Cmd) ([]byte, error) {
var stderr bytes.Buffer
cmd.Stderr = &stderr
output, err := cmd.Output()
if err != nil {
fmt.Println(cmd.Args)
fmt.Println(stderr.String())
fmt.Println(string(output))
}
output, err := cmd.CombinedOutput()
return output, err
}
integration/dockerfiles/Dockerfile_onbuild_base
+1 -1
View File
@@ -1,4 +1,4 @@
FROM gcr.io/google-appengine/debian9@sha256:1d6a9a6d106bd795098f60f4abb7083626354fa6735e81743c7f8cfca11259f0
FROM marketplace.gcr.io/google/ubuntu1804@sha256:4649ae6b381090fba6db38137eb05e03f44bf43c40149f734241c9f96aa0e001
ENV dir /tmp/dir/
ONBUILD RUN echo "onbuild" > /tmp/onbuild
ONBUILD RUN mkdir $dir
integration/dockerfiles/Dockerfile_test_add
+4 -1
View File
@@ -24,4 +24,7 @@ COPY $file /arg
# Finally, test adding a remote URL, concurrently with a normal file
ADD https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.4.3/docker-credential-gcr_linux_386-1.4.3.tar.gz context/foo /test/all/
ADD https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.4.3-static/docker-credential-gcr_linux_amd64-1.4.3.tar.gz /destination
# Test environment replacement in the URL
ENV VERSION=v1.4.3
ADD https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/${VERSION}-static/docker-credential-gcr_linux_amd64-1.4.3.tar.gz /destination
integration/dockerfiles/Dockerfile_test_arg_multi
+12
View File
@@ -0,0 +1,12 @@
ARG FILE_NAME=myFile
FROM busybox:latest AS builder
ARG FILE_NAME
RUN echo $FILE_NAME && touch /$FILE_NAME.txt && stat /$FILE_NAME.txt;
FROM busybox:latest
ARG FILE_NAME
RUN echo $FILE_NAME && touch /$FILE_NAME.txt && stat /$FILE_NAME.txt;
COPY --from=builder /$FILE_NAME.txt /
integration/dockerfiles/Dockerfile_test_cache_perm
+8
View File
@@ -0,0 +1,8 @@
# Test to make sure the cache works with special file permissions properly.
# If the image is built twice, directory foo should have the sticky bit,
# and file bar should have the setuid and setgid bits.
FROM busybox
RUN mkdir foo && chmod +t foo
RUN touch bar && chmod u+s,g+s bar
integration/dockerfiles/Dockerfile_test_daemons
+1 -1
View File
@@ -1,3 +1,3 @@
FROM busybox
RUN while true; do dd if=/dev/zero of=file`date +%s`.txt count=16000 bs=256 > /dev/null 2>&1; done &
RUN (while true; do sleep 10; dd if=/dev/zero of=file`date +%s`.txt count=16000 bs=256 > /dev/null 2>&1; done &); sleep 1
RUN echo "wait a second..." && sleep 2 && ls -lrat file*.txt || echo "test passed."
integration/dockerfiles/Dockerfile_test_dockerignore
+5
View File
@@ -0,0 +1,5 @@
# This dockerfile makes sure the .dockerignore is working
# If so then ignore/foo should copy to /foo
# If not, then this image won't build because it will attempt to copy three files to /foo, which is a file not a directory
FROM scratch
COPY ignore/* /foo
integration/dockerfiles/Dockerfile_test_hardlink
+7
View File
@@ -1,3 +1,10 @@
FROM composer@sha256:4598feb4b58b4370893a29cbc654afa9420b4debed1d574531514b78a24cd608 AS composer
FROM php@sha256:13813f20fec7ded7bf3a4305ea0ccd4df3cea900e263f7f86c3d5737f86669eb
COPY --from=composer /usr/bin/composer /usr/bin/composer
# make sure hardlink extracts correctly
FROM jboss/base-jdk@sha256:138591422fdab93a5844c13f6cbcc685631b37a16503675e9f340d2503617a41
FROM gcr.io/kaniko-test/hardlink-base:latest
RUN ls -al /usr/libexec/git-core/git /usr/bin/git /usr/libexec/git-core/git-diff
RUN stat /usr/bin/git
integration/dockerfiles/Dockerfile_test_meta_arg
+2 -1
View File
@@ -14,4 +14,5 @@ FROM ${REGISTRY}/${REPO}/debian9
COPY --from=stage1 /hello /tmp
# /tmp/hey should not get created without the ARG statement
RUN touch /tmp/${WORD2}
# Use -d 0 to force a time change because of stat resolution
RUN touch -d 0 /tmp/${WORD2}
integration/dockerfiles/Dockerfile_test_multistage
+9 -3
View File
@@ -1,17 +1,23 @@
FROM gcr.io/distroless/base@sha256:628939ac8bf3f49571d05c6c76b8688cb4a851af6c7088e599388259875bde20 as base
FROM gcr.io/google-appengine/debian9@sha256:f0159d14385afcb58a9b2fa8955c0cb64bd3abc365e8589f8c2dd38150fbfdbe as base
COPY . .
FROM scratch as second
ENV foopath context/foo
COPY --from=0 $foopath context/b* /foo/
FROM second
FROM second as third
COPY --from=base /context/foo /new/foo
FROM base as fourth
# Make sure that we snapshot intermediate images correctly
RUN date > /date
ENV foo bar
# This base image contains symlinks with relative paths to whitelisted directories
# We need to test they're extracted correctly
FROM fedora@sha256:c4cc32b09c6ae3f1353e7e33a8dda93dc41676b923d6d89afa996b421cc5aa48
FROM base
FROM fourth
ARG file
COPY --from=second /foo ${file}
COPY --from=gcr.io/google-appengine/debian9@sha256:00109fa40230a081f5ecffe0e814725042ff62a03e2d1eae0563f1f82eaeae9b /etc/os-release /new
integration/dockerfiles/Dockerfile_test_parent_dir_perms
+8
View File
@@ -0,0 +1,8 @@
FROM busybox
RUN adduser --disabled-password --gecos "" --uid 1000 user
RUN mkdir -p /home/user/foo
RUN chown -R user /home/user
RUN chmod 700 /home/user/foo
ADD https://raw.githubusercontent.com/GoogleContainerTools/kaniko/master/README.md /home/user/foo/README.md
RUN chown -R user /home/user
integration/dockerfiles/Dockerfile_test_user
+17
View File
@@ -0,0 +1,17 @@
# Copyright 2018 Google, Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM gcr.io/google-appengine/debian9@sha256:1d6a9a6d106bd795098f60f4abb7083626354fa6735e81743c7f8cfca11259f0
USER testuser:testgroup
integration/dockerfiles/Dockerfile_test_volume_3
+5
View File
@@ -0,0 +1,5 @@
FROM scratch as one
VOLUME /vol
FROM alpine@sha256:5ce5f501c457015c4b91f91a15ac69157d9b06f1a75cf9107bf2b62e0843983a as two
RUN mkdir /vol && echo hey > /vol/foo
integration/dockerfiles/Dockerfile_test_volume_4
+5
View File
@@ -0,0 +1,5 @@
FROM rabbitmq@sha256:57b028a4bb9592ece3915e3e9cdbbaecb3eb82b753aaaf5250f8d25d81d318e2
# This base image has a volume declared at /var/lib/rabbitmq
# This is important because it should not exist in the child image.
COPY context/foo /usr/local/bin/
CMD ["script.sh"]
integration/dockerfiles/Dockerfile_test_whitelist
+11
View File
@@ -0,0 +1,11 @@
# Make sure that whitelisting (specifically, filepath.SkipDir) works correctly, and that /var/test/testfile and
# /etc/test/testfile end up in the final image
FROM debian@sha256:38236c068c393272ad02db100e09cac36a5465149e2924a035ee60d6c60c38fe
RUN mkdir -p /var/test \
&& mkdir -p /etc/test \
&& touch /var/test/testfile \
&& touch /etc/test/testfile \
&& ls -lah /var/test \
&& ls -lah /etc/test;
integration/dockerignore.go
-112
View File
@@ -1,112 +0,0 @@
/*
Copyright 2018 Google LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package integration
import (
"fmt"
"os"
"os/exec"
"path"
"path/filepath"
"runtime"
"strings"
)
var filesToIgnore = []string{"ignore/fo*", "!ignore/foobar", "ignore/Dockerfile_test_ignore"}
const (
ignoreDir = "ignore"
ignoreDockerfile = "Dockerfile_test_ignore"
ignoreDockerfileContents = `FROM scratch
COPY . .`
)
// Set up a test dir to ignore with the structure:
// ignore
// -- Dockerfile_test_ignore
// -- foo
// -- foobar
func setupIgnoreTestDir() error {
if err := os.MkdirAll(ignoreDir, 0750); err != nil {
return err
}
// Create and write contents to dockerfile
path := filepath.Join(ignoreDir, ignoreDockerfile)
f, err := os.Create(path)
if err != nil {
return err
}
defer f.Close()
if _, err := f.Write([]byte(ignoreDockerfileContents)); err != nil {
return err
}
additionalFiles := []string{"ignore/foo", "ignore/foobar"}
for _, add := range additionalFiles {
a, err := os.Create(add)
if err != nil {
return err
}
defer a.Close()
}
return generateDockerIgnore()
}
// generate the .dockerignore file
func generateDockerIgnore() error {
f, err := os.Create(".dockerignore")
if err != nil {
return err
}
defer f.Close()
contents := strings.Join(filesToIgnore, "\n")
if _, err := f.Write([]byte(contents)); err != nil {
return err
}
return nil
}
func generateDockerignoreImages(imageRepo string) error {
dockerfilePath := filepath.Join(ignoreDir, ignoreDockerfile)
dockerImage := strings.ToLower(imageRepo + dockerPrefix + ignoreDockerfile)
dockerCmd := exec.Command("docker", "build",
"-t", dockerImage,
"-f", path.Join(dockerfilePath),
".")
_, err := RunCommandWithoutTest(dockerCmd)
if err != nil {
return fmt.Errorf("Failed to build image %s with docker command \"%s\": %s", dockerImage, dockerCmd.Args, err)
}
_, ex, _, _ := runtime.Caller(0)
cwd := filepath.Dir(ex)
kanikoImage := GetKanikoImage(imageRepo, ignoreDockerfile)
kanikoCmd := exec.Command("docker",
"run",
"-v", os.Getenv("HOME")+"/.config/gcloud:/root/.config/gcloud",
"-v", cwd+":/workspace",
ExecutorImage,
"-f", path.Join(buildContextPath, dockerfilePath),
"-d", kanikoImage,
"-c", buildContextPath)
_, err = RunCommandWithoutTest(kanikoCmd)
return err
}
integration/gcs.go
+8 -6
View File
@@ -22,7 +22,6 @@ import (
"log"
"os"
"os/exec"
"path/filepath"
"time"
)
@@ -49,16 +48,19 @@ func CreateIntegrationTarball() (string, error) {
// UploadFileToBucket will upload the at filePath to gcsBucket. It will return the path
// of the file in gcsBucket.
func UploadFileToBucket(gcsBucket string, filePath string) (string, error) {
log.Printf("Uploading file at %s to GCS bucket at %s\n", filePath, gcsBucket)
func UploadFileToBucket(gcsBucket string, filePath string, gcsPath string) (string, error) {
dst := fmt.Sprintf("%s/%s", gcsBucket, gcsPath)
log.Printf("Uploading file at %s to GCS bucket at %s\n", filePath, dst)
cmd := exec.Command("gsutil", "cp", filePath, gcsBucket)
_, err := RunCommandWithoutTest(cmd)
cmd := exec.Command("gsutil", "cp", filePath, dst)
out, err := RunCommandWithoutTest(cmd)
if err != nil {
log.Printf("Error uploading file %s to GCS at %s: %s", filePath, dst, err)
log.Println(string(out))
return "", fmt.Errorf("Failed to copy tarball to GCS bucket %s: %s", gcsBucket, err)
}
return filepath.Join(gcsBucket, filePath), err
return dst, nil
}
// DeleteFromBucket will remove the content at path. path should be the full path
integration/ignore/bar
View File
integration/ignore/baz
View File
integration/ignore/foo
View File
integration/images.go
+40 -7
View File
@@ -18,6 +18,7 @@ package integration
import (
"fmt"
"io/ioutil"
"os"
"os/exec"
"path"
@@ -25,12 +26,16 @@ import (
"runtime"
"strconv"
"strings"
"time"
"github.com/GoogleContainerTools/kaniko/pkg/timing"
)
const (
// ExecutorImage is the name of the kaniko executor image
ExecutorImage = "executor-image"
WarmerImage = "warmer-image"
//WarmerImage is the name of the kaniko cache warmer image
WarmerImage = "warmer-image"
dockerPrefix = "docker-"
kanikoPrefix = "kaniko-"
@@ -129,6 +134,7 @@ func NewDockerFileBuilder(dockerfiles []string) *DockerFileBuilder {
d.TestCacheDockerfiles = map[string]struct{}{
"Dockerfile_test_cache": {},
"Dockerfile_test_cache_install": {},
"Dockerfile_test_cache_perm": {},
}
return &d
}
@@ -159,9 +165,11 @@ func (d *DockerFileBuilder) BuildImage(imageRepo, gcsBucket, dockerfilesPath, do
additionalFlags...)...,
)
_, err := RunCommandWithoutTest(dockerCmd)
timer := timing.Start(dockerfile + "_docker")
out, err := RunCommandWithoutTest(dockerCmd)
timing.DefaultRun.Stop(timer)
if err != nil {
return fmt.Errorf("Failed to build image %s with docker command \"%s\": %s", dockerImage, dockerCmd.Args, err)
return fmt.Errorf("Failed to build image %s with docker command \"%s\": %s %s", dockerImage, dockerCmd.Args, err, string(out))
}
contextFlag := "-c"
@@ -170,7 +178,6 @@ func (d *DockerFileBuilder) BuildImage(imageRepo, gcsBucket, dockerfilesPath, do
if d == dockerfile {
contextFlag = "-b"
contextPath = gcsBucket
break
}
}
@@ -182,13 +189,28 @@ func (d *DockerFileBuilder) BuildImage(imageRepo, gcsBucket, dockerfilesPath, do
}
}
benchmarkEnv := "BENCHMARK_FILE=false"
benchmarkDir, err := ioutil.TempDir("", "")
if err != nil {
return err
}
if b, err := strconv.ParseBool(os.Getenv("BENCHMARK")); err == nil && b {
benchmarkEnv = "BENCHMARK_FILE=/kaniko/benchmarks/" + dockerfile
benchmarkFile := path.Join(benchmarkDir, dockerfile)
fileName := fmt.Sprintf("run_%s_%s", time.Now().Format("2006-01-02-15:04"), dockerfile)
dst := path.Join("benchmarks", fileName)
defer UploadFileToBucket(gcsBucket, benchmarkFile, dst)
}
// build kaniko image
additionalFlags = append(buildArgs, additionalKanikoFlagsMap[dockerfile]...)
kanikoImage := GetKanikoImage(imageRepo, dockerfile)
kanikoCmd := exec.Command("docker",
append([]string{"run",
"-v", os.Getenv("HOME") + "/.config/gcloud:/root/.config/gcloud",
"-v", benchmarkDir + ":/kaniko/benchmarks",
"-v", cwd + ":/workspace",
"-e", benchmarkEnv,
ExecutorImage,
"-f", path.Join(buildContextPath, dockerfilesPath, dockerfile),
"-d", kanikoImage, reproducibleFlag,
@@ -196,9 +218,11 @@ func (d *DockerFileBuilder) BuildImage(imageRepo, gcsBucket, dockerfilesPath, do
additionalFlags...)...,
)
_, err = RunCommandWithoutTest(kanikoCmd)
timer = timing.Start(dockerfile + "_kaniko")
out, err = RunCommandWithoutTest(kanikoCmd)
timing.DefaultRun.Stop(timer)
if err != nil {
return fmt.Errorf("Failed to build image %s with kaniko command \"%s\": %s", dockerImage, kanikoCmd.Args, err)
return fmt.Errorf("Failed to build image %s with kaniko command \"%s\": %s %s", dockerImage, kanikoCmd.Args, err, string(out))
}
d.FilesBuilt[dockerfile] = true
@@ -233,11 +257,17 @@ func (d *DockerFileBuilder) buildCachedImages(imageRepo, cacheRepo, dockerfilesP
cacheFlag := "--cache=true"
for dockerfile := range d.TestCacheDockerfiles {
benchmarkEnv := "BENCHMARK_FILE=false"
if b, err := strconv.ParseBool(os.Getenv("BENCHMARK")); err == nil && b {
os.Mkdir("benchmarks", 0755)
benchmarkEnv = "BENCHMARK_FILE=/workspace/benchmarks/" + dockerfile
}
kanikoImage := GetVersionedKanikoImage(imageRepo, dockerfile, version)
kanikoCmd := exec.Command("docker",
append([]string{"run",
"-v", os.Getenv("HOME") + "/.config/gcloud:/root/.config/gcloud",
"-v", cwd + ":/workspace",
"-e", benchmarkEnv,
ExecutorImage,
"-f", path.Join(buildContextPath, dockerfilesPath, dockerfile),
"-d", kanikoImage,
@@ -247,7 +277,10 @@ func (d *DockerFileBuilder) buildCachedImages(imageRepo, cacheRepo, dockerfilesP
"--cache-dir", cacheDir})...,
)
if _, err := RunCommandWithoutTest(kanikoCmd); err != nil {
timer := timing.Start(dockerfile + "_kaniko_cached_" + strconv.Itoa(version))
_, err := RunCommandWithoutTest(kanikoCmd)
timing.DefaultRun.Stop(timer)
if err != nil {
return fmt.Errorf("Failed to build cached image %s with kaniko command \"%s\": %s", kanikoImage, kanikoCmd.Args, err)
}
}
integration/integration_test.go
+172 -74
View File
@@ -25,13 +25,16 @@ import (
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1/daemon"
"golang.org/x/sync/errgroup"
"github.com/GoogleContainerTools/kaniko/pkg/timing"
"github.com/GoogleContainerTools/kaniko/pkg/util"
"github.com/GoogleContainerTools/kaniko/testutil"
)
@@ -123,7 +126,7 @@ func TestMain(m *testing.M) {
os.Exit(1)
}
fileInBucket, err := UploadFileToBucket(config.gcsBucket, contextFile)
fileInBucket, err := UploadFileToBucket(config.gcsBucket, contextFile, contextFile)
if err != nil {
fmt.Println("Failed to upload build context", err)
os.Exit(1)
@@ -138,73 +141,83 @@ func TestMain(m *testing.M) {
RunOnInterrupt(func() { DeleteFromBucket(fileInBucket) })
defer DeleteFromBucket(fileInBucket)
fmt.Println("Building kaniko image")
cmd := exec.Command("docker", "build", "-t", ExecutorImage, "-f", "../deploy/Dockerfile", "..")
if _, err = RunCommandWithoutTest(cmd); err != nil {
fmt.Printf("Building kaniko failed: %s", err)
os.Exit(1)
setupCommands := []struct {
name string
command []string
}{
{
name: "Building kaniko image",
command: []string{"docker", "build", "-t", ExecutorImage, "-f", "../deploy/Dockerfile", ".."},
},
{
name: "Building cache warmer image",
command: []string{"docker", "build", "-t", WarmerImage, "-f", "../deploy/Dockerfile_warmer", ".."},
},
{
name: "Building onbuild base image",
command: []string{"docker", "build", "-t", config.onbuildBaseImage, "-f", "dockerfiles/Dockerfile_onbuild_base", "."},
},
{
name: "Pushing onbuild base image",
command: []string{"docker", "push", config.onbuildBaseImage},
},
{
name: "Building hardlink base image",
command: []string{"docker", "build", "-t", config.hardlinkBaseImage, "-f", "dockerfiles/Dockerfile_hardlink_base", "."},
},
{
name: "Pushing hardlink base image",
command: []string{"docker", "push", config.hardlinkBaseImage},
},
}
fmt.Println("Building cache warmer image")
cmd = exec.Command("docker", "build", "-t", WarmerImage, "-f", "../deploy/Dockerfile_warmer", "..")
if _, err = RunCommandWithoutTest(cmd); err != nil {
fmt.Printf("Building kaniko's cache warmer failed: %s", err)
os.Exit(1)
for _, setupCmd := range setupCommands {
fmt.Println(setupCmd.name)
cmd := exec.Command(setupCmd.command[0], setupCmd.command[1:]...)
if out, err := RunCommandWithoutTest(cmd); err != nil {
fmt.Printf("%s failed: %s", setupCmd.name, err)
fmt.Println(string(out))
os.Exit(1)
}
}
fmt.Println("Building onbuild base image")
buildOnbuildBase := exec.Command("docker", "build", "-t", config.onbuildBaseImage, "-f", "dockerfiles/Dockerfile_onbuild_base", ".")
if err := buildOnbuildBase.Run(); err != nil {
fmt.Printf("error building onbuild base: %v", err)
os.Exit(1)
}
pushOnbuildBase := exec.Command("docker", "push", config.onbuildBaseImage)
if err := pushOnbuildBase.Run(); err != nil {
fmt.Printf("error pushing onbuild base %s: %v", config.onbuildBaseImage, err)
os.Exit(1)
}
fmt.Println("Building hardlink base image")
buildHardlinkBase := exec.Command("docker", "build", "-t", config.hardlinkBaseImage, "-f", "dockerfiles/Dockerfile_hardlink_base", ".")
if err := buildHardlinkBase.Run(); err != nil {
fmt.Printf("error building hardlink base: %v", err)
os.Exit(1)
}
pushHardlinkBase := exec.Command("docker", "push", config.hardlinkBaseImage)
if err := pushHardlinkBase.Run(); err != nil {
fmt.Printf("error pushing hardlink base %s: %v", config.hardlinkBaseImage, err)
os.Exit(1)
}
dockerfiles, err := FindDockerFiles(dockerfilesPath)
if err != nil {
fmt.Printf("Coudn't create map of dockerfiles: %s", err)
os.Exit(1)
}
imageBuilder = NewDockerFileBuilder(dockerfiles)
g := errgroup.Group{}
for dockerfile := range imageBuilder.FilesBuilt {
df := dockerfile
g.Go(func() error {
return imageBuilder.BuildImage(config.imageRepo, config.gcsBucket, dockerfilesPath, df)
})
}
if err := g.Wait(); err != nil {
fmt.Printf("Error building images: %s", err)
os.Exit(1)
}
os.Exit(m.Run())
}
func TestRun(t *testing.T) {
for dockerfile, built := range imageBuilder.FilesBuilt {
for dockerfile := range imageBuilder.FilesBuilt {
t.Run("test_"+dockerfile, func(t *testing.T) {
dockerfile := dockerfile
t.Parallel()
if _, ok := imageBuilder.DockerfilesToIgnore[dockerfile]; ok {
t.SkipNow()
}
if _, ok := imageBuilder.TestCacheDockerfiles[dockerfile]; ok {
t.SkipNow()
}
if !built {
err := imageBuilder.BuildImage(config.imageRepo, config.gcsBucket, dockerfilesPath, dockerfile)
if err != nil {
t.Fatalf("Failed to build kaniko and docker images for %s: %s", dockerfile, err)
}
}
dockerImage := GetDockerImage(config.imageRepo, dockerfile)
kanikoImage := GetKanikoImage(config.imageRepo, dockerfile)
// container-diff
daemonDockerImage := daemonPrefix + dockerImage
containerdiffCmd := exec.Command("container-diff", "diff",
containerdiffCmd := exec.Command("container-diff", "diff", "--no-cache",
daemonDockerImage, kanikoImage,
"-q", "--type=file", "--type=metadata", "--json")
diff := RunCommand(containerdiffCmd, t)
@@ -215,6 +228,97 @@ func TestRun(t *testing.T) {
})
}
err := logBenchmarks("benchmark")
if err != nil {
t.Logf("Failed to create benchmark file: %v", err)
}
}
func TestGitBuildcontext(t *testing.T) {
repo := "github.com/GoogleContainerTools/kaniko"
dockerfile := "integration/dockerfiles/Dockerfile_test_run_2"
// Build with docker
dockerImage := GetDockerImage(config.imageRepo, "Dockerfile_test_git")
dockerCmd := exec.Command("docker",
append([]string{"build",
"-t", dockerImage,
"-f", dockerfile,
repo})...)
out, err := RunCommandWithoutTest(dockerCmd)
if err != nil {
t.Errorf("Failed to build image %s with docker command \"%s\": %s %s", dockerImage, dockerCmd.Args, err, string(out))
}
// Build with kaniko
kanikoImage := GetKanikoImage(config.imageRepo, "Dockerfile_test_git")
kanikoCmd := exec.Command("docker",
append([]string{"run",
"-v", os.Getenv("HOME") + "/.config/gcloud:/root/.config/gcloud",
ExecutorImage,
"-f", dockerfile,
"-d", kanikoImage,
"-c", fmt.Sprintf("git://%s", repo)})...)
out, err = RunCommandWithoutTest(kanikoCmd)
if err != nil {
t.Errorf("Failed to build image %s with kaniko command \"%s\": %v %s", dockerImage, kanikoCmd.Args, err, string(out))
}
// container-diff
daemonDockerImage := daemonPrefix + dockerImage
containerdiffCmd := exec.Command("container-diff", "diff", "--no-cache",
daemonDockerImage, kanikoImage,
"-q", "--type=file", "--type=metadata", "--json")
diff := RunCommand(containerdiffCmd, t)
t.Logf("diff = %s", string(diff))
expected := fmt.Sprintf(emptyContainerDiff, dockerImage, kanikoImage, dockerImage, kanikoImage)
checkContainerDiffOutput(t, diff, expected)
}
func TestGitBuildContextWithBranch(t *testing.T) {
repo := "github.com/GoogleContainerTools/kaniko#refs/tags/v0.10.0"
dockerfile := "integration/dockerfiles/Dockerfile_test_run_2"
// Build with docker
dockerImage := GetDockerImage(config.imageRepo, "Dockerfile_test_git")
dockerCmd := exec.Command("docker",
append([]string{"build",
"-t", dockerImage,
"-f", dockerfile,
repo})...)
out, err := RunCommandWithoutTest(dockerCmd)
if err != nil {
t.Errorf("Failed to build image %s with docker command \"%s\": %s %s", dockerImage, dockerCmd.Args, err, string(out))
}
// Build with kaniko
kanikoImage := GetKanikoImage(config.imageRepo, "Dockerfile_test_git")
kanikoCmd := exec.Command("docker",
append([]string{"run",
"-v", os.Getenv("HOME") + "/.config/gcloud:/root/.config/gcloud",
ExecutorImage,
"-f", dockerfile,
"-d", kanikoImage,
"-c", fmt.Sprintf("git://%s", repo)})...)
out, err = RunCommandWithoutTest(kanikoCmd)
if err != nil {
t.Errorf("Failed to build image %s with kaniko command \"%s\": %v %s", dockerImage, kanikoCmd.Args, err, string(out))
}
// container-diff
daemonDockerImage := daemonPrefix + dockerImage
containerdiffCmd := exec.Command("container-diff", "diff", "--no-cache",
daemonDockerImage, kanikoImage,
"-q", "--type=file", "--type=metadata", "--json")
diff := RunCommand(containerdiffCmd, t)
t.Logf("diff = %s", string(diff))
expected := fmt.Sprintf(emptyContainerDiff, dockerImage, kanikoImage, dockerImage, kanikoImage)
checkContainerDiffOutput(t, diff, expected)
}
func TestLayers(t *testing.T) {
@@ -222,17 +326,13 @@ func TestLayers(t *testing.T) {
"Dockerfile_test_add": 11,
"Dockerfile_test_scratch": 3,
}
for dockerfile, built := range imageBuilder.FilesBuilt {
for dockerfile := range imageBuilder.FilesBuilt {
t.Run("test_layer_"+dockerfile, func(t *testing.T) {
dockerfile := dockerfile
t.Parallel()
if _, ok := imageBuilder.DockerfilesToIgnore[dockerfile]; ok {
t.SkipNow()
}
if !built {
err := imageBuilder.BuildImage(config.imageRepo, config.gcsBucket, dockerfilesPath, dockerfile)
if err != nil {
t.Fatalf("Failed to build kaniko and docker images for %s: %s", dockerfile, err)
}
}
// Pull the kaniko image
dockerImage := GetDockerImage(config.imageRepo, dockerfile)
kanikoImage := GetKanikoImage(config.imageRepo, dockerfile)
@@ -241,6 +341,11 @@ func TestLayers(t *testing.T) {
checkLayers(t, dockerImage, kanikoImage, offset[dockerfile])
})
}
err := logBenchmarks("benchmark_layers")
if err != nil {
t.Logf("Failed to create benchmark file: %v", err)
}
}
// Build each image with kaniko twice, and then make sure they're exactly the same
@@ -248,6 +353,8 @@ func TestCache(t *testing.T) {
populateVolumeCache()
for dockerfile := range imageBuilder.TestCacheDockerfiles {
t.Run("test_cache_"+dockerfile, func(t *testing.T) {
dockerfile := dockerfile
t.Parallel()
cache := filepath.Join(config.imageRepo, "cache", fmt.Sprintf("%v", time.Now().UnixNano()))
// Build the initial image which will cache layers
if err := imageBuilder.buildCachedImages(config.imageRepo, cache, dockerfilesPath, 0); err != nil {
@@ -273,31 +380,10 @@ func TestCache(t *testing.T) {
checkContainerDiffOutput(t, diff, expected)
})
}
}
func TestDockerignore(t *testing.T) {
t.Run(fmt.Sprintf("test_%s", ignoreDockerfile), func(t *testing.T) {
if err := setupIgnoreTestDir(); err != nil {
t.Fatalf("error setting up ignore test dir: %v", err)
}
if err := generateDockerignoreImages(config.imageRepo); err != nil {
t.Fatalf("error generating dockerignore test images: %v", err)
}
dockerImage := GetDockerImage(config.imageRepo, ignoreDockerfile)
kanikoImage := GetKanikoImage(config.imageRepo, ignoreDockerfile)
// container-diff
daemonDockerImage := daemonPrefix + dockerImage
containerdiffCmd := exec.Command("container-diff", "diff",
daemonDockerImage, kanikoImage,
"-q", "--type=file", "--type=metadata", "--json")
diff := RunCommand(containerdiffCmd, t)
t.Logf("diff = %s", string(diff))
expected := fmt.Sprintf(emptyContainerDiff, dockerImage, kanikoImage, dockerImage, kanikoImage)
checkContainerDiffOutput(t, diff, expected)
})
if err := logBenchmarks("benchmark_cache"); err != nil {
t.Logf("Failed to create benchmark file: %v", err)
}
}
type fileDiff struct {
@@ -400,3 +486,15 @@ func getImageDetails(image string) (*imageDetails, error) {
digest: digest.Hex,
}, nil
}
func logBenchmarks(benchmark string) error {
if b, err := strconv.ParseBool(os.Getenv("BENCHMARK")); err == nil && b {
f, err := os.Create(benchmark)
if err != nil {
return err
}
f.WriteString(timing.Summary())
defer f.Close()
}
return nil
}
pkg/buildcontext/buildcontext.go
+3 -1
View File
@@ -42,6 +42,8 @@ func GetBuildContext(srcContext string) (BuildContext, error) {
return &S3{context: context}, nil
case constants.LocalDirBuildContextPrefix:
return &Dir{context: context}, nil
case constants.GitBuildContextPrefix:
return &Git{context: context}, nil
}
return nil, errors.New("unknown build context prefix provided, please use one of the following: gs://, dir://, s3://")
return nil, errors.New("unknown build context prefix provided, please use one of the following: gs://, dir://, s3://, git://")
}
pkg/buildcontext/git.go
+46
View File
@@ -0,0 +1,46 @@
/*
Copyright 2018 Google LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package buildcontext
import (
"os"
"strings"
"github.com/GoogleContainerTools/kaniko/pkg/constants"
git "gopkg.in/src-d/go-git.v4"
"gopkg.in/src-d/go-git.v4/plumbing"
)
// Git unifies calls to download and unpack the build context.
type Git struct {
context string
}
// UnpackTarFromBuildContext will provide the directory where Git Repository is Cloned
func (g *Git) UnpackTarFromBuildContext() (string, error) {
directory := constants.BuildContextDir
parts := strings.Split(g.context, "#")
options := git.CloneOptions{
URL: "https://" + parts[0],
Progress: os.Stdout,
}
if len(parts) > 1 {
options.ReferenceName = plumbing.ReferenceName(parts[1])
}
_, err := git.PlainClone(directory, false, &options)
return directory, err
}
pkg/cache/cache.go
Vendored
+107 -9
View File
@@ -17,12 +17,16 @@ limitations under the License.
package cache
import (
"crypto/tls"
"fmt"
"net/http"
"os"
"path"
"path/filepath"
"time"
"github.com/GoogleContainerTools/kaniko/pkg/config"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/authn/k8schain"
"github.com/GoogleContainerTools/kaniko/pkg/creds"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/remote"
@@ -31,14 +35,17 @@ import (
"github.com/sirupsen/logrus"
)
// LayerCache is the layer cache
type LayerCache interface {
RetrieveLayer(string) (v1.Image, error)
}
// RegistryCache is the registry cache
type RegistryCache struct {
Opts *config.KanikoOptions
}
// RetrieveLayer retrieves a layer from the cache given the cache key ck.
func (rc *RegistryCache) RetrieveLayer(ck string) (v1.Image, error) {
cache, err := Destination(rc.Opts, ck)
if err != nil {
@@ -50,15 +57,40 @@ func (rc *RegistryCache) RetrieveLayer(ck string) (v1.Image, error) {
if err != nil {
return nil, errors.Wrap(err, fmt.Sprintf("getting reference for %s", cache))
}
k8sc, err := k8schain.NewNoClient()
registryName := cacheRef.Repository.Registry.Name()
if rc.Opts.InsecureRegistries.Contains(registryName) {
newReg, err := name.NewRegistry(registryName, name.WeakValidation, name.Insecure)
if err != nil {
return nil, err
}
cacheRef.Repository.Registry = newReg
}
tr := http.DefaultTransport.(*http.Transport)
if rc.Opts.SkipTLSVerifyRegistries.Contains(registryName) {
tr.TLSClientConfig = &tls.Config{
InsecureSkipVerify: true,
}
}
img, err := remote.Image(cacheRef, remote.WithTransport(tr), remote.WithAuthFromKeychain(creds.GetKeychain()))
if err != nil {
return nil, err
}
kc := authn.NewMultiKeychain(authn.DefaultKeychain, k8sc)
img, err := remote.Image(cacheRef, remote.WithAuthFromKeychain(kc))
cf, err := img.ConfigFile()
if err != nil {
return nil, err
return nil, errors.Wrap(err, fmt.Sprintf("retrieving config file for %s", cache))
}
expiry := cf.Created.Add(rc.Opts.CacheTTL)
// Layer is stale, rebuild it.
if expiry.Before(time.Now()) {
logrus.Infof("Cache entry expired: %s", cache)
return nil, errors.New(fmt.Sprintf("Cache entry expired: %s", cache))
}
// Force the manifest to be populated
if _, err := img.RawManifest(); err != nil {
return nil, err
@@ -81,6 +113,7 @@ func Destination(opts *config.KanikoOptions, cacheKey string) (string, error) {
return fmt.Sprintf("%s:%s", cache, cacheKey), nil
}
// LocalSource retieves a source image from a local cache given cacheKey
func LocalSource(opts *config.KanikoOptions, cacheKey string) (v1.Image, error) {
cache := opts.CacheDir
if cache == "" {
@@ -89,11 +122,76 @@ func LocalSource(opts *config.KanikoOptions, cacheKey string) (v1.Image, error)
path := path.Join(cache, cacheKey)
imgTar, err := tarball.ImageFromPath(path, nil)
fi, err := os.Stat(path)
if err != nil {
return nil, errors.Wrap(err, "getting file info")
}
// A stale cache is a bad cache
expiry := fi.ModTime().Add(opts.CacheTTL)
if expiry.Before(time.Now()) {
logrus.Debugf("Cached image is too old: %v", fi.ModTime())
return nil, nil
}
logrus.Infof("Found %s in local cache", cacheKey)
return cachedImageFromPath(path)
}
// cachedImage represents a v1.Tarball that is cached locally in a CAS.
// Computing the digest for a v1.Tarball is very expensive. If the tarball
// is named with the digest we can store this and return it directly rather
// than recompute it.
type cachedImage struct {
digest string
v1.Image
mfst *v1.Manifest
}
func (c *cachedImage) Digest() (v1.Hash, error) {
return v1.NewHash(c.digest)
}
func (c *cachedImage) Manifest() (*v1.Manifest, error) {
if c.mfst == nil {
return c.Image.Manifest()
}
return c.mfst, nil
}
func mfstFromPath(p string) (*v1.Manifest, error) {
f, err := os.Open(p)
if err != nil {
return nil, err
}
defer f.Close()
return v1.ParseManifest(f)
}
func cachedImageFromPath(p string) (v1.Image, error) {
imgTar, err := tarball.ImageFromPath(p, nil)
if err != nil {
return nil, errors.Wrap(err, "getting image from path")
}
logrus.Infof("Found %s in local cache", cacheKey)
return imgTar, nil
// Manifests may be present next to the tar, named with a ".json" suffix
mfstPath := p + ".json"
var mfst *v1.Manifest
if _, err := os.Stat(mfstPath); err != nil {
logrus.Debugf("Manifest does not exist at file: %s", mfstPath)
} else {
mfst, err = mfstFromPath(mfstPath)
if err != nil {
logrus.Debugf("Error parsing manifest from file: %s", mfstPath)
} else {
logrus.Infof("Found manifest at %s", mfstPath)
}
}
return &cachedImage{
digest: filepath.Base(p),
Image: imgTar,
mfst: mfst,
}, nil
}
pkg/cache/warm.go
Vendored
+14 -6
View File
@@ -18,6 +18,7 @@ package cache
import (
"fmt"
"io/ioutil"
"path"
"github.com/GoogleContainerTools/kaniko/pkg/config"
@@ -37,25 +38,32 @@ func WarmCache(opts *config.WarmerOptions) error {
for _, image := range images {
cacheRef, err := name.NewTag(image, name.WeakValidation)
if err != nil {
errors.Wrap(err, fmt.Sprintf("Failed to verify image name: %s", image))
return errors.Wrap(err, fmt.Sprintf("Failed to verify image name: %s", image))
}
img, err := remote.Image(cacheRef)
if err != nil {
errors.Wrap(err, fmt.Sprintf("Failed to retrieve image: %s", image))
return errors.Wrap(err, fmt.Sprintf("Failed to retrieve image: %s", image))
}
digest, err := img.Digest()
if err != nil {
errors.Wrap(err, fmt.Sprintf("Failed to retrieve digest: %s", image))
return errors.Wrap(err, fmt.Sprintf("Failed to retrieve digest: %s", image))
}
cachePath := path.Join(cacheDir, digest.String())
err = tarball.WriteToFile(cachePath, cacheRef, img)
if err != nil {
errors.Wrap(err, fmt.Sprintf("Failed to write %s to cache", image))
} else {
logrus.Debugf("Wrote %s to cache", image)
return errors.Wrap(err, fmt.Sprintf("Failed to write %s to cache", image))
}
mfst, err := img.RawManifest()
if err != nil {
return errors.Wrap(err, fmt.Sprintf("Failed to retrieve manifest for %s", image))
}
mfstPath := cachePath + ".json"
if err := ioutil.WriteFile(mfstPath, mfst, 0666); err != nil {
return errors.Wrap(err, fmt.Sprintf("Failed to save manifest for %s", image))
}
logrus.Debugf("Wrote %s to cache", image)
}
return nil
}
pkg/commands/add.go
+7 -4
View File
@@ -43,11 +43,11 @@ type AddCommand struct {
// - If remote file has HTTP Last-Modified header, we set the mtime of the file to that timestamp
// - If dest doesn't end with a slash, the filepath is inferred to be <dest>/<filename>
// 2. If <src> is a local tar archive:
// -If <src> is a local tar archive, it is unpacked at the dest, as 'tar -x' would
// - it is unpacked at the dest, as 'tar -x' would
func (a *AddCommand) ExecuteCommand(config *v1.Config, buildArgs *dockerfile.BuildArgs) error {
replacementEnvs := buildArgs.ReplacementEnvs(config.Env)
srcs, dest, err := resolveEnvAndWildcards(a.cmd.SourcesAndDest, a.buildcontext, replacementEnvs)
srcs, dest, err := util.ResolveEnvAndWildcards(a.cmd.SourcesAndDest, a.buildcontext, replacementEnvs)
if err != nil {
return err
}
@@ -61,7 +61,10 @@ func (a *AddCommand) ExecuteCommand(config *v1.Config, buildArgs *dockerfile.Bui
for _, src := range srcs {
fullPath := filepath.Join(a.buildcontext, src)
if util.IsSrcRemoteFileURL(src) {
urlDest := util.URLDestinationFilepath(src, dest, config.WorkingDir)
urlDest, err := util.URLDestinationFilepath(src, dest, config.WorkingDir, replacementEnvs)
if err != nil {
return err
}
logrus.Infof("Adding remote URL %s to %s", src, urlDest)
if err := util.DownloadFileToDest(src, urlDest); err != nil {
return err
@@ -111,7 +114,7 @@ func (a *AddCommand) String() string {
func (a *AddCommand) FilesUsedFromContext(config *v1.Config, buildArgs *dockerfile.BuildArgs) ([]string, error) {
replacementEnvs := buildArgs.ReplacementEnvs(config.Env)
srcs, _, err := resolveEnvAndWildcards(a.cmd.SourcesAndDest, a.buildcontext, replacementEnvs)
srcs, _, err := util.ResolveEnvAndWildcards(a.cmd.SourcesAndDest, a.buildcontext, replacementEnvs)
if err != nil {
return nil, err
}
pkg/commands/arg.go
+17 -9
View File
@@ -30,27 +30,35 @@ type ArgCommand struct {
// ExecuteCommand only needs to add this ARG key/value as seen
func (r *ArgCommand) ExecuteCommand(config *v1.Config, buildArgs *dockerfile.BuildArgs) error {
replacementEnvs := buildArgs.ReplacementEnvs(config.Env)
resolvedKey, err := util.ResolveEnvironmentReplacement(r.cmd.Key, replacementEnvs, false)
key, val, err := ParseArg(r.cmd.Key, r.cmd.Value, config.Env, buildArgs)
if err != nil {
return err
}
buildArgs.AddArg(key, val)
return nil
}
func ParseArg(key string, val *string, env []string, ba *dockerfile.BuildArgs) (string, *string, error) {
replacementEnvs := ba.ReplacementEnvs(env)
resolvedKey, err := util.ResolveEnvironmentReplacement(key, replacementEnvs, false)
if err != nil {
return "", nil, err
}
var resolvedValue *string
if r.cmd.Value != nil {
value, err := util.ResolveEnvironmentReplacement(*r.cmd.Value, replacementEnvs, false)
if val != nil {
value, err := util.ResolveEnvironmentReplacement(*val, replacementEnvs, false)
if err != nil {
return err
return "", nil, err
}
resolvedValue = &value
} else {
meta := buildArgs.GetAllMeta()
meta := ba.GetAllMeta()
if value, ok := meta[resolvedKey]; ok {
resolvedValue = &value
}
}
buildArgs.AddArg(resolvedKey, resolvedValue)
return nil
return resolvedKey, resolvedValue, nil
}
// String returns some information about the command for the image config history
pkg/commands/copy.go
+13 -17
View File
@@ -45,7 +45,7 @@ func (c *CopyCommand) ExecuteCommand(config *v1.Config, buildArgs *dockerfile.Bu
replacementEnvs := buildArgs.ReplacementEnvs(config.Env)
srcs, dest, err := resolveEnvAndWildcards(c.cmd.SourcesAndDest, c.buildcontext, replacementEnvs)
srcs, dest, err := util.ResolveEnvAndWildcards(c.cmd.SourcesAndDest, c.buildcontext, replacementEnvs)
if err != nil {
return err
}
@@ -70,40 +70,36 @@ func (c *CopyCommand) ExecuteCommand(config *v1.Config, buildArgs *dockerfile.Bu
// we need to add '/' to the end to indicate the destination is a directory
dest = filepath.Join(cwd, dest) + "/"
}
copiedFiles, err := util.CopyDir(fullPath, dest)
copiedFiles, err := util.CopyDir(fullPath, dest, c.buildcontext)
if err != nil {
return err
}
c.snapshotFiles = append(c.snapshotFiles, copiedFiles...)
} else if fi.Mode()&os.ModeSymlink != 0 {
// If file is a symlink, we want to create the same relative symlink
if err := util.CopySymlink(fullPath, destPath); err != nil {
exclude, err := util.CopySymlink(fullPath, destPath, c.buildcontext)
if err != nil {
return err
}
if exclude {
continue
}
c.snapshotFiles = append(c.snapshotFiles, destPath)
} else {
// ... Else, we want to copy over a file
if err := util.CopyFile(fullPath, destPath); err != nil {
exclude, err := util.CopyFile(fullPath, destPath, c.buildcontext)
if err != nil {
return err
}
if exclude {
continue
}
c.snapshotFiles = append(c.snapshotFiles, destPath)
}
}
return nil
}
func resolveEnvAndWildcards(sd instructions.SourcesAndDest, buildcontext string, envs []string) ([]string, string, error) {
// First, resolve any environment replacement
resolvedEnvs, err := util.ResolveEnvironmentReplacementList(sd, envs, true)
if err != nil {
return nil, "", err
}
dest := resolvedEnvs[len(resolvedEnvs)-1]
// Resolve wildcards and get a list of resolved sources
srcs, err := util.ResolveSources(resolvedEnvs, buildcontext)
return srcs, dest, err
}
// FilesToSnapshot should return an empty array if still nil; no files were changed
func (c *CopyCommand) FilesToSnapshot() []string {
return c.snapshotFiles
@@ -121,7 +117,7 @@ func (c *CopyCommand) FilesUsedFromContext(config *v1.Config, buildArgs *dockerf
}
replacementEnvs := buildArgs.ReplacementEnvs(config.Env)
srcs, _, err := resolveEnvAndWildcards(c.cmd.SourcesAndDest, c.buildcontext, replacementEnvs)
srcs, _, err := util.ResolveEnvAndWildcards(c.cmd.SourcesAndDest, c.buildcontext, replacementEnvs)
if err != nil {
return nil, err
}
pkg/commands/expose.go
+1 -1
View File
@@ -54,7 +54,7 @@ func (r *ExposeCommand) ExecuteCommand(config *v1.Config, buildArgs *dockerfile.
}
protocol := strings.Split(p, "/")[1]
if !validProtocol(protocol) {
return fmt.Errorf("Invalid protocol: %s", protocol)
return fmt.Errorf("invalid protocol: %s", protocol)
}
logrus.Infof("Adding exposed port: %s", p)
existingPorts[p] = struct{}{}
pkg/commands/user.go
-5
View File
@@ -48,11 +48,6 @@ func (r *UserCommand) ExecuteCommand(config *v1.Config, buildArgs *dockerfile.Bu
}
}
_, _, err = util.GetUserFromUsername(userStr, groupStr)
if err != nil {
return err
}
if groupStr != "" {
userStr = userStr + ":" + groupStr
}
pkg/commands/user_test.go
+2 -17
View File
@@ -28,57 +28,42 @@ import (
var userTests = []struct {
user string
expectedUID string
shouldError bool
}{
{
user: "root",
expectedUID: "root",
shouldError: false,
},
{
user: "0",
expectedUID: "0",
shouldError: false,
},
{
user: "fakeUser",
expectedUID: "",
shouldError: true,
expectedUID: "fakeUser",
},
{
user: "root:root",
expectedUID: "root:root",
shouldError: false,
},
{
user: "0:root",
expectedUID: "0:root",
shouldError: false,
},
{
user: "root:0",
expectedUID: "root:0",
shouldError: false,
},
{
user: "0:0",
expectedUID: "0:0",
shouldError: false,
},
{
user: "root:fakeGroup",
expectedUID: "",
shouldError: true,
},
{
user: "$envuser",
expectedUID: "root",
shouldError: false,
},
{
user: "root:$envgroup",
expectedUID: "root:root",
shouldError: false,
},
}
@@ -97,6 +82,6 @@ func TestUpdateUser(t *testing.T) {
}
buildArgs := dockerfile.NewBuildArgs([]string{})
err := cmd.ExecuteCommand(cfg, buildArgs)
testutil.CheckErrorAndDeepEqual(t, test.shouldError, err, test.expectedUID, cfg.User)
testutil.CheckErrorAndDeepEqual(t, false, err, test.expectedUID, cfg.User)
}
}
pkg/commands/volume.go
+2 -5
View File
@@ -48,16 +48,13 @@ func (v *VolumeCommand) ExecuteCommand(config *v1.Config, buildArgs *dockerfile.
for _, volume := range resolvedVolumes {
var x struct{}
existingVolumes[volume] = x
err := util.AddVolumePathToWhitelist(volume)
if err != nil {
return err
}
util.AddVolumePathToWhitelist(volume)
// Only create and snapshot the dir if it didn't exist already
if _, err := os.Stat(volume); os.IsNotExist(err) {
logrus.Infof("Creating directory %s", volume)
if err := os.MkdirAll(volume, 0755); err != nil {
return fmt.Errorf("Could not create directory for volume %s: %s", volume, err)
return fmt.Errorf("could not create directory for volume %s: %s", volume, err)
}
}
}
pkg/commands/workdir.go
+6 -2
View File
@@ -34,6 +34,9 @@ type WorkdirCommand struct {
snapshotFiles []string
}
// For testing
var mkdir = os.MkdirAll
func (w *WorkdirCommand) ExecuteCommand(config *v1.Config, buildArgs *dockerfile.BuildArgs) error {
logrus.Info("cmd: workdir")
workdirPath := w.cmd.Path
@@ -50,10 +53,11 @@ func (w *WorkdirCommand) ExecuteCommand(config *v1.Config, buildArgs *dockerfile
logrus.Infof("Changed working directory to %s", config.WorkingDir)
// Only create and snapshot the dir if it didn't exist already
w.snapshotFiles = []string{}
if _, err := os.Stat(config.WorkingDir); os.IsNotExist(err) {
logrus.Infof("Creating directory %s", config.WorkingDir)
w.snapshotFiles = []string{config.WorkingDir}
return os.MkdirAll(config.WorkingDir, 0755)
w.snapshotFiles = append(w.snapshotFiles, config.WorkingDir)
return mkdir(config.WorkingDir, 0755)
}
return nil
}
pkg/commands/workdir_test.go
+44 -17
View File
@@ -16,6 +16,7 @@ limitations under the License.
package commands
import (
"os"
"testing"
"github.com/GoogleContainerTools/kaniko/pkg/dockerfile"
@@ -30,41 +31,66 @@ import (
// This is needed to make sure WorkingDir handles paths correctly
// For example, if WORKDIR specifies a non-absolute path, it should be appended to the current WORKDIR
var workdirTests = []struct {
path string
expectedPath string
path string
expectedPath string
snapshotFiles []string
}{
{
path: "/a",
expectedPath: "/a",
path: "/a",
expectedPath: "/a",
snapshotFiles: []string{"/a"},
},
{
path: "b",
expectedPath: "/a/b",
path: "b",
expectedPath: "/a/b",
snapshotFiles: []string{"/a/b"},
},
{
path: "c",
expectedPath: "/a/b/c",
path: "c",
expectedPath: "/a/b/c",
snapshotFiles: []string{"/a/b/c"},
},
{
path: "/d",
expectedPath: "/d",
path: "/d",
expectedPath: "/d",
snapshotFiles: []string{"/d"},
},
{
path: "$path",
expectedPath: "/d/usr",
path: "$path",
expectedPath: "/d/usr",
snapshotFiles: []string{"/d/usr"},
},
{
path: "$home",
expectedPath: "/root",
path: "$home",
expectedPath: "/root",
snapshotFiles: []string{},
},
{
path: "$path/$home",
expectedPath: "/root/usr/root",
path: "/foo/$path/$home",
expectedPath: "/foo/usr/root",
snapshotFiles: []string{"/foo/usr/root"},
},
{
path: "/tmp",
expectedPath: "/tmp",
snapshotFiles: []string{},
},
}
// For testing
func mockDir(p string, fi os.FileMode) error {
return nil
}
func TestWorkdirCommand(t *testing.T) {
// Mock out mkdir for testing.
oldMkdir := mkdir
mkdir = mockDir
defer func() {
mkdir = oldMkdir
}()
cfg := &v1.Config{
WorkingDir: "/",
Env: []string{
@@ -78,10 +104,11 @@ func TestWorkdirCommand(t *testing.T) {
cmd: &instructions.WorkdirCommand{
Path: test.path,
},
snapshotFiles: []string{},
snapshotFiles: nil,
}
buildArgs := dockerfile.NewBuildArgs([]string{})
cmd.ExecuteCommand(cfg, buildArgs)
testutil.CheckErrorAndDeepEqual(t, false, nil, test.expectedPath, cfg.WorkingDir)
testutil.CheckErrorAndDeepEqual(t, false, nil, test.snapshotFiles, cmd.snapshotFiles)
}
}
pkg/config/args.go
+9
View File
@@ -42,3 +42,12 @@ func (b *multiArg) Set(value string) error {
func (b *multiArg) Type() string {
return "multi-arg type"
}
func (b *multiArg) Contains(v string) bool {
for _, s := range *b {
if s == v {
return true
}
}
return false
}
pkg/config/options.go
+27 -19
View File
@@ -16,27 +16,35 @@ limitations under the License.
package config
import (
"time"
)
// KanikoOptions are options that are set by command line arguments
type KanikoOptions struct {
DockerfilePath string
SrcContext string
SnapshotMode string
Bucket string
TarPath string
Target string
CacheRepo string
CacheDir string
Destinations multiArg
BuildArgs multiArg
Insecure bool
SkipTLSVerify bool
InsecurePull bool
SkipTLSVerifyPull bool
SingleSnapshot bool
Reproducible bool
NoPush bool
Cache bool
Cleanup bool
DockerfilePath string
SrcContext string
SnapshotMode string
Bucket string
TarPath string
Target string
CacheRepo string
CacheDir string
DigestFile string
Destinations multiArg
BuildArgs multiArg
Insecure bool
SkipTLSVerify bool
InsecurePull bool
SkipTLSVerifyPull bool
SingleSnapshot bool
Reproducible bool
NoPush bool
Cache bool
Cleanup bool
CacheTTL time.Duration
InsecureRegistries multiArg
SkipTLSVerifyRegistries multiArg
}
// WarmerOptions are options that are set by command line arguments to the cache warmer.
pkg/config/stage.go
+1
View File
@@ -26,4 +26,5 @@ type KanikoStage struct {
BaseImageStoredLocally bool
SaveStage bool
MetaArgs []instructions.ArgCommand
Index int
}
pkg/constants/constants.go
+4 -7
View File
@@ -57,6 +57,7 @@ const (
GCSBuildContextPrefix = "gs://"
S3BuildContextPrefix = "s3://"
LocalDirBuildContextPrefix = "dir://"
GitBuildContextPrefix = "git://"
HOME = "HOME"
// DefaultHOMEValue is the default value Docker sets for $HOME
@@ -66,14 +67,10 @@ const (
// Docker command names
Cmd = "cmd"
Entrypoint = "entrypoint"
)
// KanikoBuildFiles is the list of files required to build kaniko
var KanikoBuildFiles = []string{"/kaniko/executor",
"/kaniko/ssl/certs/ca-certificates.crt",
"/kaniko/docker-credential-gcr",
"/kaniko/docker-credential-ecr-login",
"/kaniko/.docker/config.json"}
// Name of the .dockerignore file
Dockerignore = ".dockerignore"
)
// ScratchEnvVars are the default environment variables needed for a scratch image.
var ScratchEnvVars = []string{"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"}
pkg/creds/creds_darwin.go
+36
View File
@@ -0,0 +1,36 @@
/*
Copyright 2018 Google LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package creds
import (
"sync"
"github.com/google/go-containerregistry/pkg/authn"
)
var (
setupKeyChainOnce sync.Once
keyChain authn.Keychain
)
// GetKeychain returns a keychain for accessing container registries.
func GetKeychain() authn.Keychain {
setupKeyChainOnce.Do(func() {
keyChain = authn.NewMultiKeychain(authn.DefaultKeychain)
})
return keyChain
}
pkg/creds/creds_linux.go
+54
View File
@@ -0,0 +1,54 @@
/*
Copyright 2018 Google LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package creds
import (
"sync"
"github.com/genuinetools/amicontained/container"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/authn/k8schain"
"github.com/sirupsen/logrus"
)
var (
setupKeyChainOnce sync.Once
keyChain authn.Keychain
)
// GetKeychain returns a keychain for accessing container registries.
func GetKeychain() authn.Keychain {
setupKeyChainOnce.Do(func() {
keyChain = authn.NewMultiKeychain(authn.DefaultKeychain)
// Add the Kubernetes keychain if we're on Kubernetes
r, err := container.DetectRuntime()
if err != nil {
logrus.Warnf("Error detecting container runtime. Using default keychain: %s", err)
return
}
if r == container.RuntimeKubernetes {
k8sc, err := k8schain.NewNoClient()
if err != nil {
logrus.Warnf("Error setting up k8schain. Using default keychain %s", err)
return
}
keyChain = authn.NewMultiKeychain(keyChain, k8sc)
}
})
return keyChain
}
pkg/dockerfile/dockerfile.go
+22 -29
View File
@@ -20,13 +20,15 @@ import (
"bytes"
"fmt"
"io/ioutil"
"path/filepath"
"net/http"
"regexp"
"strconv"
"strings"
"github.com/sirupsen/logrus"
"github.com/GoogleContainerTools/kaniko/pkg/config"
"github.com/GoogleContainerTools/kaniko/pkg/util"
"github.com/docker/docker/builder/dockerignore"
"github.com/moby/buildkit/frontend/dockerfile/instructions"
"github.com/moby/buildkit/frontend/dockerfile/parser"
"github.com/pkg/errors"
@@ -34,10 +36,23 @@ import (
// Stages parses a Dockerfile and returns an array of KanikoStage
func Stages(opts *config.KanikoOptions) ([]config.KanikoStage, error) {
d, err := ioutil.ReadFile(opts.DockerfilePath)
var err error
var d []uint8
match, _ := regexp.MatchString("^https?://", opts.DockerfilePath)
if match {
response, e := http.Get(opts.DockerfilePath)
if e != nil {
return nil, e
}
d, err = ioutil.ReadAll(response.Body)
} else {
d, err = ioutil.ReadFile(opts.DockerfilePath)
}
if err != nil {
return nil, errors.Wrap(err, fmt.Sprintf("reading dockerfile at path %s", opts.DockerfilePath))
}
stages, metaArgs, err := Parse(d)
if err != nil {
return nil, errors.Wrap(err, "parsing dockerfile")
@@ -54,6 +69,7 @@ func Stages(opts *config.KanikoOptions) ([]config.KanikoStage, error) {
return nil, errors.Wrap(err, "resolving base name")
}
stage.Name = resolvedBaseName
logrus.Infof("Resolved base name %s to %s", stage.BaseName, stage.Name)
kanikoStages = append(kanikoStages, config.KanikoStage{
Stage: stage,
BaseImageIndex: baseImageIndex(index, stages),
@@ -61,6 +77,7 @@ func Stages(opts *config.KanikoOptions) ([]config.KanikoStage, error) {
SaveStage: saveStage(index, stages),
Final: index == targetStage,
MetaArgs: metaArgs,
Index: index,
})
if index == targetStage {
break
@@ -94,7 +111,7 @@ func Parse(b []byte) ([]instructions.Stage, []instructions.ArgCommand, error) {
if err != nil {
return nil, nil, err
}
return stages, metaArgs, err
return stages, metaArgs, nil
}
// targetStage returns the index of the target stage kaniko is trying to build
@@ -126,6 +143,7 @@ func resolveStages(stages []instructions.Stage) {
if val, ok := nameToIndex[c.From]; ok {
c.From = val
}
}
}
}
@@ -161,31 +179,6 @@ func saveStage(index int, stages []instructions.Stage) bool {
return true
}
}
for _, cmd := range stage.Commands {
switch c := cmd.(type) {
case *instructions.CopyCommand:
if c.From == strconv.Itoa(index) {
return true
}
}
}
}
return false
}
// DockerignoreExists returns true if .dockerignore exists in the source context
func DockerignoreExists(opts *config.KanikoOptions) bool {
path := filepath.Join(opts.SrcContext, ".dockerignore")
return util.FilepathExists(path)
}
// ParseDockerignore returns a list of all paths in .dockerignore
func ParseDockerignore(opts *config.KanikoOptions) ([]string, error) {
path := filepath.Join(opts.SrcContext, ".dockerignore")
contents, err := ioutil.ReadFile(path)
if err != nil {
return nil, errors.Wrap(err, "parsing .dockerignore")
}
reader := bytes.NewBuffer(contents)
return dockerignore.ReadAll(reader)
}
pkg/dockerfile/dockerfile_test.go
+1 -1
View File
@@ -114,7 +114,7 @@ func Test_SaveStage(t *testing.T) {
{
name: "reference stage in later copy command",
index: 0,
expected: true,
expected: false,
},
{
name: "reference stage in later from command",
pkg/executor/build.go
+251 -56
View File
@@ -23,10 +23,17 @@ import (
"strconv"
"time"
"github.com/otiai10/copy"
"github.com/google/go-containerregistry/pkg/v1/partial"
"github.com/moby/buildkit/frontend/dockerfile/instructions"
"golang.org/x/sync/errgroup"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/empty"
"github.com/google/go-containerregistry/pkg/v1/mutate"
"github.com/google/go-containerregistry/pkg/v1/tarball"
"github.com/pkg/errors"
@@ -38,6 +45,7 @@ import (
"github.com/GoogleContainerTools/kaniko/pkg/constants"
"github.com/GoogleContainerTools/kaniko/pkg/dockerfile"
"github.com/GoogleContainerTools/kaniko/pkg/snapshot"
"github.com/GoogleContainerTools/kaniko/pkg/timing"
"github.com/GoogleContainerTools/kaniko/pkg/util"
)
@@ -52,21 +60,27 @@ type stageBuilder struct {
snapshotter *snapshot.Snapshotter
baseImageDigest string
opts *config.KanikoOptions
cmds []commands.DockerCommand
args *dockerfile.BuildArgs
crossStageDeps map[int][]string
}
// newStageBuilder returns a new type stageBuilder which contains all the information required to build the stage
func newStageBuilder(opts *config.KanikoOptions, stage config.KanikoStage) (*stageBuilder, error) {
func newStageBuilder(opts *config.KanikoOptions, stage config.KanikoStage, crossStageDeps map[int][]string) (*stageBuilder, error) {
sourceImage, err := util.RetrieveSourceImage(stage, opts)
if err != nil {
return nil, err
}
imageConfig, err := util.RetrieveConfigFile(sourceImage)
imageConfig, err := initializeConfig(sourceImage)
if err != nil {
return nil, err
}
if err := resolveOnBuild(&stage, &imageConfig.Config); err != nil {
return nil, err
}
hasher, err := getHasher(opts.SnapshotMode)
if err != nil {
return nil, err
@@ -78,17 +92,45 @@ func newStageBuilder(opts *config.KanikoOptions, stage config.KanikoStage) (*sta
if err != nil {
return nil, err
}
return &stageBuilder{
s := &stageBuilder{
stage: stage,
image: sourceImage,
cf: imageConfig,
snapshotter: snapshotter,
baseImageDigest: digest.String(),
opts: opts,
}, nil
crossStageDeps: crossStageDeps,
}
for _, cmd := range s.stage.Commands {
command, err := commands.GetCommand(cmd, opts.SrcContext)
if err != nil {
return nil, err
}
if command == nil {
continue
}
s.cmds = append(s.cmds, command)
}
s.args = dockerfile.NewBuildArgs(s.opts.BuildArgs)
s.args.AddMetaArgs(s.stage.MetaArgs)
return s, nil
}
func (s *stageBuilder) optimize(compositeKey CompositeCache, cfg v1.Config, cmds []commands.DockerCommand, args *dockerfile.BuildArgs) error {
func initializeConfig(img partial.WithConfigFile) (*v1.ConfigFile, error) {
imageConfig, err := img.ConfigFile()
if err != nil {
return nil, err
}
if img == empty.Image {
imageConfig.Config.Env = constants.ScratchEnvVars
}
return imageConfig, nil
}
func (s *stageBuilder) optimize(compositeKey CompositeCache, cfg v1.Config) error {
if !s.opts.Cache {
return nil
}
@@ -101,13 +143,13 @@ func (s *stageBuilder) optimize(compositeKey CompositeCache, cfg v1.Config, cmds
// We walk through all the commands, running any commands that only operate on metadata.
// We throw the metadata away after, but we need it to properly track command dependencies
// for things like COPY ${FOO} or RUN commands that use environment variables.
for i, command := range cmds {
for i, command := range s.cmds {
if command == nil {
continue
}
compositeKey.AddKey(command.String())
// If the command uses files from the context, add them.
files, err := command.FilesUsedFromContext(&cfg, args)
files, err := command.FilesUsedFromContext(&cfg, s.args)
if err != nil {
return err
}
@@ -124,19 +166,21 @@ func (s *stageBuilder) optimize(compositeKey CompositeCache, cfg v1.Config, cmds
if command.ShouldCacheOutput() {
img, err := layerCache.RetrieveLayer(ck)
if err != nil {
logrus.Debugf("Failed to retrieve layer: %s", err)
logrus.Infof("No cached layer found for cmd %s", command.String())
logrus.Debugf("Key missing was: %s", compositeKey.Key())
break
}
if cacheCmd := command.CacheCommand(img); cacheCmd != nil {
logrus.Infof("Using caching version of cmd: %s", command.String())
cmds[i] = cacheCmd
s.cmds[i] = cacheCmd
}
}
// Mutate the config for any commands that require it.
if command.MetadataOnly() {
if err := command.ExecuteCommand(&cfg, args); err != nil {
if err := command.ExecuteCommand(&cfg, s.args); err != nil {
return err
}
}
@@ -147,61 +191,55 @@ func (s *stageBuilder) optimize(compositeKey CompositeCache, cfg v1.Config, cmds
func (s *stageBuilder) build() error {
// Set the initial cache key to be the base image digest, the build args and the SrcContext.
compositeKey := NewCompositeCache(s.baseImageDigest)
compositeKey.AddKey(s.opts.BuildArgs...)
cmds := []commands.DockerCommand{}
for _, cmd := range s.stage.Commands {
command, err := commands.GetCommand(cmd, s.opts.SrcContext)
if err != nil {
return err
}
if command == nil {
continue
}
cmds = append(cmds, command)
}
args := dockerfile.NewBuildArgs(s.opts.BuildArgs)
args.AddMetaArgs(s.stage.MetaArgs)
// Apply optimizations to the instructions.
if err := s.optimize(*compositeKey, s.cf.Config, cmds, args); err != nil {
if err := s.optimize(*compositeKey, s.cf.Config); err != nil {
return err
}
// Unpack file system to root if we need to.
shouldUnpack := false
for _, cmd := range cmds {
for _, cmd := range s.cmds {
if cmd.RequiresUnpackedFS() {
logrus.Infof("Unpacking rootfs as cmd %s requires it.", cmd.String())
shouldUnpack = true
break
}
}
if len(s.crossStageDeps[s.stage.Index]) > 0 {
shouldUnpack = true
}
if shouldUnpack {
t := timing.Start("FS Unpacking")
if _, err := util.GetFSFromImage(constants.RootDir, s.image); err != nil {
return err
}
timing.DefaultRun.Stop(t)
} else {
logrus.Info("Skipping unpacking as no commands require it.")
}
if err := util.DetectFilesystemWhitelist(constants.WhitelistPath); err != nil {
return err
}
// Take initial snapshot
t := timing.Start("Initial FS snapshot")
if err := s.snapshotter.Init(); err != nil {
return err
}
timing.DefaultRun.Stop(t)
cacheGroup := errgroup.Group{}
for index, command := range cmds {
for index, command := range s.cmds {
if command == nil {
continue
}
// Add the next command to the cache key.
compositeKey.AddKey(command.String())
t := timing.Start("Command: " + command.String())
// If the command uses files from the context, add them.
files, err := command.FilesUsedFromContext(&s.cf.Config, args)
files, err := command.FilesUsedFromContext(&s.cf.Config, s.args)
if err != nil {
return err
}
@@ -212,10 +250,11 @@ func (s *stageBuilder) build() error {
}
logrus.Info(command.String())
if err := command.ExecuteCommand(&s.cf.Config, args); err != nil {
if err := command.ExecuteCommand(&s.cf.Config, s.args); err != nil {
return err
}
files = command.FilesToSnapshot()
timing.DefaultRun.Stop(t)
if !s.shouldTakeSnapshot(index, files) {
continue
@@ -247,30 +286,33 @@ func (s *stageBuilder) build() error {
}
func (s *stageBuilder) takeSnapshot(files []string) (string, error) {
var snapshot string
var err error
t := timing.Start("Snapshotting FS")
if files == nil || s.opts.SingleSnapshot {
return s.snapshotter.TakeSnapshotFS()
snapshot, err = s.snapshotter.TakeSnapshotFS()
} else {
// Volumes are very weird. They get snapshotted in the next command.
files = append(files, util.Volumes()...)
snapshot, err = s.snapshotter.TakeSnapshot(files)
}
// Volumes are very weird. They get created in their command, but snapshotted in the next one.
// Add them to the list of files to snapshot.
for v := range s.cf.Config.Volumes {
files = append(files, v)
}
return s.snapshotter.TakeSnapshot(files)
timing.DefaultRun.Stop(t)
return snapshot, err
}
func (s *stageBuilder) shouldTakeSnapshot(index int, files []string) bool {
isLastCommand := index == len(s.stage.Commands)-1
// We only snapshot the very end of intermediate stages.
if !s.stage.Final {
return isLastCommand
}
// We only snapshot the very end with single snapshot mode on.
if s.opts.SingleSnapshot {
return isLastCommand
}
// Always take snapshots if we're using the cache.
if s.opts.Cache {
return true
}
// nil means snapshot everything.
if files == nil {
return true
@@ -280,6 +322,7 @@ func (s *stageBuilder) shouldTakeSnapshot(index int, files []string) bool {
if len(files) == 0 {
return false
}
return true
}
@@ -313,17 +356,94 @@ func (s *stageBuilder) saveSnapshotToImage(createdBy string, tarPath string) err
}
func CalculateDependencies(opts *config.KanikoOptions) (map[int][]string, error) {
stages, err := dockerfile.Stages(opts)
if err != nil {
return nil, err
}
images := []v1.Image{}
depGraph := map[int][]string{}
for _, s := range stages {
ba := dockerfile.NewBuildArgs(opts.BuildArgs)
ba.AddMetaArgs(s.MetaArgs)
var image v1.Image
var err error
if s.BaseImageStoredLocally {
image = images[s.BaseImageIndex]
} else if s.Name == constants.NoBaseImage {
image = empty.Image
} else {
image, err = util.RetrieveSourceImage(s, opts)
if err != nil {
return nil, err
}
}
cfg, err := initializeConfig(image)
if err != nil {
return nil, err
}
for _, c := range s.Commands {
switch cmd := c.(type) {
case *instructions.CopyCommand:
if cmd.From != "" {
i, err := strconv.Atoi(cmd.From)
if err != nil {
continue
}
resolved, err := util.ResolveEnvironmentReplacementList(cmd.SourcesAndDest, ba.ReplacementEnvs(cfg.Config.Env), true)
if err != nil {
return nil, err
}
depGraph[i] = append(depGraph[i], resolved[0:len(resolved)-1]...)
}
case *instructions.EnvCommand:
if err := util.UpdateConfigEnv(cmd.Env, &cfg.Config, ba.ReplacementEnvs(cfg.Config.Env)); err != nil {
return nil, err
}
image, err = mutate.Config(image, cfg.Config)
if err != nil {
return nil, err
}
case *instructions.ArgCommand:
k, v, err := commands.ParseArg(cmd.Key, cmd.Value, cfg.Config.Env, ba)
if err != nil {
return nil, err
}
ba.AddArg(k, v)
}
}
images = append(images, image)
}
return depGraph, nil
}
// DoBuild executes building the Dockerfile
func DoBuild(opts *config.KanikoOptions) (v1.Image, error) {
t := timing.Start("Total Build Time")
// Parse dockerfile and unpack base image to root
stages, err := dockerfile.Stages(opts)
if err != nil {
return nil, err
}
if err := util.GetExcludedFiles(opts.SrcContext); err != nil {
return nil, err
}
// Some stages may refer to other random images, not previous stages
if err := fetchExtraStages(stages, opts); err != nil {
return nil, err
}
crossStageDependencies, err := CalculateDependencies(opts)
if err != nil {
return nil, err
}
logrus.Infof("Built cross stage deps: %v", crossStageDependencies)
for index, stage := range stages {
sb, err := newStageBuilder(opts, stage)
sb, err := newStageBuilder(opts, stage, crossStageDependencies)
if err != nil {
return nil, errors.Wrap(err, fmt.Sprintf("getting stage builder for stage %d", index))
return nil, err
}
if err := sb.build(); err != nil {
return nil, errors.Wrap(err, "error building stage")
@@ -349,26 +469,99 @@ func DoBuild(opts *config.KanikoOptions) (v1.Image, error) {
return nil, err
}
}
timing.DefaultRun.Stop(t)
return sourceImage, nil
}
if stage.SaveStage {
if err := saveStageAsTarball(index, sourceImage); err != nil {
return nil, err
}
if err := extractImageToDependecyDir(index, sourceImage); err != nil {
if err := saveStageAsTarball(strconv.Itoa(index), sourceImage); err != nil {
return nil, err
}
}
filesToSave, err := filesToSave(crossStageDependencies[index])
if err != nil {
return nil, err
}
dstDir := filepath.Join(constants.KanikoDir, strconv.Itoa(index))
if err := os.MkdirAll(dstDir, 0644); err != nil {
return nil, err
}
for _, p := range filesToSave {
logrus.Infof("Saving file %s for later use.", p)
copy.Copy(p, filepath.Join(dstDir, p))
}
// Delete the filesystem
if err := util.DeleteFilesystem(); err != nil {
return nil, err
}
}
return nil, err
}
func extractImageToDependecyDir(index int, image v1.Image) error {
dependencyDir := filepath.Join(constants.KanikoDir, strconv.Itoa(index))
func filesToSave(deps []string) ([]string, error) {
allFiles := []string{}
for _, src := range deps {
srcs, err := filepath.Glob(src)
if err != nil {
return nil, err
}
allFiles = append(allFiles, srcs...)
}
return allFiles, nil
}
func fetchExtraStages(stages []config.KanikoStage, opts *config.KanikoOptions) error {
t := timing.Start("Fetching Extra Stages")
defer timing.DefaultRun.Stop(t)
var names = []string{}
for stageIndex, s := range stages {
for _, cmd := range s.Commands {
c, ok := cmd.(*instructions.CopyCommand)
if !ok || c.From == "" {
continue
}
// FROMs at this point are guaranteed to be either an integer referring to a previous stage,
// the name of a previous stage, or a name of a remote image.
// If it is an integer stage index, validate that it is actually a previous index
if fromIndex, err := strconv.Atoi(c.From); err == nil && stageIndex > fromIndex && fromIndex >= 0 {
continue
}
// Check if the name is the alias of a previous stage
for _, name := range names {
if name == c.From {
continue
}
}
// This must be an image name, fetch it.
logrus.Debugf("Found extra base image stage %s", c.From)
sourceImage, err := util.RetrieveRemoteImage(c.From, opts)
if err != nil {
return err
}
if err := saveStageAsTarball(c.From, sourceImage); err != nil {
return err
}
if err := extractImageToDependencyDir(c.From, sourceImage); err != nil {
return err
}
}
// Store the name of the current stage in the list with names, if applicable.
if s.Name != "" {
names = append(names, s.Name)
}
}
return nil
}
func extractImageToDependencyDir(name string, image v1.Image) error {
t := timing.Start("Extracting Image to Dependency Dir")
defer timing.DefaultRun.Stop(t)
dependencyDir := filepath.Join(constants.KanikoDir, name)
if err := os.MkdirAll(dependencyDir, 0755); err != nil {
return err
}
@@ -377,16 +570,18 @@ func extractImageToDependecyDir(index int, image v1.Image) error {
return err
}
func saveStageAsTarball(stageIndex int, image v1.Image) error {
func saveStageAsTarball(path string, image v1.Image) error {
t := timing.Start("Saving stage as tarball")
defer timing.DefaultRun.Stop(t)
destRef, err := name.NewTag("temp/tag", name.WeakValidation)
if err != nil {
return err
}
if err := os.MkdirAll(constants.KanikoIntermediateStagesDir, 0750); err != nil {
tarPath := filepath.Join(constants.KanikoIntermediateStagesDir, path)
logrus.Infof("Storing source image from stage %s at path %s", path, tarPath)
if err := os.MkdirAll(filepath.Dir(tarPath), 0750); err != nil {
return err
}
tarPath := filepath.Join(constants.KanikoIntermediateStagesDir, strconv.Itoa(stageIndex))
logrus.Infof("Storing source image from stage %d at path %s", stageIndex, tarPath)
return tarball.WriteToFile(tarPath, destRef, image)
}
pkg/executor/build_test.go
+332 -1
View File
@@ -17,12 +17,19 @@ limitations under the License.
package executor
import (
"io/ioutil"
"os"
"path/filepath"
"reflect"
"sort"
"testing"
"github.com/GoogleContainerTools/kaniko/pkg/config"
"github.com/GoogleContainerTools/kaniko/pkg/dockerfile"
"github.com/GoogleContainerTools/kaniko/testutil"
"github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-cmp/cmp"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/moby/buildkit/frontend/dockerfile/instructions"
)
func Test_reviewConfig(t *testing.T) {
@@ -74,3 +81,327 @@ func stage(t *testing.T, d string) config.KanikoStage {
Stage: stages[0],
}
}
type MockCommand struct {
name string
}
func (m *MockCommand) Name() string {
return m.name
}
func Test_stageBuilder_shouldTakeSnapshot(t *testing.T) {
commands := []instructions.Command{
&MockCommand{name: "command1"},
&MockCommand{name: "command2"},
&MockCommand{name: "command3"},
}
stage := instructions.Stage{
Commands: commands,
}
type fields struct {
stage config.KanikoStage
opts *config.KanikoOptions
}
type args struct {
index int
files []string
}
tests := []struct {
name string
fields fields
args args
want bool
}{
{
name: "final stage not last command",
fields: fields{
stage: config.KanikoStage{
Final: true,
Stage: stage,
},
},
args: args{
index: 1,
},
want: true,
},
{
name: "not final stage last command",
fields: fields{
stage: config.KanikoStage{
Final: false,
Stage: stage,
},
},
args: args{
index: len(commands) - 1,
},
want: true,
},
{
name: "not final stage not last command",
fields: fields{
stage: config.KanikoStage{
Final: false,
Stage: stage,
},
},
args: args{
index: 0,
},
want: true,
},
{
name: "caching enabled intermediate container",
fields: fields{
stage: config.KanikoStage{
Final: false,
Stage: stage,
},
opts: &config.KanikoOptions{Cache: true},
},
args: args{
index: 0,
},
want: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if tt.fields.opts == nil {
tt.fields.opts = &config.KanikoOptions{}
}
s := &stageBuilder{
stage: tt.fields.stage,
opts: tt.fields.opts,
}
if got := s.shouldTakeSnapshot(tt.args.index, tt.args.files); got != tt.want {
t.Errorf("stageBuilder.shouldTakeSnapshot() = %v, want %v", got, tt.want)
}
})
}
}
func TestCalculateDependencies(t *testing.T) {
type args struct {
dockerfile string
}
tests := []struct {
name string
args args
want map[int][]string
}{
{
name: "no deps",
args: args{
dockerfile: `
FROM debian as stage1
RUN foo
FROM stage1
RUN bar
`,
},
want: map[int][]string{},
},
{
name: "args",
args: args{
dockerfile: `
ARG myFile=foo
FROM debian as stage1
RUN foo
FROM stage1
ARG myFile
COPY --from=stage1 /tmp/$myFile.txt .
RUN bar
`,
},
want: map[int][]string{
0: {"/tmp/foo.txt"},
},
},
{
name: "simple deps",
args: args{
dockerfile: `
FROM debian as stage1
FROM alpine
COPY --from=stage1 /foo /bar
`,
},
want: map[int][]string{
0: {"/foo"},
},
},
{
name: "two sets deps",
args: args{
dockerfile: `
FROM debian as stage1
FROM ubuntu as stage2
RUN foo
COPY --from=stage1 /foo /bar
FROM alpine
COPY --from=stage2 /bar /bat
`,
},
want: map[int][]string{
0: {"/foo"},
1: {"/bar"},
},
},
{
name: "double deps",
args: args{
dockerfile: `
FROM debian as stage1
FROM ubuntu as stage2
RUN foo
COPY --from=stage1 /foo /bar
FROM alpine
COPY --from=stage1 /baz /bat
`,
},
want: map[int][]string{
0: {"/foo", "/baz"},
},
},
{
name: "envs in deps",
args: args{
dockerfile: `
FROM debian as stage1
FROM ubuntu as stage2
RUN foo
ENV key1 val1
ENV key2 val2
COPY --from=stage1 /foo/$key1 /foo/$key2 /bar
FROM alpine
COPY --from=stage2 /bar /bat
`,
},
want: map[int][]string{
0: {"/foo/val1", "/foo/val2"},
1: {"/bar"},
},
},
{
name: "envs from base image in deps",
args: args{
dockerfile: `
FROM debian as stage1
ENV key1 baseval1
FROM stage1 as stage2
RUN foo
ENV key2 val2
COPY --from=stage1 /foo/$key1 /foo/$key2 /bar
FROM alpine
COPY --from=stage2 /bar /bat
`,
},
want: map[int][]string{
0: {"/foo/baseval1", "/foo/val2"},
1: {"/bar"},
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
f, _ := ioutil.TempFile("", "")
ioutil.WriteFile(f.Name(), []byte(tt.args.dockerfile), 0755)
opts := &config.KanikoOptions{
DockerfilePath: f.Name(),
}
got, err := CalculateDependencies(opts)
if err != nil {
t.Errorf("got error: %s,", err)
}
if !reflect.DeepEqual(got, tt.want) {
diff := cmp.Diff(got, tt.want)
t.Errorf("CalculateDependencies() = %v, want %v, diff %v", got, tt.want, diff)
}
})
}
}
func Test_filesToSave(t *testing.T) {
tests := []struct {
name string
args []string
want []string
files []string
}{
{
name: "simple",
args: []string{"foo"},
files: []string{"foo"},
want: []string{"foo"},
},
{
name: "glob",
args: []string{"foo*"},
files: []string{"foo", "foo2", "fooooo", "bar"},
want: []string{"foo", "foo2", "fooooo"},
},
{
name: "complex glob",
args: []string{"foo*", "bar?"},
files: []string{"foo", "foo2", "fooooo", "bar", "bar1", "bar2", "bar33"},
want: []string{"foo", "foo2", "fooooo", "bar1", "bar2"},
},
{
name: "dir",
args: []string{"foo"},
files: []string{"foo/bar", "foo/baz", "foo/bat/baz"},
want: []string{"foo"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
tmpDir, err := ioutil.TempDir("", "")
if err != nil {
t.Errorf("error creating tmpdir: %s", err)
}
defer os.RemoveAll(tmpDir)
for _, f := range tt.files {
p := filepath.Join(tmpDir, f)
dir := filepath.Dir(p)
if dir != "." {
if err := os.MkdirAll(dir, 0755); err != nil {
t.Errorf("error making dir: %s", err)
}
}
fp, err := os.Create(p)
if err != nil {
t.Errorf("error making file: %s", err)
}
fp.Close()
}
args := []string{}
for _, arg := range tt.args {
args = append(args, filepath.Join(tmpDir, arg))
}
got, err := filesToSave(args)
if err != nil {
t.Errorf("got err: %s", err)
}
want := []string{}
for _, w := range tt.want {
want = append(want, filepath.Join(tmpDir, w))
}
sort.Strings(want)
sort.Strings(got)
if !reflect.DeepEqual(got, want) {
t.Errorf("filesToSave() = %v, want %v", got, want)
}
})
}
}
pkg/executor/foo
View File
pkg/executor/push.go
+76 -21
View File
@@ -19,14 +19,16 @@ package executor
import (
"crypto/tls"
"fmt"
"io/ioutil"
"net/http"
"time"
"github.com/GoogleContainerTools/kaniko/pkg/cache"
"github.com/GoogleContainerTools/kaniko/pkg/config"
"github.com/GoogleContainerTools/kaniko/pkg/constants"
"github.com/GoogleContainerTools/kaniko/pkg/creds"
"github.com/GoogleContainerTools/kaniko/pkg/timing"
"github.com/GoogleContainerTools/kaniko/pkg/version"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/authn/k8schain"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/empty"
@@ -46,12 +48,49 @@ func (w *withUserAgent) RoundTrip(r *http.Request) (*http.Response, error) {
return w.t.RoundTrip(r)
}
// DoPush is responsible for pushing image to the destinations specified in opts
func DoPush(image v1.Image, opts *config.KanikoOptions) error {
// CheckPushPermissionos checks that the configured credentials can be used to
// push to every specified destination.
func CheckPushPermissions(opts *config.KanikoOptions) error {
if opts.NoPush {
logrus.Info("Skipping push to container registry due to --no-push flag")
return nil
}
checked := map[string]bool{}
for _, destination := range opts.Destinations {
destRef, err := name.NewTag(destination, name.WeakValidation)
if err != nil {
return errors.Wrap(err, "getting tag for destination")
}
if checked[destRef.Context().RepositoryStr()] {
continue
}
registryName := destRef.Repository.Registry.Name()
tr := makeTransport(opts, registryName)
if err := remote.CheckPushPermission(destRef, creds.GetKeychain(), tr); err != nil {
return errors.Wrapf(err, "checking push permission for %q", destRef)
}
checked[destRef.Context().RepositoryStr()] = true
}
return nil
}
// DoPush is responsible for pushing image to the destinations specified in opts
func DoPush(image v1.Image, opts *config.KanikoOptions) error {
t := timing.Start("Total Push Time")
if opts.DigestFile != "" {
digest, err := image.Digest()
if err != nil {
return errors.Wrap(err, "error fetching digest")
}
digestByteArray := []byte(digest.String())
err = ioutil.WriteFile(opts.DigestFile, digestByteArray, 0644)
if err != nil {
return errors.Wrap(err, "writing digest to file failed")
}
}
destRefs := []name.Tag{}
for _, destination := range opts.Destinations {
destRef, err := name.NewTag(destination, name.WeakValidation)
@@ -69,42 +108,49 @@ func DoPush(image v1.Image, opts *config.KanikoOptions) error {
return tarball.MultiWriteToFile(opts.TarPath, tagToImage)
}
if opts.NoPush {
logrus.Info("Skipping push to container registry due to --no-push flag")
return nil
}
// continue pushing unless an error occurs
for _, destRef := range destRefs {
if opts.Insecure {
newReg, err := name.NewInsecureRegistry(destRef.Repository.Registry.Name(), name.WeakValidation)
registryName := destRef.Repository.Registry.Name()
if opts.Insecure || opts.InsecureRegistries.Contains(registryName) {
newReg, err := name.NewRegistry(registryName, name.WeakValidation, name.Insecure)
if err != nil {
return errors.Wrap(err, "getting new insecure registry")
}
destRef.Repository.Registry = newReg
}
k8sc, err := k8schain.NewNoClient()
if err != nil {
return errors.Wrap(err, "getting k8schain client")
}
kc := authn.NewMultiKeychain(authn.DefaultKeychain, k8sc)
pushAuth, err := kc.Resolve(destRef.Context().Registry)
pushAuth, err := creds.GetKeychain().Resolve(destRef.Context().Registry)
if err != nil {
return errors.Wrap(err, "resolving pushAuth")
}
// Create a transport to set our user-agent.
tr := http.DefaultTransport
if opts.SkipTLSVerify {
tr.(*http.Transport).TLSClientConfig = &tls.Config{
InsecureSkipVerify: true,
}
}
tr := makeTransport(opts, registryName)
rt := &withUserAgent{t: tr}
if err := remote.Write(destRef, image, pushAuth, rt); err != nil {
if err := remote.Write(destRef, image, remote.WithAuth(pushAuth), remote.WithTransport(rt)); err != nil {
return errors.Wrap(err, fmt.Sprintf("failed to push to destination %s", destRef))
}
}
timing.DefaultRun.Stop(t)
return nil
}
func makeTransport(opts *config.KanikoOptions, registryName string) http.RoundTripper {
// Create a transport to set our user-agent.
tr := http.DefaultTransport
if opts.SkipTLSVerify || opts.SkipTLSVerifyRegistries.Contains(registryName) {
tr.(*http.Transport).TLSClientConfig = &tls.Config{
InsecureSkipVerify: true,
}
}
return tr
}
// pushLayerToCache pushes layer (tagged with cacheKey) to opts.Cache
// if opts.Cache doesn't exist, infer the cache from the given destination
func pushLayerToCache(opts *config.KanikoOptions, cacheKey string, tarPath string, createdBy string) error {
@@ -118,6 +164,11 @@ func pushLayerToCache(opts *config.KanikoOptions, cacheKey string, tarPath strin
}
logrus.Infof("Pushing layer %s to cache now", cache)
empty := empty.Image
empty, err = mutate.CreatedAt(empty, v1.Time{Time: time.Now()})
if err != nil {
return errors.Wrap(err, "setting empty image created time")
}
empty, err = mutate.Append(empty,
mutate.Addendum{
Layer: layer,
@@ -131,6 +182,10 @@ func pushLayerToCache(opts *config.KanikoOptions, cacheKey string, tarPath strin
return errors.Wrap(err, "appending layer onto empty image")
}
cacheOpts := *opts
cacheOpts.TarPath = "" // tarPath doesn't make sense for Docker layers
cacheOpts.NoPush = false // we want to push cached layers
cacheOpts.Destinations = []string{cache}
cacheOpts.InsecureRegistries = opts.InsecureRegistries
cacheOpts.SkipTLSVerifyRegistries = opts.SkipTLSVerifyRegistries
return DoPush(empty, &cacheOpts)
}
pkg/snapshot/layered_map.go
+10 -19
View File
@@ -23,13 +23,13 @@ import (
"path/filepath"
"strings"
"github.com/GoogleContainerTools/kaniko/pkg/timing"
"github.com/GoogleContainerTools/kaniko/pkg/util"
)
type LayeredMap struct {
layers []map[string]string
whiteouts []map[string]string
added []map[string]string
hasher func(string) (string, error)
// cacheHasher doesn't include mtime in it's hash so that filesystem cache keys are stable
cacheHasher func(string) (string, error)
@@ -47,26 +47,23 @@ func NewLayeredMap(h func(string) (string, error), c func(string) (string, error
func (l *LayeredMap) Snapshot() {
l.whiteouts = append(l.whiteouts, map[string]string{})
l.layers = append(l.layers, map[string]string{})
l.added = append(l.added, map[string]string{})
}
// Key returns a hash for added files
func (l *LayeredMap) Key() (string, error) {
c := bytes.NewBuffer([]byte{})
enc := json.NewEncoder(c)
enc.Encode(l.added)
enc.Encode(l.layers)
return util.SHA256(c)
}
// GetFlattenedPathsForWhiteOut returns all paths in the current FS
func (l *LayeredMap) GetFlattenedPathsForWhiteOut() map[string]struct{} {
func (l *LayeredMap) getFlattenedPathsForWhiteOut() map[string]struct{} {
paths := map[string]struct{}{}
for _, l := range l.layers {
for p := range l {
if strings.HasPrefix(filepath.Base(p), ".wh.") {
delete(paths, p)
} else {
paths[p] = struct{}{}
}
paths[p] = struct{}{}
}
@@ -106,24 +103,19 @@ func (l *LayeredMap) Add(s string) error {
// Use hash function and add to layers
newV, err := l.hasher(s)
if err != nil {
return fmt.Errorf("Error creating hash for %s: %v", s, err)
return fmt.Errorf("error creating hash for %s: %v", s, err)
}
l.layers[len(l.layers)-1][s] = newV
// Use cache hash function and add to added
cacheV, err := l.cacheHasher(s)
if err != nil {
return fmt.Errorf("Error creating cache hash for %s: %v", s, err)
}
l.added[len(l.added)-1][s] = cacheV
return nil
}
// MaybeAdd will add the specified file s to the layered map if
// the layered map's hashing function determines it has changed. If
// it has not changed, it will not be added. Returns true if the file
// was added.
func (l *LayeredMap) MaybeAdd(s string) (bool, error) {
// CheckFileChange checks whether a given file changed
// from the current layered map by its hashing function.
// Returns true if the file is changed.
func (l *LayeredMap) CheckFileChange(s string) (bool, error) {
oldV, ok := l.Get(s)
t := timing.Start("Hashing files")
defer timing.DefaultRun.Stop(t)
newV, err := l.hasher(s)
if err != nil {
return false, err
@@ -131,6 +123,5 @@ func (l *LayeredMap) MaybeAdd(s string) (bool, error) {
if ok && newV == oldV {
return false, nil
}
l.layers[len(l.layers)-1][s] = newV
return true, nil
}
pkg/snapshot/layered_map_test.go
+2 -2
View File
@@ -61,8 +61,8 @@ func Test_CacheKey(t *testing.T) {
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
lm1 := LayeredMap{added: []map[string]string{test.map1}}
lm2 := LayeredMap{added: []map[string]string{test.map2}}
lm1 := LayeredMap{layers: []map[string]string{test.map1}}
lm2 := LayeredMap{layers: []map[string]string{test.map2}}
k1, err := lm1.Key()
if err != nil {
t.Fatalf("error getting key for map 1: %v", err)
pkg/snapshot/snapshot.go
+114 -71
View File
@@ -19,10 +19,13 @@ package snapshot
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"syscall"
"github.com/GoogleContainerTools/kaniko/pkg/timing"
"github.com/karrick/godirwalk"
"github.com/GoogleContainerTools/kaniko/pkg/constants"
"github.com/GoogleContainerTools/kaniko/pkg/util"
@@ -45,7 +48,7 @@ func NewSnapshotter(l *LayeredMap, d string) *Snapshotter {
// Init initializes a new snapshotter
func (s *Snapshotter) Init() error {
_, err := s.TakeSnapshotFS()
_, _, err := s.scanFullFilesystem()
return err
}
@@ -70,50 +73,21 @@ func (s *Snapshotter) TakeSnapshot(files []string) (string, error) {
}
logrus.Info("Taking snapshot of files...")
logrus.Debugf("Taking snapshot of files %v", files)
snapshottedFiles := make(map[string]bool)
// Also add parent directories to keep the permission of them correctly.
filesToAdd := filesWithParentDirs(files)
// Add files to the layered map
for _, file := range filesToAdd {
if err := s.l.Add(file); err != nil {
return "", fmt.Errorf("unable to add file %s to layered map: %s", file, err)
}
}
t := util.NewTar(f)
defer t.Close()
// First add to the tar any parent directories that haven't been added
parentDirs := []string{}
for _, file := range files {
parents := util.ParentDirectories(file)
parentDirs = append(parentDirs, parents...)
}
for _, file := range parentDirs {
file = filepath.Clean(file)
if val, ok := snapshottedFiles[file]; ok && val {
continue
}
snapshottedFiles[file] = true
fileAdded, err := s.l.MaybeAdd(file)
if err != nil {
return "", fmt.Errorf("Unable to add parent dir %s to layered map: %s", file, err)
}
if fileAdded {
err = t.AddFileToTar(file)
if err != nil {
return "", fmt.Errorf("Error adding parent dir %s to tar: %s", file, err)
}
}
}
// Next add the files themselves to the tar
for _, file := range files {
file = filepath.Clean(file)
if val, ok := snapshottedFiles[file]; ok && val {
continue
}
snapshottedFiles[file] = true
if err := s.l.Add(file); err != nil {
return "", fmt.Errorf("Unable to add file %s to layered map: %s", file, err)
}
if err := t.AddFileToTar(file); err != nil {
return "", fmt.Errorf("Error adding file %s to tar: %s", file, err)
}
if err := writeToTar(t, filesToAdd, nil); err != nil {
return "", err
}
return f.Name(), nil
}
@@ -121,13 +95,28 @@ func (s *Snapshotter) TakeSnapshot(files []string) (string, error) {
// TakeSnapshotFS takes a snapshot of the filesystem, avoiding directories in the whitelist, and creates
// a tarball of the changed files.
func (s *Snapshotter) TakeSnapshotFS() (string, error) {
logrus.Info("Taking snapshot of full filesystem...")
f, err := ioutil.TempFile(snapshotPathPrefix, "")
if err != nil {
return "", err
}
defer f.Close()
t := util.NewTar(f)
defer t.Close()
filesToAdd, filesToWhiteOut, err := s.scanFullFilesystem()
if err != nil {
return "", err
}
if err := writeToTar(t, filesToAdd, filesToWhiteOut); err != nil {
return "", err
}
return f.Name(), nil
}
func (s *Snapshotter) scanFullFilesystem() ([]string, []string, error) {
logrus.Info("Taking snapshot of full filesystem...")
// Some of the operations that follow (e.g. hashing) depend on the file system being synced,
// for example the hashing function that determines if files are equal uses the mtime of the files,
@@ -136,57 +125,111 @@ func (s *Snapshotter) TakeSnapshotFS() (string, error) {
syscall.Sync()
s.l.Snapshot()
existingPaths := s.l.GetFlattenedPathsForWhiteOut()
t := util.NewTar(f)
defer t.Close()
timer := timing.Start("Walking filesystem")
// Save the fs state in a map to iterate over later.
memFs := map[string]os.FileInfo{}
filepath.Walk(s.directory, func(path string, info os.FileInfo, err error) error {
memFs[path] = info
return nil
})
memFs := map[string]*godirwalk.Dirent{}
godirwalk.Walk(s.directory, &godirwalk.Options{
Callback: func(path string, ent *godirwalk.Dirent) error {
if util.IsInWhitelist(path) {
if util.IsDestDir(path) {
logrus.Debugf("Skipping paths under %s, as it is a whitelisted directory", path)
return filepath.SkipDir
}
return nil
}
memFs[path] = ent
return nil
},
Unsorted: true,
},
)
timing.DefaultRun.Stop(timer)
// First handle whiteouts
// Get a list of all the files that existed before this layer
existingPaths := s.l.getFlattenedPathsForWhiteOut()
// Find the delta by removing everything left in this layer.
for p := range memFs {
delete(existingPaths, p)
}
// The paths left here are the ones that have been deleted in this layer.
filesToWhiteOut := []string{}
for path := range existingPaths {
// Only add the whiteout if the directory for the file still exists.
dir := filepath.Dir(path)
if _, ok := memFs[dir]; ok {
if s.l.MaybeAddWhiteout(path) {
logrus.Infof("Adding whiteout for %s", path)
if err := t.Whiteout(path); err != nil {
return "", err
}
filesToWhiteOut = append(filesToWhiteOut, path)
}
}
}
// Now create the tar.
filesToAdd := []string{}
for path := range memFs {
whitelisted, err := util.CheckWhitelist(path)
if err != nil {
return "", err
}
if whitelisted {
if util.CheckWhitelist(path) {
logrus.Debugf("Not adding %s to layer, as it's whitelisted", path)
continue
}
// Only add to the tar if we add it to the layeredmap.
maybeAdd, err := s.l.MaybeAdd(path)
// Only add changed files.
fileChanged, err := s.l.CheckFileChange(path)
if err != nil {
return "", err
return nil, nil, err
}
if maybeAdd {
if fileChanged {
logrus.Debugf("Adding %s to layer, because it was changed.", path)
if err := t.AddFileToTar(path); err != nil {
return "", err
}
filesToAdd = append(filesToAdd, path)
}
}
return f.Name(), nil
// Also add parent directories to keep their permissions correctly.
filesToAdd = filesWithParentDirs(filesToAdd)
// Add files to the layered map
for _, file := range filesToAdd {
if err := s.l.Add(file); err != nil {
return nil, nil, fmt.Errorf("unable to add file %s to layered map: %s", file, err)
}
}
return filesToAdd, filesToWhiteOut, nil
}
func writeToTar(t util.Tar, files, whiteouts []string) error {
timer := timing.Start("Writing tar file")
defer timing.DefaultRun.Stop(timer)
// Now create the tar.
for _, path := range whiteouts {
if err := t.Whiteout(path); err != nil {
return err
}
}
for _, path := range files {
if err := t.AddFileToTar(path); err != nil {
return err
}
}
return nil
}
func filesWithParentDirs(files []string) []string {
filesSet := map[string]bool{}
for _, file := range files {
file = filepath.Clean(file)
filesSet[file] = true
for _, dir := range util.ParentDirectories(file) {
dir = filepath.Clean(dir)
filesSet[dir] = true
}
}
newFiles := []string{}
for file := range filesSet {
newFiles = append(newFiles, file)
}
return newFiles
}
pkg/snapshot/snapshot_test.go
+18 -3
View File
@@ -21,6 +21,7 @@ import (
"io/ioutil"
"os"
"path/filepath"
"sort"
"testing"
"github.com/GoogleContainerTools/kaniko/pkg/util"
@@ -60,6 +61,12 @@ func TestSnapshotFSFileChange(t *testing.T) {
fooPath: "newbaz1",
batPath: "baz",
}
for _, dir := range util.ParentDirectories(fooPath) {
snapshotFiles[dir] = ""
}
for _, dir := range util.ParentDirectories(batPath) {
snapshotFiles[dir] = ""
}
numFiles := 0
for {
hdr, err := tr.Next()
@@ -75,7 +82,7 @@ func TestSnapshotFSFileChange(t *testing.T) {
t.Fatalf("Contents of %s incorrect, expected: %s, actual: %s", hdr.Name, snapshotFiles[hdr.Name], string(contents))
}
}
if numFiles != 2 {
if numFiles != len(snapshotFiles) {
t.Fatalf("Incorrect number of files were added, expected: 2, actual: %v", numFiles)
}
}
@@ -105,6 +112,9 @@ func TestSnapshotFSChangePermissions(t *testing.T) {
snapshotFiles := map[string]string{
batPath: "baz2",
}
for _, dir := range util.ParentDirectories(batPath) {
snapshotFiles[dir] = ""
}
numFiles := 0
for {
hdr, err := tr.Next()
@@ -120,7 +130,7 @@ func TestSnapshotFSChangePermissions(t *testing.T) {
t.Fatalf("Contents of %s incorrect, expected: %s, actual: %s", hdr.Name, snapshotFiles[hdr.Name], string(contents))
}
}
if numFiles != 1 {
if numFiles != len(snapshotFiles) {
t.Fatalf("Incorrect number of files were added, expected: 1, got: %v", numFiles)
}
}
@@ -147,7 +157,10 @@ func TestSnapshotFiles(t *testing.T) {
}
defer os.Remove(tarPath)
expectedFiles := []string{"/", "/tmp", filepath.Join(testDir, "foo")}
expectedFiles := []string{
filepath.Join(testDir, "foo"),
}
expectedFiles = append(expectedFiles, util.ParentDirectories(filepath.Join(testDir, "foo"))...)
f, err := os.Open(tarPath)
if err != nil {
@@ -166,6 +179,8 @@ func TestSnapshotFiles(t *testing.T) {
}
actualFiles = append(actualFiles, hdr.Name)
}
sort.Strings(expectedFiles)
sort.Strings(actualFiles)
testutil.CheckErrorAndDeepEqual(t, false, nil, expectedFiles, actualFiles)
}
pkg/timing/timing.go
+15 -4
View File
@@ -18,7 +18,7 @@ package timing
import (
"bytes"
"fmt"
"encoding/json"
"sync"
"text/template"
"time"
@@ -39,12 +39,11 @@ type TimedRun struct {
// Stop stops the specified timer and increments the time spent in that category.
func (tr *TimedRun) Stop(t *Timer) {
stop := currentTimeFunc()
tr.cl.Lock()
defer tr.cl.Unlock()
if _, ok := tr.categories[t.category]; !ok {
tr.categories[t.category] = 0
}
fmt.Println(stop)
tr.cl.Lock()
defer tr.cl.Unlock()
tr.categories[t.category] += stop.Sub(t.startTime)
}
@@ -79,6 +78,10 @@ func Summary() string {
return DefaultRun.Summary()
}
func JSON() (string, error) {
return DefaultRun.JSON()
}
// Summary outputs a summary of the specified TimedRun.
func (tr *TimedRun) Summary() string {
b := bytes.Buffer{}
@@ -88,3 +91,11 @@ func (tr *TimedRun) Summary() string {
DefaultFormat.Execute(&b, tr.categories)
return b.String()
}
func (tr *TimedRun) JSON() (string, error) {
b, err := json.Marshal(tr.categories)
if err != nil {
return "", err
}
return string(b), nil
}
pkg/util/bucket_util.go
+1 -1
View File
@@ -24,7 +24,7 @@ import (
func GetBucketAndItem(context string) (string, string) {
split := strings.SplitN(context, "/", 2)
if len(split) == 2 {
if len(split) == 2 && split[1] != "" {
return split[0], split[1]
}
return split[0], constants.ContextTar
pkg/util/bucket_util_test.go
+67
View File
@@ -0,0 +1,67 @@
/*
Copyright 2018 Google LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package util
import (
"testing"
"github.com/GoogleContainerTools/kaniko/pkg/constants"
"github.com/GoogleContainerTools/kaniko/testutil"
)
func Test_GetBucketAndItem(t *testing.T) {
tests := []struct {
name string
context string
expectedBucket string
expectedItem string
}{
{
name: "three slashes",
context: "test1/test2/test3",
expectedBucket: "test1",
expectedItem: "test2/test3",
},
{
name: "two slashes",
context: "test1/test2",
expectedBucket: "test1",
expectedItem: "test2",
},
{
name: "one slash",
context: "test1/",
expectedBucket: "test1",
expectedItem: constants.ContextTar,
},
{
name: "zero slash",
context: "test1",
expectedBucket: "test1",
expectedItem: constants.ContextTar,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
gotBucket, gotItem := GetBucketAndItem(test.context)
testutil.CheckDeepEqual(t, test.expectedBucket, gotBucket)
testutil.CheckDeepEqual(t, test.expectedItem, gotItem)
})
}
}
pkg/util/command_util.go
+69 -38
View File
@@ -37,11 +37,13 @@ import (
func ResolveEnvironmentReplacementList(values, envs []string, isFilepath bool) ([]string, error) {
var resolvedValues []string
for _, value := range values {
var resolved string
var err error
if IsSrcRemoteFileURL(value) {
resolvedValues = append(resolvedValues, value)
continue
resolved, err = ResolveEnvironmentReplacement(value, envs, false)
} else {
resolved, err = ResolveEnvironmentReplacement(value, envs, isFilepath)
}
resolved, err := ResolveEnvironmentReplacement(value, envs, isFilepath)
logrus.Debugf("Resolved %s to %s", value, resolved)
if err != nil {
return nil, err
@@ -53,7 +55,7 @@ func ResolveEnvironmentReplacementList(values, envs []string, isFilepath bool) (
// ResolveEnvironmentReplacement resolves replacing env variables in some text from envs
// It takes in a string representation of the command, the value to be resolved, and a list of envs (config.Env)
// Ex: fp = $foo/newdir, envs = [foo=/foodir], then this should return /foodir/newdir
// Ex: value = $foo/newdir, envs = [foo=/foodir], then this should return /foodir/newdir
// The dockerfile/shell package handles processing env values
// It handles escape characters and supports expansion from the config.Env array
// Shlex handles some of the following use cases (these and more are tested in integration tests)
@@ -76,6 +78,22 @@ func ResolveEnvironmentReplacement(value string, envs []string, isFilepath bool)
return fp, nil
}
func ResolveEnvAndWildcards(sd instructions.SourcesAndDest, buildcontext string, envs []string) ([]string, string, error) {
// First, resolve any environment replacement
resolvedEnvs, err := ResolveEnvironmentReplacementList(sd, envs, true)
if err != nil {
return nil, "", err
}
dest := resolvedEnvs[len(resolvedEnvs)-1]
// Resolve wildcards and get a list of resolved sources
srcs, err := ResolveSources(resolvedEnvs[0:len(resolvedEnvs)-1], buildcontext)
if err != nil {
return nil, "", err
}
err = IsSrcsValid(sd, srcs, buildcontext)
return srcs, dest, err
}
// ContainsWildcards returns true if any entry in paths contains wildcards
func ContainsWildcards(paths []string) bool {
for _, path := range paths {
@@ -88,23 +106,22 @@ func ContainsWildcards(paths []string) bool {
// ResolveSources resolves the given sources if the sources contains wildcards
// It returns a list of resolved sources
func ResolveSources(srcsAndDest instructions.SourcesAndDest, root string) ([]string, error) {
srcs := srcsAndDest[:len(srcsAndDest)-1]
func ResolveSources(srcs []string, root string) ([]string, error) {
// If sources contain wildcards, we first need to resolve them to actual paths
if ContainsWildcards(srcs) {
logrus.Debugf("Resolving srcs %v...", srcs)
files, err := RelativeFiles("", root)
if err != nil {
return nil, err
}
srcs, err = matchSources(srcs, files)
if err != nil {
return nil, err
}
logrus.Debugf("Resolved sources to %v", srcs)
if !ContainsWildcards(srcs) {
return srcs, nil
}
// Check to make sure the sources are valid
return srcs, IsSrcsValid(srcsAndDest, srcs, root)
logrus.Infof("Resolving srcs %v...", srcs)
files, err := RelativeFiles("", root)
if err != nil {
return nil, err
}
resolved, err := matchSources(srcs, files)
if err != nil {
return nil, err
}
logrus.Debugf("Resolved sources to %v", resolved)
return resolved, nil
}
// matchSources returns a list of sources that match wildcards
@@ -165,20 +182,24 @@ func DestinationFilepath(src, dest, cwd string) (string, error) {
}
// URLDestinationFilepath gives the destination a file from a remote URL should be saved to
func URLDestinationFilepath(rawurl, dest, cwd string) string {
func URLDestinationFilepath(rawurl, dest, cwd string, envs []string) (string, error) {
if !IsDestDir(dest) {
if !filepath.IsAbs(dest) {
return filepath.Join(cwd, dest)
return filepath.Join(cwd, dest), nil
}
return dest
return dest, nil
}
urlBase := filepath.Base(rawurl)
urlBase, err := ResolveEnvironmentReplacement(urlBase, envs, true)
if err != nil {
return "", err
}
destPath := filepath.Join(dest, urlBase)
if !filepath.IsAbs(dest) {
destPath = filepath.Join(cwd, destPath)
}
return destPath
return destPath, nil
}
func IsSrcsValid(srcsAndDest instructions.SourcesAndDest, resolvedSources []string, root string) error {
@@ -186,7 +207,14 @@ func IsSrcsValid(srcsAndDest instructions.SourcesAndDest, resolvedSources []stri
dest := srcsAndDest[len(srcsAndDest)-1]
if !ContainsWildcards(srcs) {
if len(srcs) > 1 && !IsDestDir(dest) {
totalSrcs := 0
for _, src := range srcs {
if excludeFile(src, root) {
continue
}
totalSrcs++
}
if totalSrcs > 1 && !IsDestDir(dest) {
return errors.New("when specifying multiple sources in a COPY command, destination must be a directory and end in '/'")
}
}
@@ -216,7 +244,12 @@ func IsSrcsValid(srcsAndDest instructions.SourcesAndDest, resolvedSources []stri
if err != nil {
return err
}
totalFiles += len(files)
for _, file := range files {
if excludeFile(file, root) {
continue
}
totalFiles++
}
}
if totalFiles == 0 {
return errors.New("copy failed: no source files specified")
@@ -292,13 +325,12 @@ func GetUserFromUsername(userStr string, groupStr string) (string, string, error
// Lookup by username
userObj, err := user.Lookup(userStr)
if err != nil {
if _, ok := err.(user.UnknownUserError); ok {
// Lookup by id
userObj, err = user.LookupId(userStr)
if err != nil {
return "", "", err
}
} else {
if _, ok := err.(user.UnknownUserError); !ok {
return "", "", err
}
// Lookup by id
userObj, err = user.LookupId(userStr)
if err != nil {
return "", "", err
}
}
@@ -308,12 +340,11 @@ func GetUserFromUsername(userStr string, groupStr string) (string, string, error
if groupStr != "" {
group, err = user.LookupGroup(groupStr)
if err != nil {
if _, ok := err.(user.UnknownGroupError); ok {
group, err = user.LookupGroupId(groupStr)
if err != nil {
return "", "", err
}
} else {
if _, ok := err.(user.UnknownGroupError); !ok {
return "", "", err
}
group, err = user.LookupGroupId(groupStr)
if err != nil {
return "", "", err
}
}
pkg/util/command_util_test.go
+121 -30
View File
@@ -17,6 +17,7 @@ limitations under the License.
package util
import (
"reflect"
"sort"
"testing"
@@ -27,14 +28,12 @@ var testURL = "https://github.com/GoogleContainerTools/runtimes-common/blob/mast
var testEnvReplacement = []struct {
path string
command string
envs []string
isFilepath bool
expectedPath string
}{
{
path: "/simple/path",
command: "WORKDIR /simple/path",
path: "/simple/path",
envs: []string{
"simple=/path/",
},
@@ -42,8 +41,7 @@ var testEnvReplacement = []struct {
expectedPath: "/simple/path",
},
{
path: "/simple/path/",
command: "WORKDIR /simple/path/",
path: "/simple/path/",
envs: []string{
"simple=/path/",
},
@@ -51,8 +49,7 @@ var testEnvReplacement = []struct {
expectedPath: "/simple/path/",
},
{
path: "${a}/b",
command: "WORKDIR ${a}/b",
path: "${a}/b",
envs: []string{
"a=/path/",
"b=/path2/",
@@ -61,8 +58,7 @@ var testEnvReplacement = []struct {
expectedPath: "/path/b",
},
{
path: "/$a/b",
command: "COPY ${a}/b /c/",
path: "/$a/b",
envs: []string{
"a=/path/",
"b=/path2/",
@@ -71,8 +67,7 @@ var testEnvReplacement = []struct {
expectedPath: "/path/b",
},
{
path: "/$a/b/",
command: "COPY /${a}/b /c/",
path: "/$a/b/",
envs: []string{
"a=/path/",
"b=/path2/",
@@ -81,8 +76,7 @@ var testEnvReplacement = []struct {
expectedPath: "/path/b/",
},
{
path: "\\$foo",
command: "COPY \\$foo /quux",
path: "\\$foo",
envs: []string{
"foo=/path/",
},
@@ -90,8 +84,14 @@ var testEnvReplacement = []struct {
expectedPath: "$foo",
},
{
path: "8080/$protocol",
command: "EXPOSE 8080/$protocol",
path: "8080/$protocol",
envs: []string{
"protocol=udp",
},
expectedPath: "8080/udp",
},
{
path: "8080/$protocol",
envs: []string{
"protocol=udp",
},
@@ -183,6 +183,7 @@ var urlDestFilepathTests = []struct {
cwd string
dest string
expectedDest string
envs []string
}{
{
url: "https://something/something",
@@ -202,12 +203,19 @@ var urlDestFilepathTests = []struct {
dest: "/dest/",
expectedDest: "/dest/something",
},
{
url: "https://something/$foo.tar.gz",
cwd: "/test",
dest: "/foo/",
expectedDest: "/foo/bar.tar.gz",
envs: []string{"foo=bar"},
},
}
func Test_UrlDestFilepath(t *testing.T) {
for _, test := range urlDestFilepathTests {
actualDest := URLDestinationFilepath(test.url, test.dest, test.cwd)
testutil.CheckErrorAndDeepEqual(t, false, nil, test.expectedDest, actualDest)
actualDest, err := URLDestinationFilepath(test.url, test.dest, test.cwd, test.envs)
testutil.CheckErrorAndDeepEqual(t, false, err, test.expectedDest, actualDest)
}
}
@@ -249,11 +257,13 @@ func Test_MatchSources(t *testing.T) {
}
var isSrcValidTests = []struct {
name string
srcsAndDest []string
resolvedSources []string
shouldErr bool
}{
{
name: "dest isn't directory",
srcsAndDest: []string{
"context/foo",
"context/bar",
@@ -266,6 +276,7 @@ var isSrcValidTests = []struct {
shouldErr: true,
},
{
name: "dest is directory",
srcsAndDest: []string{
"context/foo",
"context/bar",
@@ -278,6 +289,7 @@ var isSrcValidTests = []struct {
shouldErr: false,
},
{
name: "copy file to file",
srcsAndDest: []string{
"context/bar/bam",
"dest",
@@ -288,16 +300,7 @@ var isSrcValidTests = []struct {
shouldErr: false,
},
{
srcsAndDest: []string{
"context/foo",
"dest",
},
resolvedSources: []string{
"context/foo",
},
shouldErr: false,
},
{
name: "copy files with wildcards to dir",
srcsAndDest: []string{
"context/foo",
"context/b*",
@@ -310,6 +313,7 @@ var isSrcValidTests = []struct {
shouldErr: false,
},
{
name: "copy multilple files with wildcards to file",
srcsAndDest: []string{
"context/foo",
"context/b*",
@@ -322,6 +326,7 @@ var isSrcValidTests = []struct {
shouldErr: true,
},
{
name: "copy two files to file, one of which doesn't exist",
srcsAndDest: []string{
"context/foo",
"context/doesntexist*",
@@ -333,6 +338,7 @@ var isSrcValidTests = []struct {
shouldErr: false,
},
{
name: "copy dir to dest not specified as dir",
srcsAndDest: []string{
"context/",
"dest",
@@ -343,6 +349,7 @@ var isSrcValidTests = []struct {
shouldErr: false,
},
{
name: "copy url to file",
srcsAndDest: []string{
testURL,
"dest",
@@ -352,12 +359,43 @@ var isSrcValidTests = []struct {
},
shouldErr: false,
},
{
name: "copy two srcs, one excluded, to file",
srcsAndDest: []string{
"ignore/foo",
"ignore/bar",
"dest",
},
resolvedSources: []string{
"ignore/foo",
"ignore/bar",
},
shouldErr: false,
},
{
name: "copy two srcs, both excluded, to file",
srcsAndDest: []string{
"ignore/baz",
"ignore/bar",
"dest",
},
resolvedSources: []string{
"ignore/baz",
"ignore/bar",
},
shouldErr: true,
},
}
func Test_IsSrcsValid(t *testing.T) {
for _, test := range isSrcValidTests {
err := IsSrcsValid(test.srcsAndDest, test.resolvedSources, buildContextPath)
testutil.CheckError(t, test.shouldErr, err)
t.Run(test.name, func(t *testing.T) {
if err := GetExcludedFiles(buildContextPath); err != nil {
t.Fatalf("error getting excluded files: %v", err)
}
err := IsSrcsValid(test.srcsAndDest, test.resolvedSources, buildContextPath)
testutil.CheckError(t, test.shouldErr, err)
})
}
}
@@ -370,7 +408,6 @@ var testResolveSources = []struct {
"context/foo",
"context/b*",
testURL,
"dest/",
},
expectedList: []string{
"context/foo",
@@ -418,3 +455,57 @@ func Test_RemoteUrls(t *testing.T) {
}
}
func TestResolveEnvironmentReplacementList(t *testing.T) {
type args struct {
values []string
envs []string
isFilepath bool
}
tests := []struct {
name string
args args
want []string
wantErr bool
}{
{
name: "url",
args: args{
values: []string{
"https://google.com/$foo", "$bar",
},
envs: []string{
"foo=baz",
"bar=bat",
},
},
want: []string{"https://google.com/baz", "bat"},
},
{
name: "mixed",
args: args{
values: []string{
"$foo", "$bar$baz", "baz",
},
envs: []string{
"foo=FOO",
"bar=BAR",
"baz=BAZ",
},
},
want: []string{"FOO", "BARBAZ", "baz"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := ResolveEnvironmentReplacementList(tt.args.values, tt.args.envs, tt.args.isFilepath)
if (err != nil) != tt.wantErr {
t.Errorf("ResolveEnvironmentReplacementList() error = %v, wantErr %v", err, tt.wantErr)
return
}
if !reflect.DeepEqual(got, tt.want) {
t.Errorf("ResolveEnvironmentReplacementList() = %v, want %v", got, tt.want)
}
})
}
}
pkg/util/fs_util.go
+142 -82
View File
@@ -19,7 +19,9 @@ package util
import (
"archive/tar"
"bufio"
"bytes"
"io"
"io/ioutil"
"net/http"
"os"
"path/filepath"
@@ -27,10 +29,11 @@ import (
"syscall"
"time"
"github.com/GoogleContainerTools/kaniko/pkg/constants"
"github.com/docker/docker/builder/dockerignore"
"github.com/docker/docker/pkg/fileutils"
"github.com/google/go-containerregistry/pkg/v1"
"github.com/pkg/errors"
"github.com/GoogleContainerTools/kaniko/pkg/constants"
"github.com/sirupsen/logrus"
)
@@ -39,7 +42,7 @@ type WhitelistEntry struct {
PrefixMatchOnly bool
}
var whitelist = []WhitelistEntry{
var initialWhitelist = []WhitelistEntry{
{
Path: "/kaniko",
PrefixMatchOnly: false,
@@ -59,6 +62,12 @@ var whitelist = []WhitelistEntry{
},
}
var whitelist = initialWhitelist
var volumes = []string{}
var excluded []string
// GetFSFromImage extracts the layers of img to root
// It returns a list of all files extracted
func GetFSFromImage(root string, img v1.Image) ([]string, error) {
@@ -111,14 +120,17 @@ func GetFSFromImage(root string, img v1.Image) ([]string, error) {
func DeleteFilesystem() error {
logrus.Info("Deleting filesystem...")
return filepath.Walk(constants.RootDir, func(path string, info os.FileInfo, _ error) error {
whitelisted, err := CheckWhitelist(path)
if err != nil {
return err
}
if whitelisted || ChildDirInWhitelist(path, constants.RootDir) {
if CheckWhitelist(path) {
if info.IsDir() {
return filepath.SkipDir
}
logrus.Debugf("Not deleting %s, as it's whitelisted", path)
return nil
}
if childDirInWhitelist(path) {
logrus.Debugf("Not deleting %s, as it contains a whitelisted path", path)
return nil
}
if path == constants.RootDir {
return nil
}
@@ -127,16 +139,9 @@ func DeleteFilesystem() error {
}
// ChildDirInWhitelist returns true if there is a child file or directory of the path in the whitelist
func ChildDirInWhitelist(path, directory string) bool {
for _, d := range constants.KanikoBuildFiles {
dirPath := filepath.Join(directory, d)
if HasFilepathPrefix(dirPath, path, false) {
return true
}
}
func childDirInWhitelist(path string) bool {
for _, d := range whitelist {
dirPath := filepath.Join(directory, d.Path)
if HasFilepathPrefix(dirPath, path, d.PrefixMatchOnly) {
if HasFilepathPrefix(d.Path, path, d.PrefixMatchOnly) {
return true
}
}
@@ -171,18 +176,19 @@ func extractFile(dest string, hdr *tar.Header, tr io.Reader) error {
uid := hdr.Uid
gid := hdr.Gid
whitelisted, err := CheckWhitelist(path)
abs, err := filepath.Abs(path)
if err != nil {
return err
}
if whitelisted && !checkWhitelistRoot(dest) {
if CheckWhitelist(abs) && !checkWhitelistRoot(dest) {
logrus.Debugf("Not adding %s because it is whitelisted", path)
return nil
}
switch hdr.Typeflag {
case tar.TypeReg:
logrus.Debugf("creating file %s", path)
// It's possible a file is in the tar before it's directory.
// It's possible a file is in the tar before its directory.
if _, err := os.Stat(dir); os.IsNotExist(err) {
logrus.Debugf("base %s for file %s does not exist. Creating.", base, path)
if err := os.MkdirAll(dir, 0755); err != nil {
@@ -200,37 +206,26 @@ func extractFile(dest string, hdr *tar.Header, tr io.Reader) error {
if err != nil {
return err
}
// manually set permissions on file, since the default umask (022) will interfere
if err = os.Chmod(path, mode); err != nil {
return err
}
if _, err = io.Copy(currFile, tr); err != nil {
return err
}
if err = currFile.Chown(uid, gid); err != nil {
if err = setFilePermissions(path, mode, uid, gid); err != nil {
return err
}
currFile.Close()
case tar.TypeDir:
logrus.Debugf("creating dir %s", path)
if err := os.MkdirAll(path, mode); err != nil {
return err
}
// In some cases, MkdirAll doesn't change the permissions, so run Chmod
if err := os.Chmod(path, mode); err != nil {
return err
}
if err := os.Chown(path, uid, gid); err != nil {
if err := mkdirAllWithPermissions(path, mode, uid, gid); err != nil {
return err
}
case tar.TypeLink:
logrus.Debugf("link from %s to %s", hdr.Linkname, path)
whitelisted, err := CheckWhitelist(hdr.Linkname)
abs, err := filepath.Abs(hdr.Linkname)
if err != nil {
return err
}
if whitelisted {
if CheckWhitelist(abs) {
logrus.Debugf("skipping symlink from %s to %s because %s is whitelisted", hdr.Linkname, path, hdr.Linkname)
return nil
}
@@ -245,8 +240,8 @@ func extractFile(dest string, hdr *tar.Header, tr io.Reader) error {
return errors.Wrapf(err, "error removing %s to make way for new link", hdr.Name)
}
}
if err := os.Link(filepath.Clean(filepath.Join("/", hdr.Linkname)), path); err != nil {
link := filepath.Clean(filepath.Join(dest, hdr.Linkname))
if err := os.Link(link, path); err != nil {
return err
}
@@ -270,30 +265,30 @@ func extractFile(dest string, hdr *tar.Header, tr io.Reader) error {
return nil
}
func CheckWhitelist(path string) (bool, error) {
abs, err := filepath.Abs(path)
if err != nil {
logrus.Infof("unable to get absolute path for %s", path)
return false, err
}
func IsInWhitelist(path string) bool {
for _, wl := range whitelist {
if HasFilepathPrefix(abs, wl.Path, wl.PrefixMatchOnly) {
return true, nil
if !wl.PrefixMatchOnly && path == wl.Path {
return true
}
}
return false, nil
return false
}
func CheckWhitelist(path string) bool {
for _, wl := range whitelist {
if HasFilepathPrefix(path, wl.Path, wl.PrefixMatchOnly) {
return true
}
}
return false
}
func checkWhitelistRoot(root string) bool {
if root == constants.RootDir {
return false
}
for _, wl := range whitelist {
if HasFilepathPrefix(root, wl.Path, wl.PrefixMatchOnly) {
return true
}
}
return false
return CheckWhitelist(root)
}
// Get whitelist from roots of mounted files
@@ -303,6 +298,8 @@ func checkWhitelistRoot(root string) bool {
// Where (5) is the mount point relative to the process's root
// From: https://www.kernel.org/doc/Documentation/filesystems/proc.txt
func DetectFilesystemWhitelist(path string) error {
whitelist = initialWhitelist
volumes = []string{}
f, err := os.Open(path)
if err != nil {
return err
@@ -347,11 +344,7 @@ func RelativeFiles(fp string, root string) ([]string, error) {
if err != nil {
return err
}
whitelisted, err := CheckWhitelist(path)
if err != nil {
return err
}
if whitelisted && !HasFilepathPrefix(path, root, false) {
if CheckWhitelist(path) && !HasFilepathPrefix(path, root, false) {
return nil
}
if err != nil {
@@ -408,22 +401,17 @@ func CreateFile(path string, reader io.Reader, perm os.FileMode, uid uint32, gid
if _, err := io.Copy(dest, reader); err != nil {
return err
}
if err := dest.Chmod(perm); err != nil {
return err
}
return dest.Chown(int(uid), int(gid))
return setFilePermissions(path, perm, int(uid), int(gid))
}
// AddVolumePathToWhitelist adds the given path to the whitelist with
// PrefixMatchOnly set to true. Snapshotting will ignore paths prefixed
// with the volume, but the volume itself will not be ignored.
func AddVolumePathToWhitelist(path string) error {
// AddVolumePath adds the given path to the volume whitelist.
func AddVolumePathToWhitelist(path string) {
logrus.Infof("adding volume %s to whitelist", path)
whitelist = append(whitelist, WhitelistEntry{
Path: path,
PrefixMatchOnly: true,
})
return nil
volumes = append(volumes, path)
}
// DownloadFileToDest downloads the file at rawurl to the given dest for the ADD command
@@ -453,7 +441,7 @@ func DownloadFileToDest(rawurl, dest string) error {
// CopyDir copies the file or directory at src to dest
// It returns a list of files it copied over
func CopyDir(src, dest string) ([]string, error) {
func CopyDir(src, dest, buildcontext string) ([]string, error) {
files, err := RelativeFiles("", src)
if err != nil {
return nil, err
@@ -465,27 +453,29 @@ func CopyDir(src, dest string) ([]string, error) {
if err != nil {
return nil, err
}
if excludeFile(fullPath, buildcontext) {
logrus.Debugf("%s found in .dockerignore, ignoring", src)
continue
}
destPath := filepath.Join(dest, file)
if fi.IsDir() {
logrus.Debugf("Creating directory %s", destPath)
mode := fi.Mode()
uid := int(fi.Sys().(*syscall.Stat_t).Uid)
gid := int(fi.Sys().(*syscall.Stat_t).Gid)
if err := os.MkdirAll(destPath, fi.Mode()); err != nil {
return nil, err
}
if err := os.Chown(destPath, uid, gid); err != nil {
if err := mkdirAllWithPermissions(destPath, mode, uid, gid); err != nil {
return nil, err
}
} else if fi.Mode()&os.ModeSymlink != 0 {
// If file is a symlink, we want to create the same relative symlink
if err := CopySymlink(fullPath, destPath); err != nil {
if _, err := CopySymlink(fullPath, destPath, buildcontext); err != nil {
return nil, err
}
} else {
// ... Else, we want to copy over a file
if err := CopyFile(fullPath, destPath); err != nil {
if _, err := CopyFile(fullPath, destPath, buildcontext); err != nil {
return nil, err
}
}
@@ -495,33 +485,78 @@ func CopyDir(src, dest string) ([]string, error) {
}
// CopySymlink copies the symlink at src to dest
func CopySymlink(src, dest string) error {
func CopySymlink(src, dest, buildcontext string) (bool, error) {
if excludeFile(src, buildcontext) {
logrus.Debugf("%s found in .dockerignore, ignoring", src)
return true, nil
}
link, err := os.Readlink(src)
if err != nil {
return err
return false, err
}
linkDst := filepath.Join(dest, link)
return os.Symlink(linkDst, dest)
if FilepathExists(dest) {
if err := os.RemoveAll(dest); err != nil {
return false, err
}
}
return false, os.Symlink(link, dest)
}
// CopyFile copies the file at src to dest
func CopyFile(src, dest string) error {
func CopyFile(src, dest, buildcontext string) (bool, error) {
if excludeFile(src, buildcontext) {
logrus.Debugf("%s found in .dockerignore, ignoring", src)
return true, nil
}
fi, err := os.Stat(src)
if err != nil {
return err
return false, err
}
logrus.Debugf("Copying file %s to %s", src, dest)
srcFile, err := os.Open(src)
if err != nil {
return err
return false, err
}
defer srcFile.Close()
uid := fi.Sys().(*syscall.Stat_t).Uid
gid := fi.Sys().(*syscall.Stat_t).Gid
return CreateFile(dest, srcFile, fi.Mode(), uid, gid)
return false, CreateFile(dest, srcFile, fi.Mode(), uid, gid)
}
// HasFilepathPrefix checks if the given file path begins with prefix
// GetExcludedFiles gets a list of files to exclude from the .dockerignore
func GetExcludedFiles(buildcontext string) error {
path := filepath.Join(buildcontext, ".dockerignore")
if !FilepathExists(path) {
return nil
}
contents, err := ioutil.ReadFile(path)
if err != nil {
return errors.Wrap(err, "parsing .dockerignore")
}
reader := bytes.NewBuffer(contents)
excluded, err = dockerignore.ReadAll(reader)
return err
}
// excludeFile returns true if the .dockerignore specified this file should be ignored
func excludeFile(path, buildcontext string) bool {
if HasFilepathPrefix(path, buildcontext, false) {
var err error
path, err = filepath.Rel(buildcontext, path)
if err != nil {
logrus.Errorf("unable to get relative path, including %s in build: %v", path, err)
return false
}
}
match, err := fileutils.Matches(path, excluded)
if err != nil {
logrus.Errorf("error matching, including %s in build: %v", path, err)
return false
}
return match
}
// HasFilepathPrefix checks if the given file path begins with prefix
func HasFilepathPrefix(path, prefix string, prefixMatchOnly bool) bool {
path = filepath.Clean(path)
prefix = filepath.Clean(prefix)
@@ -542,3 +577,28 @@ func HasFilepathPrefix(path, prefix string, prefixMatchOnly bool) bool {
}
return true
}
func Volumes() []string {
return volumes
}
func mkdirAllWithPermissions(path string, mode os.FileMode, uid, gid int) error {
if err := os.MkdirAll(path, mode); err != nil {
return err
}
if err := os.Chown(path, uid, gid); err != nil {
return err
}
// In some cases, MkdirAll doesn't change the permissions, so run Chmod
// Must chmod after chown because chown resets the file mode.
return os.Chmod(path, mode)
}
func setFilePermissions(path string, mode os.FileMode, uid, gid int) error {
if err := os.Chown(path, uid, gid); err != nil {
return err
}
// manually set permissions on file, since the default umask (022) will interfere
// Must chmod after chown because chown resets the file mode.
return os.Chmod(path, mode)
}
pkg/util/fs_util_test.go
+133 -7
View File
@@ -227,10 +227,7 @@ func Test_CheckWhitelist(t *testing.T) {
whitelist = original
}()
whitelist = tt.args.whitelist
got, err := CheckWhitelist(tt.args.path)
if err != nil {
t.Fatalf("error checking whitelist: %v", err)
}
got := CheckWhitelist(tt.args.path)
if got != tt.want {
t.Errorf("CheckWhitelist() = %v, want %v", got, tt.want)
}
@@ -367,7 +364,7 @@ func filesAreHardlinks(first, second string) checker {
if err != nil {
t.Fatalf("error getting file %s", first)
}
fi2, err := os.Stat(filepath.Join(second))
fi2, err := os.Stat(filepath.Join(root, second))
if err != nil {
t.Fatalf("error getting file %s", second)
}
@@ -499,11 +496,35 @@ func TestExtractFile(t *testing.T) {
tmpdir: "/tmp/hardlink",
hdrs: []*tar.Header{
fileHeader("/bin/gzip", "gzip-binary", 0751),
hardlinkHeader("/bin/uncompress", "/tmp/hardlink/bin/gzip"),
hardlinkHeader("/bin/uncompress", "/bin/gzip"),
},
checkers: []checker{
fileExists("/bin/gzip"),
filesAreHardlinks("/bin/uncompress", "/tmp/hardlink/bin/gzip"),
filesAreHardlinks("/bin/uncompress", "/bin/gzip"),
},
},
{
name: "file with setuid bit",
contents: []byte("helloworld"),
hdrs: []*tar.Header{fileHeader("./bar", "helloworld", 04644)},
checkers: []checker{
fileExists("/bar"),
fileMatches("/bar", []byte("helloworld")),
permissionsMatch("/bar", 0644|os.ModeSetuid),
},
},
{
name: "dir with sticky bit",
contents: []byte("helloworld"),
hdrs: []*tar.Header{
dirHeader("./foo", 01755),
fileHeader("./foo/bar", "helloworld", 0644),
},
checkers: []checker{
fileExists("/foo/bar"),
fileMatches("/foo/bar", []byte("helloworld")),
permissionsMatch("/foo/bar", 0644),
permissionsMatch("/foo", 0755|os.ModeDir|os.ModeSticky),
},
},
}
@@ -536,3 +557,108 @@ func TestExtractFile(t *testing.T) {
})
}
}
func TestCopySymlink(t *testing.T) {
type tc struct {
name string
linkTarget string
dest string
beforeLink func(r string) error
}
tcs := []tc{{
name: "absolute symlink",
linkTarget: "/abs/dest",
}, {
name: "relative symlink",
linkTarget: "rel",
}, {
name: "symlink copy overwrites existing file",
linkTarget: "/abs/dest",
dest: "overwrite_me",
beforeLink: func(r string) error {
return ioutil.WriteFile(filepath.Join(r, "overwrite_me"), nil, 0644)
},
}}
for _, tc := range tcs {
t.Run(tc.name, func(t *testing.T) {
tc := tc
t.Parallel()
r, err := ioutil.TempDir("", "")
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(r)
if tc.beforeLink != nil {
if err := tc.beforeLink(r); err != nil {
t.Fatal(err)
}
}
link := filepath.Join(r, "link")
dest := filepath.Join(r, "copy")
if tc.dest != "" {
dest = filepath.Join(r, tc.dest)
}
if err := os.Symlink(tc.linkTarget, link); err != nil {
t.Fatal(err)
}
if _, err := CopySymlink(link, dest, ""); err != nil {
t.Fatal(err)
}
got, err := os.Readlink(dest)
if err != nil {
t.Fatalf("error reading link %s: %s", link, err)
}
if got != tc.linkTarget {
t.Errorf("link target does not match: %s != %s", got, tc.linkTarget)
}
})
}
}
func Test_childDirInWhitelist(t *testing.T) {
type args struct {
path string
whitelist []WhitelistEntry
}
tests := []struct {
name string
args args
want bool
}{
{
name: "not in whitelist",
args: args{
path: "/foo",
},
want: false,
},
{
name: "child in whitelist",
args: args{
path: "/foo",
whitelist: []WhitelistEntry{
{
Path: "/foo/bar",
},
},
},
want: true,
},
}
oldWhitelist := whitelist
defer func() {
whitelist = oldWhitelist
}()
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
whitelist = tt.args.whitelist
if got := childDirInWhitelist(tt.args.path); got != tt.want {
t.Errorf("childDirInWhitelist() = %v, want %v", got, tt.want)
}
})
}
}
pkg/util/image_util.go
+17 -30
View File
@@ -23,12 +23,13 @@ import (
"path/filepath"
"strconv"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/authn/k8schain"
"github.com/GoogleContainerTools/kaniko/pkg/timing"
"github.com/GoogleContainerTools/kaniko/pkg/creds"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/empty"
"github.com/google/go-containerregistry/pkg/v1/partial"
"github.com/google/go-containerregistry/pkg/v1/remote"
"github.com/google/go-containerregistry/pkg/v1/tarball"
"github.com/sirupsen/logrus"
@@ -39,13 +40,15 @@ import (
)
var (
// For testing
retrieveRemoteImage = remoteImage
// RetrieveRemoteImage downloads an image from a remote location
RetrieveRemoteImage = remoteImage
retrieveTarImage = tarballImage
)
// RetrieveSourceImage returns the base image of the stage at index
func RetrieveSourceImage(stage config.KanikoStage, opts *config.KanikoOptions) (v1.Image, error) {
t := timing.Start("Retrieving Source Image")
defer timing.DefaultRun.Stop(t)
buildArgs := opts.BuildArgs
var metaArgsString []string
for _, arg := range stage.MetaArgs {
@@ -67,33 +70,21 @@ func RetrieveSourceImage(stage config.KanikoStage, opts *config.KanikoOptions) (
return retrieveTarImage(stage.BaseImageIndex)
}
// Next, check if local caching is enabled
// Finally, check if local caching is enabled
// If so, look in the local cache before trying the remote registry
if opts.Cache && opts.CacheDir != "" {
if opts.CacheDir != "" {
cachedImage, err := cachedImage(opts, currentBaseName)
if cachedImage != nil {
return cachedImage, nil
}
if err != nil {
logrus.Warnf("Error while retrieving image from cache: %v", err)
logrus.Infof("Error while retrieving image from cache: %v", err)
}
}
// Otherwise, initialize image as usual
return retrieveRemoteImage(currentBaseName, opts)
}
// RetrieveConfigFile returns the config file for an image
func RetrieveConfigFile(sourceImage partial.WithConfigFile) (*v1.ConfigFile, error) {
imageConfig, err := sourceImage.ConfigFile()
if err != nil {
return nil, err
}
if sourceImage == empty.Image {
imageConfig.Config.Env = constants.ScratchEnvVars
}
return imageConfig, nil
return RetrieveRemoteImage(currentBaseName, opts)
}
func tarballImage(index int) (v1.Image, error) {
@@ -109,8 +100,9 @@ func remoteImage(image string, opts *config.KanikoOptions) (v1.Image, error) {
return nil, err
}
if opts.InsecurePull {
newReg, err := name.NewInsecureRegistry(ref.Context().RegistryStr(), name.WeakValidation)
registryName := ref.Context().RegistryStr()
if opts.InsecurePull || opts.InsecureRegistries.Contains(registryName) {
newReg, err := name.NewRegistry(registryName, name.WeakValidation, name.Insecure)
if err != nil {
return nil, err
}
@@ -125,18 +117,13 @@ func remoteImage(image string, opts *config.KanikoOptions) (v1.Image, error) {
}
tr := http.DefaultTransport.(*http.Transport)
if opts.SkipTLSVerifyPull {
if opts.SkipTLSVerifyPull || opts.SkipTLSVerifyRegistries.Contains(registryName) {
tr.TLSClientConfig = &tls.Config{
InsecureSkipVerify: true,
}
}
k8sc, err := k8schain.NewNoClient()
if err != nil {
return nil, err
}
kc := authn.NewMultiKeychain(authn.DefaultKeychain, k8sc)
return remote.Image(ref, remote.WithTransport(tr), remote.WithAuthFromKeychain(kc))
return remote.Image(ref, remote.WithTransport(tr), remote.WithAuthFromKeychain(creds.GetKeychain()))
}
func cachedImage(opts *config.KanikoOptions, image string) (v1.Image, error) {
pkg/util/image_util_test.go
+3 -3
View File
@@ -47,14 +47,14 @@ func Test_StandardImage(t *testing.T) {
if err != nil {
t.Error(err)
}
original := retrieveRemoteImage
original := RetrieveRemoteImage
defer func() {
retrieveRemoteImage = original
RetrieveRemoteImage = original
}()
mock := func(image string, opts *config.KanikoOptions) (v1.Image, error) {
return nil, nil
}
retrieveRemoteImage = mock
RetrieveRemoteImage = mock
actual, err := RetrieveSourceImage(config.KanikoStage{
Stage: stages[0],
}, &config.KanikoOptions{})
run_in_docker.sh
+10 -4
View File
@@ -15,15 +15,19 @@
#!/bin/bash
set -e
if [ $# -ne 3 ];
then echo "Usage: run_in_docker.sh <path to Dockerfile> <context directory> <image tag>"
exit 1
if [ $# -lt 3 ];
then echo "Usage: run_in_docker.sh <path to Dockerfile> <context directory> <image tag> <cache>"
fi
dockerfile=$1
context=$2
destination=$3
cache="false"
if [[ ! -z "$4" ]]; then
cache=$4
fi
if [[ ! -e $HOME/.config/gcloud/application_default_credentials.json ]]; then
echo "Application Default Credentials do not exist. Run [gcloud auth application-default login] to configure them"
exit 1
@@ -33,4 +37,6 @@ docker run \
-v $HOME/.config/gcloud:/root/.config/gcloud \
-v ${context}:/workspace \
gcr.io/kaniko-project/executor:latest \
--dockerfile ${dockerfile} --destination ${destination} --context dir:///workspace/
--dockerfile ${dockerfile} --destination ${destination} --context dir:///workspace/ \
--cache=${cache}
testutil/util.go
+8
View File
@@ -41,6 +41,14 @@ func SetupFiles(path string, files map[string]string) error {
return nil
}
func CheckDeepEqual(t *testing.T, expected, actual interface{}) {
t.Helper()
if diff := cmp.Diff(actual, expected); diff != "" {
t.Errorf("%T differ (-got, +want): %s", expected, diff)
return
}
}
func CheckErrorAndDeepEqual(t *testing.T, shouldErr bool, err error, expected, actual interface{}) {
t.Helper()
if err := checkErr(shouldErr, err); err != nil {
vendor/github.com/emirpasic/gods/LICENSE
Generated Vendored
+41
View File
@@ -0,0 +1,41 @@
Copyright (c) 2015, Emir Pasic
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-------------------------------------------------------------------------------
AVL Tree:
Copyright (c) 2017 Benjamin Scher Purcell <benjapurcell@gmail.com>
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
vendor/github.com/emirpasic/gods/containers/containers.go
Generated Vendored
+35
View File
@@ -0,0 +1,35 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package containers provides core interfaces and functions for data structures.
//
// Container is the base interface for all data structures to implement.
//
// Iterators provide stateful iterators.
//
// Enumerable provides Ruby inspired (each, select, map, find, any?, etc.) container functions.
//
// Serialization provides serializers (marshalers) and deserializers (unmarshalers).
package containers
import "github.com/emirpasic/gods/utils"
// Container is base interface that all data structures implement.
type Container interface {
Empty() bool
Size() int
Clear()
Values() []interface{}
}
// GetSortedValues returns sorted container's elements with respect to the passed comparator.
// Does not effect the ordering of elements within the container.
func GetSortedValues(container Container, comparator utils.Comparator) []interface{} {
values := container.Values()
if len(values) < 2 {
return values
}
utils.Sort(values, comparator)
return values
}
vendor/github.com/emirpasic/gods/containers/enumerable.go
Generated Vendored
+61
View File
@@ -0,0 +1,61 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package containers
// EnumerableWithIndex provides functions for ordered containers whose values can be fetched by an index.
type EnumerableWithIndex interface {
// Each calls the given function once for each element, passing that element's index and value.
Each(func(index int, value interface{}))
// Map invokes the given function once for each element and returns a
// container containing the values returned by the given function.
// TODO need help on how to enforce this in containers (don't want to type assert when chaining)
// Map(func(index int, value interface{}) interface{}) Container
// Select returns a new container containing all elements for which the given function returns a true value.
// TODO need help on how to enforce this in containers (don't want to type assert when chaining)
// Select(func(index int, value interface{}) bool) Container
// Any passes each element of the container to the given function and
// returns true if the function ever returns true for any element.
Any(func(index int, value interface{}) bool) bool
// All passes each element of the container to the given function and
// returns true if the function returns true for all elements.
All(func(index int, value interface{}) bool) bool
// Find passes each element of the container to the given function and returns
// the first (index,value) for which the function is true or -1,nil otherwise
// if no element matches the criteria.
Find(func(index int, value interface{}) bool) (int, interface{})
}
// EnumerableWithKey provides functions for ordered containers whose values whose elements are key/value pairs.
type EnumerableWithKey interface {
// Each calls the given function once for each element, passing that element's key and value.
Each(func(key interface{}, value interface{}))
// Map invokes the given function once for each element and returns a container
// containing the values returned by the given function as key/value pairs.
// TODO need help on how to enforce this in containers (don't want to type assert when chaining)
// Map(func(key interface{}, value interface{}) (interface{}, interface{})) Container
// Select returns a new container containing all elements for which the given function returns a true value.
// TODO need help on how to enforce this in containers (don't want to type assert when chaining)
// Select(func(key interface{}, value interface{}) bool) Container
// Any passes each element of the container to the given function and
// returns true if the function ever returns true for any element.
Any(func(key interface{}, value interface{}) bool) bool
// All passes each element of the container to the given function and
// returns true if the function returns true for all elements.
All(func(key interface{}, value interface{}) bool) bool
// Find passes each element of the container to the given function and returns
// the first (key,value) for which the function is true or nil,nil otherwise if no element
// matches the criteria.
Find(func(key interface{}, value interface{}) bool) (interface{}, interface{})
}
vendor/github.com/emirpasic/gods/containers/iterator.go
Generated Vendored
+109
View File
@@ -0,0 +1,109 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package containers
// IteratorWithIndex is stateful iterator for ordered containers whose values can be fetched by an index.
type IteratorWithIndex interface {
// Next moves the iterator to the next element and returns true if there was a next element in the container.
// If Next() returns true, then next element's index and value can be retrieved by Index() and Value().
// If Next() was called for the first time, then it will point the iterator to the first element if it exists.
// Modifies the state of the iterator.
Next() bool
// Value returns the current element's value.
// Does not modify the state of the iterator.
Value() interface{}
// Index returns the current element's index.
// Does not modify the state of the iterator.
Index() int
// Begin resets the iterator to its initial state (one-before-first)
// Call Next() to fetch the first element if any.
Begin()
// First moves the iterator to the first element and returns true if there was a first element in the container.
// If First() returns true, then first element's index and value can be retrieved by Index() and Value().
// Modifies the state of the iterator.
First() bool
}
// IteratorWithKey is a stateful iterator for ordered containers whose elements are key value pairs.
type IteratorWithKey interface {
// Next moves the iterator to the next element and returns true if there was a next element in the container.
// If Next() returns true, then next element's key and value can be retrieved by Key() and Value().
// If Next() was called for the first time, then it will point the iterator to the first element if it exists.
// Modifies the state of the iterator.
Next() bool
// Value returns the current element's value.
// Does not modify the state of the iterator.
Value() interface{}
// Key returns the current element's key.
// Does not modify the state of the iterator.
Key() interface{}
// Begin resets the iterator to its initial state (one-before-first)
// Call Next() to fetch the first element if any.
Begin()
// First moves the iterator to the first element and returns true if there was a first element in the container.
// If First() returns true, then first element's key and value can be retrieved by Key() and Value().
// Modifies the state of the iterator.
First() bool
}
// ReverseIteratorWithIndex is stateful iterator for ordered containers whose values can be fetched by an index.
//
// Essentially it is the same as IteratorWithIndex, but provides additional:
//
// Prev() function to enable traversal in reverse
//
// Last() function to move the iterator to the last element.
//
// End() function to move the iterator past the last element (one-past-the-end).
type ReverseIteratorWithIndex interface {
// Prev moves the iterator to the previous element and returns true if there was a previous element in the container.
// If Prev() returns true, then previous element's index and value can be retrieved by Index() and Value().
// Modifies the state of the iterator.
Prev() bool
// End moves the iterator past the last element (one-past-the-end).
// Call Prev() to fetch the last element if any.
End()
// Last moves the iterator to the last element and returns true if there was a last element in the container.
// If Last() returns true, then last element's index and value can be retrieved by Index() and Value().
// Modifies the state of the iterator.
Last() bool
IteratorWithIndex
}
// ReverseIteratorWithKey is a stateful iterator for ordered containers whose elements are key value pairs.
//
// Essentially it is the same as IteratorWithKey, but provides additional:
//
// Prev() function to enable traversal in reverse
//
// Last() function to move the iterator to the last element.
type ReverseIteratorWithKey interface {
// Prev moves the iterator to the previous element and returns true if there was a previous element in the container.
// If Prev() returns true, then previous element's key and value can be retrieved by Key() and Value().
// Modifies the state of the iterator.
Prev() bool
// End moves the iterator past the last element (one-past-the-end).
// Call Prev() to fetch the last element if any.
End()
// Last moves the iterator to the last element and returns true if there was a last element in the container.
// If Last() returns true, then last element's key and value can be retrieved by Key() and Value().
// Modifies the state of the iterator.
Last() bool
IteratorWithKey
}
vendor/github.com/emirpasic/gods/containers/serialization.go
Generated Vendored
+17
View File
@@ -0,0 +1,17 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package containers
// JSONSerializer provides JSON serialization
type JSONSerializer interface {
// ToJSON outputs the JSON representation of containers's elements.
ToJSON() ([]byte, error)
}
// JSONDeserializer provides JSON deserialization
type JSONDeserializer interface {
// FromJSON populates containers's elements from the input JSON representation.
FromJSON([]byte) error
}
vendor/github.com/emirpasic/gods/lists/arraylist/arraylist.go
Generated Vendored
+200
View File
@@ -0,0 +1,200 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package arraylist implements the array list.
//
// Structure is not thread safe.
//
// Reference: https://en.wikipedia.org/wiki/List_%28abstract_data_type%29
package arraylist
import (
"fmt"
"github.com/emirpasic/gods/lists"
"github.com/emirpasic/gods/utils"
"strings"
)
func assertListImplementation() {
var _ lists.List = (*List)(nil)
}
// List holds the elements in a slice
type List struct {
elements []interface{}
size int
}
const (
growthFactor = float32(2.0) // growth by 100%
shrinkFactor = float32(0.25) // shrink when size is 25% of capacity (0 means never shrink)
)
// New instantiates a new empty list
func New() *List {
return &List{}
}
// Add appends a value at the end of the list
func (list *List) Add(values ...interface{}) {
list.growBy(len(values))
for _, value := range values {
list.elements[list.size] = value
list.size++
}
}
// Get returns the element at index.
// Second return parameter is true if index is within bounds of the array and array is not empty, otherwise false.
func (list *List) Get(index int) (interface{}, bool) {
if !list.withinRange(index) {
return nil, false
}
return list.elements[index], true
}
// Remove removes one or more elements from the list with the supplied indices.
func (list *List) Remove(index int) {
if !list.withinRange(index) {
return
}
list.elements[index] = nil // cleanup reference
copy(list.elements[index:], list.elements[index+1:list.size]) // shift to the left by one (slow operation, need ways to optimize this)
list.size--
list.shrink()
}
// Contains checks if elements (one or more) are present in the set.
// All elements have to be present in the set for the method to return true.
// Performance time complexity of n^2.
// Returns true if no arguments are passed at all, i.e. set is always super-set of empty set.
func (list *List) Contains(values ...interface{}) bool {
for _, searchValue := range values {
found := false
for _, element := range list.elements {
if element == searchValue {
found = true
break
}
}
if !found {
return false
}
}
return true
}
// Values returns all elements in the list.
func (list *List) Values() []interface{} {
newElements := make([]interface{}, list.size, list.size)
copy(newElements, list.elements[:list.size])
return newElements
}
// Empty returns true if list does not contain any elements.
func (list *List) Empty() bool {
return list.size == 0
}
// Size returns number of elements within the list.
func (list *List) Size() int {
return list.size
}
// Clear removes all elements from the list.
func (list *List) Clear() {
list.size = 0
list.elements = []interface{}{}
}
// Sort sorts values (in-place) using.
func (list *List) Sort(comparator utils.Comparator) {
if len(list.elements) < 2 {
return
}
utils.Sort(list.elements[:list.size], comparator)
}
// Swap swaps the two values at the specified positions.
func (list *List) Swap(i, j int) {
if list.withinRange(i) && list.withinRange(j) {
list.elements[i], list.elements[j] = list.elements[j], list.elements[i]
}
}
// Insert inserts values at specified index position shifting the value at that position (if any) and any subsequent elements to the right.
// Does not do anything if position is negative or bigger than list's size
// Note: position equal to list's size is valid, i.e. append.
func (list *List) Insert(index int, values ...interface{}) {
if !list.withinRange(index) {
// Append
if index == list.size {
list.Add(values...)
}
return
}
l := len(values)
list.growBy(l)
list.size += l
// Shift old to right
for i := list.size - 1; i >= index+l; i-- {
list.elements[i] = list.elements[i-l]
}
// Insert new
for i, value := range values {
list.elements[index+i] = value
}
}
// String returns a string representation of container
func (list *List) String() string {
str := "ArrayList\n"
values := []string{}
for _, value := range list.elements[:list.size] {
values = append(values, fmt.Sprintf("%v", value))
}
str += strings.Join(values, ", ")
return str
}
// Check that the index is within bounds of the list
func (list *List) withinRange(index int) bool {
return index >= 0 && index < list.size
}
func (list *List) resize(cap int) {
newElements := make([]interface{}, cap, cap)
copy(newElements, list.elements)
list.elements = newElements
}
// Expand the array if necessary, i.e. capacity will be reached if we add n elements
func (list *List) growBy(n int) {
// When capacity is reached, grow by a factor of growthFactor and add number of elements
currentCapacity := cap(list.elements)
if list.size+n >= currentCapacity {
newCapacity := int(growthFactor * float32(currentCapacity+n))
list.resize(newCapacity)
}
}
// Shrink the array if necessary, i.e. when size is shrinkFactor percent of current capacity
func (list *List) shrink() {
if shrinkFactor == 0.0 {
return
}
// Shrink when size is at shrinkFactor * capacity
currentCapacity := cap(list.elements)
if list.size <= int(float32(currentCapacity)*shrinkFactor) {
list.resize(list.size)
}
}
vendor/github.com/emirpasic/gods/lists/arraylist/enumerable.go
Generated Vendored
+79
View File
@@ -0,0 +1,79 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package arraylist
import "github.com/emirpasic/gods/containers"
func assertEnumerableImplementation() {
var _ containers.EnumerableWithIndex = (*List)(nil)
}
// Each calls the given function once for each element, passing that element's index and value.
func (list *List) Each(f func(index int, value interface{})) {
iterator := list.Iterator()
for iterator.Next() {
f(iterator.Index(), iterator.Value())
}
}
// Map invokes the given function once for each element and returns a
// container containing the values returned by the given function.
func (list *List) Map(f func(index int, value interface{}) interface{}) *List {
newList := &List{}
iterator := list.Iterator()
for iterator.Next() {
newList.Add(f(iterator.Index(), iterator.Value()))
}
return newList
}
// Select returns a new container containing all elements for which the given function returns a true value.
func (list *List) Select(f func(index int, value interface{}) bool) *List {
newList := &List{}
iterator := list.Iterator()
for iterator.Next() {
if f(iterator.Index(), iterator.Value()) {
newList.Add(iterator.Value())
}
}
return newList
}
// Any passes each element of the collection to the given function and
// returns true if the function ever returns true for any element.
func (list *List) Any(f func(index int, value interface{}) bool) bool {
iterator := list.Iterator()
for iterator.Next() {
if f(iterator.Index(), iterator.Value()) {
return true
}
}
return false
}
// All passes each element of the collection to the given function and
// returns true if the function returns true for all elements.
func (list *List) All(f func(index int, value interface{}) bool) bool {
iterator := list.Iterator()
for iterator.Next() {
if !f(iterator.Index(), iterator.Value()) {
return false
}
}
return true
}
// Find passes each element of the container to the given function and returns
// the first (index,value) for which the function is true or -1,nil otherwise
// if no element matches the criteria.
func (list *List) Find(f func(index int, value interface{}) bool) (int, interface{}) {
iterator := list.Iterator()
for iterator.Next() {
if f(iterator.Index(), iterator.Value()) {
return iterator.Index(), iterator.Value()
}
}
return -1, nil
}
vendor/github.com/emirpasic/gods/lists/arraylist/iterator.go
Generated Vendored
+83
View File
@@ -0,0 +1,83 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package arraylist
import "github.com/emirpasic/gods/containers"
func assertIteratorImplementation() {
var _ containers.ReverseIteratorWithIndex = (*Iterator)(nil)
}
// Iterator holding the iterator's state
type Iterator struct {
list *List
index int
}
// Iterator returns a stateful iterator whose values can be fetched by an index.
func (list *List) Iterator() Iterator {
return Iterator{list: list, index: -1}
}
// Next moves the iterator to the next element and returns true if there was a next element in the container.
// If Next() returns true, then next element's index and value can be retrieved by Index() and Value().
// If Next() was called for the first time, then it will point the iterator to the first element if it exists.
// Modifies the state of the iterator.
func (iterator *Iterator) Next() bool {
if iterator.index < iterator.list.size {
iterator.index++
}
return iterator.list.withinRange(iterator.index)
}
// Prev moves the iterator to the previous element and returns true if there was a previous element in the container.
// If Prev() returns true, then previous element's index and value can be retrieved by Index() and Value().
// Modifies the state of the iterator.
func (iterator *Iterator) Prev() bool {
if iterator.index >= 0 {
iterator.index--
}
return iterator.list.withinRange(iterator.index)
}
// Value returns the current element's value.
// Does not modify the state of the iterator.
func (iterator *Iterator) Value() interface{} {
return iterator.list.elements[iterator.index]
}
// Index returns the current element's index.
// Does not modify the state of the iterator.
func (iterator *Iterator) Index() int {
return iterator.index
}
// Begin resets the iterator to its initial state (one-before-first)
// Call Next() to fetch the first element if any.
func (iterator *Iterator) Begin() {
iterator.index = -1
}
// End moves the iterator past the last element (one-past-the-end).
// Call Prev() to fetch the last element if any.
func (iterator *Iterator) End() {
iterator.index = iterator.list.size
}
// First moves the iterator to the first element and returns true if there was a first element in the container.
// If First() returns true, then first element's index and value can be retrieved by Index() and Value().
// Modifies the state of the iterator.
func (iterator *Iterator) First() bool {
iterator.Begin()
return iterator.Next()
}
// Last moves the iterator to the last element and returns true if there was a last element in the container.
// If Last() returns true, then last element's index and value can be retrieved by Index() and Value().
// Modifies the state of the iterator.
func (iterator *Iterator) Last() bool {
iterator.End()
return iterator.Prev()
}
vendor/github.com/emirpasic/gods/lists/arraylist/serialization.go
Generated Vendored
+29
View File
@@ -0,0 +1,29 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package arraylist
import (
"encoding/json"
"github.com/emirpasic/gods/containers"
)
func assertSerializationImplementation() {
var _ containers.JSONSerializer = (*List)(nil)
var _ containers.JSONDeserializer = (*List)(nil)
}
// ToJSON outputs the JSON representation of list's elements.
func (list *List) ToJSON() ([]byte, error) {
return json.Marshal(list.elements[:list.size])
}
// FromJSON populates list's elements from the input JSON representation.
func (list *List) FromJSON(data []byte) error {
err := json.Unmarshal(data, &list.elements)
if err == nil {
list.size = len(list.elements)
}
return err
}
vendor/github.com/emirpasic/gods/lists/lists.go
Generated Vendored
+32
View File
@@ -0,0 +1,32 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package lists provides an abstract List interface.
//
// In computer science, a list or sequence is an abstract data type that represents an ordered sequence of values, where the same value may occur more than once. An instance of a list is a computer representation of the mathematical concept of a finite sequence; the (potentially) infinite analog of a list is a stream. Lists are a basic example of containers, as they contain other values. If the same value occurs multiple times, each occurrence is considered a distinct item.
//
// Reference: https://en.wikipedia.org/wiki/List_%28abstract_data_type%29
package lists
import (
"github.com/emirpasic/gods/containers"
"github.com/emirpasic/gods/utils"
)
// List interface that all lists implement
type List interface {
Get(index int) (interface{}, bool)
Remove(index int)
Add(values ...interface{})
Contains(values ...interface{}) bool
Sort(comparator utils.Comparator)
Swap(index1, index2 int)
Insert(index int, values ...interface{})
containers.Container
// Empty() bool
// Size() int
// Clear()
// Values() []interface{}
}
vendor/github.com/emirpasic/gods/trees/binaryheap/binaryheap.go
Generated Vendored
+163
View File
@@ -0,0 +1,163 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package binaryheap implements a binary heap backed by array list.
//
// Comparator defines this heap as either min or max heap.
//
// Structure is not thread safe.
//
// References: http://en.wikipedia.org/wiki/Binary_heap
package binaryheap
import (
"fmt"
"github.com/emirpasic/gods/lists/arraylist"
"github.com/emirpasic/gods/trees"
"github.com/emirpasic/gods/utils"
"strings"
)
func assertTreeImplementation() {
var _ trees.Tree = (*Heap)(nil)
}
// Heap holds elements in an array-list
type Heap struct {
list *arraylist.List
Comparator utils.Comparator
}
// NewWith instantiates a new empty heap tree with the custom comparator.
func NewWith(comparator utils.Comparator) *Heap {
return &Heap{list: arraylist.New(), Comparator: comparator}
}
// NewWithIntComparator instantiates a new empty heap with the IntComparator, i.e. elements are of type int.
func NewWithIntComparator() *Heap {
return &Heap{list: arraylist.New(), Comparator: utils.IntComparator}
}
// NewWithStringComparator instantiates a new empty heap with the StringComparator, i.e. elements are of type string.
func NewWithStringComparator() *Heap {
return &Heap{list: arraylist.New(), Comparator: utils.StringComparator}
}
// Push adds a value onto the heap and bubbles it up accordingly.
func (heap *Heap) Push(values ...interface{}) {
if len(values) == 1 {
heap.list.Add(values[0])
heap.bubbleUp()
} else {
// Reference: https://en.wikipedia.org/wiki/Binary_heap#Building_a_heap
for _, value := range values {
heap.list.Add(value)
}
size := heap.list.Size()/2 + 1
for i := size; i >= 0; i-- {
heap.bubbleDownIndex(i)
}
}
}
// Pop removes top element on heap and returns it, or nil if heap is empty.
// Second return parameter is true, unless the heap was empty and there was nothing to pop.
func (heap *Heap) Pop() (value interface{}, ok bool) {
value, ok = heap.list.Get(0)
if !ok {
return
}
lastIndex := heap.list.Size() - 1
heap.list.Swap(0, lastIndex)
heap.list.Remove(lastIndex)
heap.bubbleDown()
return
}
// Peek returns top element on the heap without removing it, or nil if heap is empty.
// Second return parameter is true, unless the heap was empty and there was nothing to peek.
func (heap *Heap) Peek() (value interface{}, ok bool) {
return heap.list.Get(0)
}
// Empty returns true if heap does not contain any elements.
func (heap *Heap) Empty() bool {
return heap.list.Empty()
}
// Size returns number of elements within the heap.
func (heap *Heap) Size() int {
return heap.list.Size()
}
// Clear removes all elements from the heap.
func (heap *Heap) Clear() {
heap.list.Clear()
}
// Values returns all elements in the heap.
func (heap *Heap) Values() []interface{} {
return heap.list.Values()
}
// String returns a string representation of container
func (heap *Heap) String() string {
str := "BinaryHeap\n"
values := []string{}
for _, value := range heap.list.Values() {
values = append(values, fmt.Sprintf("%v", value))
}
str += strings.Join(values, ", ")
return str
}
// Performs the "bubble down" operation. This is to place the element that is at the root
// of the heap in its correct place so that the heap maintains the min/max-heap order property.
func (heap *Heap) bubbleDown() {
heap.bubbleDownIndex(0)
}
// Performs the "bubble down" operation. This is to place the element that is at the index
// of the heap in its correct place so that the heap maintains the min/max-heap order property.
func (heap *Heap) bubbleDownIndex(index int) {
size := heap.list.Size()
for leftIndex := index<<1 + 1; leftIndex < size; leftIndex = index<<1 + 1 {
rightIndex := index<<1 + 2
smallerIndex := leftIndex
leftValue, _ := heap.list.Get(leftIndex)
rightValue, _ := heap.list.Get(rightIndex)
if rightIndex < size && heap.Comparator(leftValue, rightValue) > 0 {
smallerIndex = rightIndex
}
indexValue, _ := heap.list.Get(index)
smallerValue, _ := heap.list.Get(smallerIndex)
if heap.Comparator(indexValue, smallerValue) > 0 {
heap.list.Swap(index, smallerIndex)
} else {
break
}
index = smallerIndex
}
}
// Performs the "bubble up" operation. This is to place a newly inserted
// element (i.e. last element in the list) in its correct place so that
// the heap maintains the min/max-heap order property.
func (heap *Heap) bubbleUp() {
index := heap.list.Size() - 1
for parentIndex := (index - 1) >> 1; index > 0; parentIndex = (index - 1) >> 1 {
indexValue, _ := heap.list.Get(index)
parentValue, _ := heap.list.Get(parentIndex)
if heap.Comparator(parentValue, indexValue) <= 0 {
break
}
heap.list.Swap(index, parentIndex)
index = parentIndex
}
}
// Check that the index is within bounds of the list
func (heap *Heap) withinRange(index int) bool {
return index >= 0 && index < heap.list.Size()
}
vendor/github.com/emirpasic/gods/trees/binaryheap/iterator.go
Generated Vendored
+84
View File
@@ -0,0 +1,84 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package binaryheap
import "github.com/emirpasic/gods/containers"
func assertIteratorImplementation() {
var _ containers.ReverseIteratorWithIndex = (*Iterator)(nil)
}
// Iterator returns a stateful iterator whose values can be fetched by an index.
type Iterator struct {
heap *Heap
index int
}
// Iterator returns a stateful iterator whose values can be fetched by an index.
func (heap *Heap) Iterator() Iterator {
return Iterator{heap: heap, index: -1}
}
// Next moves the iterator to the next element and returns true if there was a next element in the container.
// If Next() returns true, then next element's index and value can be retrieved by Index() and Value().
// If Next() was called for the first time, then it will point the iterator to the first element if it exists.
// Modifies the state of the iterator.
func (iterator *Iterator) Next() bool {
if iterator.index < iterator.heap.Size() {
iterator.index++
}
return iterator.heap.withinRange(iterator.index)
}
// Prev moves the iterator to the previous element and returns true if there was a previous element in the container.
// If Prev() returns true, then previous element's index and value can be retrieved by Index() and Value().
// Modifies the state of the iterator.
func (iterator *Iterator) Prev() bool {
if iterator.index >= 0 {
iterator.index--
}
return iterator.heap.withinRange(iterator.index)
}
// Value returns the current element's value.
// Does not modify the state of the iterator.
func (iterator *Iterator) Value() interface{} {
value, _ := iterator.heap.list.Get(iterator.index)
return value
}
// Index returns the current element's index.
// Does not modify the state of the iterator.
func (iterator *Iterator) Index() int {
return iterator.index
}
// Begin resets the iterator to its initial state (one-before-first)
// Call Next() to fetch the first element if any.
func (iterator *Iterator) Begin() {
iterator.index = -1
}
// End moves the iterator past the last element (one-past-the-end).
// Call Prev() to fetch the last element if any.
func (iterator *Iterator) End() {
iterator.index = iterator.heap.Size()
}
// First moves the iterator to the first element and returns true if there was a first element in the container.
// If First() returns true, then first element's index and value can be retrieved by Index() and Value().
// Modifies the state of the iterator.
func (iterator *Iterator) First() bool {
iterator.Begin()
return iterator.Next()
}
// Last moves the iterator to the last element and returns true if there was a last element in the container.
// If Last() returns true, then last element's index and value can be retrieved by Index() and Value().
// Modifies the state of the iterator.
func (iterator *Iterator) Last() bool {
iterator.End()
return iterator.Prev()
}
vendor/github.com/emirpasic/gods/trees/binaryheap/serialization.go
Generated Vendored
+22
View File
@@ -0,0 +1,22 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package binaryheap
import "github.com/emirpasic/gods/containers"
func assertSerializationImplementation() {
var _ containers.JSONSerializer = (*Heap)(nil)
var _ containers.JSONDeserializer = (*Heap)(nil)
}
// ToJSON outputs the JSON representation of list's elements.
func (heap *Heap) ToJSON() ([]byte, error) {
return heap.list.ToJSON()
}
// FromJSON populates list's elements from the input JSON representation.
func (heap *Heap) FromJSON(data []byte) error {
return heap.list.FromJSON(data)
}
vendor/github.com/emirpasic/gods/trees/trees.go
Generated Vendored
+21
View File
@@ -0,0 +1,21 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package trees provides an abstract Tree interface.
//
// In computer science, a tree is a widely used abstract data type (ADT) or data structure implementing this ADT that simulates a hierarchical tree structure, with a root value and subtrees of children with a parent node, represented as a set of linked nodes.
//
// Reference: https://en.wikipedia.org/wiki/Tree_%28data_structure%29
package trees
import "github.com/emirpasic/gods/containers"
// Tree interface that all trees implement
type Tree interface {
containers.Container
// Empty() bool
// Size() int
// Clear()
// Values() []interface{}
}
vendor/github.com/emirpasic/gods/utils/comparator.go
Generated Vendored
+251
View File
@@ -0,0 +1,251 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package utils
import "time"
// Comparator will make type assertion (see IntComparator for example),
// which will panic if a or b are not of the asserted type.
//
// Should return a number:
// negative , if a < b
// zero , if a == b
// positive , if a > b
type Comparator func(a, b interface{}) int
// StringComparator provides a fast comparison on strings
func StringComparator(a, b interface{}) int {
s1 := a.(string)
s2 := b.(string)
min := len(s2)
if len(s1) < len(s2) {
min = len(s1)
}
diff := 0
for i := 0; i < min && diff == 0; i++ {
diff = int(s1[i]) - int(s2[i])
}
if diff == 0 {
diff = len(s1) - len(s2)
}
if diff < 0 {
return -1
}
if diff > 0 {
return 1
}
return 0
}
// IntComparator provides a basic comparison on int
func IntComparator(a, b interface{}) int {
aAsserted := a.(int)
bAsserted := b.(int)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// Int8Comparator provides a basic comparison on int8
func Int8Comparator(a, b interface{}) int {
aAsserted := a.(int8)
bAsserted := b.(int8)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// Int16Comparator provides a basic comparison on int16
func Int16Comparator(a, b interface{}) int {
aAsserted := a.(int16)
bAsserted := b.(int16)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// Int32Comparator provides a basic comparison on int32
func Int32Comparator(a, b interface{}) int {
aAsserted := a.(int32)
bAsserted := b.(int32)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// Int64Comparator provides a basic comparison on int64
func Int64Comparator(a, b interface{}) int {
aAsserted := a.(int64)
bAsserted := b.(int64)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// UIntComparator provides a basic comparison on uint
func UIntComparator(a, b interface{}) int {
aAsserted := a.(uint)
bAsserted := b.(uint)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// UInt8Comparator provides a basic comparison on uint8
func UInt8Comparator(a, b interface{}) int {
aAsserted := a.(uint8)
bAsserted := b.(uint8)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// UInt16Comparator provides a basic comparison on uint16
func UInt16Comparator(a, b interface{}) int {
aAsserted := a.(uint16)
bAsserted := b.(uint16)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// UInt32Comparator provides a basic comparison on uint32
func UInt32Comparator(a, b interface{}) int {
aAsserted := a.(uint32)
bAsserted := b.(uint32)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// UInt64Comparator provides a basic comparison on uint64
func UInt64Comparator(a, b interface{}) int {
aAsserted := a.(uint64)
bAsserted := b.(uint64)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// Float32Comparator provides a basic comparison on float32
func Float32Comparator(a, b interface{}) int {
aAsserted := a.(float32)
bAsserted := b.(float32)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// Float64Comparator provides a basic comparison on float64
func Float64Comparator(a, b interface{}) int {
aAsserted := a.(float64)
bAsserted := b.(float64)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// ByteComparator provides a basic comparison on byte
func ByteComparator(a, b interface{}) int {
aAsserted := a.(byte)
bAsserted := b.(byte)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// RuneComparator provides a basic comparison on rune
func RuneComparator(a, b interface{}) int {
aAsserted := a.(rune)
bAsserted := b.(rune)
switch {
case aAsserted > bAsserted:
return 1
case aAsserted < bAsserted:
return -1
default:
return 0
}
}
// TimeComparator provides a basic comparison on time.Time
func TimeComparator(a, b interface{}) int {
aAsserted := a.(time.Time)
bAsserted := b.(time.Time)
switch {
case aAsserted.After(bAsserted):
return 1
case aAsserted.Before(bAsserted):
return -1
default:
return 0
}
}
vendor/github.com/emirpasic/gods/utils/sort.go
Generated Vendored
+29
View File
@@ -0,0 +1,29 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package utils
import "sort"
// Sort sorts values (in-place) with respect to the given comparator.
//
// Uses Go's sort (hybrid of quicksort for large and then insertion sort for smaller slices).
func Sort(values []interface{}, comparator Comparator) {
sort.Sort(sortable{values, comparator})
}
type sortable struct {
values []interface{}
comparator Comparator
}
func (s sortable) Len() int {
return len(s.values)
}
func (s sortable) Swap(i, j int) {
s.values[i], s.values[j] = s.values[j], s.values[i]
}
func (s sortable) Less(i, j int) bool {
return s.comparator(s.values[i], s.values[j]) < 0
}
vendor/github.com/emirpasic/gods/utils/utils.go
Generated Vendored
+47
View File
@@ -0,0 +1,47 @@
// Copyright (c) 2015, Emir Pasic. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package utils provides common utility functions.
//
// Provided functionalities:
// - sorting
// - comparators
package utils
import (
"fmt"
"strconv"
)
// ToString converts a value to string.
func ToString(value interface{}) string {
switch value.(type) {
case string:
return value.(string)
case int8:
return strconv.FormatInt(int64(value.(int8)), 10)
case int16:
return strconv.FormatInt(int64(value.(int16)), 10)
case int32:
return strconv.FormatInt(int64(value.(int32)), 10)
case int64:
return strconv.FormatInt(int64(value.(int64)), 10)
case uint8:
return strconv.FormatUint(uint64(value.(uint8)), 10)
case uint16:
return strconv.FormatUint(uint64(value.(uint16)), 10)
case uint32:
return strconv.FormatUint(uint64(value.(uint32)), 10)
case uint64:
return strconv.FormatUint(uint64(value.(uint64)), 10)
case float32:
return strconv.FormatFloat(float64(value.(float32)), 'g', -1, 64)
case float64:
return strconv.FormatFloat(float64(value.(float64)), 'g', -1, 64)
case bool:
return strconv.FormatBool(value.(bool))
default:
return fmt.Sprintf("%+v", value)
}
}
vendor/github.com/google/go-containerregistry/cmd/ko/test/kodata/kenobi
Generated Vendored
-1
View File
@@ -1 +0,0 @@
../kenobi
vendor/github.com/google/go-containerregistry/pkg/authn/helper.go
Generated Vendored
+11 -5
View File
@@ -72,7 +72,7 @@ func (h *helper) Authorization() (string, error) {
var out bytes.Buffer
cmd.Stdout = &out
err := h.r.Run(cmd)
cmdErr := h.r.Run(cmd)
// If we see this specific message, it means the domain wasn't found
// and we should fall back on anonymous auth.
@@ -81,16 +81,22 @@ func (h *helper) Authorization() (string, error) {
return Anonymous.Authorization()
}
if err != nil {
return "", err
}
// Any other output should be parsed as JSON and the Username / Secret
// fields used for Basic authentication.
ho := helperOutput{}
if err := json.Unmarshal([]byte(output), &ho); err != nil {
if cmdErr != nil {
// If we failed to parse output, it won't contain Secret, so returning it
// in an error should be fine.
return "", fmt.Errorf("invoking %s: %v; output: %s", helperName, cmdErr, output)
}
return "", err
}
if cmdErr != nil {
return "", fmt.Errorf("invoking %s: %v", helperName, cmdErr)
}
b := Basic{Username: ho.Username, Password: ho.Secret}
return b.Authorization()
}
vendor/github.com/google/go-containerregistry/pkg/authn/k8schain/doc.go
Generated Vendored
+1 -1
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
// package k8schain exposes an implementation of the authn.Keychain interface
// Package k8schain exposes an implementation of the authn.Keychain interface
// based on the semantics the Kubelet follows when pulling the images for a
// Pod in Kubernetes.
package k8schain
vendor/github.com/google/go-containerregistry/pkg/authn/k8schain/k8schain.go
Generated Vendored
+1 -1
View File
@@ -17,7 +17,7 @@ package k8schain
import (
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/name"
"k8s.io/api/core/v1"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
vendor/github.com/google/go-containerregistry/pkg/authn/keychain.go
Generated Vendored
+4 -4
View File
@@ -19,11 +19,11 @@ import (
"errors"
"fmt"
"io/ioutil"
"log"
"os"
"path/filepath"
"runtime"
"github.com/google/go-containerregistry/pkg/logs"
"github.com/google/go-containerregistry/pkg/name"
)
@@ -100,19 +100,19 @@ var (
func (dk *defaultKeychain) Resolve(reg name.Registry) (Authenticator, error) {
dir, err := configDir()
if err != nil {
log.Printf("Unable to determine config dir: %v", err)
logs.Warn.Printf("Unable to determine config dir: %v", err)
return Anonymous, nil
}
file := filepath.Join(dir, "config.json")
content, err := ioutil.ReadFile(file)
if err != nil {
log.Printf("Unable to read %q: %v", file, err)
logs.Warn.Printf("Unable to read %q: %v", file, err)
return Anonymous, nil
}
var cf cfg
if err := json.Unmarshal(content, &cf); err != nil {
log.Printf("Unable to parse %q: %v", file, err)
logs.Warn.Printf("Unable to parse %q: %v", file, err)
return Anonymous, nil
}

Some files were not shown because too many files have changed in this diff Show More