Commit Graph

16 Commits

Author SHA1 Message Date
Joël Pepper 14ea7c4071
Fix Integration tests (#2425)
* fix(ci): Bump golangci-lint to 1.51.1

* chore(lint): fix gofmt and goimport issues

* chore(lint): fix linter issues

- Adapted error comparison according to linter recommendation
- Disabled noctx linting for http request where canceling makes no sense
- Disabled nilerror linting where nil error is returned on purpose
- Disabled makezero linter where slice is explicitly deepcopied

* chore(ci): Update go version in tests workflows

* fix(ci): Allow boilerplate years from 2000-2099

Previously the regex only allowed the copyright notice to contain the
years 2018,2019,2020,2021, or 2022. This commit widens to regex to
20\d\d allowing any year in the range [2000-2099]

* feat(ci): Replace minikube with k3s for intregration tests

The existing setup for minikube is very complicated, replicating most of
the setup steps for a full kubernetes cluster in an only partially
supported minikube configuration (driver=none). Furthermore the existing
setup has been broken for sometime, likely, at least in part due to the
changes to CNI and CRI in recent kubernetes versions.

Since what we actually need is only a running Kubernetes cluster on the
node and access to a registry on localhost:5000, we can switch the
extremely complicated minikube setup for a lightweight cluster using
k3s. Minikube came with a default addon for running a registry on every
node, but the same is not the case for k3s, instead we make use of the
package helm controller and its HelmChart CR to deploy twuni/docker-registry.helm
and expose it on localhost using the integrated LoadBalancer controller.

* fix(test-684): pin base container version

The dockerfile for the regression test connected to issue 684 used a
rolling tag as base image, making it flaky and fail since it was
introduced.

This commit pins the base image to the digest of bionic-20200219, which,
based on the date of the commit that introduced to the dockerfile would
be the most newest ubuntu build and likely what the "rolling" tag
resolved to back then. Since this also an image from the pre-oci days of
ubuntu, this circumvents a bug in container-diff as well
(https://github.com/GoogleContainerTools/container-diff/issues/389)
2023-03-21 12:30:54 -04:00
Lukas aad03dc285
fix: getUIDandGID is able to resolve non-existing users and groups (#2106)
* fix: getUIDandGID is able to resolve non-existing users and groups

A common pattern in dockerfiles is to provide a plain uid and gid number, which doesn't neccesarily exist inside the os.

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* test: add chown dockerfile

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* chore: format

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* chore: add comment

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* tests: fix chown dockerfile

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* refactor: split up getIdsFromUsernameAndGroup func

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix: implement raw uid logic for LookupUser

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* test: add dockerfiles for integration test

* fix: lookup user error message

* test: add dockerfiles for non-existing user testcase

* fix: forgot error check

* tests: fix syscall credentials test

* chore: add debug output for copy command

* tests: set specific gid for integration dockerfile

* tests: fix syscall credentials test

github runner had the exact uid that i was testing on, so the groups were not empty

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* tests: fix test script

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* chore: apply golangci lint checks

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix: reset file ownership in createFile if not root owned

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* chore: logrus.Debugf missed format variable

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* chore(test-script): remove go html coverage

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* test(k8s): increase wait timeout

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
2022-07-12 10:21:37 -04:00
Lukas 8710ce3311
fix(Dockerfile): use temporary busybox mount to create /kaniko directory (#2155)
* ci(minikube): always install latest binary and run as root

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* chore: remove docker unmask because minikube handles this

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* debug: view run folder and user ids

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* debug: log minikube if start fails

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* debug: start cri-docker before starting minikube cluster

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* Revert "debug: start cri-docker before starting minikube cluster"

This reverts commit 8923941358.

* debug: include verbose minikube output

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix(minikube): cri-dockerd move command

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* chore(minikube): copy minikube profile from root to user

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix(minikube): change dirs to $HOME directory in config.json

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix(minikube): enable registry on start

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* chore: run as non root

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* cleanup

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* docs(minikube-script): update

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* deploy: create kaniko.tar

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* deploy: create kaniko folder in dockerfiles with 777 permissions via mount

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* chore(Dockerfile): remove ARG

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* docs(Dockerfile): add comment to kaniko dir creation

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix(Dockerfile): mount whole busybox image

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix(Dockerfile): use musl busybox

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* chore(Dockerfile): use musl busybox in debug image

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* typo

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
2022-07-08 09:34:31 -04:00
Lukas 4338275220
ci(setup-minikube): use cri-dockerd (#2149)
* fix(setup-minikube): use cri-dockerd

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix: add fix-broken flag to apt

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix: use focal deb file

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* chore: rerun apt-get update before installing

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix: install containerd with apt

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix: install runc with apt

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* debug: systemd docker service masked

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix: unmask docker service

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix: use binary and copy systemd unit files from repo

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix: remove port-forward of registry

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>
2022-06-27 09:50:16 -04:00
Lukas 679c71c907
refactor: simpler local integration tests (#2110)
* tests(integration): add fake gcs server and use sdk instead of gsutil

* tests(integration): add flag to run tests for a subset of dockerfiles

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* tests(integration): conditionally create gcs client

* refactor: create package for gcs bucket interaction

* tests(integration): use util.Tar for integration tarball creation

* refactor: create TarballOfDirectory func

* chore: add dockerignore for faster builds

* docs: add docs for dockerfile pattern

* fix: issue during personal review

* chore: cleanup

Signed-off-by: Höhl, Lukas <lukas.hoehl@accso.de>

* fix(integration-tests): remove default bucket

Signed-off-by: Lukas Hoehl <ludi.origin@gmail.com>
2022-06-14 13:38:01 -04:00
Jason Hall 59c2d2d08f
Collapse integration test workflows into one config (#1855)
* Collapse integration test workflows into one config

Remove last traces of Travis, including env vars and `make travis-setup`
and badges in README

* fix lint

* make integration tests pass on PRs
2021-12-22 20:04:40 -05:00
Jason Hall b1b6962726
Share the Go build cache when building in Dockerfiles (#1853)
On my machine this makes image rebuilds go from ~5m to 1.5s.

This also required setting DOCKER_BUILDKIT=1 in integration test image
builds.

I also took the opportunity to bump the Go version used in those tests
to Go 1.17 to match the image build processes, and tidied up the
workflow files a bit too (renaming for consistency, typos, whitespace,
etc.)
2021-12-22 12:43:12 -05:00
Matt Moore 49f0154b3f
Fix the e2e K8s test (#1842)
* Debugging the integration testing.

It seems to be consistently failing (also fix a typo in the file/leg name).

I'm disabling `-v` for `go test` because it interleaves the test output in ways that make it painful to read a single failing test's output when `t.Parallel` is involved.

* Try swapping registry:2 for the registry add-on

* Drop logf and make Logger private
2021-12-17 14:43:38 -05:00
wwade 82fc94d930
git: accept explicit commit hash for git context (#1765)
* git: accept explicit commit hash for git context

When checking out code from non-github repositories, the typical
assumptions may not be valid, e.g. that the only interesting
non-branch commits have ref names starting with refs/pull. A specific
example is fetching an un-merged commit from a gerrit repository by
commit hash.

This change just looks at the second part of the git context path and
checks if it's a SHA commit hash, and if so, will fetch and check out
this commit after cloning the repository.

Sample context argument:

    https://github.repo/project#e1772f228e06d15facdf175e5385e265b57068c0

* ci: fix test script to recognize any non-zero exit as an error

hack/linter.sh didn't properly install golangci-lint in hack/bin as I
already have another version of golangci-lint on my PATH, but then it
failed to execute because it was looking for it specifically in
hack/bin.

When the executable is not found, the exit code is 127 instead of 1,
and so test.sh ignored the error.

Two fixes:

1. `test.sh`:
  - Use `if (script) ...` instead of assigning / checking a result
    variable to determine if each validation script passed or failed.

2. `hack/linter.sh`:
  - Instead of checking for golangci-lint on the path, just
    specifically check for an executable file (`test -x`) in the
    expected location.

Co-authored-by: Wade Carpenter <wwade@users.noreply.github.com>
2021-10-19 19:27:15 -07:00
Tejal Desai 1da17b6e9b
chore: add workflows for pr tests (#1766)
* chore: add workflows for pr tests

* fix unit tests

* fix formatting

* chore: fix gobuild

* change minikube script

* chore: fix lint install script

* chore: ignore and fix tests

* fix lint and run gofmt

* lint fixes

* k8s executor image only

* fix Makefile

* fix travis env variables

* more info on k8s tests

* fix travis run

* fix

* fix

* fix

* fix log

* some more changes

* increase timeout

* delete travis.yml and fix multiple copy tests

* fix registry mirror

* fix lint

* add concurency

* last attemot to fix k8 integrations

* diff id for diff workflows
2021-10-18 23:22:44 -07:00
Tejal Desai ffd35dbe12
Mutli-arch support (#1531)
* initial commit

* remove bazel jobs

* fix arch

* more fixes after testing and code review comments

* fix build platform

* add individual cloudbuild.yaml as its taking 45 mins for a cloud build trigger

* add buildx plugin

* add more debugging

* update busybox version to fix CVE-2018-1000500

* fix

* lint + more debug

* fix

* fix

* fix

* remove images from cloudbiuld

* move CI job back to docker

* one more fix

* lets see

* bring it back

* move CI job back to docker

* remove aerg from top

* live restart config

* remove live restore as minikube setup failed

* add --force-systemd

* add --force-systemd and docker driver none

* change the --run flag

* docker info and some logs removed

* fix docker command

* upgrade version for buildx to 0.5.1

* remove docker service from travis.yml and add systemd cgroup config

* move the docker config up

* move them back to docker build

* fix

* fix all dockerfiles

* fix warmer

* fix

* rm bazel jobs

* add more logs

* fix debug

Co-authored-by: ankitm123 <ankitmohapatra123@gmail.com>
2021-01-25 17:49:19 -08:00
Tejal Desai f8bf02ae3c rm benchmark 2020-06-06 20:43:35 -07:00
Tejal Desai 5d013626fc benchmark project added 2020-05-25 21:30:07 -07:00
Tejal Desai 54c2a7abf0
Add conntrack to minikube setup script. 2020-03-27 10:03:37 -07:00
Sam Stoelinga 882793b755 Add K8s integration tests
- Use minikube for deploying a lightweight K8s on Travis CI
- Build and push Kaniko image to local docker registry
- Build dockerfiles with context on both docker and K8s and check
  the diff between the 2.
2020-03-11 11:49:22 -07:00
Sam Stoelinga cdc8c60c52 Move testing scripts to scripts directory
Since we recently started adding more scripts it makes sense to move to
a separate directory. This aligns with a common practice across golang
projects: https://github.com/golang-standards/project-layout
2020-03-07 13:59:57 -08:00