Kind 0.9.0 accesses the "bazel-out" directory with a relative path,
which only works when the command is invoked inside the Kubernetes
source code
directory (https://github.com/kubernetes-sigs/kind/issues/1910).
Commit 7bcee13d79 added alpha feature gates for Kubernetes 1.19 in
the CSI_PROW_E2E_ALPHA_GATES_LATEST variable based on the comment in
https://github.com/kubernetes-csi/external-provisioner/pull/493#discussion_r502663402
that alpha testing only runs for the latest Kubernetes.
But some components (like external-health-monitor) are configured with
a single Prow job which runs the default set of tests on a stable
Kubernetes release (currently 1.17). Those tests used to include alpha
testing which then broke during Kind cluster startup because the
Kubernetes 1.19 feature gates weren't recognized by 1.17.
The solution is to disable alpha testing for Kubernetes != latest in
the default set of tests.
kind 0.9 adds support for recent Kubernetes releases like 1.19 and
simplifies configuration of feature gates and runtime config.
With Kubernetes 1.19, new feature gates were introduced which might
become relevant for the alpha Prow jobs.
The updated kind release comes with images for different Kubernetes
releases than the one before. To avoid breaking existing jobs,
the script now picks kind images automatically. As an additional
bonus, it then uses images with hash, i.e. it's less likely to break
if those image tags change because of a future kind release.
Bazel makes sense in the Prow jobs because those often get invoked
with a pre-populated Bazel cache. But local invocation don't need it
and now can turn it off with CSI_PROW_USE_BAZEL=false.
It used to be necessary to override from where the E2E suite came on a
case-by-case basis (initially, testing was using a more recent suite
against an older Kubernetes). This should never become necessary again
and the lack of a specific entry for 1.18 already had the unintended
effect that Kubernetes 1.18 was tested with the suite from master, so
overall it is better to always use the E2E suite which matches
Kubernetes.
Kubernetes 1.19.0 uses Go 1.15, but refers to it as 1.15.0. This broke
both the check whether we need to install 1.15 (because "go version"
reports 1.15, which didn't match 1.15.0) and then downloading the
release archive (because the URL also only uses 1.15).
Go 1.15 was released and is the major version that Kubernetes 1.19.0
is going to use. There are probably bugs in the older 1.13.3 that were
fixed, so we should update.
If the Dockerfile needs to run some command, that step fails unless
QEMU is set up properly first:
failed to solve: rpc error: code = Unknown desc = failed to load
LLB: runtime execution on platform linux/ppc64le not supported
Most repos inherit the default BUILD_PLATFORMS, which includes
Windows, but don't have the necessary Dockerfile.Windows yet. To
simplify the rollout of multiarch image builds, Windows binary
building continues to be tested (i.e. BUILD_PLATFORMS remains
unchanged), but push-multiarch skips Windows if the Dockerfile.Windows
is missing.
"make push-multiarch" matched both push-multiarch and push-%. This
seems to be none-deterministic and in at least one
repo (external-provisioner), make picked the wildcard rule which then
failed because there is no "multiarch" command.
This ambiguity gets resolved by instantiating the wildcard rules only
for existing commands. The advantage also is that "make
push-no-such-command" will fail with an obvious "No rule to make
target 'push-no-such-command'" instead of attempting to build the
command.
The approach taken here extends the existing support for
cross-compiling binaries on the build host and specifying the Go
compiler: Go is installed if needed (as in Prow testing), binaries are
build on the host, then one image is created for each platform, and
finally those are combined into a single multi-architecture image.
Kubernetes Prow Robot
xing-yang
Patrick Ohly
Abhisek Banerjee
Michelle Au
namrata-ibm
Animesh Kumar
Yibo Cai