diff --git a/bitnami/spark/4.1/debian-12/Dockerfile b/bitnami/spark/4.1/debian-12/Dockerfile index 02ddbc901f01..d011e1bf75ea 100644 --- a/bitnami/spark/4.1/debian-12/Dockerfile +++ b/bitnami/spark/4.1/debian-12/Dockerfile @@ -8,7 +8,7 @@ ARG JAVA_EXTRA_SECURITY_DIR="/bitnami/java/extra-security" ARG TARGETARCH LABEL org.opencontainers.image.base.name="docker.io/bitnami/minideb:bookworm" \ - org.opencontainers.image.created="2026-04-09T19:57:39Z" \ + org.opencontainers.image.created="2026-05-09T21:04:42Z" \ org.opencontainers.image.description="Application packaged by Broadcom, Inc." \ org.opencontainers.image.documentation="https://github.com/bitnami/containers/tree/main/bitnami/spark/README.md" \ org.opencontainers.image.source="https://github.com/bitnami/containers/tree/main/bitnami/spark" \ @@ -31,8 +31,8 @@ RUN --mount=type=secret,id=downloads_url,env=SECRET_DOWNLOADS_URL \ mkdir -p /tmp/bitnami/pkg/cache/ ; cd /tmp/bitnami/pkg/cache/ || exit 1 ; \ COMPONENTS=( \ "nss-wrapper-1.1.16-0-linux-${OS_ARCH}-debian-12" \ - "python-3.12.13-4-linux-${OS_ARCH}-debian-12" \ - "jre-17.0.18-10-1-linux-${OS_ARCH}-debian-12" \ + "python-3.12.13-9-linux-${OS_ARCH}-debian-12" \ + "jre-17.0.19-11-1-linux-${OS_ARCH}-debian-12" \ "spark-4.1.1-4-linux-${OS_ARCH}-debian-12" \ ) ; \ for COMPONENT in "${COMPONENTS[@]}"; do \ @@ -56,7 +56,7 @@ RUN /opt/bitnami/scripts/spark/postunpack.sh RUN /opt/bitnami/scripts/java/postunpack.sh ENV APP_VERSION="4.1.1" \ BITNAMI_APP_NAME="spark" \ - IMAGE_REVISION="11" \ + IMAGE_REVISION="12" \ JAVA_HOME="/opt/bitnami/java" \ LD_LIBRARY_PATH="/opt/bitnami/python/lib:/opt/bitnami/spark/venv/lib/python3.12/site-packages/numpy.libs:$LD_LIBRARY_PATH" \ LIBNSS_WRAPPER_PATH="/opt/bitnami/common/lib/libnss_wrapper.so" \ diff --git a/bitnami/spark/README.md b/bitnami/spark/README.md index f7259332d0dc..221d59ca5a95 100644 --- a/bitnami/spark/README.md +++ b/bitnami/spark/README.md @@ -7,13 +7,17 @@ Trademarks: This software listing is packaged by Bitnami. The respective tradema ## TL;DR -### Docker Compose - ```console docker run --name spark bitnami/spark:latest ``` -You can find the available configuration options in the [Environment Variables](#environment-variables) section. +## Using `docker-compose.yml` + +The docker-compose.yaml file of this container can be found in the [Bitnami Containers repository](https://github.com/bitnami/containers/). + +[https://github.com/bitnami/containers/tree/main/bitnami/spark/docker-compose.yml](https://github.com/bitnami/containers/tree/main/bitnami/spark/docker-compose.yml) + +Please be aware this file has not undergone internal testing. Consequently, we advise its use exclusively for development or testing purposes. For production-ready deployments, we highly recommend utilizing its associated [Bitnami Helm chart](https://github.com/bitnami/charts/tree/main/bitnami/spark). ## Why use Bitnami Secure Images? @@ -48,10 +52,6 @@ Learn more about the Bitnami tagging policy and the difference between rolling t The Bitnami Apache Spark Docker image is only available to [Bitnami Secure Images](https://bitnami.com) customers. -## Using `docker-compose.yaml` - -Please be aware this file has not undergone internal testing. Consequently, we advise its use exclusively for development or testing purposes. For production-ready deployments, we highly recommend utilizing its associated [Bitnami Helm chart](https://github.com/bitnami/charts/tree/main/bitnami/spark). - ## Configuration The following sections describe environment variables and related settings. @@ -103,8 +103,6 @@ Additionally, more environment variables natively supported by Apache Spark can For example, you could still use `SPARK_WORKER_CORES` or `SPARK_WORKER_MEMORY` to configure the number of cores and the amount of memory to be used by a worker machine. -When you start the spark image, you can adjust the configuration of the instance by passing one or more environment variables either on the docker-compose file or on the `docker run` command line. - ### Security The Bitnami Apache Spark docker image supports enabling RPC authentication, RPC encryption and local storage encryption easily using the following environment variables in all the nodes of the cluster. @@ -136,18 +134,6 @@ Additionally, SSL configuration can be easily activated following the next steps 2. You need to mount your spark `keystore` and `truststore` files to `/opt/bitnami/spark/conf/certs`. Please note they should be called `spark-keystore.jks` and `spark-truststore.jks` and they should be in JKS format. -### Setting up an Apache Spark cluster - -A Apache Spark cluster can easily be set up with the default docker-compose.yml file from the root of this repository. The docker-compose includes two different services, `spark-master` and `spark-worker.` - -By default, when you deploy the docker-compose file you will get an Apache Spark cluster with 1 master and 1 worker. - -If you want N workers, all you need to do is start the docker-compose deployment with the following command: - -```console -docker-compose up --scale spark-worker=3 -``` - ### Mount a custom configuration file The image looks for configuration in the `conf/` directory of `/opt/bitnami/spark`. @@ -197,74 +183,7 @@ The Bitnami Apache Spark Docker image from the [Bitnami Secure Images](https://g ## Logging -The Bitnami Apache Spark Docker image sends the container logs to the `stdout`. To view the logs: - -```console -docker logs spark -``` - -or using Docker Compose: - -```console -docker-compose logs spark -``` - -You can configure the containers [logging driver](https://docs.docker.com/engine/admin/logging/overview/) using the `--log-driver` option if you wish to consume the container logs differently. In the default configuration docker uses the `json-file` driver. - -## Maintenance - -The following sections describe how to back up, restore, and upgrade the image. - -### Backing up your container - -To backup your data, configuration and logs, follow these simple steps: - -1. Stop the currently running container - - ```console - docker stop spark - ``` - - or using Docker Compose: - - ```console - docker-compose stop spark - ``` - -2. Run the backup command - - We need to mount two volumes in a container we will use to create the backup: a directory on your host to store the backup in, and the volumes from the container we just stopped so we can access the data. - - ```console - docker run --rm -v /path/to/spark-backups:/backups --volumes-from spark busybox \ - cp -a /bitnami/spark /backups/latest - ``` - - or using Docker Compose: - - ```console - docker run --rm -v /path/to/spark-backups:/backups --volumes-from `docker-compose ps -q spark` busybox \ - cp -a /bitnami/spark /backups/latest - ``` - -### Restoring your backup - -To restore your backup, mount the backup as volumes in the container. - -```console -docker run -v /path/to/spark-backups/latest:/bitnami/spark bitnami/spark:latest -``` - -or by modifying the [`docker-compose.yml`](https://github.com/bitnami/containers/blob/main/bitnami/spark/docker-compose.yml) file present in this repository: - -```yaml -services: - spark: - ... - volumes: - - /path/to/spark-backups/latest:/bitnami/spark - ... -``` +The Bitnami Apache Spark Docker image sends the container logs to the `stdout`. You can configure the containers [logging driver](https://docs.docker.com/engine/admin/logging/overview/) using the `--log-driver` option if you wish to consume the container logs differently. In the default configuration docker uses the `json-file` driver. ## Notable changes