From 2998f54cac21cf7429db90ef1147dae9cef8ae4e Mon Sep 17 00:00:00 2001 From: Bitnami Bot Date: Mon, 22 Jan 2024 16:03:14 +0100 Subject: [PATCH] [bitnami/airflow] Release 2.8.0-debian-11-r3 (#55273) Signed-off-by: Bitnami Containers --- bitnami/airflow/2/debian-11/Dockerfile | 5 +- bitnami/airflow/README.md | 139 +++++++++++-------------- 2 files changed, 65 insertions(+), 79 deletions(-) diff --git a/bitnami/airflow/2/debian-11/Dockerfile b/bitnami/airflow/2/debian-11/Dockerfile index ffd734dc9950..90fbcd6a68ff 100644 --- a/bitnami/airflow/2/debian-11/Dockerfile +++ b/bitnami/airflow/2/debian-11/Dockerfile @@ -7,10 +7,10 @@ ARG TARGETARCH LABEL com.vmware.cp.artifact.flavor="sha256:1e1b4657a77f0d47e9220f0c37b9bf7802581b93214fff7d1bd2364c8bf22e8e" \ org.opencontainers.image.base.name="docker.io/bitnami/minideb:bullseye" \ - org.opencontainers.image.created="2024-01-13T11:25:39Z" \ + org.opencontainers.image.created="2024-01-22T13:21:07Z" \ org.opencontainers.image.description="Application packaged by VMware, Inc" \ org.opencontainers.image.licenses="Apache-2.0" \ - org.opencontainers.image.ref.name="2.8.0-debian-11-r2" \ + org.opencontainers.image.ref.name="2.8.0-debian-11-r3" \ org.opencontainers.image.title="airflow" \ org.opencontainers.image.vendor="VMware, Inc." \ org.opencontainers.image.version="2.8.0" @@ -46,6 +46,7 @@ RUN apt-get autoremove --purge -y curl && \ apt-get clean && rm -rf /var/lib/apt/lists /var/cache/apt/archives RUN chmod g+rwX /opt/bitnami RUN localedef -c -f UTF-8 -i en_US en_US.UTF-8 +RUN find / -perm /6000 -type f -exec chmod a-s {} \; || true RUN update-locale LANG=C.UTF-8 LC_MESSAGES=POSIX && \ DEBIAN_FRONTEND=noninteractive dpkg-reconfigure locales RUN echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen && locale-gen diff --git a/bitnami/airflow/README.md b/bitnami/airflow/README.md index e94f0d23555f..edd0f89f18c3 100644 --- a/bitnami/airflow/README.md +++ b/bitnami/airflow/README.md @@ -9,14 +9,12 @@ Trademarks: This software listing is packaged by Bitnami. The respective tradema ## TL;DR -### Docker Compose - ```console -curl -LO https://raw.githubusercontent.com/bitnami/containers/main/bitnami/airflow/docker-compose.yml -docker-compose up +docker run --name airflow bitnami/airflow:latest ``` -**Warning**: This quick setup is only intended for development environments. You are encouraged to change the insecure default credentials and check out the available configuration options in the [Environment Variables](#environment-variables) section for a more secure deployment. +**Warning**: This quick setup is only intended for development environments. You are encouraged to change the insecure default credentials and check out the available configuration options in the [Environment Variables](#environment-variables) section for a more secure d +eployment. ## Why use Bitnami Images? @@ -45,19 +43,8 @@ To run this application you need [Docker Engine](https://www.docker.com/products Airflow requires access to a PostgreSQL database to store information. We will use our very own [PostgreSQL image](https://github.com/bitnami/containers/tree/main/bitnami/postgresql) for the database requirements. Additionally, if you pretend to use the `CeleryExecutor`, you will also need an [Airflow Scheduler](https://github.com/bitnami/containers/tree/main/bitnami/airflow-scheduler), one or more [Airflow Workers](https://github.com/bitnami/containers/tree/main/bitnami/airflow-worker) and a [Redis(R) server](https://github.com/bitnami/containers/tree/main/bitnami/redis). -### Using Docker Compose - -The main folder of this repository contains a functional [`docker-compose.yml`](https://github.com/bitnami/containers/blob/main/bitnami/airflow/docker-compose.yml) file. Run the application using it as shown below: - -```console -curl -sSL https://raw.githubusercontent.com/bitnami/containers/main/bitnami/airflow/docker-compose.yml > docker-compose.yml -docker-compose up -d -``` - ### Using the Docker Command Line -If you want to run the application manually instead of using `docker-compose`, these are the basic steps you need to run: - 1. Create a network ```console @@ -139,6 +126,17 @@ If you want to run the application manually instead of using `docker-compose`, t Access your application at `http://your-ip:8080` +### Using `docker-compose.yaml` + +```console +curl -LO https://raw.githubusercontent.com/bitnami/containers/main/bitnami/airflow/docker-compose.yml +docker-compose up +``` + +Please be aware this file has not undergone internal testing. Consequently, we advise its use exclusively for development or testing purposes. For production-ready deployments, we highly recommend utilizing its associated [Bitnami Helm chart](https://github.com/bitnami/charts/tree/main/bitnami/airflow). + +If you detect any issue in the `docker-compose.yaml` file, feel free to report it or contribute with a fix by following our [Contributing Guidelines](https://github.com/bitnami/containers/blob/main/CONTRIBUTING.md). + ### Persisting your application The Bitnami Airflow container relies on the PostgreSQL database & Redis to persist the data. This means that Airflow does not persist anything. To avoid loss of data, you should mount volumes for persistence of [PostgreSQL data](https://github.com/bitnami/containers/blob/main/bitnami/mariadb#persisting-your-database) and [Redis(R) data](https://github.com/bitnami/containers/blob/main/bitnami/redis#persisting-your-database) @@ -295,68 +293,55 @@ This container supports the installation of additional python modules at start-u ### Environment variables -| Name | Description | Default Value | Can be set | -|----------------------------------------|-----------------------------------------------------------------------|---------------------------------------------|------------| -| `$AIRFLOW_BASE_DIR` | Airflow installation directory. | `${BITNAMI_ROOT_DIR}/airflow` | | -| `$AIRFLOW_HOME` | Airflow home directory. | `${AIRFLOW_BASE_DIR}` | | -| `$AIRFLOW_BIN_DIR` | Airflow directory for binary executables. | `${AIRFLOW_BASE_DIR}/venv/bin` | | -| `$AIRFLOW_LOGS_DIR` | Airflow logs directory. | `${AIRFLOW_BASE_DIR}/logs` | | -| `$AIRFLOW_SCHEDULER_LOGS_DIR` | Airflow scheduler logs directory. | `${AIRFLOW_LOGS_DIR}/scheduler` | | -| `$AIRFLOW_LOG_FILE` | Airflow log file. | `${AIRFLOW_LOGS_DIR}/airflow-webserver.log` | | -| `$AIRFLOW_CONF_FILE` | Airflow configuration file. | `${AIRFLOW_BASE_DIR}/airflow.cfg` | | -| `$AIRFLOW_WEBSERVER_CONF_FILE` | Airflow configuration file. | `${AIRFLOW_BASE_DIR}/webserver_config.py` | | -| `$AIRFLOW_TMP_DIR` | Airflow directory temporary files. | `${AIRFLOW_BASE_DIR}/tmp` | | -| `$AIRFLOW_PID_FILE` | Path to the Airflow PID file. | `${AIRFLOW_TMP_DIR}/airflow-webserver.pid` | | -| `$AIRFLOW_DAGS_DIR` | Airflow data to be persisted. | `${AIRFLOW_BASE_DIR}/dags` | | -| `$AIRFLOW_DAEMON_USER` | Airflow system user. | `airflow` | | -| `$AIRFLOW_DAEMON_GROUP` | Airflow system group. | `airflow` | | -| `$AIRFLOW_USERNAME` | Airflow username | `user` | ✓ | -| `$AIRFLOW_PASSWORD` | Airflow password | `bitnami` | ✓ | -| `$AIRFLOW_FIRSTNAME` | Airflow firstname | `Firstname` | ✓ | -| `$AIRFLOW_LASTNAME` | Airflow lastname | `Lastname` | ✓ | -| `$AIRFLOW_EMAIL` | Airflow email | `user@example.com` | ✓ | -| `$AIRFLOW_EXECUTOR` | Airflow executor. | `SequentialExecutor` | ✓ | -| `$AIRFLOW_EXECUTOR` | Airflow executor. | `CeleryExecutor` | ✓ | -| `$AIRFLOW_RAW_FERNET_KEY` | Airflow raw/unencoded Fernet key | | ✓ | -| `$AIRFLOW_FERNET_KEY` | Airflow Fernet key | | ✓ | -| `$AIRFLOW_SECRET_KEY` | Airflow Secret key | | ✓ | -| `$AIRFLOW_WEBSERVER_HOST` | Airflow webserver host | `127.0.0.1` | ✓ | -| `$AIRFLOW_WEBSERVER_PORT_NUMBER` | Airflow webserver port. | `8080` | ✓ | -| `$AIRFLOW_LOAD_EXAMPLES` | To load example tasks into the application. | `yes` | ✓ | -| `$AIRFLOW_BASE_URL` | Airflow webserver base URL. | | ✓ | -| `$AIRFLOW_HOSTNAME_CALLABLE` | Method to obtain the hostname. | | ✓ | -| `$AIRFLOW_HOSTNAME_CALLABLE` | Method to obtain the hostname. | `socket.gethostname` | ✓ | -| `$AIRFLOW_POOL_NAME` | Pool name. | | ✓ | -| `$AIRFLOW_POOL_SIZE` | Pool size, required with AIRFLOW_POOL_NAME. | | ✓ | -| `$AIRFLOW_POOL_DESC` | Pool description, required with AIRFLOW_POOL_NAME. | | ✓ | -| `$AIRFLOW_DATABASE_HOST` | Hostname for PostgreSQL server. | `postgresql` | ✓ | -| `$AIRFLOW_DATABASE_HOST` | Hostname for PostgreSQL server. | `127.0.0.1` | ✓ | -| `$AIRFLOW_DATABASE_PORT_NUMBER` | Port used by PostgreSQL server. | `5432` | ✓ | -| `$AIRFLOW_DATABASE_NAME` | Database name that Airflow will use to connect with the database. | `bitnami_airflow` | ✓ | -| `$AIRFLOW_DATABASE_USERNAME` | Database user that Airflow will use to connect with the database. | `bn_airflow` | ✓ | -| `$AIRFLOW_DATABASE_PASSWORD` | Database password that Airflow will use to connect with the database. | | ✓ | -| `$AIRFLOW_DATABASE_USE_SSL` | Set to yes if the database is using SSL. | `no` | ✓ | -| `$AIRFLOW_REDIS_USE_SSL` | Set to yes if Redis(R) uses SSL. | `no` | ✓ | -| `$REDIS_HOST` | Hostname for Redis(R) server. | `redis` | ✓ | -| `$REDIS_HOST` | Hostname for Redis(R) server. | `127.0.0.1` | ✓ | -| `$REDIS_PORT_NUMBER` | Port used by Redis(R) server. | `6379` | ✓ | -| `$REDIS_USER` | User that Airflow will use to connect with Redis(R). | | ✓ | -| `$REDIS_PASSWORD` | Password that Airflow will use to connect with Redis(R). | | ✓ | -| `$REDIS_DATABASE` | Name of the Redis(R) database. | `1` | ✓ | -| `$AIRFLOW_LDAP_ENABLE` | Enable LDAP authentication. | `no` | ✓ | -| `$AIRFLOW_LDAP_URI` | LDAP server URI. | | ✓ | -| `$AIRFLOW_LDAP_SEARCH` | LDAP search base. | | ✓ | -| `$AIRFLOW_LDAP_UID_FIELD` | LDAP field used for uid. | | ✓ | -| `$AIRFLOW_LDAP_BIND_USER` | LDAP user name. | | ✓ | -| `$AIRFLOW_LDAP_BIND_PASSWORD` | LDAP user password. | | ✓ | -| `$AIRFLOW_LDAP_USER_REGISTRATION` | User self registration. | `True` | ✓ | -| `$AIRFLOW_LDAP_USER_REGISTRATION_ROLE` | Role name to be assign when a user registers himself. | | ✓ | -| `$AIRFLOW_LDAP_ROLES_MAPPING` | Mapping from LDAP DN to a list of Airflow roles. | | ✓ | -| `$AIRFLOW_LDAP_ROLES_SYNC_AT_LOGIN` | Replace ALL the user roles each login, or only on registration. | `True` | ✓ | -| `$AIRFLOW_LDAP_USE_TLS` | Use LDAP SSL. | `False` | ✓ | -| `$AIRFLOW_LDAP_ALLOW_SELF_SIGNED` | Allow self signed certicates in LDAP ssl. | `True` | ✓ | -| `$AIRFLOW_LDAP_TLS_CA_CERTIFICATE` | File that store the CA for LDAP ssl. | | ✓ | +#### Customizable environment variables +| Name | Description | Default Value | +|------------------------------------|-------------------------------------------------------------------|----------------------| +| `AIRFLOW_USERNAME` | Airflow username | `user` | +| `AIRFLOW_PASSWORD` | Airflow password | `bitnami` | +| `AIRFLOW_FIRSTNAME` | Airflow firstname | `Firstname` | +| `AIRFLOW_LASTNAME` | Airflow lastname | `Lastname` | +| `AIRFLOW_EMAIL` | Airflow email | `user@example.com` | +| `AIRFLOW_EXECUTOR` | Airflow executor. | `SequentialExecutor` | +| `AIRFLOW_EXECUTOR` | Airflow executor. | `CeleryExecutor` | +| `AIRFLOW_WEBSERVER_HOST` | Airflow webserver host | `127.0.0.1` | +| `AIRFLOW_WEBSERVER_PORT_NUMBER` | Airflow webserver port. | `8080` | +| `AIRFLOW_LOAD_EXAMPLES` | To load example tasks into the application. | `yes` | +| `AIRFLOW_HOSTNAME_CALLABLE` | Method to obtain the hostname. | `socket.gethostname` | +| `AIRFLOW_DATABASE_HOST` | Hostname for PostgreSQL server. | `postgresql` | +| `AIRFLOW_DATABASE_HOST` | Hostname for PostgreSQL server. | `127.0.0.1` | +| `AIRFLOW_DATABASE_PORT_NUMBER` | Port used by PostgreSQL server. | `5432` | +| `AIRFLOW_DATABASE_NAME` | Database name that Airflow will use to connect with the database. | `bitnami_airflow` | +| `AIRFLOW_DATABASE_USERNAME` | Database user that Airflow will use to connect with the database. | `bn_airflow` | +| `AIRFLOW_DATABASE_USE_SSL` | Set to yes if the database is using SSL. | `no` | +| `AIRFLOW_REDIS_USE_SSL` | Set to yes if Redis(R) uses SSL. | `no` | +| `REDIS_HOST` | Hostname for Redis(R) server. | `redis` | +| `REDIS_HOST` | Hostname for Redis(R) server. | `127.0.0.1` | +| `REDIS_PORT_NUMBER` | Port used by Redis(R) server. | `6379` | +| `REDIS_DATABASE` | Name of the Redis(R) database. | `1` | +| `AIRFLOW_LDAP_ENABLE` | Enable LDAP authentication. | `no` | +| `AIRFLOW_LDAP_USER_REGISTRATION` | User self registration. | `True` | +| `AIRFLOW_LDAP_ROLES_SYNC_AT_LOGIN` | Replace ALL the user roles each login, or only on registration. | `True` | +| `AIRFLOW_LDAP_USE_TLS` | Use LDAP SSL. | `False` | +| `AIRFLOW_LDAP_ALLOW_SELF_SIGNED` | Allow self signed certicates in LDAP ssl. | `True` | + +#### Read-only environment variables + +| Name | Description | Value | +|-------------------------------|-------------------------------------------|---------------------------------------------| +| `AIRFLOW_BASE_DIR` | Airflow installation directory. | `${BITNAMI_ROOT_DIR}/airflow` | +| `AIRFLOW_HOME` | Airflow home directory. | `${AIRFLOW_BASE_DIR}` | +| `AIRFLOW_BIN_DIR` | Airflow directory for binary executables. | `${AIRFLOW_BASE_DIR}/venv/bin` | +| `AIRFLOW_LOGS_DIR` | Airflow logs directory. | `${AIRFLOW_BASE_DIR}/logs` | +| `AIRFLOW_SCHEDULER_LOGS_DIR` | Airflow scheduler logs directory. | `${AIRFLOW_LOGS_DIR}/scheduler` | +| `AIRFLOW_LOG_FILE` | Airflow log file. | `${AIRFLOW_LOGS_DIR}/airflow-webserver.log` | +| `AIRFLOW_CONF_FILE` | Airflow configuration file. | `${AIRFLOW_BASE_DIR}/airflow.cfg` | +| `AIRFLOW_WEBSERVER_CONF_FILE` | Airflow configuration file. | `${AIRFLOW_BASE_DIR}/webserver_config.py` | +| `AIRFLOW_TMP_DIR` | Airflow directory temporary files. | `${AIRFLOW_BASE_DIR}/tmp` | +| `AIRFLOW_PID_FILE` | Path to the Airflow PID file. | `${AIRFLOW_TMP_DIR}/airflow-webserver.pid` | +| `AIRFLOW_DAGS_DIR` | Airflow data to be persisted. | `${AIRFLOW_BASE_DIR}/dags` | +| `AIRFLOW_DAEMON_USER` | Airflow system user. | `airflow` | +| `AIRFLOW_DAEMON_GROUP` | Airflow system group. | `airflow` | > In addition to the previous environment variables, all the parameters from the configuration file can be overwritten by using environment variables with this format: `AIRFLOW__{SECTION}__{KEY}`. Note the double underscores.