Added configuration for using specific SQLite versions.

.env

@@ -1,7 +1,11 @@
 PYTHON_IMPLEMENTATION=python
-PYTHON_VERSION=3.10
+PYTHON_VERSION=3.12
 MARIADB_VERSION=10.5
 MYSQL_VERSION=8.0
 ORACLE_VERSION=23.5.0.0
 POSTGRESQL_VERSION=14
 POSTGIS_VERSION=3.1
+SQLITE_VERSION=
+SQLITE_CFLAGS="-DSQLITE_ENABLE_DESERIALIZE \
+               -DSQLITE_ENABLE_JSON1 \
+               -DSQLITE_MAX_VARIABLE_NUMBER=32766"

Containerfile

@@ -1,7 +1,7 @@
 # syntax=docker/dockerfile:1.12
 
 ARG PYTHON_IMPLEMENTATION=python
-ARG PYTHON_VERSION=3.10
+ARG PYTHON_VERSION=3.12
 FROM ${PYTHON_IMPLEMENTATION}:${PYTHON_VERSION}-slim-bookworm
 
 LABEL org.opencontainers.image.authors="Django Software Foundation"

README.md

@@ -36,7 +36,7 @@ Tooling and test execution support for [Django][0] :unicorn:
 3. Build the image:
 
    ```console
-   $ docker compose build sqlite
+   $ docker compose build base
    ```
 
 4. Run the tests:
@@ -193,7 +193,7 @@ environment variables:
 | ----------------------- | ------------- | ---------------------------------------------------- |
 | `DJANGO_PATH`           | `../django`   | Path to the Django repostory on your local machine   |
 | `PYTHON_IMPLEMENTATION` | `python`      | Implementation of Python to use — `python` or `pypy` |
-| `PYTHON_VERSION`        | `3.10`        | Version of Python container image to use             |
+| `PYTHON_VERSION`        | `3.12`        | Version of Python container image to use             |
 
 The versions of various backend services can be switched by setting these environment variables:
 
@@ -204,15 +204,21 @@ The versions of various backend services can be switched by setting these enviro
 | `ORACLE_VERSION`        | `23.5.0.0`    | Version of Oracle container image to use             |
 | `POSTGRESQL_VERSION`    | `14`          | Version of PostgreSQL container image to use         |
 | `POSTGIS_VERSION`       | `3.1`         | Version of PostGIS extension to use                  |
+| `SQLITE_VERSION`        |               | Version of SQLite to compile and use                 |
 
+> [!NOTE]
+>
+> If left unspecified, the SQLite version provided by Debian will be used.
+> Using a specific SQLite version requires compiling it from source. For more
+> details, see [SQLite Versions](#SQLite-Versions).
 
 ### Python Versions
 
 The `PYTHON_VERSION` environment variable controls which version of Python you
 are running the tests against, e.g.
 
 ```console
-$ PYTHON_VERSION=3.10 docker compose run --rm sqlite
+$ PYTHON_VERSION=3.12 docker compose run --rm sqlite
 ```
 
 In addition, it's possible to select a different implementation of Python, i.e.
@@ -229,7 +235,8 @@ restrictions with respect to the range of versions available.
 ### Database Versions
 
 Most database container images are pulled from [Docker Hub][2]. Oracle database
-is pulled from the [Oracle Container Registry][3].
+is pulled from the [Oracle Container Registry][3]. Specific versions of SQLite
+are compiled directly from the tags in the [official Git mirror][11].
 
 You can switch the version of the database you test against by changing the
 appropriate environment variable. Available options and their defaults can be
@@ -273,6 +280,46 @@ To determine what database versions can be used you can check the release notes
 for the branch of Django that you have checked out, or alternatively there is
 the [supported database versions][4] page on Django's Trac Wiki.
 
+#### SQLite Versions
+
+SQLite is normally bundled in the Python installation using the version
+available on the system where Python is compiled. We use the Python Docker image
+based on Debian `bookworm`, which has SQLite 3.40.1.
+
+To use a different version, we compile SQLite from source and load the library
+dynamically using `LD_PRELOAD`. There are a few caveats as a result:
+
+- Some SQLite features are only available if certain flags are set during
+  compilation. SQLite is known to change these flags in newer releases, such as
+  to enable features by default that were previously opt-in. When Python is
+  compiled, it inspects the system's SQLite to determine features that are
+  included in the `sqlite` module. A mismatch in the module and the dynamically
+  loaded library may result in Python failing to load, which may happen if we
+  use an SQLite version that is older than the system version.
+- Debian and Ubuntu use a custom `CFLAGS` variable to compile their distributed
+  SQLite. Historically, Django's CI has only been configured with SQLite
+  versions that come with the operating system. If SQLite is compiled with
+  different flags, some tests may fail.
+
+We currently work around the above caveats by setting the simplest `CFLAGS`
+value that allows all the tests to pass. To customize the `CFLAGS` used for the
+compilation, you can set the `SQLITE_CFLAGS` environment variable. See the
+[`.env`][10] file for its default value.
+
+```
+SQLITE_VERSION=3.48.0 SQLITE_CFLAGS="-DSQLITE_OMIT_JSON -DSQLITE_MAX_VARIABLE_NUMBER=999" docker compose run --build --rm sqlite
+```
+
+> [!NOTE]
+>
+> The `--build` argument is necessary if you've changed `SQLITE_CFLAGS` since
+> the last run, as it's not part of the image tag. You can also rebuild the
+> image separately by running `docker compose build sqlite`, optionally with
+> `--no-cache` to ignore the cached build.
+
+In the future, the Django codebase may be more robust when tested against
+different SQLite configurations and the `CFLAGS` workaround may no longer be
+necessary.
 
 ### Other Versions
 
@@ -309,3 +356,5 @@ with no promises that they'll be delivered:
 [7]: https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/unit-tests/#running-the-unit-tests
 [8]: https://docs.djangoproject.com/en/stable/internals/contributing/writing-code/unit-tests/#running-the-selenium-tests
 [9]: https://docs.djangoproject.com/en/stable/ref/contrib/gis/testing/#geodjango-tests
+[10]: .env
+[11]: https://github.com/sqlite/sqlite

compose.yml

@@ -8,7 +8,7 @@ x-base: &base
     args:
       - PYTHON_IMPLEMENTATION=${PYTHON_IMPLEMENTATION}
       - PYTHON_VERSION=${PYTHON_VERSION}
-    additional_contexts:
+    additional_contexts: &additional-contexts
       src: ${DJANGO_PATH:-../django}
   volumes:
     - ${DJANGO_PATH:-../django}:/django/source:rw
@@ -139,6 +139,10 @@ volumes:
 
 services:
 
+  # Base service to allow building the image with `docker compose build base`.
+  base:
+    <<: *base
+
   # Services: Databases
 
   mariadb-db:
@@ -285,6 +289,38 @@ services:
 
   sqlite:
     <<: *base
+    image: django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION}-sqlite${SQLITE_VERSION}
+    pull_policy: never
+    build:
+      context: .
+      dockerfile_inline: |
+        FROM django-docker-box:${PYTHON_IMPLEMENTATION}-${PYTHON_VERSION}
+        SHELL ["/bin/bash", "-o", "errexit", "-o", "nounset", "-o", "pipefail", "-o", "xtrace", "-c"]
+        # Only compile SQLite and set LD_PRELOAD if a version is specified.
+        RUN <<EOF
+          if [[ "${SQLITE_VERSION}" ]]; then
+            export CFLAGS="${SQLITE_CFLAGS}"
+            git clone --depth 1 --branch version-${SQLITE_VERSION} \
+            https://github.com/sqlite/sqlite.git /tmp/sqlite
+            cd /tmp/sqlite
+            ./configure
+            make
+            if [ -f libsqlite3.so ]; then
+              cp libsqlite3.so /tmp/
+            else
+              cp .libs/libsqlite3.so /tmp/
+            fi
+            rm -rf /tmp/sqlite
+          fi
+        EOF
+        SHELL ["/bin/bash", "-c"]
+        ENV LD_PRELOAD=${SQLITE_VERSION:+/tmp/libsqlite3.so}
+      args:
+        - PYTHON_IMPLEMENTATION=${PYTHON_IMPLEMENTATION}
+        - PYTHON_VERSION=${PYTHON_VERSION}
+        - SQLITE_VERSION=${SQLITE_VERSION}
+        - SQLITE_CFLAGS=${SQLITE_CFLAGS}
+      additional_contexts: *additional-contexts
     depends_on:
       <<: *depends-on-caches
     environment:

packages.txt

@@ -14,4 +14,5 @@ libpq-dev
 libproj-dev
 libsqlite3-mod-spatialite
 pkg-config
+tcl-dev
 unzip