DBLab Engine v3.4.0

[NikolayS]

---

New name: DBLab Engine

The new name for the Database Lab Engine is "DBLab Engine". Updates are currently underway across our materials to reflect this change. To align with this change, we have introduced specific domains for the product: dblab.dev and dblab.sh. For ease of access, we have established the following short URLs:

• https://cli.dblab.dev/ and https://dblab.sh – CLI setup script (works on macOS/Linux/Windows: curl -sSL dblab.sh | bash)
• https://docs.dblab.dev/ – the docs
• https://api.dblab.dev/ – interactive API reference (powered by readme.io)
• https://demo.dblab.dev/ – DBLab 3.4 demo (token: demo-token)
• https://branching.dblab.dev/ – DBLab 4.0-alpha demo, implementing full-fledged DB branching and snapshots on demand (token: demo-token)
• https://aws.dblab.dev/ – DBLab SE page in AWS Marketplace

New contributors

🙇 Many thanks to our first-time contributors: @mbobin, @TheLiamGuy, @gromsterus, @ibash, @byx01, @davidsvenson.

By the numbers

• DBLab GitHub repository has crossed the 1700+ stars mark
• X/Twitter account @Database_Lab has more than 1300 followers
• Our LinkedIn page has amassed 1500+ followers

Thank you for the support!

New DBLab SE installer

We've expanded the installation options for DBLab SE. Besides its presence in the AWS Marketplace, you can now seamlessly install DBLab SE directly from the Postgres.ai Console.

This setup is entirely automated and can be used anywhere:

• For those who have existing machines, we support the "BYOM" (Bring Your Own Machine) method
• If you're utilizing AWS, GCP, DigitalOcean, or Hetzner Cloud, the installer handles resource provisioning—including VMs, disks, and more—automatically.

Check out the updated tutorial.

Migration notes

For those considering an upgrade, we recommend executing a fresh installation of DBLab. At present, we don't have automation in place for upgrades. If you need to update an existing DBLab installation, it's essential to stop and remove the current containers. Then, initiate a fresh start of the DBLab container using the 3.4.0 tag. More details can be found in the docs. Before initiation, ensure you've adjusted the DBLab configuration as outlined below.

Postgres images

There is a new method for running clones with customized port and socket directories. Please note that this is NOT backward compatible with previous versions of the Postgres images that DBLab utilized. It is crucial to ensure the dockerImage options utilize tags with the -0.3.0 suffix or later, such as postgresai/extended-postgres:15-0.3.0 (details can be found in this MR).

Global telemetry

The global.telemetry subsection has been deprecated and removed. The control to activate/deactivate telemetry has been relocated to the platform section:

platform:
  url: "https://postgres.ai/api/general"
  enableTelemetry: true

Platform

From version v3.4.0 onward, the platform section is enabled by default.

If either the accessToken or orgKey keys aren't empty, ensure the url parameter is specified; otherwise, the DBLab instance will not initialize. By default, neither accessToken nor orgKey are defined when in case of self-managed installations.

platform:
  url: "https://postgres.ai/api/general"
  enableTelemetry: true
#  orgKey: "org_key"
#  accessToken: "platform_access_token"

Docker mounts

Please omit the following Docker mount directives during container startup, as they are no longer necessary:

    --volume /sys/kernel/debug:/sys/kernel/debug:rw \
    --volume /lib/modules:/lib/modules:ro \
    --volume /proc:/host_proc:ro \

Changelog

New features

• Configuration: new option ignoreErrors in subsections logicalDump and logicalRestore to allow not to interrupt the process of dump/restore in case of errors - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/699, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/700, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/730
• Configuration: new option skipPolicies in subsection logicalRestore to allow to skip policies (CREATE POLICY) during restore process, to avoid errors like policy "xxx" for table "yyy" already exists: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/769
• Configuration: new option cloneAccessAddresses in section provision to control port publishing for clone containers; now limited to 127.0.0.1 by default; support multiple addresses and IPv6: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/786
• UI: Improved Logs view with log filters - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/672, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/702
• CLI: ARM / Apple silicon (M1, M2) support for DBLab CLI binaries - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/719, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/733
• API: DBLab v3.4.0 API specification updated and is now published on api.dblab.dev: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/747
• Numerous updates to Postgres images
  • supported Postgres major versions: 9.6-15
  • image tags: 9.6-0.3.0 ... 15-0.3.0; ⚠️ older tags won't work with DBLab 3.4.x
  • pgvector extension added to
  • existing extension upgraded (docs)
  • special images for DBLab SE/EE (with some extension being mocked): RDS Postgres, Aurora, GCP Cloud SQL, Supabase, Timescale, Heroku, PostGIS
• New installer for DBLab SE, available at Console.Postgres.ai, and simplified configuration for various managed PostgreSQL services such as AWS RDS, GCP Cloud SQL, Supabase, Heroku - various MRs

Improvements and fixes

• Enable debug mode by default - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/687
• UI: Instant view of configuration, not waiting for the end of retrieval (physical mode) - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/674
• UI: Collapse/expand the right-side panel - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/677, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/681
• UI: Prevent crash when project ID doesn't exist - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/684
• Add system stats (number of CPUs and size of RAM) to the EngineStarted telemetry event - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/688
• Add hint about Postgres logs - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/685
• UI: "Sign out" button - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/678
• Correctly detect clone activity / idleness for timezones different from UTC - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/692, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/734
• If full refresh fails, report it in /status with proper messages - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/696
• Mask sensitive data in logs - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/705
• UI: Extend and improve the Configuration tab - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/698
• Check in UI if retrieval mode is unknown - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/669
• Clean up data during full data refresh in logical mode - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/704
• Choose the last snapshot when reset to the latest snapshot is requested - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/708
• Remove timing estimator (experimental) - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/716
• Avoid panic with concurrent map read and map write: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/726
• Exclude timescaledb internal scheme from the restricted script: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/738
• UI: show Configuration tab if mode is unknown: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/735
• Change error message from JSON format to string: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/739
• UI: Various fixes: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/752, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/759
• Remove too detailed error message from the logs: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/777
• Exclude dump location volumes nested in the mount directory: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/778
• Ignore STDERR buffer of the process while reading exec command response (previously, DBLab considered a task failed if exit code was 0 but STDERR had messages; now only the exit code defines success/failure) https://gitlab.com/postgres-ai/database-lab/-/merge_requests/782
• Run clones with custom port and socket directory to be able to restart or upgrade Postgres instance without losing container (useful in various cases such as testing Postgres major upgrades on clones): https://gitlab.com/postgres-ai/database-lab/-/merge_requests/780
• Issue a checkpoint after successful execution of query preprocessing scripts to mitigate possible heavy writes and avoid cloning timeouts: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/784
• Configuration: make upper boundary in portPool inclusive (previously, it was mistakenly exclusive, leading to unexpected behavior): https://gitlab.com/postgres-ai/database-lab/-/merge_requests/785
• Avoid panics when collecting info about sync container and no data pool is available: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/789
• Trim slashes before comparing mount points to avoid errors like Error response from daemon: Duplicate mount point: /var/lib/dblab/dblab_pool/dataset_1/dump/: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/791
• UI: Fix active tab detection: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/794

Internal changes

• CI/CD: Rework UI pipeline rules - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/676
• CI/CD: Manage keyring files in trusted.gpg.d because apt-key is deprecated - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/701
• Various libraries updated - https://gitlab.com/postgres-ai/database-lab/-/merge_requests/721, https://gitlab.com/postgres-ai/database-lab/-/merge_requests/722
• Remove hard-coded ZFS version in Dockerfile to use default one: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/724
• CLI: in the installer script, support versions with and without prefix "v", print installed version: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/745
• CLI: Add CLI setup snippet to the CLI setup script itself: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/749
• Update linter and fix new warnings: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/768
• Improve SSH port forwarding snippet – use the actual port: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/764
• Update building tools versions and dependency packages: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/788
• DBLab ansible installer: Use Git tag for ansible repository: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/761
• Adjust ansible snippet style: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/793

Full diff

Diff between versions v3.3.0 and v3.4.0: v3.3.0...v3.4.0

Where to start / get help

• Tutorial
• DBLab 3.4 demo instance (token: demo-token)
• Docs
• Postgres.ai Console
• Contact us

Provide feedback, contribute

We greatly value your feedback. Connect with us via:

• X/Twitter @Database_Lab
• (LinkedIn](https://www.linkedin.com/company/postgres-ai/posts/)
• DBLab Community Slack

Additional resources where you can hear more information about DBLab and Postgres:

• YouTube channel PostgresTV
• Podcast Postgres.FM

Interested in giving back to the project? Here's how you can make an impact:

• Give a star to our GitHub repository
• Help us reach more enthusiasts. Share about Database Lab on Twitter (don't forget to tag @Database_Lab) or any other platform you fancy
• Multilingual? Consider translating our README.md to share the knowledge in your language
• Are you a developer? Dive in and enhance the Database Lab Engine (DLE) experience; check out our CONTRIBUTING guidelines and explore the "good first issues" list on GitLab