pax_global_header00006660000000000000000000000064152126515110014511gustar00rootroot0000000000000052 comment=a8ee041436c8e61a7b63c4ec4d9f2c388048ba7e pg_background-2.0.2/000077500000000000000000000000001521265151100143175ustar00rootroot00000000000000pg_background-2.0.2/.claude/000077500000000000000000000000001521265151100156325ustar00rootroot00000000000000pg_background-2.0.2/.claude/scheduled_tasks.lock000066400000000000000000000002011521265151100216420ustar00rootroot00000000000000{"sessionId":"51b7672a-bcc3-4770-8753-1d6aef4bd0ff","pid":4942,"procStart":"Sun Jun 7 15:29:06 2026","acquiredAt":1780846742431}pg_background-2.0.2/.editorconfig000077500000000000000000000006601521265151100170010ustar00rootroot00000000000000# EditorConfig for pg_background # https://editorconfig.org/ root = true [*] charset = utf-8 end_of_line = lf insert_final_newline = true trim_trailing_whitespace = true [*.{c,h}] indent_style = tab indent_size = 4 max_line_length = 80 [*.sql] indent_style = space indent_size = 2 [*.{yml,yaml}] indent_style = space indent_size = 2 [Makefile] indent_style = tab [*.md] trim_trailing_whitespace = false max_line_length = off pg_background-2.0.2/.github/000077500000000000000000000000001521265151100156575ustar00rootroot00000000000000pg_background-2.0.2/.github/copilot-instructions.md000066400000000000000000000406671521265151100224310ustar00rootroot00000000000000# GitHub Copilot Instructions for pg_background This file guides GitHub Copilot to produce better code reviews, suggestions, and comments for this repository. --- ## 1. Repository Context **pg_background** is a PostgreSQL extension that executes SQL commands in background worker processes. It is: - Implemented in **C** (PostgreSQL backend code) and **SQL** (extension scripts) - Tightly coupled to **PostgreSQL internals**: Background Worker API, Dynamic Shared Memory (DSM), SPI, shm_mq - **Security-sensitive**: Executes arbitrary SQL with caller's privileges - **Version-sensitive**: Supports PostgreSQL 14-18 with compatibility macros - **Not a generic application**: PostgreSQL extension patterns differ significantly from typical app code ### Key Semantics to Understand - **v1 API** (legacy): Returns bare PID, no cancellation support - **v2 API** (recommended): Returns `(pid, cookie)` handle with explicit cancel/wait/detach - **`detach` is NOT `cancel`**: `detach_v2()` removes tracking; worker continues and commits. `cancel_v2()` requests termination. - **Autonomous transactions**: Workers commit independently of the launcher's transaction - **One-time result consumption**: `result_v2()` can only be called once per handle --- ## 2. Review Expectations When reviewing PRs in this repository: ### Do - Review the **entire PR** as a coherent change, not isolated lines - Consider how changes affect **extension control files**, **SQL scripts**, **C code**, **tests**, and **documentation** together - Provide **one consolidated maintainer-grade review** rather than piece-by-piece comments - Distinguish **critical issues** (correctness, security, compatibility) from **nice-to-have** (style, minor improvements) - Verify that **tests validate the intended semantic change** - Check that **documentation matches the implemented behavior** ### Do Not - Provide shallow line-by-line feedback without understanding context - Suggest generic application patterns that don't apply to PostgreSQL extensions - Assume background-worker code should look like ordinary async application code - Suggest broad refactors without verifying PostgreSQL backend semantics - Focus on style when correctness is the real concern --- ## 3. PostgreSQL-Specific Review Checklist When reviewing changes, verify: ### Extension Packaging - [ ] `pg_background.control` version matches latest SQL script - [ ] Upgrade scripts (`pg_background--X.Y--X.Z.sql`) are correct and minimal - [ ] No hardcoded `public.` schema references (extension is relocatable) - [ ] `@extschema@` used correctly for cross-references ### Background Worker Semantics - [ ] DSM segments created/attached/detached correctly - [ ] `shm_mq_wait_for_attach()` called before returning handle (prevents NOTIFY race) - [ ] Worker cleanup happens on all exit paths - [ ] `BackgroundWorkerHandle` is never `pfree()`d (let PostgreSQL manage it) ### SPI and Transactions - [ ] `SPI_connect()`/`SPI_finish()` paired correctly - [ ] No assumptions about caller's transaction state in worker code - [ ] Error handling uses `PG_TRY`/`PG_CATCH` appropriately ### PostgreSQL Version Compatibility - [ ] `#if PG_VERSION_NUM` guards for version-specific code - [ ] Compatibility macros in `pg_background.h` used correctly - [ ] Tested on all supported versions (14-18) ### Catalog and Schema - [ ] Correct use of `pg_catalog` qualifications - [ ] No unsafe `search_path` assumptions - [ ] Extension objects have correct ownership ### Privilege Model - [ ] `SECURITY DEFINER` functions set `search_path = pg_catalog` - [ ] Workers run with caller's privileges, not elevated - [ ] `pgbackground_role` grants are correct - [ ] No accidental privilege escalation --- ## 4. Security Review Checklist Flag these issues as **critical**: ### SECURITY DEFINER Misuse - Functions using `SECURITY DEFINER` without `SET search_path = pg_catalog` - `SECURITY DEFINER` on functions that shouldn't need it - Missing `REVOKE ALL ON FUNCTION ... FROM PUBLIC` for privileged helpers ### Dynamic SQL Risks - Caller-controlled strings concatenated into SQL without proper quoting - Missing `format()` with `%L` (literals) or `%I` (identifiers) - SQL injection vectors in any code path ### Privilege Escalation - Workers executing with higher privileges than caller - Extension functions granting unintended access - Unsafe object ownership assumptions ### Resource Abuse - Unbounded worker creation without limits - Unbounded DSM allocation - Missing input validation on size/timeout parameters - Denial-of-service vectors through resource exhaustion ### Search Path Safety - Functions relying on caller's `search_path` for critical operations - Missing schema qualification for security-sensitive lookups --- ## 5. Documentation and Test Review ### Documentation Must Match Behavior - If code changes user-visible behavior, README must be updated - API reference must reflect actual function signatures - Examples must work with current implementation - Limitations section must be honest about constraints ### Tests Must Validate Semantics - New functions need regression tests - Behavioral changes need test updates - Error paths need coverage - Cancel vs detach distinction must be tested explicitly ### Common Review Mistakes - Suggesting code changes when **documentation is the actual problem** - Suggesting documentation changes when **code is the actual problem** - Approving code that changes behavior without corresponding test updates - Approving tests that don't actually validate the claimed behavior --- ## 6. Common False Positives to Avoid Do not suggest these patterns that are inappropriate for this codebase: ### Generic Application Patterns - "Use async/await" - This is C code using PostgreSQL's background worker API - "Add a thread pool" - PostgreSQL uses processes, not threads; workers are process-based - "Cache the results" - Results are intentionally one-time consumption via shm_mq - "Use a mutex" - PostgreSQL uses LWLocks and other backend primitives ### Incorrect Privilege Assumptions - "Make this function SECURITY DEFINER" - Most functions should be INVOKER; only privilege helpers use DEFINER - "Grant to PUBLIC" - Extension explicitly avoids PUBLIC grants for security - "Remove the search_path setting" - Required for SECURITY DEFINER safety ### Incorrect Schema Assumptions - "Use public.function_name" - Extension is relocatable; don't hardcode schema - "Create objects in pg_catalog" - Extension objects belong in extension's schema ### Incorrect Transaction Assumptions - "Wrap in a transaction" - Workers have their own transactions; this is the design - "Rollback on error in worker" - Worker errors trigger automatic rollback on exit - "Commit the result" - Worker commits on clean exit; explicit commits not used ### Overly Broad Suggestions - "Refactor this module" - Only if directly relevant to the PR's purpose - "Add comprehensive error handling everywhere" - Be specific about which path needs it - "Modernize the code style" - PostgreSQL has its own style; don't suggest non-PG patterns --- ## 7. Preferred Review Style ### Internal Process 1. Read the entire PR to understand the intended change 2. Identify the category: bug fix, feature, refactor, documentation, test 3. Verify the change is complete: code + tests + docs as needed 4. Check PostgreSQL-specific concerns from the checklists above 5. Distinguish critical issues from suggestions ### Output Format - **One consolidated review** when possible - **Critical issues first**, clearly marked - **Suggestions second**, with rationale - **Questions third**, when intent is unclear ### Severity Levels - **Blocker**: Security vulnerability, data corruption risk, broken upgrade path - **Critical**: Incorrect behavior, missing tests for behavioral change, broken compatibility - **Major**: Missing documentation, incomplete error handling, suboptimal patterns - **Minor**: Style issues, minor improvements, documentation polish - **Nitpick**: Preferences that don't affect correctness ### Rationale Requirements - Explain **why** something is a problem, not just **what** to change - Reference PostgreSQL documentation or extension best practices when relevant - For security issues, explain the attack vector - For compatibility issues, explain which versions are affected --- ## 8. Code Suggestion Guidelines When suggesting code changes: ### C Code - Follow PostgreSQL coding style (4-space indent, K&R braces, C comments) - Use `palloc`/`pfree`, never `malloc`/`free` - Use `ereport()` for user errors, `elog()` for internal assertions - Include `CHECK_FOR_INTERRUPTS()` in loops - Handle cleanup in error paths with `PG_TRY`/`PG_CATCH` ### SQL Code - Use `STRICT` for functions that should return NULL on NULL input - Use `VOLATILE` appropriately for functions with side effects - Include `PARALLEL UNSAFE` for functions that can't run in parallel workers - Set `search_path = pg_catalog` for `SECURITY DEFINER` functions ### Test Code - Use `\gset` to capture values for later assertions - Include `pg_sleep()` with adequate margins for async operations - Test both success and failure paths - Verify cancel actually prevents work (not just detach) --- ## 9. Repository-Specific Knowledge ### Files and Their Purposes | File | Review Focus | |------|--------------| | `src/pg_background.c` | Launcher-side implementation; watch for memory, concurrency, cleanup | | `src/pg_background_worker.c` | Worker-process implementation; SPI, error paths | | `src/pg_background.h` | Version compatibility; ensure macros work on all PG versions | | `src/pg_background_internal.h` | Cross-file declarations between launcher and worker | | `pg_background.control` | Version must match latest SQL script | | `extension/pg_background--*.sql` | Current install + upgrade scripts | | `extension/legacy/pg_background--*.sql` | Pre-1.8 base + upgrade chain (kept so older installs can still upgrade) | | `sql/pg_background.sql` | Test coverage for all behaviors | | `expected/pg_background.out` | Expected output; watch for version-sensitive differences | | `scripts/test-upgrade.sh` | Upgrade path validation; verify 1.8 → 1.9 → 1.10 transitions | | `.github/workflows/ci.yml` | CI pipeline; test matrix + relocatable + upgrade tests | ### CI Pipeline Review Checklist - [ ] Main test matrix covers all PG versions (14-18) and both Ubuntu versions - [ ] Relocatable test verifies extension works in custom schema - [ ] Upgrade test validates old → new version transitions - [ ] New functions added to upgrade tests if behavioral changes - [ ] CI job dependencies are correct (test-summary depends on all test jobs) ### Critical Invariants 1. Cookie validation prevents PID reuse attacks 2. `detach_v2()` never cancels; `cancel_v2()` never detaches 3. Results are consumed exactly once 4. Workers commit on clean exit, rollback on error 5. DSM is cleaned up on worker exit or launcher detach 6. `pgbackground_role` controls access; PUBLIC has no grants ### v1.9 Features - Worker labels: `label` parameter on `launch_v2`/`submit_v2` - Structured errors: `pg_background_error_info_v2()` returns SQLSTATE, message, detail, hint, context - Result metadata: `pg_background_result_info_v2()` returns row_count, command_tag, completed, has_error - Batch operations: `pg_background_detach_all_v2()`, `pg_background_cancel_all_v2()` ### v1.10 Features - Convenience views: `pg_background_list`, `pg_background_activity` (joins `pg_stat_activity`) - Never-raises snapshot: `pg_background_outcome_v2()` (combines list + result_info + error_info) - Synchronous one-shots: `pg_background_run_v2()` (metadata), `pg_background_run_query_v2()` (rows) - Multi-handle helpers: `pg_background_drain_v2()`, `pg_background_wait_any_v2()` - Bulk/selective cleanup: `pg_background_cancel_by_label_v2()`, `pg_background_purge_v2()` - Driver-friendly status: `pg_background_status_v2()` (jsonb); full SQL: `pg_background_full_sql_v2()` ### Known Limitations (Do Not "Fix") - Windows: `cancel_v2()` cannot interrupt running statements (OS limitation) - Cross-database: Workers connect to launcher's database only (PostgreSQL limitation) - Result streaming: No pagination; results flow through shm_mq (design choice) - Session-local tracking: `list_v2()` only shows current session's workers (design choice) --- ## 10. Review Quality Checklist When generating code review comments, verify these before suggesting changes: ### SQL/C Alignment Checks - [ ] If suggesting SQL function is "not STRICT", verify C code has NULL checks - [ ] If suggesting data is "stored but not exposed", verify it's not in `list_v2()` or other APIs - [ ] If suggesting "function doesn't exist", verify against the actual SQL definition files - [ ] DSM fields that are written but never read are unused bloat - flag for removal ### Documentation Accuracy Checks - [ ] README descriptions must reflect actual behavior, not aspirational design - [ ] Error/failure semantics should be precise: specify what types of errors are captured - [ ] When reviewing `has_error` descriptions, note it reflects SQL execution errors only, not early worker failures - [ ] Features described as "available" must actually be exposed through SQL APIs ### Test Comment Accuracy - [ ] Test comments must match what the test actually verifies - [ ] Do not assume comments are accurate—read the test code ### Cross-File Consistency - [ ] Windows exports (`windows/pg_background_win.h`) must include all SQL-callable functions - [ ] Upgrade scripts must define all functions in the base install script - [ ] README API reference must match actual function signatures ### Test Isolation - [ ] Each test section should clean up its own resources - [ ] Batch operation tests should not rely on leftover workers from earlier tests - [ ] Counts in expected output must match self-contained test expectations ### Context Before Flagging - Before flagging "incomplete implementation": trace the full data path (DSM → C → SQL API) - Before flagging "test is brittle": verify the actual test dependency chain - Before flagging "function missing": check all SQL files (base install + upgrade scripts) ### Upgrade Script Safety - [ ] Do not suggest DROP/CREATE for public functions (breaks grants) - [ ] For optional parameters: suggest adding overloads, not replacing signatures - [ ] Verify suggested changes preserve upgrade safety - [ ] Fresh install and upgrade must result in equivalent functionality ### Statistics Accounting - [ ] Stats increments should happen in ONE place (typically cleanup functions) - [ ] Check for double-counting if both triggering function and cleanup increment stats - [ ] Compare similar functions (cancel_v2 vs cancel_all_v2) for accounting consistency ### Batch Helper Function Semantics - [ ] Batch helpers (e.g., `cancel_all_v2`, `detach_all_v2`) should match semantics of single-operation equivalents - [ ] Return values should reflect actual completed operations, not snapshot counts - [ ] For cancel: set cancel flag for all workers (including not-yet-started), send signals only to started workers - [ ] Keep function header comments aligned with actual output columns after API changes ### v2 API Error Handling Consistency - [ ] All v2 functions should use the same handle-validation error pattern - [ ] Missing PID: `ERRCODE_UNDEFINED_OBJECT`, `"PID %d is not attached to this session"` - [ ] Cookie mismatch: `ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE`, message + hint about stale handle - [ ] Compare new v2 functions against established patterns (wait_v2, cancel_v2, detach_v2) before suggesting changes ### Feature Test Coverage - [ ] Feature tests must verify the actual visible value, not just exercise the code path - [ ] For metadata features (labels, result info, error info): verify the value is correctly exposed - [ ] Distinguish between "feature is exercised" and "feature value is asserted" - [ ] Avoid hardcoded schema references (e.g., `public.pg_background_handle`); rely on search_path ### Error Path Cleanup - [ ] In PG_CATCH blocks, clear partial state before publishing error flags - [ ] Result metadata (row_count, command_tag) should be reset on error to avoid stale values - [ ] DSM cleanup callbacks should use non-throwing APIs (e.g., `shm_toc_lookup(..., true)`) ### Shell Script Semantics - [ ] With `set -e`, `$?` checks after commands are dead code for failure cases - [ ] psql returns 0 even for SQL errors unless `-v ON_ERROR_STOP=1` is used - [ ] Suggest `if ! command; then` instead of `command; if [ $? -ne 0 ]` - [ ] Automated test scripts should use `psql -X -v ON_ERROR_STOP=1` to make SQL errors detectable - [ ] Cleanup should be guaranteed via `trap 'cleanup' EXIT` rather than manual calls before each exit pg_background-2.0.2/.github/workflows/000077500000000000000000000000001521265151100177145ustar00rootroot00000000000000pg_background-2.0.2/.github/workflows/ci.yml000077500000000000000000000511261521265151100210420ustar00rootroot00000000000000name: CI - PostgreSQL Extension Build & Test on: push: # Branches we want pre-merge CI on. v1.* / v2.* track release-line work; # refactor/* is the conventional staging branch for non-feature cleanups. # `improvements/*` was historical and is dropped. branches: [ master, main, develop, 'v1.*', 'v2.*', 'refactor/*' ] tags: [ 'v*' ] pull_request: branches: [ master, main ] workflow_dispatch: # Cancel in-progress runs when a new commit is pushed to the same branch concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true env: # Default PostgreSQL version for lint/security jobs DEFAULT_PG_VERSION: "17" jobs: test: name: PG ${{ matrix.pg }} on ${{ matrix.os }} runs-on: ${{ matrix.os }} timeout-minutes: 15 strategy: fail-fast: false matrix: os: [ubuntu-22.04, ubuntu-24.04] # 19 is PostgreSQL 19 beta (image tag postgres:19beta1; see the # Start PostgreSQL container step). Drop the beta mapping at 19 GA. pg: [14, 15, 16, 17, 18, 19] steps: - name: Checkout code uses: actions/checkout@v4 - name: Cache APT packages uses: actions/cache@v4 with: path: | /var/cache/apt/archives ~/apt-cache key: apt-${{ matrix.os }}-pg${{ matrix.pg }}-${{ hashFiles('.github/workflows/ci.yml') }} restore-keys: | apt-${{ matrix.os }}-pg${{ matrix.pg }}- apt-${{ matrix.os }}- - name: Start PostgreSQL container run: | set -euo pipefail # Start PostgreSQL in a container docker run --name postgres-test -d \ -e POSTGRES_PASSWORD=postgres \ -e POSTGRES_USER=postgres \ -e POSTGRES_DB=postgres \ -p 5432:5432 \ postgres:${{ matrix.pg == 19 && '19beta1' || matrix.pg }} # Wait for PostgreSQL to be ready echo "Waiting for PostgreSQL to start..." for i in {1..30}; do if docker exec postgres-test pg_isready -U postgres >/dev/null 2>&1; then echo "PostgreSQL is ready" docker exec postgres-test psql -U postgres -c "SELECT version();" break fi if [ "$i" -eq 30 ]; then echo "::error::PostgreSQL failed to start within 60 seconds" exit 1 fi echo "Still waiting... ($i/30)" sleep 2 done - name: Install build dependencies run: | set -euo pipefail # Disable problematic Microsoft repositories (known to return 403 on GitHub runners) sudo rm -f /etc/apt/sources.list.d/microsoft-prod.list /etc/apt/sources.list.d/azure-cli.list || true sudo apt-get update -qq sudo apt-get install -y -qq \ ca-certificates \ wget \ gnupg \ lsb-release \ build-essential \ libkrb5-dev # Add PGDG repo sudo install -d -m 0755 /usr/share/keyrings wget -qO- https://www.postgresql.org/media/keys/ACCC4CF8.asc \ | sudo gpg --dearmor -o /usr/share/keyrings/postgresql.gpg echo "deb [signed-by=/usr/share/keyrings/postgresql.gpg] https://apt.postgresql.org/pub/repos/apt \ $(lsb_release -cs)-pgdg main" \ | sudo tee /etc/apt/sources.list.d/pgdg.list >/dev/null # Pre-release majors (PostgreSQL 19 during its beta cycle) are not in # the main pgdg component yet; their client/server-dev packages live # in the version-scoped pgdg-testing component. Add it only for 19 so # released majors keep pulling exclusively from pgdg main. if [ "${{ matrix.pg }}" = "19" ]; then echo "deb [signed-by=/usr/share/keyrings/postgresql.gpg] https://apt.postgresql.org/pub/repos/apt \ $(lsb_release -cs)-pgdg-testing main 19" \ | sudo tee /etc/apt/sources.list.d/pgdg-testing.list >/dev/null fi sudo apt-get update -qq # Install client + server-dev headers for the target major. For the # 19 beta, target the pgdg-testing release: its packages sit at a # lower apt priority (100) than stable (500), so without -t apt would # pull stable libpq-dev 18.x and break postgresql-server-dev-19's # "libpq-dev (>= 19~~)" dependency. -t raises testing's priority so # the matching beta libpq-dev / client are pulled together. APT_TARGET="" if [ "${{ matrix.pg }}" = "19" ]; then APT_TARGET="-t $(lsb_release -cs)-pgdg-testing" fi sudo apt-get install -y -qq $APT_TARGET \ postgresql-client-${{ matrix.pg }} \ postgresql-server-dev-${{ matrix.pg }} - name: Setup clang/llvm symlinks run: | # Create symlinks for clang/llvm tools to avoid version mismatch errors # The PGXS build system may expect specific versions (e.g., clang-19, llvm-19) for target_ver in 19 18; do if [ ! -e "/usr/bin/clang-${target_ver}" ]; then sudo ln -sf /usr/bin/clang "/usr/bin/clang-${target_ver}" 2>/dev/null || true fi if [ ! -d "/usr/lib/llvm-${target_ver}" ]; then # Find the highest llvm version available LLVM_VER=$(ls -d /usr/lib/llvm-* 2>/dev/null | sort -V | tail -1 | sed 's|.*/llvm-||' || echo "") if [ -n "$LLVM_VER" ] && [ -d "/usr/lib/llvm-${LLVM_VER}" ]; then sudo mkdir -p "/usr/lib/llvm-${target_ver}/bin" if [ -f "/usr/lib/llvm-${LLVM_VER}/bin/llvm-lto" ]; then sudo ln -sf "/usr/lib/llvm-${LLVM_VER}/bin/llvm-lto" "/usr/lib/llvm-${target_ver}/bin/llvm-lto" || true fi fi fi done - name: Build extension run: | set -euo pipefail export PG_CONFIG=/usr/lib/postgresql/${{ matrix.pg }}/bin/pg_config echo "=== Build Environment ===" $PG_CONFIG --version echo "=========================" make clean make echo "=== Build artifacts ===" ls -la pg_background.so ls -la pg_background.control echo "=======================" - name: Copy extension files into container run: | set -euo pipefail # Get PostgreSQL paths from pg_config PKGLIBDIR=$(/usr/lib/postgresql/${{ matrix.pg }}/bin/pg_config --pkglibdir) SHAREDIR=$(/usr/lib/postgresql/${{ matrix.pg }}/bin/pg_config --sharedir) echo "Container PostgreSQL paths:" echo " PKGLIBDIR: $PKGLIBDIR" echo " SHAREDIR: $SHAREDIR" # Copy the shared library docker exec postgres-test mkdir -p "$PKGLIBDIR" docker cp pg_background.so postgres-test:$PKGLIBDIR/pg_background.so # Copy extension control and SQL files docker exec postgres-test mkdir -p "$SHAREDIR/extension" docker cp pg_background.control postgres-test:$SHAREDIR/extension/ # Copy all SQL files (base install and upgrade scripts). # Files now live under extension/ (current) and extension/legacy/. for sqlfile in extension/pg_background--*.sql extension/legacy/pg_background--*.sql; do [ -f "$sqlfile" ] || continue echo "Copying $sqlfile..." docker cp "$sqlfile" postgres-test:$SHAREDIR/extension/ done # Optionally copy bitcode if it was built successfully. # PGXS emits one .bc per .o, so files live next to the sources in src/. for bcfile in src/pg_background.bc src/pg_background_worker.bc; do if [ -f "$bcfile" ]; then echo "Copying $bcfile..." docker exec postgres-test mkdir -p "$PKGLIBDIR/bitcode/pg_background/src" docker cp "$bcfile" postgres-test:$PKGLIBDIR/bitcode/pg_background/src/ \ || echo "Warning: Failed to copy $bcfile (non-critical)" fi done echo "=== Verify installation in container ===" docker exec postgres-test ls -la "$PKGLIBDIR/pg_background.so" docker exec postgres-test ls -la "$SHAREDIR/extension/pg_background.control" - name: Run regression tests run: | set -euo pipefail export PGHOST=127.0.0.1 export PGPORT=5432 export PGUSER=postgres export PGPASSWORD=postgres export PGDATABASE=postgres export PATH="/usr/lib/postgresql/${{ matrix.pg }}/bin:$PATH" echo "=== Test Environment ===" psql -c "SELECT version();" psql -c "CREATE EXTENSION pg_background;" psql -c "SELECT * FROM pg_available_extensions WHERE name = 'pg_background';" echo "========================" # Run regression tests - tests connect to the containerized PostgreSQL export PG_CONFIG=/usr/lib/postgresql/${{ matrix.pg }}/bin/pg_config make installcheck REGRESS_OPTS+=" --host=$PGHOST --port=$PGPORT --user=$PGUSER" - name: Show test results on failure if: failure() run: | echo "=== Regression Diffs ===" cat regression.diffs 2>/dev/null || echo "No diffs file" echo "" echo "=== Regression Out ===" cat regression.out 2>/dev/null || echo "No out file" echo "" echo "=== Container Logs ===" docker logs postgres-test 2>&1 | tail -100 - name: Upload regression diffs (on failure) if: failure() uses: actions/upload-artifact@v4 with: name: regression-diffs-pg${{ matrix.pg }}-${{ matrix.os }} path: | regression.diffs regression.out results/ retention-days: 7 - name: Cleanup if: always() run: | docker stop postgres-test 2>/dev/null || true docker rm postgres-test 2>/dev/null || true # Test extension relocatability (custom schema installation) on the full # PG matrix — 2.0's relocatable behavior depends on the privilege-helper # rename (B4), which we want validated against every supported PG. relocatable-test: name: Relocatable Test (PG ${{ matrix.pg }}) runs-on: ubuntu-24.04 timeout-minutes: 15 strategy: fail-fast: false matrix: # 19 = beta; test-relocatable.sh maps it to postgres:19beta1. pg: [14, 15, 16, 17, 18, 19] steps: - name: Checkout code uses: actions/checkout@v4 - name: Run relocatable tests run: ./scripts/test-relocatable.sh ${{ matrix.pg }} # Test upgrade path 1.8 -> 1.9 -> 1.10 -> 2.0 on the full PG matrix. # 2.0 is a major release; an upgrade-script syntax issue specific to one # PG major (e.g. removed catalogs, MERGE quirks) must not slip through. upgrade-test: name: Upgrade Test 1.8→2.0 (PG ${{ matrix.pg }}) runs-on: ubuntu-24.04 timeout-minutes: 25 strategy: fail-fast: false matrix: # No 19: the two-binary harness builds the prior v1.10 binary, which # only supports PG 14-18. A fresh 2.0 install on 19 is covered by the # main regression + relocatable jobs. pg: [14, 15, 16, 17, 18] steps: # fetch-depth: 0 + tags so the upgrade test can check out the v1.10 tag # to build the prior-version (v1-capable) binary. 2.0 dropped the v1 C # symbols, so the pre-2.0 install must be done with a pre-2.0 binary. - name: Checkout code uses: actions/checkout@v4 with: fetch-depth: 0 fetch-tags: true - name: Run upgrade test run: ./scripts/test-upgrade.sh ${{ matrix.pg }} # Test under AddressSanitizer + UndefinedBehaviorSanitizer. # # Builds PostgreSQL from source with -fsanitize=address,undefined and # builds the extension with the matching flags so ASan can hook the # allocator and stack metadata. Catches the memory-safety class of # bugs that the assert-enabled job cannot — heap-use-after-free, # stack-buffer-overflow, signed-integer-overflow, NULL-deref. Runs on # PG 17 only because the build is expensive (~6 minutes); add more # versions if a class of issue is suspected to be PG-major-specific. sanitizer-test: name: Sanitizer Test (ASan + UBSan, PG 17) runs-on: ubuntu-24.04 timeout-minutes: 30 steps: - name: Checkout code uses: actions/checkout@v4 - name: Run sanitizer test run: ./scripts/test-sanitizer.sh 17 # Test with assert-enabled PostgreSQL (catches memory context bugs) assert-test: name: Assert-Enabled Test (PG ${{ matrix.pg }}) runs-on: ubuntu-24.04 timeout-minutes: 30 strategy: fail-fast: false matrix: # No 19 yet: test-assert.sh builds from the X.0 GA source tarball, # which does not exist during the 19 beta cycle. Add 19 at GA. pg: [14, 15, 16, 17, 18] steps: - name: Checkout code uses: actions/checkout@v4 - name: Run assert-enabled test run: ./scripts/test-assert.sh ${{ matrix.pg }} - name: Show logs on failure if: failure() run: | echo "=== Test failed - check logs above for assertion failures ===" echo "Common assertion: FailedAssertion(\"CurrentMemoryContext != ErrorContext\")" # Summary job that depends on all test matrix jobs test-summary: name: Test Summary runs-on: ubuntu-24.04 needs: [test, relocatable-test, upgrade-test, assert-test, sanitizer-test] if: always() steps: - name: Check test results run: | if [ "${{ needs.test.result }}" != "success" ]; then echo "::error::Main test matrix failed" exit 1 fi if [ "${{ needs.relocatable-test.result }}" != "success" ]; then echo "::error::Relocatable test failed" exit 1 fi if [ "${{ needs.upgrade-test.result }}" != "success" ]; then echo "::error::Upgrade test failed" exit 1 fi if [ "${{ needs.assert-test.result }}" != "success" ]; then echo "::error::Assert-enabled test failed (memory context bug detected)" exit 1 fi if [ "${{ needs.sanitizer-test.result }}" != "success" ]; then echo "::error::ASan/UBSan run failed (memory-safety bug detected)" exit 1 fi echo "All test jobs passed!" lint: name: Lint & Style Check runs-on: ubuntu-24.04 timeout-minutes: 10 steps: - name: Checkout code uses: actions/checkout@v4 - name: Install dependencies run: | # Disable problematic Microsoft repositories (known to return 403 on GitHub runners) sudo rm -f /etc/apt/sources.list.d/microsoft-prod.list /etc/apt/sources.list.d/azure-cli.list || true sudo apt-get update -qq sudo apt-get install -y -qq \ ca-certificates \ wget \ gnupg \ lsb-release \ clang-format \ cppcheck # Add PGDG repo for PostgreSQL dev headers sudo install -d -m 0755 /usr/share/keyrings wget -qO- https://www.postgresql.org/media/keys/ACCC4CF8.asc \ | sudo gpg --dearmor -o /usr/share/keyrings/postgresql.gpg echo "deb [signed-by=/usr/share/keyrings/postgresql.gpg] https://apt.postgresql.org/pub/repos/apt \ $(lsb_release -cs)-pgdg main" \ | sudo tee /etc/apt/sources.list.d/pgdg.list >/dev/null sudo apt-get update -qq sudo apt-get install -y -qq postgresql-server-dev-${{ env.DEFAULT_PG_VERSION }} - name: Run cppcheck # BLOCKING. Previously the `|| echo "::warning"` swallowed cppcheck # exit codes so the job stayed green even when cppcheck found # actual issues. With --error-exitcode=1 set, any cppcheck error # now fails the lint job and gates merge. run: | echo "=== Running cppcheck ===" # -D PG_VERSION_NUM: cppcheck has no PostgreSQL headers, so without a # value the version gate in pg_background.c (#if PG_VERSION_NUM # < 140000 || >= 200000) sees an undefined macro (=0), trips the # intentional #error, and cppcheck cannot extract a valid # configuration for the file (preprocessorErrorDirective / # noValidConfiguration, which fail --error-exitcode). Pin a # supported major so cppcheck analyzes a real build configuration. # nullPointerRedundantCheck / ctunullpointer: false positives. The # code dereferences pointers from shm_toc_lookup(toc, KEY, # /*missing_ok=*/false) (never returns NULL) and from hash lookups # that are NULL-checked before use; the guards exit via # ereport(ERROR) and check_rights(), which are noreturn. cppcheck # models neither, so it flags the defensive checks as redundant. # missingInclude / missingIncludeSystem: cppcheck has none of the # PostgreSQL headers on the include path, so every "..." and <...> # server header reads as missing. That is expected (we are linting # our own .c, not the server) and counts toward --error-exitcode, # so both are suppressed. cppcheck --enable=all --inconclusive --std=c11 \ -DPG_VERSION_NUM=170000 \ --suppress=missingInclude \ --suppress=missingIncludeSystem \ --suppress=unusedFunction \ --suppress=unmatchedSuppression \ --suppress=nullPointerRedundantCheck \ --suppress=ctunullpointer \ --error-exitcode=1 \ src/pg_background.c src/pg_background_worker.c - name: Check code formatting run: | echo "=== Checking code formatting ===" clang-format --dry-run --Werror src/pg_background.c src/pg_background_worker.c 2>&1 \ || echo "::notice::Code formatting differs from clang-format style (informational)" security: name: Security Scan runs-on: ubuntu-24.04 timeout-minutes: 20 permissions: security-events: write actions: read contents: read steps: - name: Checkout code uses: actions/checkout@v4 - name: Initialize CodeQL uses: github/codeql-action/init@v3 with: languages: cpp - name: Install PostgreSQL headers run: | # Disable problematic Microsoft repositories (known to return 403 on GitHub runners) sudo rm -f /etc/apt/sources.list.d/microsoft-prod.list /etc/apt/sources.list.d/azure-cli.list || true sudo apt-get update -qq sudo apt-get install -y -qq \ ca-certificates \ wget \ gnupg \ lsb-release \ libkrb5-dev \ build-essential # Add PGDG repo for PostgreSQL dev headers sudo install -d -m 0755 /usr/share/keyrings wget -qO- https://www.postgresql.org/media/keys/ACCC4CF8.asc \ | sudo gpg --dearmor -o /usr/share/keyrings/postgresql.gpg echo "deb [signed-by=/usr/share/keyrings/postgresql.gpg] https://apt.postgresql.org/pub/repos/apt \ $(lsb_release -cs)-pgdg main" \ | sudo tee /etc/apt/sources.list.d/pgdg.list >/dev/null sudo apt-get update -qq sudo apt-get install -y -qq postgresql-server-dev-${{ env.DEFAULT_PG_VERSION }} - name: Setup clang/llvm symlinks run: | for target_ver in 19 18; do if [ ! -e "/usr/bin/clang-${target_ver}" ]; then sudo ln -sf /usr/bin/clang "/usr/bin/clang-${target_ver}" 2>/dev/null || true fi if [ ! -d "/usr/lib/llvm-${target_ver}" ]; then LLVM_VER=$(ls -d /usr/lib/llvm-* 2>/dev/null | sort -V | tail -1 | sed 's|.*/llvm-||' || echo "") if [ -n "$LLVM_VER" ] && [ -d "/usr/lib/llvm-${LLVM_VER}" ]; then sudo mkdir -p "/usr/lib/llvm-${target_ver}/bin" if [ -f "/usr/lib/llvm-${LLVM_VER}/bin/llvm-lto" ]; then sudo ln -sf "/usr/lib/llvm-${LLVM_VER}/bin/llvm-lto" "/usr/lib/llvm-${target_ver}/bin/llvm-lto" || true fi fi fi done - name: Build for CodeQL run: | export PATH=/usr/lib/postgresql/${{ env.DEFAULT_PG_VERSION }}/bin:$PATH export PG_CONFIG=/usr/lib/postgresql/${{ env.DEFAULT_PG_VERSION }}/bin/pg_config make clean make - name: Perform CodeQL Analysis uses: github/codeql-action/analyze@v3 with: category: "/language:cpp" pg_background-2.0.2/.gitignore000077500000000000000000000150471521265151100163210ustar00rootroot00000000000000# PostgreSQL Extension Build Artifacts # Build outputs *.o *.so *.so.* *.a *.dylib *.dll *.bc *.gcda *.gcno *.gcov # PGXS build artifacts .deps/ # PostgreSQL regression test outputs results/ regression.diffs regression.out tmp_check/ tmp_check_iso/ log/ output_iso/ *.log # Editor and IDE files # Vim *.swp *.swo *~ .*.swp .*.swo # Emacs \#*\# .\#* # VS Code .vscode/ # JetBrains IDEs .idea/ # Sublime Text *.sublime-project *.sublime-workspace # OS-specific files .DS_Store .DS_Store? ._* .Spotlight-V100 .Trashes ehthumbs.db Thumbs.db desktop.ini # Tooling and development # clangd .clangd/ compile_commands.json # ccache .ccache/ # Coverage coverage/ htmlcov/ .coverage .coverage.* # Temporary files tmp/ temp/ *.tmp *.bak *.orig ## Visual Studio temporary files, build results, and ## files generated by popular Visual Studio add-ons. ## ## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore # User-specific files *.suo *.user *.userosscache *.sln.docstates # User-specific files (MonoDevelop/Xamarin Studio) *.userprefs # Build results [Dd]ebug/ [Dd]ebugPublic/ [Rr]elease/ [Rr]eleases/ x64/ x86/ bld/ [Bb]in/ [Oo]bj/ [Ll]og/ # Visual Studio 2015/2017 cache/options directory .vs/ # Uncomment if you have tasks that create the project's static files in wwwroot #wwwroot/ # Visual Studio 2017 auto generated files Generated\ Files/ # MSTest test Results [Tt]est[Rr]esult*/ [Bb]uild[Ll]og.* # NUNIT *.VisualState.xml TestResult.xml # Build Results of an ATL Project [Dd]ebugPS/ [Rr]eleasePS/ dlldata.c # Benchmark Results BenchmarkDotNet.Artifacts/ # .NET Core project.lock.json project.fragment.lock.json artifacts/ # StyleCop StyleCopReport.xml # Files built by Visual Studio *_i.c *_p.c *_i.h *.ilk *.meta *.obj *.iobj *.pch *.pdb *.ipdb *.pgc *.pgd *.rsp *.sbr *.tlb *.tli *.tlh *.tmp *.tmp_proj *.log *.vspscc *.vssscc .builds *.pidb *.svclog *.scc # Chutzpah Test files _Chutzpah* # Visual C++ cache files ipch/ *.aps *.ncb *.opendb *.opensdf *.sdf *.cachefile *.VC.db *.VC.VC.opendb # Visual Studio profiler *.psess *.vsp *.vspx *.sap # Visual Studio Trace Files *.e2e # TFS 2012 Local Workspace $tf/ # Guidance Automation Toolkit *.gpState # ReSharper is a .NET coding add-in _ReSharper*/ *.[Rr]e[Ss]harper *.DotSettings.user # JustCode is a .NET coding add-in .JustCode # TeamCity is a build add-in _TeamCity* # DotCover is a Code Coverage Tool *.dotCover # AxoCover is a Code Coverage Tool .axoCover/* !.axoCover/settings.json # Visual Studio code coverage results *.coverage *.coveragexml # NCrunch _NCrunch_* .*crunch*.local.xml nCrunchTemp_* # MightyMoose *.mm.* AutoTest.Net/ # Web workbench (sass) .sass-cache/ # Installshield output folder [Ee]xpress/ # DocProject is a documentation generator add-in DocProject/buildhelp/ DocProject/Help/*.HxT DocProject/Help/*.HxC DocProject/Help/*.hhc DocProject/Help/*.hhk DocProject/Help/*.hhp DocProject/Help/Html2 DocProject/Help/html # Click-Once directory publish/ # Publish Web Output *.[Pp]ublish.xml *.azurePubxml # Note: Comment the next line if you want to checkin your web deploy settings, # but database connection strings (with potential passwords) will be unencrypted *.pubxml *.publishproj # Microsoft Azure Web App publish settings. Comment the next line if you want to # checkin your Azure Web App publish settings, but sensitive information contained # in these scripts will be unencrypted PublishScripts/ # NuGet Packages *.nupkg # The packages folder can be ignored because of Package Restore **/[Pp]ackages/* # except build/, which is used as an MSBuild target. !**/[Pp]ackages/build/ # Uncomment if necessary however generally it will be regenerated when needed #!**/[Pp]ackages/repositories.config # NuGet v3's project.json files produces more ignorable files *.nuget.props *.nuget.targets # Microsoft Azure Build Output csx/ *.build.csdef # Microsoft Azure Emulator ecf/ rcf/ # Windows Store app package directories and files AppPackages/ BundleArtifacts/ Package.StoreAssociation.xml _pkginfo.txt *.appx # Visual Studio cache files # files ending in .cache can be ignored *.[Cc]ache # but keep track of directories ending in .cache !*.[Cc]ache/ # Others ClientBin/ ~$* *~ *.dbmdl *.dbproj.schemaview *.jfm *.pfx *.publishsettings orleans.codegen.cs # Including strong name files can present a security risk # (https://github.com/github/gitignore/pull/2483#issue-259490424) #*.snk # Since there are multiple workflows, uncomment next line to ignore bower_components # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) #bower_components/ # RIA/Silverlight projects Generated_Code/ # Backup & report files from converting an old project file # to a newer Visual Studio version. Backup files are not needed, # because we have git ;-) _UpgradeReport_Files/ Backup*/ UpgradeLog*.XML UpgradeLog*.htm ServiceFabricBackup/ *.rptproj.bak # SQL Server files *.mdf *.ldf *.ndf # Business Intelligence projects *.rdl.data *.bim.layout *.bim_*.settings *.rptproj.rsuser # Microsoft Fakes FakesAssemblies/ # GhostDoc plugin setting file *.GhostDoc.xml # Node.js Tools for Visual Studio .ntvs_analysis.dat node_modules/ # Visual Studio 6 build log *.plg # Visual Studio 6 workspace options file *.opt # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) *.vbw # Visual Studio LightSwitch build output **/*.HTMLClient/GeneratedArtifacts **/*.DesktopClient/GeneratedArtifacts **/*.DesktopClient/ModelManifest.xml **/*.Server/GeneratedArtifacts **/*.Server/ModelManifest.xml _Pvt_Extensions # Paket dependency manager .paket/paket.exe paket-files/ # FAKE - F# Make .fake/ # JetBrains Rider .idea/ *.sln.iml # CodeRush .cr/ # Python Tools for Visual Studio (PTVS) __pycache__/ *.pyc # Cake - Uncomment if you are using it # tools/** # !tools/packages.config # Tabs Studio *.tss # Telerik's JustMock configuration file *.jmconfig # BizTalk build output *.btp.cs *.btm.cs *.odx.cs *.xsd.cs # OpenCover UI analysis results OpenCover/ # Azure Stream Analytics local run output ASALocalRun/ # MSBuild Binary and Structured Log *.binlog # NVidia Nsight GPU debugger configuration file *.nvuser # MFractors (Xamarin productivity tool) working folder .mfractor/ pg_background-2.0.2/CLAUDE.md000066400000000000000000000672561521265151100156160ustar00rootroot00000000000000# CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. --- ## 1. Repository Purpose pg_background is a PostgreSQL extension that executes SQL commands in background worker processes. Workers run inside the PostgreSQL server with their own transactions, enabling: - Asynchronous SQL execution without blocking client sessions - Autonomous transactions that commit/rollback independently of the caller - Observable worker lifecycle with explicit launch/wait/cancel/detach semantics **Appropriate changes include:** - Bug fixes to worker lifecycle, DSM handling, or result streaming - Improvements to observability, error handling, or resource cleanup - PostgreSQL version compatibility updates - Security hardening - API ergonomics improvements that preserve backward compatibility - Documentation and test coverage improvements **Changes that require careful consideration:** - New SQL-callable functions (API surface expansion) - Changes to handle/cookie semantics - Modifications to worker transaction behavior - Anything affecting upgrade paths --- ## 2. Core Design Principles ### Preserve PostgreSQL-native design This extension uses PostgreSQL's native Background Worker API, Dynamic Shared Memory, and SPI. Do not introduce external dependencies or non-PostgreSQL patterns. ### Prefer simple APIs over feature creep The extension provides async SQL execution primitives. Resist adding orchestration, scheduling, or workflow features that belong in application code or dedicated tools like pg_cron. ### Keep background worker behavior understandable Workers execute SQL in independent transactions. This autonomy is the feature, not a bug. Do not add implicit coordination that obscures transaction boundaries. ### Prefer explicit behavior over magic - `detach_v2()` removes tracking; it does NOT cancel - `cancel_v2()` requests termination; it does NOT guarantee immediate stop - Results are consumed once; there is no hidden caching Document behavioral semantics precisely. Users should never be surprised by what a function does. ### One canonical API; `_v2` retired (2.0) - The cookie-protected, explicit-lifecycle API is now the only API. Its **canonical names are unsuffixed** (`pg_background_launch`, `pg_background_wait`, `pg_background_run`, …). New features go here. - The v1 API (unsuffixed names returning bare PIDs) was removed in 2.0. The unsuffixed names were reused for the cookie-protected API. - Every `_v2` name that shipped through 1.10 is kept as a **thin deprecated alias** (identical behavior, forwards to the canonical function) and is **removed in 3.0**. Do not add new `_v2` names; new functions ship under their unsuffixed name only. - Names introduced in 2.0 (`pg_background_report_progress`, `pg_background_record_timeout`, the privilege helpers) have **no `_v2` alias** — no released `_v2` name ever existed for them. - The canonical function names `pg_background_list` / `pg_background_stats` / `pg_background_outcome` coexist with a same-named view / type / type; PostgreSQL resolves them by call syntax. This is intentional. --- ## 3. PostgreSQL Extension Rules ### Control file semantics - `pg_background.control` declares `relocatable = true` - `default_version` must match the latest `pg_background--X.Y.sql` - Do not add `requires` unless genuinely needed ### Install and upgrade scripts - Base install scripts: `pg_background--X.Y.sql` (complete, standalone) - Upgrade scripts: `pg_background--X.Y--X.Z.sql` (incremental changes only) - Upgrade scripts must be idempotent where possible - Never break upgrade paths from supported prior versions - Test upgrades explicitly: `ALTER EXTENSION pg_background UPDATE TO 'X.Y'` ### Version support - Supported: PostgreSQL 14, 15, 16, 17, 18, 19 (19 = beta) - Version-specific code uses `#if PG_VERSION_NUM` guards in C - Compatibility macros live in `pg_background.h` - Do not add version-specific SQL without strong justification ### Schema and ownership - Extension objects belong to the installing superuser - The extension is relocatable; do not hardcode `public.` in SQL scripts - Use `@extschema@` or dynamic schema lookup for cross-references - `pgbackground_role` is created for privilege management; do not grant to PUBLIC --- ## 4. C Code Rules ### Follow PostgreSQL backend coding patterns - 4-space indentation, no tabs - K&R brace style - C-style comments only (`/* */`) - Function names: `lowercase_with_underscores` - Macros: `UPPERCASE_WITH_UNDERSCORES` ### Memory context discipline - Use `palloc`/`pfree`, never `malloc`/`free` - Long-lived allocations use dedicated memory contexts (e.g., `PgBackgroundWorkerContext`) - Worker info hash entries are context-managed to prevent session memory bloat - Clean up in error paths using `PG_TRY`/`PG_CATCH`/`PG_FINALLY` - In PG_CATCH, clear partial state (e.g., result metadata) before publishing error flags ### SPI and transaction handling - Workers use `SPI_connect()`/`SPI_finish()` for SQL execution - Workers run in their own transaction; do not assume caller's transaction state - Commit happens automatically when worker exits cleanly - Explicit `SPI_commit()` calls are not used; worker exit triggers commit ### Worker lifecycle and cleanup - DSM segments are created by launcher, attached by worker - Worker attaches DSM on startup, detaches on exit (automatic cleanup) - Launcher tracks workers in session-local hash table - Never `pfree()` the `BackgroundWorkerHandle`; let PostgreSQL manage it - Use `shm_mq_wait_for_attach()` before returning handle to SQL (prevents NOTIFY race) ### Concurrency and state - Worker hash table is session-local; no cross-session visibility - Cookie validation prevents PID reuse confusion - Use `pg_strong_random()` for cryptographically secure cookie generation - Polling loops use exponential backoff to reduce CPU usage - Always call `CHECK_FOR_INTERRUPTS()` in loops ### DSM synchronization patterns - Worker writes to DSM fields may be observed concurrently by launcher - For multi-field data, use a "publish flag" pattern: write data fields first, then the flag - Use `pg_write_barrier()` before setting publish flag to ensure field ordering - Readers check the flag first, then `pg_read_barrier()`, then read other fields - Example: `error_sqlstate` is the publish flag for error fields; written LAST by worker - `cleanup_worker_info` callback can read DSM before unmap; use `seg` argument ### Comments - Document non-obvious backend behavior - Explain why certain PostgreSQL APIs are used - Mark fields in structs with access patterns: `[L]` launcher, `[W]` worker, `[B]` both --- ## 5. Security Rules ### Unsafe search_path - All `SECURITY DEFINER` functions must set `search_path = pg_catalog` - Do not rely on caller's `search_path` for object resolution in privileged functions - The privilege helper functions (`grant_pg_background_privileges`, etc.) use dynamic schema lookup ### Caller-controlled SQL - Workers execute arbitrary SQL provided by the caller - This is intentional; the caller's privileges apply - Do not execute caller SQL with elevated privileges - Document SQL injection risks clearly; recommend `format()` with `%L`/`%I` ### Privilege model - Workers inherit `current_user` from launcher, not superuser - Extension functions are granted to `pgbackground_role`, not PUBLIC - Users must be explicitly granted `pgbackground_role` or function EXECUTE - `SECURITY DEFINER` is used only for privilege helper functions, not core operations ### Input validation - Validate `queue_size` bounds (min: `shm_mq_minimum_size`, max: 256MB) - Validate `grace_ms` bounds (max: 1 hour) - Validate `timeout_ms` bounds (max: 24 hours) - Truncate SQL preview safely (UTF-8 aware) to prevent buffer issues ### Resource abuse prevention - `pg_background.max_workers` GUC limits concurrent workers per session - Workers count against global `max_worker_processes` - DSM segments are bounded by `queue_size` parameter - Long-running workers should use `statement_timeout` ### Keep internals private - Internal helper functions in C are `static` - Internal SQL functions use naming conventions that discourage direct use - Do not expose DSM handles or internal state to SQL callers --- ## 6. API Design Rules ### Keep the SQL API simple and explicit - Function names clearly indicate behavior: `launch`, `result`, `detach`, `cancel`, `wait`, `submit` - Canonical functions are **unsuffixed**: `pg_background_` - Return types are explicit: `pg_background_handle`, `pg_background_stats`, etc. ### Naming consistency - All functions: `pg_background_` (canonical, unsuffixed) - Options are extra parameters with sensible defaults, not name variants (e.g., `cancel(pid, cookie, grace_ms DEFAULT 0)`, not `cancel_grace`) - `_v2`-suffixed names are deprecated aliases only (removed in 3.0); never add new ones - Type names: `pg_background_` ### Backward compatibility - v1 API is frozen; do not change signatures or behavior - v2 API additions must not break existing v2 callers - New optional parameters should have sensible defaults - Deprecate, do not remove, unless security requires it ### Document behavioral semantics - `detach` vs `cancel`: different operations, document the distinction everywhere - Result consumption: one-time only, document clearly - Worker state values: `running`, `stopped`, `canceled`, `error` - defined meanings ### Runtime behavior changes - Any change to when/how workers commit requires documentation update - Any change to error handling requires test update - Do not silently change timeout, cleanup, or cancellation behavior --- ## 7. Testing Rules ### Regression tests for new behavior - All new SQL functions need tests in `sql/pg_background.sql` - Expected output in `expected/pg_background.out` - Alternative outputs (version differences) in `expected/pg_background_1.out` ### Test coverage requirements - Happy path: launch, result retrieval, cleanup - Cancel path: verify work is not committed after cancel - Detach path: verify work IS committed after detach (detach != cancel) - Error conditions: invalid handles, cookie mismatches, resource exhaustion - Privilege paths: verify `pgbackground_role` grants work correctly - Timeout behavior: `wait_v2_timeout` returns false on timeout, true on completion ### Version-sensitive testing - `./scripts/test-local.sh all` tests PostgreSQL 14-19 - CI matrix covers ubuntu-22.04 and ubuntu-24.04 with all PG versions - Version-specific expected outputs when necessary ### CI pipeline coverage - **Main test matrix**: Runs `make installcheck` on PG 14-19 × ubuntu-22.04/24.04 - **Relocatable test**: Verifies extension works in custom schema (not just `public`) - **Upgrade test**: Validates upgrade path 1.8 → 1.9 → 1.10 using `scripts/test-upgrade.sh` - All three test types must pass before merge ### Upgrade path testing - `./scripts/test-upgrade.sh [PG_VERSION]` tests extension upgrades in Docker - Validates: old version installs, old functionality works, upgrade succeeds, new features work, old features preserved - Always test upgrade paths when adding new functions or types - Upgrade scripts must be additive; never remove objects in upgrade scripts ### Failure and cleanup testing - Test worker crash behavior - Test launcher session termination - Test DSM cleanup after abnormal exit - Test `max_workers` limit enforcement ### Test maintenance - Do not remove test coverage without equivalent replacement - Flaky tests (timing-dependent) should use adequate sleep margins ### Test isolation and independence - Each test section should be self-contained and not rely on state from other sections - Explicitly clean up resources (detach workers) after each test - Do not rely on batch operations (like `detach_all_v2`) to clean up from earlier tests - Test comments must accurately describe what the test verifies - Tests must be deterministic; avoid race conditions in assertions --- ## 8. Documentation Rules ### README accuracy - README.md must match actual current behavior - API reference must list all functions with correct signatures - Examples must be tested and working ### Honest limitations - Document Windows cancel limitations clearly - Document `max_worker_processes` exhaustion behavior - Document one-time result consumption - Document autonomous transaction implications ### Version documentation - Supported PostgreSQL versions in README and control file - Migration guide for version upgrades - Breaking changes documented in version sections ### Operational guidance - Document GUC settings and their effects - Document resource implications (DSM, worker slots) - Document monitoring approaches (`list_v2`, `stats_v2`, `pg_stat_activity`) ### Distinguish usage from internals - User-facing API documentation in main README sections - Architecture and design details in dedicated section - Internal implementation notes in code comments, not user docs ### Licensing and attribution - This project is licensed under the PostgreSQL License (see LICENSE file) - Copyright belongs to "Vibhor Kumar and contributors" - PostgreSQL-derived code (compatibility macros, API patterns) includes "Portions Copyright (c) PostgreSQL Global Development Group" - File headers should use concise form: copyright line(s) + "Licensed under the PostgreSQL License. See LICENSE file for details." - Do not duplicate full license text in file headers (keep headers short) - README license section should reference LICENSE file, not duplicate its text - When adding new source files, use the same header format as existing files - Preserve upstream copyright notices (PGDG, third-party) when code is derived from external sources - Do not remove or replace contributor attribution without valid reason --- ## 9. Review Rules ### Prefer minimal reviewable patches - One logical change per PR - Separate refactoring from behavioral changes - Separate documentation from code changes when substantial ### Avoid unrelated refactors - Do not "clean up" code unrelated to the PR's purpose - Style-only changes should be separate PRs - Do not move code around without clear justification ### Explain breaking changes explicitly - PR description must call out any API changes - PR description must call out any behavioral changes - Upgrade path implications must be documented ### Correctness over cleverness - Prefer straightforward code that's obviously correct - Avoid clever optimizations without measured justification - Avoid premature abstraction ### Review checklist - [ ] Compiles without warnings on all supported PG versions - [ ] Regression tests pass (`make installcheck`) - [ ] New behavior has test coverage - [ ] Documentation updated if user-visible - [ ] No memory leaks (check with valgrind if uncertain) - [ ] No resource leaks (DSM, worker slots) - [ ] Security implications considered ### Validating AI-generated code reviews (GitHub Copilot, etc.) - Always validate review comments against actual code and repository context - Do not assume AI suggestions are correct just because they sound plausible - Skip reviews that are incorrect or do not fit the PostgreSQL extension context - Prefer simple, practical fixes over clever or invasive changes - When a pattern issue is identified, check nearby code for similar problems ### SQL/C alignment requirements - If SQL function is not STRICT, C code must check `PG_ARGISNULL()` for required parameters - If C code stores data in DSM, ensure it's exposed through observable SQL APIs - DSM fields that are written but never read are unused bloat - remove them - New features must be testable via SQL (not just internal state changes) - Windows exports (`windows/pg_background_win.h`) must include all SQL-callable functions ### Documentation accuracy - README descriptions must reflect actual behavior, not aspirational design - Error/failure semantics should be precise: specify what types of errors are captured - When describing `has_error`, note it reflects SQL execution errors only, not early worker failures - Features described as "available" must actually be exposed through SQL APIs ### Upgrade script safety - Do not DROP and recreate public functions in upgrade scripts (breaks grants/OIDs) - When adding optional parameters, add function overloads rather than dropping/recreating - Keep fresh install and upgrade paths aligned in functionality - Preserve grants by adding new signatures alongside existing ones ### Statistics accounting consistency - Stats increments should happen in one place, typically cleanup functions - Avoid double-counting: if cleanup increments a stat, don't also increment in the triggering function - Keep cancel/complete/fail paths consistent in their stats accounting ### Batch helper function semantics - Batch functions (e.g., `cancel_all_v2`, `detach_all_v2`) should stay semantically aligned with their single-operation equivalents - Return values should reflect actual completed operations, not snapshot counts (workers may be cleaned up between snapshot and processing) - For cancel operations: set cancel flag for all workers (including not-yet-started), but only send signals to started workers - Keep internal comments aligned with actual column lists when output shape changes ### v2 API error handling consistency - All v2 functions should use the same handle-validation pattern: - Missing PID: `ERRCODE_UNDEFINED_OBJECT`, `"PID %d is not attached to this session"` - Cookie mismatch: `ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE`, `"cookie mismatch for PID %d"`, with hint `"The worker may have been restarted or the handle is stale."` - New v2 functions must follow this established convention unless there is a documented reason to differ ### Test coverage for new features - Feature tests must verify the actual visible value of the feature, not just exercise it - For metadata features (row_count, command_tag, etc.), assert the actual values returned - Avoid hardcoded schema references (e.g., `public.pg_background_handle`) in tests; rely on search_path - For metadata features (labels, result info, error info), add direct assertions that query and verify the value - Example: if adding a label parameter, add a test that queries `list_v2()` and verifies the label is visible ### Shell script patterns (scripts/test-upgrade.sh, scripts/test-local.sh) - With `set -e`, do not rely on `$?` checks after commands that would already abort - Use `if ! command; then` instead of `command; if [ $? -ne 0 ]` - Use `psql -X -v ON_ERROR_STOP=1` in automated test scripts to ensure SQL errors propagate as non-zero exit codes - Ensure cleanup runs on all exit paths via `trap 'cleanup' EXIT` rather than manual cleanup calls before each exit - Test scripts should rely on output string matching for SQL success when ON_ERROR_STOP is not used (psql returns 0 even for SQL errors) --- ## 10. Feature Scope Guidance ### Good fit for pg_background | Feature Type | Examples | |--------------|----------| | Safer async execution | Better error propagation, structured error returns | | Observability | Progress reporting, execution statistics, worker introspection, labels | | Result/error handling | Improved result metadata, error context preservation | | Resource management | Better worker limits, queue size tuning, timeout enforcement | | API ergonomics | Convenience wrappers, batch operations, better handle management | | Security hardening | Privilege model improvements, input validation | ### v1.9 Features | Feature | Function/Type | |---------|---------------| | Worker labels | `label` parameter on `launch_v2`/`submit_v2` | | Structured errors | `pg_background_error_info_v2()`, `pg_background_error` type | | Result metadata | `pg_background_result_info_v2()`, `pg_background_result_info` type | | Batch operations | `pg_background_detach_all_v2()`, `pg_background_cancel_all_v2()` | ### v1.10 Features (Current) | Feature | Function/Type | |---------|---------------| | Convenience views | `pg_background_list`, `pg_background_activity` (joins `pg_stat_activity`) | | Never-raises status | `pg_background_outcome_v2()`, `pg_background_outcome` type | | Synchronous one-shot | `pg_background_run_v2()`, `pg_background_run_result` type | ### Bad fit for pg_background | Feature Type | Why it doesn't belong | |--------------|----------------------| | Full job scheduler | Use pg_cron; pg_background is for ad-hoc async execution | | Distributed queue | Application-layer concern; adds complexity and dependencies | | Workflow orchestration | Out of scope; pg_background executes SQL, not workflows | | Cross-database execution | PostgreSQL limitation; use dblink within workers if needed | | Persistent job storage | Requires tables, state management; not extension's purpose | | Retry logic | Application-layer concern; extension provides primitives | | Result caching | Complicates semantics; results are intentionally one-time | ### Gray areas (discuss before implementing) - Worker pools with pre-forked processes - Priority queues for worker scheduling - Cross-session worker visibility - Automatic cleanup policies - Integration hooks for external monitoring --- ## Build Commands ```bash # Build (requires PostgreSQL dev headers, pg_config in PATH) make clean && make # Install (requires appropriate privileges) sudo make install # Run regression tests make installcheck # Clean test artifacts make installcheckclean # Docker-based testing (no local PostgreSQL required) ./scripts/test-local.sh # Test with PostgreSQL 17 (default) ./scripts/test-local.sh 14 # Test with specific version ./scripts/test-local.sh all # Test all supported versions (14-19) # Upgrade path testing ./scripts/test-upgrade.sh # Test 1.8 → 1.9 → 1.10 upgrade path on PG 17 ./scripts/test-upgrade.sh 16 # Test upgrade on specific PG version ``` --- ## Architecture Quick Reference ``` Launcher Session Background Worker | | | pg_background_launch_v2() | | - Allocate DSM segment | | - Write SQL, GUCs, metadata | | - RegisterDynamicBackgroundWorker() | - Wait for shm_mq attach | | | |<---- (pid, cookie) handle --------| | | | Worker starts: | - Attach DSM | - Connect to database | - SPI_execute(SQL) | - Stream results via shm_mq | - Exit (auto-commit) | | | pg_background_result_v2() | | - Read from shm_mq | | - Return result rows | | | | pg_background_detach_v2() | | - Remove from tracking hash | | - DSM cleanup | v v ``` ### Key Files | File | Purpose | |------|---------| | `src/pg_background.c` | Launcher-side C implementation | | `src/pg_background_worker.c` | Worker-process C implementation (worker_main, execute_sql_string, error_exit) | | `src/pg_background.h` | Version compatibility macros (public to other modules) | | `src/pg_background_internal.h` | Cross-file declarations between launcher and worker | | `pg_background.control` | Extension metadata (version 1.10) | | `extension/pg_background--1.10.sql` | Current version install script | | `extension/pg_background--1.9--1.10.sql` | Upgrade from 1.9 | | `extension/pg_background--1.9.sql` | 1.9 install script (kept for installs that pin 1.9) | | `extension/pg_background--1.8--1.9.sql` | Upgrade from 1.8 | | `extension/pg_background--1.8.sql` | 1.8 install script | | `extension/legacy/` | Pre-1.8 base + upgrade scripts kept so older installs can still upgrade to 1.10 | | `sql/pg_background.sql` | Regression tests | | `expected/pg_background.out` | Expected test output | | `scripts/test-local.sh` | Docker-based multi-version testing | | `scripts/test-upgrade.sh` | Docker-based upgrade path testing | | `scripts/test-relocatable.sh` | Docker-based relocatable-schema test | | `scripts/test-assert.sh` | Docker-based assert-enabled PG test | | `docs/CONTRIBUTING.md`, `docs/SECURITY.md`, `docs/CI.md` | Contributor / ops docs | | `windows/pg_background_win.h` | Windows DLL symbol-export shim | | `.github/workflows/ci.yml` | CI pipeline (test matrix, relocatable, upgrade) | ## 11. Incremental Prompt Mode (AI Workflow Contract) This repository supports **incremental prompt mode** to reduce repetitive instructions. When a prompt contains only: - GitHub Copilot feedback - CI failures - Regression diffs - Small feature requests - Documentation fixes Claude should automatically: - Apply all rules from this `CLAUDE.md` - Treat the prompt as **delta-only instructions** - Avoid requiring repeated instructions ### Default Workflow for Incremental Prompts When receiving incremental prompts, Claude should automatically: #### 1. Validate Suggestions First - Validate GitHub Copilot feedback against actual repository context - Validate CI failure root cause before changing code - Validate regression diffs before changing expected output - Confirm SQL/C/test/doc consistency before applying fixes Do **not** blindly apply suggestions. #### 2. Prefer Minimal Practical Fixes - Avoid large refactors unless explicitly requested - Avoid unrelated cleanup - Prefer small, targeted, maintainable fixes - Preserve existing behavior unless a behavior change is intentional and documented #### 3. Check Nearby Similar Areas When fixing one issue: - Inspect nearby code for the same issue pattern - Fix similar occurrences when low risk and clearly useful - Avoid broad invasive changes #### 4. Maintain Cross-Component Consistency When modifying code, keep these aligned: - C code - SQL install scripts - SQL upgrade scripts - Regression tests - Expected output files - Docker tests - CI workflows - README and other docs #### 5. Run Required Validation After changes, run what is relevant: - `make` - `make installcheck` - Docker/local test scripts if affected - Upgrade tests if affected - Relocatable tests if affected - CI-equivalent checks if practical Be explicit about what was actually run versus only inspected. #### 6. Keep Documentation Current Update when necessary: - `README.md` - `CLAUDE.md` - `.github/copilot-instructions.md` Documentation must match actual behavior. #### 7. Branch Discipline - Stay on the current branch - Do **not** create a new branch unless explicitly requested - Keep changes logically grouped and reviewable ### Prompt Contract When a user provides a short prompt with only incremental items, assume: - `CLAUDE.md` is the standing base instruction set - The prompt contains only the new delta - All repository rules apply automatically ### Example Incremental Prompt Example minimal prompt: ```text Review the following Copilot feedback: 1. ... 2. ... 3. ... ``` Fix where appropriate. Claude should automatically: - Validate feedback - Apply minimal fixes - Update tests if needed - Update docs if needed - Run validation #### Example CI Failure Prompt ```text Fix the following CI failure: ``` Claude should automatically: - Identify root cause - Apply minimal correct fix - Update tests/docs if required - Validate the result #### Example Regression Diff Prompt ```text Fix regression diff: ``` Claude should automatically: - Determine whether code or expected output is wrong - Apply the minimal correct fix - Validate with tests #### Default Response Structure (When Useful) When providing a structured response, use: 1. Current State 2. Review of Issues 3. Fix Strategy 4. Changes Made 5. Files Updated 6. Validation Results 7. Remaining Follow-Up ### Final Rule For incremental prompts, Claude should assume: - Base rules come from `CLAUDE.md` - Only the new issue needs to be provided in the prompt - Repeated instructions should not be required pg_background-2.0.2/LICENSE000077500000000000000000000020601521265151100153250ustar00rootroot00000000000000PostgreSQL License Copyright (c) 2014-2026, Vibhor Kumar and contributors Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group Permission to use, copy, modify, and distribute this software and its documentation for any purpose, without fee, and without a written agreement is hereby granted, provided that the above copyright notice and this paragraph and the following two paragraphs appear in all copies. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION, EVEN IF THE COPYRIGHT HOLDER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. THE COPYRIGHT HOLDER SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS ON AN "AS IS" BASIS, AND THE COPYRIGHT HOLDER HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. pg_background-2.0.2/META.json000066400000000000000000000035021521265151100157400ustar00rootroot00000000000000{ "name": "pg_background", "abstract": "Run SQL queries in background workers with autonomous transactions", "description": "pg_background is a PostgreSQL extension that executes SQL commands in background worker processes. Workers run inside the PostgreSQL server with their own transactions, enabling asynchronous SQL execution without blocking the client session, autonomous transactions that commit or roll back independently of the caller, and an observable worker lifecycle with explicit launch, wait, cancel, and detach semantics. Includes a v2 cookie-protected API, structured error returns, worker labels, batch operations, and convenience helpers like pg_background_run_v2() and pg_background_outcome_v2().", "version": "2.0.0", "maintainer": [ "Vibhor Kumar " ], "license": "postgresql", "provides": { "pg_background": { "abstract": "Run SQL queries in background workers", "file": "pg_background--2.0.sql", "docfile": "README.md", "version": "2.0.0" } }, "prereqs": { "runtime": { "requires": { "PostgreSQL": "14.0.0" } } }, "resources": { "homepage": "https://github.com/vibhorkum/pg_background", "bugtracker": { "web": "https://github.com/vibhorkum/pg_background/issues" }, "repository": { "url": "git://github.com/vibhorkum/pg_background.git", "web": "https://github.com/vibhorkum/pg_background", "type": "git" } }, "generated_by": "Vibhor Kumar", "meta-spec": { "version": "1.0.0", "url": "https://pgxn.org/meta/spec.txt" }, "tags": [ "background", "worker", "asynchronous", "autonomous transaction", "background worker", "async", "spi" ] } pg_background-2.0.2/Makefile000077500000000000000000000033261521265151100157660ustar00rootroot00000000000000MODULE_big = pg_background OBJS = src/pg_background.o src/pg_background_worker.o EXTENSION = pg_background # Allow C sources in src/ to find local headers and the Windows shim header # (windows/pg_background_win.h is included as "pg_background_win.h"). PG_CPPFLAGS += -I$(srcdir)/src -I$(srcdir)/windows # Ship the 2.0 base script plus the upgrade scripts. # # We deliberately do NOT ship the pre-2.0 *base* install scripts # (1.8/1.9/1.10.sql): 2.0 dropped the v1 C functions, so those scripts # (which CREATE FUNCTION pg_background_launch ... LANGUAGE C) cannot resolve # their symbols against the 2.0 .so and a fresh `CREATE EXTENSION VERSION # '1.8'` would fail. Existing pre-2.0 installs upgrade via the --X--Y scripts # below (PostgreSQL only needs the upgrade scripts, not the old base scripts, # to migrate an installed extension). Anyone on a pre-1.8 install must first # reach 1.8 on the 1.10 release line before moving to 2.0. DATA = \ extension/pg_background--2.0.sql \ extension/pg_background--1.10--2.0.sql \ extension/pg_background--1.9--1.10.sql \ extension/pg_background--1.8--1.9.sql # Regression REGRESS = pg_background # Note: The test SQL file handles CREATE EXTENSION itself, # so we don't use --load-extension here. # REGRESS_OPTS = --load-extension=$(EXTENSION) # If your regression needs longer than default (yours has pg_sleep), # you can tune timeouts via PGOPTIONS if needed. # Example: # REGRESS_OPTS += --launcher="env PGOPTIONS='-c statement_timeout=0'" PG_CONFIG ?= pg_config PGXS := $(shell $(PG_CONFIG) --pgxs) include $(PGXS) .PHONY: test installcheckclean test: installcheck # Sometimes tmp_check residue causes confusion during iteration installcheckclean: rm -rf tmp_check regression.diffs results pg_background-2.0.2/README.md000066400000000000000000002170341521265151100156050ustar00rootroot00000000000000# pg_background: Production-Grade Background SQL for PostgreSQL [![PostgreSQL](https://img.shields.io/badge/PostgreSQL-14--19-blue.svg)](https://www.postgresql.org/) [![Version](https://img.shields.io/badge/version-2.0-brightgreen.svg)](https://github.com/vibhorkum/pg_background) [![License](https://img.shields.io/badge/license-PostgreSQL-green.svg)](LICENSE) [![CI](https://github.com/vibhorkum/pg_background/actions/workflows/ci.yml/badge.svg)](https://github.com/vibhorkum/pg_background/actions/workflows/ci.yml) Execute arbitrary SQL commands in **background worker processes** within PostgreSQL. Built for production workloads requiring asynchronous execution, autonomous transactions, and long-running operations without blocking client sessions. ### 30-second tour ```sql CREATE EXTENSION pg_background; -- Simplest case: run something in an autonomous transaction, get the outcome. SELECT completed, has_error, sqlstate, error_message, row_count, command_tag, elapsed_ms FROM pg_background_run( 'INSERT INTO audit_log (ts, who) VALUES (now(), current_user)', queue_size := 0, timeout_ms := 30000, label := 'audit-login' ); -- See every worker tracked by this session. SELECT pid, state, label, sql_preview FROM pg_background_list; ``` When you need the actual result rows, swap `run` for the `launch → wait → result` pattern shown in [Quick Start](#quick-start) or [Cookbook recipe 2](#cookbook). > **Naming:** 2.0 retired the `_v2` suffix — the unsuffixed names shown here > are canonical. The `_v2` names (e.g. `pg_background_run_v2`) still work as > deprecated aliases through the 2.x line and are removed in 3.0. See > [`docs/MIGRATION.md`](docs/MIGRATION.md). **Where to go next** | If you want to… | Read | |---|---| | See it in 5 minutes | [Quick Start](#quick-start) | | Copy a working pattern | [Cookbook](#cookbook) — three battle-tested templates | | Look up a function | [API Reference](#complete-api-reference) | | Understand the `cancel` vs `detach` distinction | [Critical Semantic Distinctions](#critical-semantic-distinctions) | | Decide whether this fits your problem | [When to use this — and when not to](#when-to-use-this--and-when-not-to) | --- ## Table of Contents - [Overview](#overview) - [When to use this — and when not to](#when-to-use-this--and-when-not-to) - [Architecture (at a glance)](#architecture-at-a-glance) · deep dive: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) - [Key Features](#key-features) - [PostgreSQL Version Compatibility](#postgresql-version-compatibility) - [Installation](#installation) - [Quick Start](#quick-start) - [Complete API Reference](#complete-api-reference) - [Critical Semantic Distinctions](#critical-semantic-distinctions) - Cancel vs Detach · V1 vs V2 (v1 removed in 2.0) · PID reuse · NOTIFY semantics - [Security Model](#security-model) - [Operational Guidance](#operational-guidance) - [Troubleshooting](#troubleshooting) - [Known Limitations](#known-limitations) - [Best Practices](#best-practices) - [Cookbook](#cookbook--see-docscookbookmd) — full content in [`docs/COOKBOOK.md`](docs/COOKBOOK.md) - [Migration Guide](#migration-guide--see-docsmigrationmd) — full content in [`docs/MIGRATION.md`](docs/MIGRATION.md) - [Testing](#testing) - [Contributing](#contributing) · [License](#license) · [Author](#author) --- ## Overview `pg_background` enables PostgreSQL to execute SQL commands asynchronously in dedicated background worker processes. Unlike `dblink` (which creates a separate connection) or client-side async patterns, `pg_background` workers run **inside** the database server with full access to local resources while operating in **independent transactions**. **Production-Critical Benefits:** - **Non-blocking operations**: Launch long-running queries without holding client connections - **Autonomous transactions**: Commit/rollback independently of the caller's transaction - **Resource isolation**: Workers have their own memory context and error handling - **Observable lifecycle**: Track, cancel, and wait for completion with explicit operations - **Security-hardened**: NOLOGIN role-based access, SECURITY DEFINER helpers, no PUBLIC grants **Typical Production Use Cases:** - Background maintenance (VACUUM, ANALYZE, REINDEX) - Asynchronous audit logging - Long-running ETL pipelines - Independent notification delivery - Parallel query pattern implementation --- ## When to use this — and when not to ### Good fit ✅ - **Autonomous transactions** — log audit events, send notifications, or update counters that must commit even if the parent transaction rolls back. - **Ad-hoc async maintenance** — kick off a `VACUUM`, `REINDEX`, or backfill from a SQL session without blocking it. - **Pre-known fan-out** — split a workload into N independent SQL statements and gather their outcomes (see [Cookbook recipe 3](#cookbook)). - **Bounded long-running queries** with a deadline — `pg_background_run(sql, queue_size, timeout_ms, label)` gives you a single SQL call with timeout and cancel-on-overrun. ### Not a fit ❌ | You want… | Use instead | |---|---| | A cron-style job scheduler | [`pg_cron`](https://github.com/citusdata/pg_cron) | | Cross-server SQL execution | `dblink` or `postgres_fdw` | | Cross-database execution | Workers are per-database; use `dblink` from inside a worker if you must | | Workflow orchestration with retries / DAGs | An application-layer job runner | | Persistent job queue with state across restarts | A real queue (Redis, RabbitMQ) or table-backed queue with explicit polling | | Result caching / re-fetching | Workers stream results once; persist them to a table yourself | `pg_background` provides primitives, not orchestration. If you need durable queueing, retries, scheduling, or coordination across sessions, build it on top — or use a tool that specializes in it. ### Side-by-side: `pg_background` vs neighboring tools A 30-second decision table. Pick the row that matches your job, not the column you've used before. | Capability | `pg_background` | `pg_cron` | `dblink` | `postgres_fdw` | |---|---|---|---|---| | Run SQL **in the background**, in the same db, in its own transaction | ✅ | ❌ runs on a schedule | ❌ runs in caller's flow | ❌ runs in caller's flow | | **Autonomous transactions** (commit independently of caller) | ✅ | ✅ | ✅ (separate connection) | ❌ | | **Scheduled / cron-style** execution | ❌ | ✅ | ❌ | ❌ | | Run SQL on a **different host** | ❌ | ❌ | ✅ | ✅ | | Run SQL in a **different database** of same cluster | ❌ | ✅ (per-db jobs) | ✅ | ✅ | | **Cookie-protected** lifecycle (cancel/wait/list with PID-reuse safety) | ✅ | ❌ | ❌ | ❌ | | **Structured error returns** (real SQLSTATE + detail/hint/context) | ✅ | partial | partial | partial | | Persistent job state / **survives restart** | ❌ session-local | ✅ | ❌ | ❌ | | **DAG / retry / dependency** orchestration | ❌ | ❌ | ❌ | ❌ | **Common patterns** - **Audit logging that must commit even on rollback** → `pg_background_run` with `submit`-style fire-and-forget. Don't use `dblink` (callable but heavier per call). - **Nightly maintenance at 02:00** → `pg_cron`. Don't use `pg_background` (no scheduler). - **Read from another host's table** → `postgres_fdw`. Don't use `pg_background` (single-host). - **Synchronous fan-out: launch N updates, wait for all** → `pg_background_drain`. Don't use `dblink` (no batch primitive). - **Cancel a long-running job from another session** — none of these tools is great. `pg_background_cancel` works but only from the launching session today; cluster-wide cancel needs a manual `pg_cancel_backend` against the worker PID. --- ## Architecture (at a glance) The launcher allocates a DSM segment, registers a dynamic background worker, and waits for it to attach a shared-memory queue. The worker restores the launcher's GUCs, runs the SQL via SPI, streams rows back through the queue, and writes structured metadata (row count, command tag, error fields) into a launcher-readable struct in DSM. The launcher consumes rows via `pg_background_result` and tears down through `pg_background_detach`; an `on_dsm_detach` callback ensures leaks are impossible even on abnormal exit. The full sequence diagram, TOC layout, concurrency-race history, and publish-flag patterns are in **[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)**. --- ## Key Features ### Core capabilities - **Async SQL execution** — offload queries to background workers running inside the server - **Autonomous transactions** — workers commit (or roll back) independently of the caller - **Explicit lifecycle** — `launch`, `wait`, `cancel`, `detach`, and `list` operations with documented semantics - **Cookie-protected handles** — `(pid, cookie)` tuples prevent PID-reuse confusion in long-lived sessions - **Structured error reporting** — real `SQLSTATE`, message, detail, hint, and context propagated from worker to launcher - **Observability built in** — per-session worker registry (`pg_background_list`), counters (`pg_background_stats`), progress reporting, optional labels - **Hardened security** — NOLOGIN executor role, no PUBLIC grants, privilege helpers with pinned `search_path` - **Relocatable** — `CREATE EXTENSION pg_background WITH SCHEMA myschema` works fully ### What's new in v2.0 (major release) **Breaking changes** — see [`docs/MIGRATION.md`](docs/MIGRATION.md) for the full upgrade path: - **The `_v2` suffix is retired.** With v1 gone, the suffix no longer distinguishes anything, so the **unsuffixed names are now canonical** (`pg_background_launch`, `pg_background_wait`, `pg_background_run`, …). Every `_v2` name that shipped through 1.10 is kept as a thin **deprecated alias** (identical behavior; forwards to the canonical function) and will be **removed in 3.0**. No code change is required to upgrade — migrate at your own pace. `pg_background_list` (view) / `pg_background_stats` / `pg_background_outcome` (types) coexist with same-named functions, resolved by call syntax. - The v1 API (`pg_background_launch(sql, queue_size)` returning `int4`, `pg_background_result(pid)`, `pg_background_detach(pid)`) was **removed**. The unsuffixed names are now the cookie-protected API: `pg_background_launch` returns a `pg_background_handle` (pid + cookie), not a bare `int4`. - `pg_background_cancel_v2_grace` and `pg_background_wait_v2_timeout` were **collapsed** into the base functions with a third `grace_ms` / `timeout_ms` argument that defaults to 0. - `pg_background_wait` now returns `bool` (was `void`); `timeout_ms <= 0` blocks indefinitely (matches the 1.x default), `> 0` waits up to N ms. - `pg_background_status_v2` was removed — call `to_jsonb(pg_background_outcome(...))` directly. - `pg_background_progress` (function) renamed to **`pg_background_report_progress`** (hard rename, no alias); type `pg_background_progress` renamed to **`pg_background_progress_info`**. - Privilege helpers renamed (hard rename, no alias): **`pg_background_grant_privileges`** / **`pg_background_revoke_privileges`**. **Forward-compatibility additions** — adding columns later is painful, so 2.0 widens the composite types now: - `pg_background_stats` gains `workers_timed_out int8` (separate from `workers_canceled`; bumped by `pg_background_run` on timeout). - `pg_background_result_info` gains `started_at`, `finished_at` (timestamptz) — the worker writes these around its SPI loop. - `pg_background_error` gains `schema_name`, `table_name`, `column_name`, `constraint_name` — sourced from PG's `edata` for heap/access errors. - `pg_background_run_result` now **extends** `pg_background_outcome` (gains `cookie`, `state`, `consumed`, `label`, `launched_at`) plus `timed_out` + `elapsed_ms`. No more duplicate column shape. **Internal**: 2.0 prunes pre-1.8 upgrade scripts (`extension/legacy/` is gone). Anyone on a pre-1.8 install must reach 1.8 against the 1.10 release line first. ### Earlier milestones - **v1.10**: `pg_background_list` / `pg_background_activity` views; `pg_background_outcome` (never-raises status snapshot); `pg_background_run` synchronous one-shot. - **v1.9**: worker labels, structured errors (`pg_background_error_info`), result metadata (`pg_background_result_info`), batch ops (`detach_all`, `cancel_all`). - **v1.8**: session statistics, progress reporting, GUCs (`pg_background.max_workers`, `default_queue_size`, `worker_timeout`). - **v1.6**: cryptographically secure cookies for PID-reuse protection. The full chain is documented in [`docs/MIGRATION.md`](docs/MIGRATION.md). --- ## PostgreSQL Version Compatibility | PostgreSQL Version | Support Status | Notes | |--------------------|----------------|-------| | **19** | 🧪 Beta Support | Validated against 19beta1; explicit proc.h / latch.h / wait_event.h includes | | **18** | ✅ Fully Supported | TupleDescAttr compatibility layer | | **17** | ✅ Fully Tested | Recommended for new deployments | | **16** | ✅ Fully Tested | Production-ready | | **15** | ✅ Fully Tested | pg_analyze_and_rewrite_fixedparams | | **14** | ✅ Fully Tested | Minimum supported version | | **13** | ❌ Not Supported | Use pg_background 1.6 or earlier | | **< 13** | ❌ Not Supported | Use pg_background 1.4 or earlier | **Note**: Each PostgreSQL major version requires extension rebuild against its headers. --- ## Installation ### Prerequisites - PostgreSQL 14+ with development headers (`postgresql-server-dev-*` or `postgresql##-devel`) - `pg_config` in `$PATH` - Build essentials: `gcc`, `make` - Superuser privileges for `CREATE EXTENSION` ### Build from Source ```bash # Clone repository git clone https://github.com/vibhorkum/pg_background.git cd pg_background # Build extension make clean make # Install (requires appropriate privileges) sudo make install ``` ### Enable Extension ```sql -- Connect as superuser CREATE EXTENSION pg_background; -- Verify installation SELECT extname, extversion FROM pg_extension WHERE extname = 'pg_background'; -- Expected output: -- extname | extversion -- ---------------+------------ -- pg_background | 1.10 ``` ### Library Loading `pg_background` does **not** require `shared_preload_libraries`. Workers are registered dynamically (`RegisterDynamicBackgroundWorker`) and each worker process loads the library dynamically when it starts. Adding `pg_background` to `shared_preload_libraries` is **optional** and only needed if you want the extension's GUC parameters (`pg_background.max_workers`, `pg_background.default_queue_size`, `pg_background.worker_timeout`) available in `postgresql.conf` and visible in all sessions from the start. Without SPL, the GUCs are registered on first use (`CREATE EXTENSION`, `LOAD`, or the first `launch` call). A session `SET` before that point raises an `unrecognized configuration parameter` error. The warning behavior applies to configuration file entries (for example, `postgresql.conf` or `ALTER SYSTEM`) that are read before the library is loaded. | | Without SPL | With SPL | |---|---|---| | Extension works? | Yes | Yes | | GUCs in `postgresql.conf` | Not until first load | Immediately | | After `make install` | Workers pick up new `.so` automatically | **Restart required** (postmaster caches the library) | | Recommended for | Development, staging, simple setups | Production with tuned GUCs | ### Custom Schema Installation The extension is **relocatable**, allowing installation in any schema. This is useful for organizing extensions or avoiding namespace conflicts. ```sql -- Create custom schema CREATE SCHEMA contrib; -- Install extension in custom schema CREATE EXTENSION pg_background WITH SCHEMA contrib; -- Verify installation SELECT extname, extversion, nspname AS schema FROM pg_extension e JOIN pg_namespace n ON n.oid = e.extnamespace WHERE e.extname = 'pg_background'; -- Expected output: -- extname | extversion | schema -- ---------------+------------+--------- -- pg_background | 1.10 | contrib ``` **Using Extension in Custom Schema**: When installed in a custom schema, functions can be called with schema qualification or by adding the schema to `search_path`: ```sql -- Option 1: Schema-qualified calls SELECT * FROM contrib.pg_background_launch('SELECT 1') AS h; SELECT * FROM contrib.pg_background_result(h.pid, h.cookie) AS (result int); -- Option 2: Add schema to search_path SET search_path = contrib, public; SELECT * FROM pg_background_launch('SELECT 1') AS h; ``` **Privileges with Custom Schema**: The privilege helper functions automatically detect the extension's schema: ```sql -- Grant privileges (works regardless of installation schema) SELECT contrib.pg_background_grant_privileges('app_user', true); -- Or if schema is in search_path SELECT pg_background_grant_privileges('app_user', true); ``` **Test Cases for Custom Schema Installation**: ```sql -- Test 1: Basic installation in custom schema CREATE SCHEMA test_schema; CREATE EXTENSION pg_background WITH SCHEMA test_schema; -- Test 2: Launch worker from custom schema SELECT (h).pid, (h).cookie FROM test_schema.pg_background_launch('SELECT 42') AS h \gset -- Test 3: Retrieve results SELECT * FROM test_schema.pg_background_result(:pid, :cookie) AS (val int); -- Expected: val = 42 -- Test 4: Privilege helpers work with custom schema CREATE ROLE test_user NOLOGIN; SELECT test_schema.pg_background_grant_privileges('test_user', true); -- Should output GRANT statements with test_schema prefix -- Test 5: Revoke privileges SELECT test_schema.pg_background_revoke_privileges('test_user', true); -- Test 6: V2 types are accessible SELECT (ROW(123, 456789)::test_schema.pg_background_handle).*; -- Expected: pid=123, cookie=456789 -- Cleanup DROP ROLE test_user; DROP EXTENSION pg_background; DROP SCHEMA test_schema; ``` ### Configure PostgreSQL ```sql -- Set worker process limit (adjust based on your workload) ALTER SYSTEM SET max_worker_processes = 32; -- Reload configuration SELECT pg_reload_conf(); -- Verify setting SHOW max_worker_processes; ``` ### Extension GUC Settings (v1.8+) ```sql -- Limit concurrent workers per session (default: 16) SET pg_background.max_workers = 10; -- Set default queue size for workers (default: 64KB) SET pg_background.default_queue_size = '256KB'; -- Set worker execution timeout (default: 0 = no limit) SET pg_background.worker_timeout = '5min'; ``` | GUC Parameter | Default | Range | Description | |---------------|---------|-------|-------------| | `pg_background.max_workers` | 16 | 1-1000 | Max concurrent workers per session | | `pg_background.default_queue_size` | 65536 | 4KB-256MB | Default shared memory queue size | | `pg_background.worker_timeout` | 0 | 0-∞ | Worker execution timeout (0 = no limit) | --- ## Quick Start ### V2 API (Recommended) The v2 API provides cookie-based handle protection and explicit lifecycle semantics. If you only ever read one section, **use `pg_background_run()`** (item 0 below) — it covers the common case in one SQL call. #### 0. Easiest path: synchronous one-shot (`pg_background_run`) Use this when you want autonomous-transaction semantics and just need to know whether the SQL succeeded, how many rows it affected, and the SQLSTATE if it failed. Returns metadata only — no result rows. ```sql SELECT completed, has_error, sqlstate, error_message, row_count, command_tag, elapsed_ms, timed_out FROM pg_background_run( 'INSERT INTO audit_log (ts, who) VALUES (now(), current_user)', queue_size := 0, timeout_ms := 30000, -- 30 s cap; cancels with 1 s grace on overrun label := 'audit-login' ); -- completed | has_error | sqlstate | error_message | row_count | command_tag | elapsed_ms | timed_out -- t | f | NULL | NULL | 1 | INSERT 0 1 | 14 | f ``` When you actually need **result rows**, use the launch + wait + result pattern (items 1, 2, 5 below) or jump straight to [Cookbook recipe 2](#cookbook). #### 1. Launch a Background Job ```sql -- Launch worker and capture handle SELECT * FROM pg_background_launch( 'SELECT pg_sleep(5); SELECT count(*) FROM large_table' ) AS handle; -- Output: -- pid | cookie -- -------+------------------- -- 12345 | 1234567890123456 ``` #### 2. Retrieve Results ```sql -- Results can only be consumed ONCE SELECT * FROM pg_background_result(12345, 1234567890123456) AS (count BIGINT); -- Attempting second retrieval will error: -- ERROR: PID 12345 is not attached to this session -- (auto-detach happens after the first successful consumption) ``` #### 3. Fire-and-Forget (Submit) ```sql -- For queries with side effects only (no result consumption needed) SELECT * FROM pg_background_submit( 'INSERT INTO audit_log (ts, event) VALUES (now(), ''system_check'')' ) AS handle; -- Worker commits and exits automatically ``` #### 4. Cancel a Running Job ```sql -- Request immediate cancellation SELECT pg_background_cancel(pid, cookie); -- Or with grace period (500ms to finish current statement) SELECT pg_background_cancel(pid, cookie, 500); ``` ⚠️ **Windows Limitation**: Cancel on Windows only sets interrupts; it cannot terminate an actively running statement. Always use `statement_timeout` on Windows. #### 5. Wait for Completion ```sql -- Block until worker finishes SELECT pg_background_wait(pid, cookie); -- Or wait with timeout (returns true if completed) SELECT pg_background_wait(pid, cookie, 5000); -- 5 seconds ``` #### 6. List Active Workers ```sql SELECT * FROM pg_background_list() AS ( pid int4, cookie int8, launched_at timestamptz, user_id oid, queue_size int4, state text, sql_preview text, last_error text, consumed bool ) ORDER BY launched_at DESC; ``` **State Values**: - `running`: Actively executing SQL - `stopped`: Completed successfully - `canceled`: Terminated via `cancel()` - `error`: Failed with error (see `last_error`) #### 7. View Session Statistics (v1.8+) ```sql -- Get session-wide worker statistics SELECT * FROM pg_background_stats(); -- Output: -- workers_launched | workers_completed | workers_failed | workers_active | avg_execution_ms | max_workers -- ------------------+-------------------+----------------+----------------+------------------+------------- -- 42 | 38 | 2 | 2 | 1234.5 | 16 ``` #### 8. Progress Reporting (v1.8+) **From within worker SQL** (report progress): ```sql -- Launch a worker that reports progress SELECT * FROM pg_background_launch($$ SELECT pg_background_report_progress(0, 'Starting...'); -- Do some work... SELECT pg_background_report_progress(25, 'Phase 1 complete'); -- More work... SELECT pg_background_report_progress(50, 'Halfway done'); -- Final work... SELECT pg_background_report_progress(100, 'Complete'); $$) AS h \gset; ``` **From launcher** (check progress): ```sql -- Poll worker progress SELECT * FROM pg_background_get_progress(:'h.pid', :'h.cookie'); -- Output: -- progress_pct | progress_msg -- --------------+--------------- -- 50 | Halfway done ``` ## Complete API Reference ### Core functions > **Canonical names (2.0):** the suffix `_v2` was retired. The table below > uses the **canonical, unsuffixed** names. Every name that shipped through > 1.10 also has a deprecated `_v2` alias (identical behavior; e.g. > `pg_background_launch_v2` still calls `pg_background_launch`) kept through > the 2.x line and **removed in 3.0** — migrate at your own pace. Functions > new in 2.0 (`pg_background_report_progress`, the privilege helpers) only > ever have the unsuffixed name. | Function | Returns | Description | Use Case | |----------|---------|-------------|----------| | `pg_background_launch(sql, queue_size, label)` | `pg_background_handle` | Launch worker with optional label (v1.9) | Standard async execution | | `pg_background_submit(sql, queue_size, label)` | `pg_background_handle` | Fire-and-forget with optional label (v1.9) | Side-effect queries | | `pg_background_result(pid, cookie)` | `SETOF record` | Retrieve results (**one-time consumption**) | Collect query output | | `pg_background_result_info(pid, cookie)` | `pg_background_result_info` | Get result metadata (v1.9) | Check completion without consuming | | `pg_background_error_info(pid, cookie)` | `pg_background_error` | Get structured error details (v1.9) | Error diagnostics | | `pg_background_detach(pid, cookie)` | `void` | Stop tracking worker (worker continues) | Cleanup bookkeeping | | `pg_background_detach_all()` | `int4` | Detach all workers in session (v1.9) | Session cleanup | | `pg_background_cancel(pid, cookie, grace_ms DEFAULT 0)` | `void` | Cooperative cancel via SIGTERM; `grace_ms>0` also waits up to N ms (capped at 3600000) for the worker to stop. Never force-kills (see Limitation 10) | Terminate unwanted work | | `pg_background_cancel_all()` | `int4` | Cancel all workers in session (v1.9) | Emergency cleanup | | `pg_background_wait(pid, cookie, timeout_ms DEFAULT 0)` | `bool` | `timeout_ms<=0` blocks indefinitely (returns `true`); `>0` waits up to N ms (returns `true` if stopped, `false` on timeout) | Synchronous barrier / bounded wait | | `pg_background_list()` | `SETOF record` | List known workers in current session (column-definition list required) | Internal — prefer the view below | | `pg_background_stats()` | `pg_background_stats` | Session statistics | Monitoring, debugging | | `pg_background_report_progress(pct, msg)` | `void` | Report progress from worker (v2.0 rename of `pg_background_progress`; no `_v2` alias) | Long-running task feedback | | `pg_background_get_progress(pid, cookie)` | `pg_background_progress_info` | Get worker progress | Monitor long-running tasks | | `pg_background_outcome(pid, cookie)` | `pg_background_outcome` | Combined status snapshot — never raises | Safe status retrieval | | `pg_background_run(sql, queue_size, timeout_ms, label)` | `pg_background_run_result` | Synchronous one-shot: launch + wait + outcome + detach | Autonomous-transaction-style runs | | `pg_background_list` (view) | rows of `list()` | Convenience view; no column-definition list required | Day-to-day observation | | `pg_background_activity` (view) | join with `pg_stat_activity` | Worker registry + backend state in one query | Combined monitoring | | `pg_background_full_sql(pid, cookie)` | `text` | Full SQL the worker is running (capped at 64 KiB) | Debugging beyond the 120-char `sql_preview` | **Tier A "loop killers"** — convenience helpers built on the v2 primitives: | Function | Returns | Description | |---|---|---| | `pg_background_run_query(sql, queue_size, timeout_ms, label, col_def)` | `SETOF record` | Synchronous launch + wait + result + detach with rows; `col_def` matches the AS clause at the call site | | `pg_background_drain(handles, timeout_ms)` | `SETOF pg_background_outcome` | Wait for every handle (shared wall-clock budget), collect outcomes, detach | | `pg_background_wait_any(handles, timeout_ms)` | `pg_background_handle` | Return the first handle to finish (50–500 ms adaptive polling); NULL on timeout | | `pg_background_cancel_by_label(pattern, grace_ms)` | `int4` | Cancel every worker whose label matches the SQL `LIKE` pattern; returns count | | `pg_background_purge()` | `int4` | Detach only workers that have already stopped (vs `detach_all` which is unconditional) | **Parameters**: - `sql`: SQL command(s) to execute (multiple statements allowed) - `queue_size`: Shared memory queue size in bytes (default: 65536, min: 4096) - `pid`: Process ID from handle - `cookie`: Unique identifier from handle (prevents PID reuse) - `label`: Optional worker label for identification (v1.9, default: NULL) - `grace_ms`: Milliseconds to wait for the worker to stop cooperatively after SIGTERM (capped at 1 hour); the worker is never force-killed - `timeout_ms`: Milliseconds to wait for completion **Handle Type**: ```sql CREATE TYPE pg_background_handle AS ( pid int4, -- Process ID cookie int8 -- Unique identifier (prevents PID reuse) ); ``` **Statistics Type**: ```sql CREATE TYPE pg_background_stats AS ( workers_launched int8, -- Total workers launched this session workers_completed int8, -- Workers completed successfully workers_failed int8, -- Workers that failed with error workers_canceled int8, -- Workers explicitly canceled (v1.9) workers_timed_out int8, -- Workers canceled by run timeout (v2.0) workers_active int4, -- Currently active workers avg_execution_ms float8, -- Average execution time max_workers int4 -- Current max_workers setting ); ``` **Progress Type** (v2.0 renamed from `pg_background_progress`): ```sql CREATE TYPE pg_background_progress_info AS ( progress_pct int4, -- Progress percentage (0-100) progress_msg text -- Brief status message ); ``` **Result Info Type** (v1.9+): ```sql CREATE TYPE pg_background_result_info AS ( row_count int8, -- Number of rows returned/affected command_tag text, -- Command tag (SELECT, INSERT, etc.) completed bool, -- True if worker completed has_error bool -- True if SQL execution error was captured ); ``` > **Note**: `has_error` indicates SQL execution errors captured through structured error reporting. Early worker failures (e.g., resource exhaustion, connection issues) before SQL execution begins do not set this flag. The combination of `completed=true`, `has_error=false`, and `error_info() IS NULL` indicates likely success, but does not guarantee the worker completed without infrastructure-level failures. **Error Type** (v1.9+): ```sql CREATE TYPE pg_background_error AS ( sqlstate text, -- SQLSTATE error code (e.g., '23505') message text, -- Primary error message detail text, -- Detailed error info (if any) hint text, -- Hint for resolution (if any) context text -- Error context/stack trace ); ``` **Outcome Type** (v1.10+): ```sql CREATE TYPE pg_background_outcome AS ( pid int4, cookie int8, state text, -- starting/running/stopped/canceled/error (NULL if not in this session) consumed bool, completed bool, -- from result_info has_error bool, -- from result_info row_count int8, -- from result_info command_tag text, -- from result_info sqlstate text, -- from error_info error_message text, -- from error_info.message label text, launched_at timestamptz ); ``` `pg_background_outcome()` populates this snapshot by combining `pg_background_list`, `pg_background_result_info`, and `pg_background_error_info`. It catches all exceptions internally and leaves unavailable fields NULL — handy after `result` has consumed results, after a worker has been cleaned up, or when you simply do not want to write three nested `BEGIN ... EXCEPTION` blocks. **Run Result Type** (v1.10+): ```sql CREATE TYPE pg_background_run_result AS ( pid int4, completed bool, timed_out bool, has_error bool, row_count int8, command_tag text, sqlstate text, error_message text, elapsed_ms int8 ); ``` `pg_background_run(sql, queue_size, timeout_ms, label)` runs a single SQL command in a worker, waits up to `timeout_ms` (0 = wait forever), cancels the worker with 1 s grace if it does not finish in time, gathers the outcome, and detaches the handle. It returns metadata only — no result rows. Use the launch/wait/result pattern (see [Cookbook recipe 2](#cookbook)) when you need result rows. #### Structured Error Returns — SQLSTATE Semantics `pg_background_error_info(pid, cookie)` returns the **real** five-character `SQLSTATE` emitted by the worker's failed statement, not a synthesized `08006 "lost connection to worker process"`. The worker's `PG_CATCH` handler copies `ErrorData` from the caught `ereport(ERROR)`, stores the fields in DSM (with `error_sqlstate` written last as a publish flag) and calls `EmitErrorReport()` + `ReadyForQuery(DestRemote)` + `pq_flush()` so the launcher sees the actual `'E'` error frame over `shm_mq`. Typical codes returned end-to-end (v1.9+): | Trigger SQL | Returned `sqlstate` | Path | |--------------------------------------------------|---------------------|-----------------| | `SELECT 1/0` | `22012` | execute | | `RAISE EXCEPTION 'custom error'` | `P0001` | execute | | `INSERT NULL` into `NOT NULL` column | `23502` | execute | | `INSERT` violating `INITIALLY DEFERRED` FK | `23503` | commit | | `pg_background_cancel()` during `pg_sleep()` | `57014` | execute | **Recommended pattern**: call `error_info` from the same PL/pgSQL `EXCEPTION` block that observes the failure. Once the launcher's transaction aborts, `cleanup_worker_info` removes the hash entry and the next transaction will see `ERRCODE_UNDEFINED_OBJECT` ("PID N is not attached to this session"). ```sql DO $$ DECLARE h pg_background_handle; s text; BEGIN h := pg_background_launch('SELECT 1/0'); PERFORM pg_background_wait(h.pid, h.cookie); SELECT sqlstate INTO s FROM pg_background_error_info(h.pid, h.cookie); RAISE NOTICE 'worker sqlstate=%', s; -- 22012 PERFORM pg_background_detach(h.pid, h.cookie); END$$; ``` > **Important — do not call `result()` on an error path.** `result()` > re-raises the worker's error in the launcher via `ereport(ERROR)`, which > aborts the current transaction and triggers `cleanup_worker_info` before you > can inspect `error_info()`. For error diagnosis, the supported pattern is > `launch -> wait -> error_info -> detach` (no `result`). > **`08006` is now reserved for infra-level failures only.** The launcher > synthesizes `ERRCODE_CONNECTION_FAILURE "lost connection to worker process"` > only when the worker died before it could propagate a real error (see > [Known Limitations — Early worker failures](#9-early-worker-failures-before-pq_redirect_to_shm_mq)). > Under normal operation, any SQL-level error inside the worker surfaces as > the concrete SQLSTATE shown in the table above. ### Deployment Order The fix for end-to-end SQLSTATE propagation lives in the compiled `pg_background.so`. Whether a server restart is required depends on how the library is loaded (see [Library Loading](#library-loading)): - **With `shared_preload_libraries`**: the postmaster dlopens the library once at startup and every forked background worker inherits the cached handle. After replacing the `.so` on disk you must restart PostgreSQL — a plain `pg_reload_conf()` is not sufficient. - **Without SPL** (the default): each background worker dlopens the library in its own process, so a fresh `pg_background_launch(...)` call picks up the new binary automatically. No server restart is needed; at most, reconnect long-lived client sessions. 1. Build and install: `make clean && make && sudo make install`. 2. **Reload the library:** - SPL setup → `pg_ctl restart` / systemd / platform equivalent. - On-demand setup → no action required (optionally reconnect clients). 3. Verify on staging that real SQLSTATEs propagate: ```sql DO $$ DECLARE h pg_background_handle; s text; BEGIN h := pg_background_launch('SELECT 1/0'); PERFORM pg_background_wait(h.pid, h.cookie); SELECT sqlstate INTO s FROM pg_background_error_info(h.pid, h.cookie); ASSERT s = '22012', 'expected 22012, got ' || s; PERFORM pg_background_detach(h.pid, h.cookie); END$$; ``` 4. **Only after step 3 succeeds**, remove any PL/pgSQL workarounds that read `error_info` as a fallback after catching `08006`. Before the fix ships they were the only way to get a usable SQLSTATE; after the fix they become dead code, but keeping them in place during the rollout is harmless. ### Rollback Order To roll back to a pre-fix `.so` (for example if another extension in the same image regresses): 1. **First** restore the PL/pgSQL workarounds in user code (they expect `SQLERRM` to degrade to `08006` and then read `error_info` out of band). 2. **Only after step 1**, install the old `.so` and restart PostgreSQL. Doing rollback in the reverse order (old `.so` first) causes user functions to see raw `08006` errors without the fallback path, which can manifest as `WHEN others` branches swallowing what used to be diagnosable SQLSTATEs. ## Critical Semantic Distinctions ### Cancel vs Detach **These operations are NOT interchangeable.** Confusion between them is a common source of production issues. | Operation | Stops Execution | Prevents Commit | Removes Tracking | |-----------|-----------------|-----------------|------------------| | **`cancel()`** | ⚠️ Best-effort (immediate on Unix, limited on Windows) | ⚠️ Best-effort | ❌ No | | **`detach()`** | ❌ No | ❌ No | ✅ Yes | **Rule of Thumb**: - Use **`cancel()`** to **stop work** (terminate execution, prevent commit/notify) - Use **`detach()`** to **stop tracking** (free bookkeeping memory while worker continues) #### Example: Detach Does NOT Prevent NOTIFY ```sql -- Launch worker that sends notification SELECT * FROM pg_background_launch( $$SELECT pg_notify('alerts', 'system_event')$$ ) AS h \gset -- Detach only removes launcher's tracking SELECT pg_background_detach(:'h.pid', :'h.cookie'); -- Worker STILL runs and sends notification! -- To actually prevent notification, use: SELECT pg_background_cancel(:'h.pid', :'h.cookie'); ``` #### When to Use Each **Use `cancel()`**: - User-initiated cancellation - Timeout enforcement - Rollback of unwanted side effects - Immediate resource reclamation **Use `detach()`**: - Long-running maintenance (don't need to track VACUUM for hours) - Fire-and-forget after successful submission - Session cleanup before disconnect - Reducing launcher session memory usage ### V1 → V2 (historical context) The v1 API (bare `int4` PID, no cookie, no cancel/wait) was removed in 2.0. See [`docs/MIGRATION.md`](docs/MIGRATION.md) for the v1→v2 mapping table and a port-this-into-that example. ### PID Reuse Protection **The Problem**: Operating systems recycle process IDs. On busy systems, a PID can be reused within minutes. **Pre-2.0 risk** (the deleted v1 API used a PID-only reference): ```sql -- Day 1: Launch worker SELECT pg_background_launch('slow_query()') AS pid \gset -- Day 2: Session still alive, but worker PID may be reused -- This could attach to a DIFFERENT worker with the SAME PID! SELECT pg_background_result(:pid); -- ⚠️ DANGEROUS ``` **V2 API Fix** (PID + Cookie): ```sql -- Launch with cookie SELECT * FROM pg_background_launch('slow_query()') AS h \gset -- Days later: cookie validation prevents mismatch SELECT pg_background_result(:'h.pid', :'h.cookie'); -- If PID reused, cookie won't match → safe error ``` **Implementation**: Each worker generates a random 64-bit cookie at launch. All operations validate `(pid, cookie)` tuple matches. ### NOTIFY and Autonomous Commits Workers execute in **separate transactions** from the launcher. This has critical implications: #### Autonomous Transaction Behavior ```sql BEGIN; -- Launcher transaction starts SELECT * FROM pg_background_launch( 'INSERT INTO audit_log VALUES (now(), ''user_action'')' ) AS h \gset; -- Main work UPDATE users SET status = 'active' WHERE id = 123; -- If we ROLLBACK, the audit_log INSERT still commits! ROLLBACK; -- audit_log entry exists despite rollback ``` **Implications**: - ✅ **Good for**: Audit logging, NOTIFY, stats collection - ⚠️ **Bad for**: Interdependent data modifications requiring ACID #### NOTIFY Delivery with Detach ```sql -- Worker sends notification SELECT * FROM pg_background_launch( $$SELECT pg_notify('channel', 'message')$$ ) AS h \gset; -- Detach removes tracking but does NOT cancel SELECT pg_background_detach(:'h.pid', :'h.cookie'); -- Notification WILL be delivered (worker commits independently) ``` To **prevent** notification delivery: ```sql -- Cancel before worker commits SELECT pg_background_cancel(:'h.pid', :'h.cookie'); ``` --- ## Security Model ### Privilege Architecture `pg_background` uses a role-based security model with zero PUBLIC access by default. #### Default Setup (Automatic) ```sql -- Extension creates this role automatically: CREATE ROLE pgbackground_role NOLOGIN INHERIT; -- All pg_background functions granted to this role -- PUBLIC has NO access by default ``` #### Grant Access to Users ```sql -- Method 1: Direct role grant (recommended) GRANT pgbackground_role TO app_user; -- Method 2: Helper function (explicit EXECUTE grants) SELECT pg_background_grant_privileges('app_user', true); ``` #### Revoke Access ```sql -- Method 1: Revoke role membership REVOKE pgbackground_role FROM app_user; -- Method 2: Helper function SELECT pg_background_revoke_privileges('app_user', true); ``` ### Security Considerations #### 1. SQL Injection Prevention ❌ **Unsafe** (vulnerable to SQL injection): ```sql CREATE FUNCTION unsafe_launch(user_input text) RETURNS void AS $$ BEGIN -- NEVER concatenate untrusted input! PERFORM pg_background_launch( 'SELECT * FROM users WHERE name = ''' || user_input || '''' ); END; $$ LANGUAGE plpgsql; ``` ✅ **Safe** (parameterized with `format()`): ```sql CREATE FUNCTION safe_launch(user_input text) RETURNS void AS $$ BEGIN -- Use %L for literal quoting PERFORM pg_background_launch( format('SELECT * FROM users WHERE name = %L', user_input) ); END; $$ LANGUAGE plpgsql; ``` #### 2. Resource Exhaustion Protection ```sql -- Application-level quota enforcement CREATE OR REPLACE FUNCTION launch_with_limit(sql text) RETURNS pg_background_handle AS $$ DECLARE active_count int; h pg_background_handle; BEGIN -- Count active workers for current user SELECT count(*) INTO active_count FROM pg_background_list WHERE user_id = current_user::regrole::oid AND state IN ('running'); IF active_count >= 5 THEN RAISE EXCEPTION 'User worker limit exceeded (max 5 concurrent)'; END IF; SELECT * INTO h FROM pg_background_launch(sql); RETURN h; END; $$ LANGUAGE plpgsql SECURITY DEFINER; ``` #### 3. Privilege Isolation - ✅ Workers inherit **current_user** from launcher (not superuser escalation) - ✅ `SECURITY DEFINER` helpers use pinned `search_path = pg_catalog` - ✅ No ambient PUBLIC grants - ⚠️ Workers can access all databases launcher can access #### 4. Information Disclosure Risks ```sql -- list() exposes SQL previews (first 120 chars) and error messages -- For sensitive deployments, create restricted view: CREATE VIEW safe_worker_list AS SELECT pid, cookie, state, consumed, launched_at FROM pg_background_list WHERE user_id = current_user::regrole::oid; -- Omit sql_preview and last_error GRANT SELECT ON safe_worker_list TO app_users; ``` ### Security Best Practices 1. **Never grant `pgbackground_role` to PUBLIC** 2. **Use v2 API exclusively** (cookie protection) 3. **Set `statement_timeout`** to bound execution time 4. **Implement application-level quotas** (max workers per user/database) 5. **Sanitize all dynamic SQL** with `format()` or `quote_literal()` 6. **Monitor `list()`** for suspicious activity 7. **Audit `pg_stat_activity`** for background worker usage 8. **Test disaster recovery** with active workers --- ## Use Cases — see the Cookbook End-to-end examples (background maintenance, autonomous audit logging with retry/fallback, async notification delivery, long-running ETL with job tracking, parallel-query simulation, timeout enforcement) plus quick-recipe templates live in **[`docs/COOKBOOK.md`](docs/COOKBOOK.md)**. --- ## Operational Guidance ### Resource Management #### max_worker_processes Limit Background workers count against PostgreSQL's global `max_worker_processes` limit. **Check Current Usage**: ```sql SELECT count(*) AS bgworker_count FROM pg_stat_activity WHERE backend_type LIKE '%background%'; ``` **Recommended Configuration**: ```sql -- Formula: autovacuum_workers + max_parallel_workers + pg_background_estimate + buffer ALTER SYSTEM SET max_worker_processes = 64; -- Adjust per workload SELECT pg_reload_conf(); ``` **Operational Limits**: - Default `max_worker_processes`: 8 (often insufficient) - Recommended minimum for pg_background: 16-32 - Enterprise workloads: 64-128 - Each worker: ~10MB memory overhead #### Dynamic Shared Memory (DSM) Usage Each worker allocates one DSM segment for IPC. **Monitor DSM**: ```sql SELECT name, size, allocated_size FROM pg_shmem_allocations WHERE name LIKE '%pg_background%' ORDER BY size DESC; ``` **DSM Size**: - Default queue_size: 65536 bytes (~64KB) - Minimum queue_size: 4096 bytes (enforced by `shm_mq`) - Large result sets: increase queue_size parameter **Example**: ```sql -- Small results (default) SELECT pg_background_launch('SELECT id FROM small_table', 65536); -- Large results (1MB queue) SELECT pg_background_launch('SELECT * FROM huge_table', 1048576); ``` #### Worker Lifecycle and Cleanup **Automatic Cleanup**: - Worker exits → DSM detached → hash entry removed - Launcher session ends → all tracked workers detached **Manual Cleanup**: ```sql -- Detach all completed workers DO $$ DECLARE r record; BEGIN FOR r IN SELECT * FROM pg_background_list WHERE state IN ('stopped', 'canceled', 'error') LOOP PERFORM pg_background_detach(r.pid, r.cookie); END LOOP; END; $$; ``` ### Performance Tuning #### 1. Queue Size Optimization **Rule of Thumb**: - Small queries (< 1000 rows): 65536 (64KB, default) - Medium queries (< 10000 rows): 262144 (256KB) - Large queries (>= 10000 rows): 1048576+ (1MB+) **Trade-offs**: - Larger queue → less blocking on result production - Larger queue → more shared memory consumption - Too small → worker blocks waiting for launcher to consume **Measure Contention**: ```sql -- Check if worker is blocking on queue send SELECT pid, state, wait_event_type, wait_event FROM pg_stat_activity WHERE backend_type LIKE '%background%' AND wait_event = 'SHM_MQ_SEND'; ``` #### 2. Statement Timeout Workers inherit `statement_timeout` from launcher session. **Set Per-Worker Timeout**: ```sql -- Temporarily increase timeout SET statement_timeout = '30min'; SELECT pg_background_launch('slow_aggregation_query()'); RESET statement_timeout; ``` **Set Database-Wide Default**: ```sql ALTER DATABASE production SET statement_timeout = '10min'; ``` #### 3. Work Memory **Important**: Workers do NOT inherit `work_mem` from launcher. **Workaround**: ```sql -- Include SET in worker SQL SELECT pg_background_launch($$ SET work_mem = '256MB'; SELECT * FROM large_table ORDER BY col; $$); ``` #### 4. Parallel Workers Background workers are separate from `max_parallel_workers`. **Configuration**: ```sql -- Both settings are independent ALTER SYSTEM SET max_worker_processes = 64; -- Total pool ALTER SYSTEM SET max_parallel_workers = 16; -- Parallel query subset ``` ### Monitoring #### Real-Time Worker Status ```sql CREATE VIEW pg_background_status AS SELECT w.pid, w.cookie, w.state, left(w.sql_preview, 60) AS sql_snippet, w.launched_at, (now() - w.launched_at) AS age, w.consumed, a.state AS pg_state, a.wait_event_type, a.wait_event, a.query AS current_query FROM pg_background_list w LEFT JOIN pg_stat_activity a USING (pid) ORDER BY w.launched_at DESC; -- Query it SELECT * FROM pg_background_status; ``` #### Alerting on Long-Running Workers ```sql -- Workers running > 1 hour SELECT pid, cookie, sql_preview, (now() - launched_at) AS duration FROM pg_background_list WHERE state = 'running' AND (now() - launched_at) > interval '1 hour'; ``` #### Prometheus-Style Metrics ```sql -- Export metrics for monitoring systems SELECT 'pg_background_active_workers' AS metric, count(*) AS value, state AS labels FROM pg_background_list GROUP BY state; ``` --- ## Troubleshooting ### Common Issues #### Issue 1: "could not register background process" **Symptom**: ``` ERROR: could not register background process HINT: You may need to increase max_worker_processes. ``` **Cause**: `max_worker_processes` limit reached. **Solution**: ```sql -- Check current limit and usage SHOW max_worker_processes; SELECT count(*) FROM pg_stat_activity WHERE backend_type LIKE '%worker%'; -- Increase limit (requires restart for some versions) ALTER SYSTEM SET max_worker_processes = 32; SELECT pg_reload_conf(); -- Or restart PostgreSQL ``` #### Issue 2: "cookie mismatch for PID XXXXX" **Symptom**: ``` ERROR: cookie mismatch for PID 12345: expected 1234567890123456, got 9876543210987654 ``` **Cause**: PID reused after worker exit, or stale handle. **Solution**: - Always use fresh handles from `launch()` - Never hardcode PID/cookie values - Don't cache handles across long time periods ```sql -- ❌ Bad: Reusing old handle -- h was from hours ago, worker exited, PID reused -- ✅ Good: Fresh handle per operation SELECT * FROM pg_background_launch('...') AS h \gset SELECT pg_background_wait(:'h.pid', :'h.cookie'); ``` #### Issue 3: Worker Hangs Indefinitely **Symptom**: Worker shows `running` state for hours without progress. **Cause**: Lock contention, infinite loop, or missing `CHECK_FOR_INTERRUPTS`. **Diagnosis**: ```sql -- Check what worker is waiting on SELECT w.pid, w.sql_preview, a.wait_event_type, a.wait_event, a.state, a.query FROM pg_background_list w JOIN pg_stat_activity a USING (pid) WHERE w.state = 'running'; -- Check locks SELECT l.pid, l.locktype, l.relation::regclass, l.mode, l.granted FROM pg_locks l WHERE l.pid = ; ``` **Solution**: ```sql -- Cancel with grace period SELECT pg_background_cancel(, , 5000); -- Force cancel if grace period expires SELECT pg_background_cancel(, ); ``` #### Issue 4: "PID is not attached to this session" after a successful result **Symptom**: ``` ERROR: PID 12345 is not attached to this session ``` raised by `pg_background_result(pid, cookie)` after a previous call on the same handle returned rows successfully. **Cause**: results are one-time consumption. The first successful `result` call auto-detaches the worker; the second call sees no tracked entry for that PID and raises `ERRCODE_UNDEFINED_OBJECT`. Note: this is the same SQLSTATE you'd get from a wrong PID, so the message is worded the same. **Solution**: Results are **one-time consumption**. Use CTE to reuse: ```sql -- ✅ Correct: Use CTE to consume once WITH worker_results AS ( SELECT * FROM pg_background_result(, ) AS (col text) ) SELECT * FROM worker_results UNION ALL SELECT * FROM worker_results; ``` #### Issue 5: DSM Allocation Failure **Symptom**: ``` ERROR: could not allocate dynamic shared memory ``` **Cause**: Insufficient shared memory or too many DSM segments. **Solution**: ```sql -- Check DSM usage SELECT count(*), sum(size) AS total_bytes FROM pg_shmem_allocations WHERE name LIKE '%dsm%'; -- Increase shared memory (postgresql.conf) -- dynamic_shared_memory_type = posix (or sysv, mmap) -- Restart PostgreSQL ``` #### Issue 6: Custom Schema Installation Errors (Fixed in v1.7+) **Symptom** (in versions before fix): ``` CREATE EXTENSION pg_background WITH SCHEMA contrib; ERROR: function public.pg_background_grant_privileges(unknown, boolean) does not exist ``` **Cause**: Hardcoded `public.` schema references in SQL scripts when extension is relocatable. **Status**: **Fixed in v1.7+** for fresh installations. The extension now properly supports custom schema installation. **Solution for fresh install**: ```sql -- Install directly in custom schema (v1.7+) CREATE SCHEMA myschema; CREATE EXTENSION pg_background WITH SCHEMA myschema; -- Verify SELECT * FROM myschema.pg_background_launch('SELECT 1') AS h; ``` **⚠️ Limitation for upgrades**: If you have v1.4, v1.5, or v1.6 already installed, upgrading to v1.7/v1.8 will NOT move the extension to a custom schema. The upgrade scripts for older versions contain hardcoded `public.` references because those versions only supported the public schema. **To relocate an existing installation**: ```sql -- 1. Drop existing extension DROP EXTENSION pg_background; -- 2. Reinstall in desired schema CREATE EXTENSION pg_background WITH SCHEMA myschema; ``` #### Issue 7: "column definition list is required" / "rowtype does not match" **Symptom**: ``` ERROR: function returning record called in context that cannot accept type record HINT: Try calling the function in the FROM clause using a column definition list. -- Or when columns don't match what the worker actually emitted: ERROR: remote query result rowtype does not match the specified FROM clause rowtype ``` **Cause**: `pg_background_result()` and `pg_background_list()` return `SETOF record`. PostgreSQL needs the row shape declared at parse time, either via `AS (col1 type, col2 type, ...)` or by reading from a view/wrapper that has a fixed row type. **Solutions**: ```sql -- Match the AS list to the worker's actual columns: SELECT * FROM pg_background_result(:pid, :cookie) AS (result text); SELECT * FROM pg_background_result(:pid, :cookie) AS (col1 text, col2 text); ``` If you only need to wait for completion (no rows), use `wait`: ```sql SELECT pg_background_wait(:pid, :cookie); -- block forever SELECT pg_background_wait(:pid, :cookie, 5000); -- bounded ``` If you only need *metadata* (row count, command tag, error info), use `pg_background_run()` or `pg_background_outcome()` — neither requires a column-definition list. For monitoring, prefer the **`pg_background_list`** view (introduced in 1.10) over `pg_background_list()` directly — the view declares the row type so callers don't have to. ### Platform-Specific Issues #### Windows: Cancel Limitations **Problem**: On Windows, `cancel()` cannot interrupt actively running statements. **Explanation**: Windows lacks signal-based interrupts. Cancel only sets interrupt flags checked between statements. **Workaround**: ```sql -- Always set statement_timeout on Windows ALTER DATABASE mydb SET statement_timeout = '5min'; -- Or per-worker: SELECT pg_background_launch($$ SET statement_timeout = '5min'; SELECT slow_function(); $$); ``` **Affected Operations**: - Long-running CPU-bound queries - Infinite loops in PL/pgSQL - Queries with no yielding points **See**: `windows/ReadMe.md` for details. ### Debug Logging ```sql -- Enable verbose logging SET client_min_messages = DEBUG1; SET log_min_messages = DEBUG1; -- Launch worker (check logs for DSM info) SELECT * FROM pg_background_launch('SELECT 1') AS h \gset; -- Check PostgreSQL logs for: -- - "registered dynamic background worker" -- - "DSM segment attached" -- - Worker execution details ``` --- ## Architecture & Design — see docs/ARCHITECTURE.md The deep architecture write-up (sequence diagram, DSM TOC layout, key components, concurrency-race history, publish-flag patterns) is in **[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)**. ## Known Limitations ### 1. Windows Cancel Limitations **Limitation**: `cancel()` on Windows cannot interrupt running statements. **Details**: - Windows lacks `SIGUSR1` equivalent for query cancellation - Cancel only sets `InterruptPending` flag - Flag checked between statements, not during execution **Impact**: - Infinite loops in PL/pgSQL cannot be interrupted - Long-running aggregate functions cannot be interrupted mid-execution - `pg_sleep()` DOES check interrupts (interruptible) **Workarounds**: 1. Always set `statement_timeout`: ```sql ALTER DATABASE mydb SET statement_timeout = '5min'; ``` 2. Avoid infinite loops in worker SQL 3. Test cancellation on Unix/Linux platforms first **Reference**: See `windows/ReadMe.md` for implementation details. ### 2. No Cross-Database Workers **Limitation**: Workers can only connect to the **same database** as launcher. **Reason**: `BackgroundWorker` API requires database OID at registration. **Workaround**: Use `dblink` for cross-database operations: ```sql SELECT pg_background_launch($$ SELECT * FROM dblink('dbname=other_db', 'SELECT ...') $$); ``` ### 3. Per-Session Worker Limits (v1.8+) **v1.8 Improvement**: Built-in `pg_background.max_workers` GUC limits concurrent workers per session. ```sql -- Limit to 10 concurrent workers per session SET pg_background.max_workers = 10; ``` **Remaining Limitation**: No per-user or per-database quotas across sessions. **Workaround**: Implement application-level quotas for cross-session limits (see [Security](#security-model)). ### 4. Worker Exhaustion (INSUFFICIENT_RESOURCES) **Limitation**: When `max_worker_processes` is exhausted, `pg_background_launch()` throws `INSUFFICIENT_RESOURCES`. **Error Message**: ``` ERROR: could not register background process HINT: You may need to increase max_worker_processes. ``` **Impact**: This is particularly problematic for **autonomous logging** use cases: 1. **Data Loss**: The message intended for logging is lost 2. **Cascading Failures**: The calling transaction may fail unexpectedly 3. **Unpredictable**: Failures occur sporadically under high load **Why This Happens**: Background workers share the global `max_worker_processes` pool with: - Parallel query workers (`max_parallel_workers`) - Autovacuum workers (`autovacuum_max_workers`) - Logical replication workers - Custom background workers from other extensions **Mitigation Strategies**: 1. **Increase worker pool** (reduces frequency, doesn't eliminate): ```sql ALTER SYSTEM SET max_worker_processes = 64; -- Requires PostgreSQL restart ``` 2. **Implement retry with backoff**: ```sql BEGIN SELECT pg_background_launch(...); EXCEPTION WHEN insufficient_resources THEN PERFORM pg_sleep(0.1); -- Backoff -- Retry or fallback END; ``` 3. **Fallback to synchronous execution** (for critical operations): ```sql EXCEPTION WHEN insufficient_resources THEN -- Execute synchronously as fallback INSERT INTO audit_log VALUES (...); END; ``` 4. **Pre-check worker availability** (advisory, not guaranteed): ```sql SELECT count(*) < current_setting('max_worker_processes')::int FROM pg_stat_activity WHERE backend_type LIKE '%worker%'; ``` 5. **Reserve capacity** by setting conservative `pg_background.max_workers`: ```sql -- Leave headroom for other workers SET pg_background.max_workers = 8; -- Even if pool is 64 ``` **Recommendation**: For mission-critical logging, always implement a synchronous fallback. Autonomous transactions via pg_background are **best-effort**, not guaranteed. **See Also**: [Autonomous Audit Logging](#2-autonomous-audit-logging) for robust implementation patterns. ### 5. Result Consumption is One-Time **Limitation**: `result()` can only be called **once** per handle. **Reason**: Results streamed from DSM; no persistent storage. **Workaround**: Use CTE or temporary table: ```sql -- Store results in temp table CREATE TEMP TABLE worker_output AS SELECT * FROM pg_background_result(, ) AS (col text); -- Query multiple times SELECT * FROM worker_output WHERE col LIKE '%foo%'; SELECT count(*) FROM worker_output; ``` ### 6. No Result Pagination **Limitation**: Cannot retrieve results in chunks (all-or-nothing). **Reason**: shm_mq is streaming; no cursor support. **Impact**: Large result sets (> queue_size) may block worker. **Workaround**: - Increase `queue_size` parameter - Use `LIMIT` in worker SQL - Process results incrementally in launcher ### 7. Limited Observability **Limitation**: `list()` only shows workers in **current session**. **Reason**: Hash table is session-local (not shared memory). **Impact**: Cannot observe other sessions' workers. **Workaround**: Query `pg_stat_activity`: ```sql SELECT pid, backend_type, state, query, backend_start FROM pg_stat_activity WHERE backend_type LIKE '%background%'; ``` ### 8. No Transaction Pinning **Limitation**: Worker transactions are **fully autonomous** (cannot join launcher's transaction). **Reason**: PostgreSQL does not support distributed transactions. **Impact**: Cannot implement 2PC-like patterns natively. **Workaround**: Use `dblink` with `PREPARE TRANSACTION` for XA-like semantics. ### 9. Early Worker Failures (Before `pq_redirect_to_shm_mq`) **Limitation**: Errors raised in the worker **before** `pq_redirect_to_shm_mq()` installs the shm_mq destination cannot be captured as a structured error. **What is "early"**: The small window between worker startup and the `pq_redirect_to_shm_mq()` call in `pg_background_worker_main` — primarily: - Failure to attach the DSM segment (`dsm_attach` returning NULL). - `shm_toc_lookup` failure (missing TOC entry — implies an internally inconsistent DSM, typically a sign of server misconfiguration). - Out-of-memory during the initial worker setup allocations. **Observable behavior for the launcher**: - `pg_background_result()` raises `SQLSTATE 08006 "lost connection to worker process"` when it tries to read results from the detached shm_mq. `pg_background_wait()` blocks on `WaitForBackgroundWorkerShutdown` and returns silently — it does not raise; the early worker exit leaves no structured error on the wire for it to observe. - `pg_background_error_info()` returns `NULL` row (no structured info). - `pg_background_result_info()` reports `completed=true, has_error=false` since the worker never got far enough to run SQL. **Why it cannot be captured**: the worker's error-propagation contract (`EmitErrorReport` over shm_mq, `ReadyForQuery(DestRemote)`, `pq_flush`) requires the shm_mq destination to already be installed. Before `pq_redirect_to_shm_mq`, `ereport(ERROR)` goes to the server log only; the launcher observes the worker exit and synthesizes `08006`. **Impact in practice**: these are infrastructure-level failures (DSM OOM, misconfigured `dynamic_shared_memory_type`, missing `shm_toc` entry). They are rare in a correctly configured server and do not indicate user-level SQL problems. **Recommended handling**: treat a `08006` from `pg_background_result()` as an infra signal — do not attempt to parse an `error_info` row that may be `NULL`. All ordinary SQL errors (syntax, constraint violation, division-by-zero, `RAISE EXCEPTION`, statement cancel) propagate through the normal path and appear as their real SQLSTATE, not `08006`. ### 10. Cancellation is cooperative — workers are never force-killed **Behavior**: `pg_background_cancel(pid, cookie, grace_ms)` (and the batch/by-label variants) cancel a worker **cooperatively** by sending `SIGTERM`. The worker converts that into a catchable query cancel at its next interrupt check and exits via `proc_exit(1)`. When `grace_ms > 0` the launcher additionally waits up to `grace_ms` for the worker to stop, purely so the caller can observe whether it did. **pg_background never escalates to `SIGKILL`.** **Why no force-kill**: PostgreSQL's postmaster treats *any* child that dies from an uncaught signal (`SIGKILL`, `SIGSEGV`, …) as a crash and responds by terminating every other backend and reinitializing shared memory — a cluster-wide restart that drops all sessions. There is no signal that force-stops a background worker without tripping crash recovery, so a worker that ignores the cooperative cancel cannot be force-killed safely. **History (fixed in 2.0)**: earlier 2.0 development builds *did* escalate to `kill(pid, SIGKILL)` after the grace period expired. This was benign on fast machines (the worker almost always exited cooperatively within `grace_ms`, so the `SIGKILL` never fired) but caused a hard-to-trace **cluster crash on slow or loaded hosts** — most visibly on the Debian/pgdg build farm and under Valgrind, where a worker still in PostgreSQL's bgworker *startup* path when the grace timer expired was `SIGKILL`ed before it ever reached `pg_background_worker_main`. Because `SIGKILL` is uncatchable, the dying worker emitted **zero log output**, which made the failure look like an upstream SIGSEGV in PG's startup machinery. It was not: it was this extension force-killing its own worker. Removing the `SIGKILL` escalation resolves it. **Guidance for unresponsive workers**: a worker stuck in CPU-bound SQL that never reaches an interrupt check will not stop on cancel. Bound such work ahead of time with `statement_timeout` or the `pg_background.worker_timeout` GUC so it self-cancels. --- ## Best Practices ### 1. Always Use v2 API in Production ✅ **Correct**: ```sql SELECT * FROM pg_background_launch('...') AS h \gset SELECT pg_background_result(:'h.pid', :'h.cookie'); ``` ❌ **Avoid**: ```sql SELECT pg_background_launch('...') AS pid \gset -- No PID reuse protection SELECT pg_background_result(:pid); ``` ### 2. Set Timeouts for All Workers ```sql -- Database-wide default ALTER DATABASE production SET statement_timeout = '10min'; -- Or per-worker SELECT pg_background_launch($$ SET statement_timeout = '5min'; SELECT slow_query(); $$); ``` ### 3. Use submit() for Fire-and-Forget ```sql -- ✅ Idiomatic: submit + detach SELECT * FROM pg_background_submit('INSERT INTO log ...') AS h \gset; SELECT pg_background_detach(:'h.pid', :'h.cookie'); -- ❌ Verbose: launch + detach without result retrieval SELECT * FROM pg_background_launch('INSERT INTO log ...') AS h \gset; SELECT pg_background_detach(:'h.pid', :'h.cookie'); ``` > **NOTIFY note for `submit`**: NOTIFY frames raised inside a worker > are forwarded to the launcher's protocol output only while the launcher > is parked inside `pg_background_result(...)`. A `submit` worker > never has a result reader, so any `NOTIFY` it emits is effectively > dropped — the row hits the worker's transaction (`pg_listener` / > `pg_notify_queue`) but the *message* on the wire is not relayed back to > a session that called `LISTEN`. If a NOTIFY needs to reach a listening > session, use `launch` + `result`, or write the notification to a > table the listener polls. ### Worker GUC propagation `pg_background_launch` / `submit` propagate the launching session's **full** GUC state to the worker via `SerializeGUCState` (the same mechanism PostgreSQL uses for parallel workers). The worker calls `RestoreGUCState` at startup and inherits everything: `search_path`, `statement_timeout`, `lock_timeout`, role-based settings, custom GUCs. This is intentional, but it means session-local knobs you may not want inside a long-running worker (e.g. `idle_in_transaction_session_timeout`) are inherited too. If you need a tighter GUC profile for a worker, set the GUC in the launching session **before** calling `launch`/`submit`: ```sql SET LOCAL statement_timeout = '30s'; SELECT pg_background_run('SELECT slow_thing()'); ``` ### 4. Monitor Worker State Regularly ```sql -- Scheduled cleanup of stale workers CREATE OR REPLACE FUNCTION cleanup_stale_workers() RETURNS void AS $$ DECLARE r record; BEGIN FOR r IN SELECT * FROM pg_background_list WHERE state IN ('stopped', 'error') AND (now() - launched_at) > interval '1 hour' LOOP PERFORM pg_background_detach(r.pid, r.cookie); END LOOP; END; $$ LANGUAGE plpgsql; -- Run periodically SELECT cleanup_stale_workers(); ``` ### 5. Sanitize All Dynamic SQL ```sql -- ✅ Safe: Use format() with %L CREATE FUNCTION safe_worker(table_name text) RETURNS void AS $$ BEGIN PERFORM pg_background_launch( format('VACUUM %I', table_name) -- %I for identifiers ); END; $$ LANGUAGE plpgsql; ``` ### 6. Handle Errors Gracefully ```sql DO $$ DECLARE h pg_background_handle; result_val text; BEGIN SELECT * INTO h FROM pg_background_launch('SELECT 1/0'); BEGIN SELECT * INTO result_val FROM pg_background_result(h.pid, h.cookie) AS (r text); EXCEPTION WHEN OTHERS THEN RAISE NOTICE 'Worker failed: %', SQLERRM; -- Cleanup PERFORM pg_background_detach(h.pid, h.cookie); END; END; $$; ``` ### 7. Document Worker Purpose ```sql -- ✅ Good: Clear intent SELECT * FROM pg_background_launch($$ /* Background VACUUM for nightly maintenance */ VACUUM (VERBOSE, ANALYZE) user_activity; $$) AS h \gset; -- Comment visible in list() sql_preview ``` ### 8. Test Disaster Recovery Ensure application handles: - PostgreSQL restart (all workers lost) - Worker crashes (orphaned handles) - Launcher session termination (workers detached) ```sql -- Simulate crash: check handle invalidation SELECT * FROM pg_background_launch('SELECT pg_sleep(100)') AS h \gset; -- Restart PostgreSQL SELECT pg_background_wait(:'h.pid', :'h.cookie'); -- Should error gracefully ``` --- ## Cookbook — see docs/COOKBOOK.md Copy-paste templates (synchronous run with metadata, wait-with-timeout + result-or-error, fan-out + drain, wait-any, cancel-by-label) plus the end-to-end use cases live in **[`docs/COOKBOOK.md`](docs/COOKBOOK.md)**. --- ## Migration Guide — see docs/MIGRATION.md The 1.10 → 2.0 migration story (dropped APIs, renamed helpers, type field additions) and every older 1.x → 1.x upgrade chain live in **[`docs/MIGRATION.md`](docs/MIGRATION.md)**. The v1 → v2 API mapping table is also there. --- ## Testing ### Local Testing (Native) If you have PostgreSQL development files installed locally: ```bash # Build and install make clean && make sudo make install # Run regression tests make installcheck # Clean test artifacts make installcheckclean ``` ### Docker-Based Testing (Recommended) Docker-based testing requires no local PostgreSQL installation: ```bash # Test with PostgreSQL 17 (default) ./scripts/test-local.sh # Test with specific PostgreSQL version ./scripts/test-local.sh 14 ./scripts/test-local.sh 16 # Test all supported versions (14-19) ./scripts/test-local.sh all ``` ### Relocatable Extension Testing Verify the extension works correctly when installed in a custom schema: ```bash # Run comprehensive relocatable tests ./scripts/test-relocatable.sh 17 ``` ### Upgrade Path Testing Validate extension upgrades work correctly: ```bash # Test 1.8 → 1.9 upgrade path ./scripts/test-upgrade.sh 17 ``` ### CI Pipeline The project uses GitHub Actions for continuous integration: | Job | Description | |-----|-------------| | **test** | Matrix: PG 14-19 × ubuntu-22.04/24.04 regression tests | | **relocatable-test** | Validates custom schema installation (PG 17) | | **upgrade-test** | Validates 1.8 → 1.9 upgrade path | | **lint** | cppcheck and clang-format checks | | **security** | CodeQL security analysis | All tests must pass before merging to main branches. --- ## Contributing We welcome contributions! Please see [docs/CONTRIBUTING.md](docs/CONTRIBUTING.md) for: - Code of conduct - Development setup - Coding standards (PostgreSQL style, `pgindent`) - Testing requirements - Pull request process **Quick Start**: ```bash git clone https://github.com/vibhorkum/pg_background.git cd pg_background make clean && make && sudo make install make installcheck ``` **Before Submitting PR**: - [ ] Code follows PostgreSQL conventions - [ ] Regression tests added/updated - [ ] Tests pass (`make installcheck`) - [ ] No compiler warnings - [ ] Documentation updated --- ## License This project is licensed under the [PostgreSQL License](LICENSE). Copyright (c) 2014-2026, Vibhor Kumar and contributors. Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group. --- ## Author **Vibhor Kumar** – Original author and maintainer **Inspiration**: - PostgreSQL Background Worker API - `dblink` extension - Oracle DBMS_JOB --- ## Related Projects - **[pg_cron](https://github.com/citusdata/pg_cron)** – Schedule periodic jobs - **[dblink](https://www.postgresql.org/docs/current/dblink.html)** – Cross-database/async queries - **[pgAgent](https://www.pgagent.org/)** – Job scheduler daemon - **[pg_task](https://github.com/RekGRpth/pg_task)** – Task queue extension --- **Production Deployments**: For critical workloads, always: 1. Use **v2 API exclusively** (cookie-protected handles) 2. Set **statement_timeout** on all workers 3. **Monitor** `pg_background_list()` and `pg_stat_activity` 4. **Test** disaster recovery scenarios (restarts, crashes) 5. **Audit** privilege grants regularly **Version**: 2.0 **Last Updated**: 2026-05-10 **Minimum PostgreSQL**: 14 **Tested Through**: PostgreSQL 18 (PostgreSQL 19 beta validated) **Companion docs**: [`docs/MIGRATION.md`](docs/MIGRATION.md) · [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) · [`docs/COOKBOOK.md`](docs/COOKBOOK.md) · [`docs/SECURITY.md`](docs/SECURITY.md) · [`docs/CI.md`](docs/CI.md) · [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md) pg_background-2.0.2/docs/000077500000000000000000000000001521265151100152475ustar00rootroot00000000000000pg_background-2.0.2/docs/ARCHITECTURE.md000066400000000000000000000306511521265151100174600ustar00rootroot00000000000000# pg_background — Architecture & Design This document describes how `pg_background` is built. For end-user API docs see [`README.md`](../README.md); for migration between versions see [`docs/MIGRATION.md`](MIGRATION.md). --- ## One-page mental model ```mermaid sequenceDiagram autonumber participant Launcher as Launcher session participant DSM as DSM segment participant SHMMQ as shm_mq participant Worker as Background worker process Launcher->>DSM: Allocate (input, output, sql, GUCs, queue) Launcher->>Worker: RegisterDynamicBackgroundWorker Launcher->>SHMMQ: shm_mq_wait_for_attach Worker->>DSM: dsm_attach + shm_toc_lookup Worker->>SHMMQ: shm_mq_attach (sender) Note over Launcher,Worker: launch returns (pid, cookie) handle Worker->>Worker: BackgroundWorkerInitializeConnection Worker->>Worker: StartTransactionCommand Worker->>Worker: execute_sql_string(sql) alt success Worker->>SHMMQ: stream rows + RowDescription Worker->>DSM: row_count, command_tag, started_at, finished_at Worker->>SHMMQ: ReadyForQuery, pq_flush Worker->>Worker: CommitTransactionCommand Worker--xLauncher: proc_exit(0) else error Worker->>DSM: error_sqlstate (publish flag, written LAST) Worker->>SHMMQ: EmitErrorReport (real 'E' frame) Worker--xLauncher: proc_exit(1) end Launcher->>SHMMQ: pg_background_result (consume rows / get error) Launcher->>Launcher: pg_background_detach (DSM cleanup callback fires) ``` **Key design points** - The DSM segment is the only shared mutable state. The launcher's session-local hash table tracks the segment and the BGW handle but never holds a long-lived pointer into it. - Errors propagate via two paths: the structured DSM fields (read by `error_info`) and the live `'E'` frame on `shm_mq` (read by `result`). Both must agree; the worker writes DSM first, then emits the frame. - The launcher's `cleanup_worker_info` callback runs at DSM detach, so leaking a handle is impossible — even abnormal exit paths reclaim resources. --- ## High-level data flow ``` ┌──────────────────┐ │ Client Session │ │ (Launcher) │ └────────┬─────────┘ │ 1. pg_background_launch(sql) ▼ ┌──────────────────────────────────┐ │ Extension C Code │ │ - Allocate DSM segment │ │ - RegisterDynamicBgWorker() │ │ - Create shm_mq │ │ - Wait for worker attach │ └────────┬─────────────────────────┘ │ 2. Postmaster fork() ▼ ┌──────────────────────────────────┐ │ Background Worker Process │ │ - Attach database │ │ - Restore session GUCs │ │ - Execute SQL via SPI │ │ - Send results via shm_mq │ │ - Exit (DSM cleanup) │ └──────────────────────────────────┘ │ 3. Results via shared memory ▼ ┌──────────────────┐ │ Launcher │ │ pg_background_ │ │ result() │ └──────────────────┘ ``` --- ## Key components ### Dynamic Shared Memory (DSM) **Purpose**: IPC mechanism for SQL text, GUC state, and result transport. **TOC layout (v2.0)**: | Key | Name | Direction | Contents | |---|---|---|---| | 0 | `INPUT` | launcher → worker | database/user OIDs, security context, cookie, label, `cancel_requested` (mutable) | | 1 | `SQL` | launcher → worker | SQL command string (null-terminated) | | 2 | `GUC` | launcher → worker | serialized session GUCs | | 3 | `QUEUE` | bidirectional | `shm_mq` for result streaming | | 4 | `OUTPUT` | worker → launcher | progress, error fields, result row count, command tag, started_at, finished_at, error-source identifiers | **v2.0 (C1)** split the previous single `FIXED_DATA` chunk into INPUT and OUTPUT under separate TOC keys. The split clarifies barrier ordering and avoids cache-line bouncing between launcher and worker. **Lifecycle**: - Created by the launcher in `launch_internal()` (allocates both INPUT and OUTPUT, zero-fills OUTPUT, populates INPUT). - Worker attaches at startup and looks up both keys. - Launcher detaches via `detach()` or session end. The worker's own `dsm_detach` on exit drops its reference; the segment is released when refcount hits zero. ### Shared memory queue (shm_mq) **Purpose**: bidirectional streaming transport for results, ReadyForQuery frames, and any NOTIFY/'A' frames the worker emits. **Flow**: 1. Worker executes the query via SPI. 2. Each result row is serialized to the shm_mq. 3. The launcher reads from the queue in `pg_background_result`. 4. Queue blocks the writer if full (backpressure). **Receive-side liveness check (v2.0 C3)**: the launcher's read loop uses non-blocking `shm_mq_receive` + `WaitLatch(... PG_WAIT_EXTENSION)` with a periodic `GetBackgroundWorkerPid` check. A worker that attaches but never sends and never exits no longer hangs the launcher session indefinitely; the launcher detects the BGW stopping and treats the queue as detached. **Tuning**: - Queue size set at launch (default 64 KiB; bounded above at 256 MiB). - Larger queues reduce launcher-side blocking on bursty result streams. - Monitor with `pg_stat_activity.wait_event = 'SHM_MQ_SEND'` / `'MessageQueueReceive'`. ### Background worker API **Registration**: ```c BackgroundWorker worker; worker.bgw_flags = BGWORKER_SHMEM_ACCESS | BGWORKER_BACKEND_DATABASE_CONNECTION; worker.bgw_start_time = BgWorkerStart_ConsistentState; worker.bgw_main = pg_background_worker_main; RegisterDynamicBackgroundWorker(&worker, &handle); ``` **Lifecycle hooks**: - `bgw_main`: entry point (`pg_background_worker_main` in `src/pg_background_worker.c`). - `bgw_notify_pid`: launcher PID (for postmaster-driven notifications). - `bgw_main_arg`: DSM handle (Datum). ### Server Programming Interface (SPI) **Execution pipeline**: parse → analyze → plan → execute via Portal. The worker calls into `pg_parse_query`, `pg_analyze_and_rewrite_*`, `pg_plan_queries`, `CreatePortal`, `PortalStart`, `PortalRun`. Results flow through a remote `DestReceiver` that writes into `shm_mq`. **Result serialization on the wire**: - `RowDescription`: column metadata (names, types, formats). - `DataRow`: binary-encoded tuple data. - `CommandComplete`: result tag (e.g., "SELECT 42"); the worker also writes the row count and command tag into the OUTPUT struct so `result_info` can report them without re-reading the queue. ### Worker hash table **Purpose**: per-session tracking of launched workers, keyed by PID. **Structure** (`pg_background_worker_info` in `src/pg_background_internal.h`): ```c typedef struct pg_background_worker_info { pid_t pid; Oid current_user_id; uint64 cookie; dsm_segment *seg; BackgroundWorkerHandle *handle; shm_mq_handle *responseq; bool consumed; bool mapping_pinned; bool result_disabled; bool canceled; TimestampTz launched_at; int32 queue_size; char sql_preview[PGBG_SQL_PREVIEW_LEN + 1]; char *last_error; char *full_sql; } pg_background_worker_info; ``` **Cleanup**: - On `dsm_detach` for the worker's segment, `cleanup_worker_info` fires via `on_dsm_detach`. It removes the hash entry and updates session stats. - v2.0 (C2) hardened: the cleanup callback NO LONGER destroys the hash or resets `WorkerInfoMemoryContext`. Both live for the duration of the session and are released by PostgreSQL at backend exit. This removes a use-after-free trap where a public C entrypoint could be holding a `pg_background_worker_info *` across the `dsm_detach` that triggers this callback. - On launcher session end: PG releases everything. - On explicit `detach()`: `detach_worker_seg(info)` clears `seg` / `mapping_pinned` before calling `dsm_detach`; the cleanup callback then sees the entry but does not double-detach. --- ## Concurrency and race conditions ### NOTIFY race (solved in 1.5+) **Problem**: launcher used to return from `launch()` before the worker had attached the shm_mq, so any `NOTIFY` the worker fired before it registered as the queue's sender would be lost. **Solution**: `shm_mq_wait_for_attach()` blocks the launcher inside `launch_internal` until the worker is the registered sender. ```c shm_mq_wait_for_attach(mqh); /* BLOCK until worker attaches */ return handle; /* safe to return now */ ``` > Note: NOTIFY frames produced by `submit` workers are written to > the response shm_mq but never read, since `submit` callers don't > consume `result`. They are effectively dropped. See README "NOTIFY > note for `submit`" for the user-facing recommendation. ### PID reuse (solved in the v2 API) **Problem**: a worker exits, the OS reuses its PID for an unrelated process, the launcher operates on the wrong worker. **Solution**: 64-bit cryptographically secure cookie generated by `pg_strong_random` at launch. Every cookie-protected v2 entrypoint validates the caller's cookie before performing any work. ```c if (info->cookie != (uint64) cookie_in) ereport(ERROR, (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), errmsg("cookie mismatch for PID %d", pid), errhint("The worker may have been restarted or the handle is stale."))); ``` ### DSM cleanup races (hardened in 1.6, again in v2.0) **1.6**: never `pfree(handle)`; let PostgreSQL manage the BGW handle's lifetime, so the launcher cannot free a handle that's still needed by postmaster's BGW machinery. **v2.0 (C2)**: `cleanup_worker_info` no longer resets the session-local `WorkerInfoMemoryContext` from inside the `on_dsm_detach` callback. A public C entrypoint can be holding a `pg_background_worker_info *` across the `dsm_detach` that triggers the callback (e.g., the result-streaming SRF detaching at end of iteration); resetting the context underneath them was a latent use-after-free. ### Result-metadata publish-flag pattern (v2.0 B5) The OUTPUT struct contains pairs of fields the worker writes that the launcher reads concurrently: - `result_row_count` + `command_tag` (paired post-SPI metadata) - The whole error block (`error_message`, `error_detail`, `error_hint`, `error_context`, `error_schema_name`, `error_table_name`, `error_column_name`, `error_constraint_name`) For each pair the worker writes the data fields first, issues `pg_write_barrier()`, then sets a "publish flag" LAST: | pair | publish flag | |---|---| | result_row_count + command_tag | `result_published` (uint8) | | error_* fields | `error_sqlstate` (non-empty) | Readers test the flag first; if set, issue `pg_read_barrier()` and only then read the other fields. This prevents the launcher from observing a fresh `row_count` paired with a stale `command_tag`, or a partially- written error. ### Worker error-exit cleanup ordering (v2.0 F) `pg_background_worker_error_exit` follows PostgreSQL's standard error-cleanup sequence: 1. `EmitErrorReport()` — emit the error frame on shm_mq. 2. `AbortCurrentTransaction()` — let abort callbacks fire with the original error still on the stack. 3. `FlushErrorState()` — clear the original worker error. 4. `pgstat_report_activity(STATE_IDLE, NULL)`, `ReadyForQuery`, `pq_flush`. 5. `proc_exit(1)` — interrupts deliberately remain held; resuming them immediately before `proc_exit` could dispatch a queued second-cancel into proc_exit's cleanup chain via `CHECK_FOR_INTERRUPTS` with no live `PG_TRY` to catch the resulting `ereport(ERROR)`. Cancellation is cooperative only: the launcher sends SIGTERM and, for a non-zero grace period, waits for the worker to stop, but it never escalates to SIGKILL. Force-killing a bgworker would make the postmaster treat the death as a crash and restart the whole cluster. (Earlier 2.0 builds did escalate to SIGKILL after the grace period and crashed the cluster on slow hosts when the grace timer beat a still-starting worker; see README "Known Limitations §10".) pg_background-2.0.2/docs/CI.md000066400000000000000000000256701521265151100160760ustar00rootroot00000000000000# CI/CD Documentation ## Overview The pg_background CI pipeline uses GitHub Actions with containerized PostgreSQL to ensure consistent, deterministic testing across multiple PostgreSQL versions (14-19, where 19 is a beta target). The workflow builds the extension on Ubuntu runners with proper development headers and copies the built artifacts into PostgreSQL Docker containers for testing. ## Quick Start ### Local Testing with Docker The easiest way to run tests locally is using the provided `scripts/test-local.sh` script: ```bash # Test with default PostgreSQL version (17) ./scripts/test-local.sh # Test with a specific version ./scripts/test-local.sh 14 ./scripts/test-local.sh 15 ./scripts/test-local.sh 16 ./scripts/test-local.sh 17 ./scripts/test-local.sh 18 ./scripts/test-local.sh 19 # PostgreSQL 19 beta1 # Test all supported versions (14-19) ./scripts/test-local.sh all ``` **Requirements**: Docker must be installed and running. No local PostgreSQL installation required. ## CI Workflow Architecture ### Jobs | Job | Purpose | Runs On | Timeout | |-----|---------|---------|---------| | **test** | Build and test against the PostgreSQL × Ubuntu matrix | ubuntu-22.04, ubuntu-24.04 (PG 14–19) | 15 min | | **relocatable-test** | Verify `CREATE EXTENSION ... WITH SCHEMA` on every supported PG | ubuntu-24.04 (PG 14–19) | 15 min | | **upgrade-test** | Validate the 1.8 → 1.9 → 1.10 → 2.0 upgrade chain | ubuntu-24.04 (PG 14–18) | 15 min | | **assert-test** | Run the regression suite against an assert-enabled PG build | ubuntu-24.04 (PG 14–18) | 30 min | | **sanitizer-test** | Run the regression suite under AddressSanitizer + UndefinedBehaviorSanitizer | ubuntu-24.04 (PG 17) | 30 min | | **test-summary** | Aggregate matrix results into a single status check | ubuntu-24.04 | — | | **lint** | Static analysis (**blocking** cppcheck + clang-format) | ubuntu-24.04 | 10 min | | **security** | CodeQL security scanning | ubuntu-latest | 20 min | ### Test Matrix The **test** job runs against all combinations: | Ubuntu Version | PostgreSQL Versions | |----------------|---------------------| | 22.04 | 14, 15, 16, 17, 18, 19 | | 24.04 | 14, 15, 16, 17, 18, 19 | PostgreSQL 19 is a beta target: the server runs from the `postgres:19beta1` image while the build uses `postgresql-server-dev-19`. The `upgrade-test` and `assert-test` jobs stay at 14–18 — the former builds the prior v1.10 binary (which only supports 14–18) and the latter builds from the X.0 GA source tarball (not published during the beta cycle). **Per-job parallelism**: - `test`: **12** (2 OS × 6 PG) - `relocatable-test`: **6** (PG 14–19) - `upgrade-test`: **5** (PG 14–18) - `assert-test`: **5** (PG 14–18) - `sanitizer-test`: **1** (PG 17 only — the build is expensive; add more versions if a class of issue is suspected to be PG-major-specific) - `test-summary`, `lint`, `security`: 1 each **Grand total: 31 jobs per CI run.** This is up from 19 in the pre-2.0 layout because the relocatable, upgrade, and sanitizer paths were either single-shot or didn't exist; 2.0's matrix expansion is deliberate so a PG-major-specific issue in any of those paths cannot slip through. ### Sanitizer build details The `sanitizer-test` job builds PostgreSQL from source with `-fsanitize=address,undefined -fno-omit-frame-pointer -fno-sanitize-recover=all -O1 -g3` and builds pg_background with the matching flags. Loading an instrumented `.so` into a vanilla PG silently misses bugs because the runtime allocator is unhooked, so we need a PG that was itself instrumented. Runtime knobs: - `ASAN_OPTIONS=detect_leaks=0:abort_on_error=1:halt_on_error=1` — leak detection is off because postmaster's small known leaks would otherwise drown out real issues; halt+abort on error so CI fails on the first real find. - `UBSAN_OPTIONS=halt_on_error=1:print_stacktrace=1` — same idea. ### Workflow Triggers - **Push**: `master`, `main`, `develop`, `v1.*`, `improvements/*` branches - **Tags**: `v*` (releases) - **Pull Requests**: To `master` or `main` - **Manual**: Via `workflow_dispatch` ### Concurrency Control The workflow automatically cancels in-progress runs when new commits are pushed to the same branch, saving CI minutes and providing faster feedback. ## Build and Test Flow ``` ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ Start Docker │────▶│ Build on Runner │────▶│ Copy to Docker │ │ PostgreSQL │ │ (with dev hdrs) │ │ Container │ └─────────────────┘ └──────────────────┘ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Run Regression │ │ Tests │ └─────────────────┘ ``` 1. **Start Container**: PostgreSQL container started with `docker run` 2. **Build on Runner**: Extension built with PGDG development headers 3. **Copy Artifacts**: Built `.so` and SQL files copied into running container 4. **Run Tests**: Regression tests connect to containerized PostgreSQL ### Key Features - **APT Package Caching**: Faster subsequent runs - **Parallel Matrix Execution**: All 12 test combinations run simultaneously - **Artifact Upload on Failure**: Regression diffs available for debugging - **clang/llvm Symlink Handling**: Automatic compatibility for PGXS requirements ## Extension Features Tested The regression tests verify all pg_background v1.8 functionality: - **Core API** (canonical, unsuffixed in 2.0): `pg_background_launch()`, `pg_background_result()`, `pg_background_detach()`, `pg_background_cancel()` - **Deprecated `_v2` aliases**: `pg_background_launch_v2()`, … (kept through 2.x, removed in 3.0) - **Wait Functions**: `pg_background_wait()` (single entrypoint; `timeout_ms` arg replaces the old `_timeout` variant) - **Progress Reporting**: `pg_background_report_progress()`, `pg_background_get_progress()` - **Statistics**: `pg_background_stats()`, `pg_background_list` (view) / `pg_background_list()` - **GUC Settings**: `pg_background.max_workers`, `pg_background.default_queue_size`, `pg_background.worker_timeout` ## Manual Local Testing If you prefer not to use `scripts/test-local.sh`, follow these steps: ### 1. Start PostgreSQL Container ```bash PG_VERSION=17 docker run --name postgres-test -d \ -e POSTGRES_PASSWORD=postgres \ -e POSTGRES_USER=postgres \ -e POSTGRES_DB=postgres \ -p 5432:5432 \ postgres:${PG_VERSION} # Wait for PostgreSQL to be ready for i in {1..30}; do if docker exec postgres-test pg_isready -U postgres >/dev/null 2>&1; then echo "PostgreSQL is ready" break fi echo "Waiting... ($i/30)" sleep 2 done ``` ### 2. Install Build Dependencies ```bash # Add PostgreSQL APT repository sudo apt-get update sudo apt-get install -y ca-certificates wget gnupg lsb-release build-essential libkrb5-dev sudo install -d -m 0755 /usr/share/keyrings wget -qO- https://www.postgresql.org/media/keys/ACCC4CF8.asc \ | sudo gpg --dearmor -o /usr/share/keyrings/postgresql.gpg echo "deb [signed-by=/usr/share/keyrings/postgresql.gpg] \ https://apt.postgresql.org/pub/repos/apt \ $(lsb_release -cs)-pgdg main" \ | sudo tee /etc/apt/sources.list.d/pgdg.list sudo apt-get update sudo apt-get install -y \ postgresql-client-${PG_VERSION} \ postgresql-server-dev-${PG_VERSION} ``` ### 3. Build the Extension ```bash export PG_CONFIG=/usr/lib/postgresql/${PG_VERSION}/bin/pg_config make clean && make ``` ### 4. Copy to Container ```bash PKGLIBDIR=$($PG_CONFIG --pkglibdir) SHAREDIR=$($PG_CONFIG --sharedir) docker exec postgres-test mkdir -p "$PKGLIBDIR" "$SHAREDIR/extension" docker cp pg_background.so postgres-test:$PKGLIBDIR/ docker cp pg_background.control postgres-test:$SHAREDIR/extension/ for f in pg_background--*.sql; do docker cp "$f" postgres-test:$SHAREDIR/extension/; done ``` ### 5. Run Tests ```bash export PGHOST=127.0.0.1 PGPORT=5432 PGUSER=postgres PGPASSWORD=postgres PGDATABASE=postgres export PATH=/usr/lib/postgresql/${PG_VERSION}/bin:$PATH make installcheck REGRESS_OPTS+=" --host=$PGHOST --port=$PGPORT --user=$PGUSER" ``` ### 6. Cleanup ```bash docker stop postgres-test && docker rm postgres-test ``` ## Troubleshooting ### Build Fails with "Cannot find postgres.h" Ensure `PG_CONFIG` points to the correct `pg_config`: ```bash export PG_CONFIG=/usr/lib/postgresql/${PG_VERSION}/bin/pg_config $PG_CONFIG --includedir-server # Should show header directory make clean && make ``` ### Build Fails with clang-19 or llvm-lto Not Found PGXS may expect specific clang/llvm versions. Create symlinks: ```bash sudo ln -sf /usr/bin/clang /usr/bin/clang-19 LLVM_VER=$(ls -d /usr/lib/llvm-* 2>/dev/null | sort -V | tail -1 | sed 's|.*/llvm-||') sudo mkdir -p /usr/lib/llvm-19/bin sudo ln -sf /usr/lib/llvm-${LLVM_VER}/bin/llvm-lto /usr/lib/llvm-19/bin/llvm-lto ``` ### Tests Fail with Connection Errors Verify container is running: ```bash docker ps docker exec postgres-test pg_isready -U postgres psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "SELECT version();" ``` ### Extension Not Found After Copying Verify paths match `pg_config`: ```bash docker exec postgres-test ls -la /usr/lib/postgresql/${PG_VERSION}/lib/pg_background.so docker exec postgres-test ls -la /usr/share/postgresql/${PG_VERSION}/extension/pg_background.control ``` ### Regression Test Diffs Check the diff output: ```bash cat regression.diffs cat regression.out ls results/ ``` ## CI Environment Variables | Variable | Description | Example | |----------|-------------|---------| | `PG_CONFIG` | Path to pg_config binary | `/usr/lib/postgresql/17/bin/pg_config` | | `PGHOST` | PostgreSQL host | `127.0.0.1` | | `PGPORT` | PostgreSQL port | `5432` | | `PGUSER` | Database user | `postgres` | | `PGPASSWORD` | Database password | `postgres` | | `PGDATABASE` | Database name | `postgres` | | `DEFAULT_PG_VERSION` | Default PG version for lint/security | `17` | ## Contributing When modifying CI: 1. Test changes locally using `./scripts/test-local.sh` first 2. Consider all matrix combinations (10 total) 3. Update this documentation if workflow changes 4. Keep YAML readable; complex logic goes in step scripts ## References - [PostgreSQL Docker Hub](https://hub.docker.com/_/postgres) - [GitHub Actions Documentation](https://docs.github.com/en/actions) - [PGXS Build System](https://www.postgresql.org/docs/current/extend-pgxs.html) - [PostgreSQL APT Repository](https://wiki.postgresql.org/wiki/Apt) - [CodeQL for C/C++](https://codeql.github.com/docs/codeql-language-guides/codeql-for-cpp/) pg_background-2.0.2/docs/CONTRIBUTING.md000066400000000000000000000204771521265151100175120ustar00rootroot00000000000000# Contributing to pg_background Thank you for your interest in contributing to pg_background! This document provides guidelines for contributing to the project. ## Table of Contents - [Code of Conduct](#code-of-conduct) - [Getting Started](#getting-started) - [Development Workflow](#development-workflow) - [Coding Standards](#coding-standards) - [Testing Requirements](#testing-requirements) - [Commit Guidelines](#commit-guidelines) - [Pull Request Process](#pull-request-process) - [Review Process](#review-process) --- ## Code of Conduct This project follows the [PostgreSQL Community Code of Conduct](https://www.postgresql.org/about/policies/coc/). Please be respectful and constructive in all interactions. --- ## Getting Started ### Prerequisites - PostgreSQL 14 or later (development headers required) - GCC or Clang compiler - `make` build system - `git` for version control ### Setting Up Development Environment ```bash # Clone the repository git clone https://github.com/vibhorkum/pg_background.git cd pg_background # Build the extension make clean make # Install to PostgreSQL (requires appropriate permissions) sudo make install # Run regression tests make installcheck ``` --- ## Development Workflow 1. **Fork** the repository on GitHub 2. **Create a branch** for your feature or bugfix: ```bash git checkout -b feature/my-feature # or git checkout -b bugfix/issue-123 ``` 3. **Make your changes** following the coding standards 4. **Test thoroughly** (see Testing Requirements) 5. **Commit** your changes with clear messages 6. **Push** to your fork: ```bash git push origin feature/my-feature ``` 7. **Submit a Pull Request** with a clear description --- ## Coding Standards ### PostgreSQL Coding Conventions pg_background follows [PostgreSQL coding conventions](https://www.postgresql.org/docs/current/source.html): #### Style Guidelines - **Indentation**: 4 spaces (no tabs) - **Line length**: Max 80 characters (soft limit; 100 hard limit) - **Braces**: K&R style (opening brace on same line) - **Comments**: Use C-style `/* */` comments, not C++ `//` #### Naming Conventions ```c /* Functions: lowercase with underscores */ static void cleanup_worker_info(dsm_segment *seg, Datum pid_datum); /* Types: lowercase with underscores */ typedef struct pg_background_worker_info { /* ... */ } pg_background_worker_info; /* Macros: UPPERCASE with underscores */ #define PG_BACKGROUND_MAGIC 0x50674267 /* Constants: UPPERCASE or lowercase depending on context */ #define PGBG_SQL_PREVIEW_LEN 120 ``` #### Memory Management - **Always** use PostgreSQL memory contexts (`palloc`, `pfree`) - **Never** use C library `malloc`/`free` - Use `TopMemoryContext` for session-lifetime allocations - Clean up resources in error paths using `PG_TRY`/`PG_CATCH` #### Error Handling ```c /* Use ereport for user-facing errors */ ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("queue size must be at least %zu bytes", shm_mq_minimum_size))); /* Use elog for internal assertions */ elog(DEBUG1, "worker_hash entry for PID %d already removed", pid); ``` ### Code Formatting We use `pgindent` for automatic formatting: ```bash # Format a single file pgindent src/pg_background.c # Check formatting without modifying pgindent --check src/pg_background.c ``` **Note**: `pgindent` requires PostgreSQL source tree. See [PostgreSQL wiki](https://wiki.postgresql.org/wiki/Running_pgindent) for setup. --- ## Testing Requirements ### Regression Tests All code changes must include regression tests in `sql/pg_background.sql`: ```sql -- Test your new feature SELECT * FROM pg_background_launch('SELECT 1') AS h \gset SELECT * FROM pg_background_result(:h_pid, :h_cookie) AS (result int); ``` Update expected output in `expected/pg_background.out` if needed. ### Running Tests ```bash # Run all regression tests make installcheck # Run specific test make installcheck REGRESS=pg_background # Clean up test artifacts make installcheckclean ``` ### Test Coverage Goals - **Critical paths**: 100% coverage (launch, result, detach, cancel) - **Edge cases**: PID reuse, DSM cleanup races, error handling - **Error conditions**: Invalid inputs, permission failures - **Concurrency**: Concurrent detach/result operations ### Manual Testing Checklist Before submitting a PR: - [ ] Tested on PostgreSQL 14 (minimum version) - [ ] Tested on latest stable PostgreSQL version - [ ] No compiler warnings (`-Wall -Wextra`) - [ ] No memory leaks (use `valgrind` if available) - [ ] No resource leaks (check `pg_stat_activity`, `pg_shmem_allocations`) - [ ] Regression tests pass - [ ] Documentation updated if API changed --- ## Commit Guidelines ### Commit Message Format ``` Short summary (50 chars max) Detailed explanation of the change, including: - What was changed and why - Any behavior changes - References to issues (e.g., "Fixes #123") (optional) Breaking changes or migration notes ``` ### Good Commit Messages ✅ **Good**: ``` Add CHECK_FOR_INTERRUPTS to result loop Allows cancellation of long-running result retrieval. Without this, Ctrl-C or pg_terminate_backend() won't interrupt the launcher session while it's blocked reading results. Fixes #456 ``` ❌ **Bad**: ``` Fix bug ``` ### Atomic Commits - One logical change per commit - Commit compiles and passes tests - Can be cherry-picked or reverted independently --- ## Pull Request Process ### Before Submitting 1. Rebase on latest `main`: ```bash git fetch origin git rebase origin/main ``` 2. Squash fixup commits if needed: ```bash git rebase -i origin/main ``` 3. Run full test suite: ```bash make clean && make && make installcheck ``` ### PR Description Template ```markdown ## Description Brief summary of changes ## Motivation Why is this change needed? ## Changes Made - Change 1 - Change 2 ## Testing - [ ] Regression tests added/updated - [ ] Manual testing performed - [ ] Tested on PG 14 - [ ] Tested on latest PG ## Breaking Changes None / List any breaking changes ## Checklist - [ ] Code follows style guidelines - [ ] Comments added for complex logic - [ ] Documentation updated - [ ] Regression tests pass - [ ] No compiler warnings ``` ### PR Size Guidelines - **Small**: < 100 lines changed (preferred) - **Medium**: 100-500 lines - **Large**: > 500 lines (split if possible) Large PRs should be split into smaller, logical commits for easier review. --- ## Review Process ### What Reviewers Look For 1. **Correctness**: Does it solve the problem? 2. **Safety**: Memory leaks? Race conditions? Resource cleanup? 3. **Style**: Follows PostgreSQL conventions? 4. **Tests**: Adequate test coverage? 5. **Documentation**: Clear comments and README updates? 6. **Compatibility**: Works on PG 14-19 (19 = beta)? ### Responding to Feedback - Be open to suggestions and constructive criticism - Respond to all comments, even if just "Fixed" or "Done" - Push new commits for changes (don't force-push during review) - Request re-review when ready ### Approval Criteria PRs require: - ✅ At least one approval from maintainer - ✅ All CI checks passing - ✅ No unresolved comments - ✅ Documentation updated (if applicable) --- ## Development Tips ### Debugging ```sql -- Enable debug logging SET client_min_messages = DEBUG1; -- Check active workers SELECT * FROM pg_background_list() AS (...); -- Check DSM usage SELECT * FROM pg_shmem_allocations WHERE name LIKE '%pg_background%'; ``` ### Common Pitfalls ❌ **Don't**: Call `pfree()` on `BackgroundWorkerHandle` ✅ **Do**: Let PostgreSQL manage handle lifetime ❌ **Don't**: Use `malloc()`/`free()` ✅ **Do**: Use `palloc()`/`pfree()` with proper memory context ❌ **Don't**: Ignore error context cleanup ✅ **Do**: Use `PG_TRY`/`PG_CATCH` for robust error handling ### Useful Resources - [PostgreSQL Backend Internals](https://www.postgresql.org/docs/current/source.html) - [Background Workers Documentation](https://www.postgresql.org/docs/current/bgworker.html) - [Dynamic Shared Memory](https://www.postgresql.org/docs/current/shm-mq.html) - [PostgreSQL Extension Best Practices](https://wiki.postgresql.org/wiki/ExtensionBestPractices) --- ## Questions? - **GitHub Issues**: https://github.com/vibhorkum/pg_background/issues - **Discussions**: https://github.com/vibhorkum/pg_background/discussions - **Mailing List**: pgsql-general@postgresql.org --- Thank you for contributing! 🎉 pg_background-2.0.2/docs/COOKBOOK.md000066400000000000000000000254531521265151100170100ustar00rootroot00000000000000# pg_background — Cookbook Copy-paste templates for the most common patterns. All recipes use the v2 API (cookie-protected handles); the v1 API was removed in 2.0. Each example assumes the extension has been installed and the calling role has been granted `pgbackground_role` (see [`README.md`](../README.md) "Security Model"). --- ## Quick recipes (small, idiomatic) ### Synchronous run with metadata When you want autonomous-transaction semantics and just need to know whether it worked, how many rows were affected, and the SQLSTATE on failure. Returns metadata only — for result rows use the next recipe. ```sql SELECT pid, completed, timed_out, has_error, row_count, command_tag, sqlstate, error_message, elapsed_ms FROM pg_background_run( 'INSERT INTO audit_log SELECT now(), current_user, ''login''', queue_size := 0, timeout_ms := 30000, -- 30s cap; cancels with 1s grace on overrun label := 'audit-login' ); ``` ### Wait-with-timeout, then capture result rows or error When you need the actual result rows. Uses `outcome` to inspect state without raising; only calls `result` on the success path. ```sql DO $$ DECLARE h pg_background_handle; o pg_background_outcome; finished bool; BEGIN h := pg_background_launch('SELECT id, name FROM big_table WHERE active', 65536, 'lookup'); finished := pg_background_wait(h.pid, h.cookie, 5000); IF NOT finished THEN PERFORM pg_background_cancel(h.pid, h.cookie, 1000); PERFORM pg_background_detach(h.pid, h.cookie); RAISE EXCEPTION 'lookup did not complete within 5s'; END IF; o := pg_background_outcome(h.pid, h.cookie); IF o.has_error THEN PERFORM pg_background_detach(h.pid, h.cookie); RAISE EXCEPTION 'lookup failed: % (sqlstate %)', o.error_message, o.sqlstate; END IF; -- Safe: only consume result rows on the success path. INSERT INTO lookup_cache (id, name) SELECT id, name FROM pg_background_result(h.pid, h.cookie) AS r(id int, name text); PERFORM pg_background_detach(h.pid, h.cookie); END $$; ``` ### Launch many, gather outcomes When you fan out N independent jobs and want a per-worker outcome row. ```sql WITH launched AS ( SELECT (pg_background_launch( format('VACUUM (ANALYZE) %I', tablename), 0, 'nightly-vacuum-' || tablename)).* FROM pg_tables WHERE schemaname = 'public' ), waited AS ( SELECT l.pid, l.cookie, pg_background_wait(l.pid, l.cookie, 60000) AS finished FROM launched l ) SELECT w.pid, w.finished, o.completed, o.has_error, o.row_count, o.command_tag, o.sqlstate, o.error_message, o.label FROM waited w, LATERAL pg_background_outcome(w.pid, w.cookie) AS o; -- Per-handle detach is preferred (CLAUDE.md §7), but for one-shot scripts -- the batch helper is fine. SELECT pg_background_detach_all(); ``` ### Drain a fan-out with a shared deadline `drain` waits on every handle in an array against a shared wall-clock budget, returns one outcome row per input handle in input order, and detaches each one. ```sql SELECT pid, completed, has_error, row_count, label FROM pg_background_drain( ARRAY(SELECT pg_background_launch( format('SELECT count(*) FROM %I', t), 0, 'count-' || t) FROM pg_tables WHERE schemaname = 'public'), 60000 -- 60s shared deadline across all handles ); ``` ### "Did any of these finish?" — wait_any Polls a set of handles, returns the first one whose worker has stopped, or `NULL` on timeout. Useful for racing tasks. ```sql DO $$ DECLARE hs pg_background_handle[]; winner pg_background_handle; BEGIN hs := ARRAY[ pg_background_launch('SELECT pg_sleep(2); SELECT 1', 0, 'race-1'), pg_background_launch('SELECT pg_sleep(0.5); SELECT 2', 0, 'race-2'), pg_background_launch('SELECT pg_sleep(5); SELECT 3', 0, 'race-3') ]; winner := pg_background_wait_any(hs, 10000); RAISE NOTICE 'first finisher: pid=% (cookie=%)', winner.pid, winner.cookie; PERFORM count(*) FROM pg_background_drain(hs, 5000); -- cleanup the rest END$$; ``` ### Cancel by label Cancel every worker in this session whose label matches a SQL `LIKE` pattern. Useful for bulk-cancelling a logical workload identified by a label prefix. ```sql SELECT pg_background_cancel_by_label('nightly-vacuum-%'); ``` --- ## End-to-end use cases ### Background maintenance operations VACUUM blocks client connections and consumes resources. Run it asynchronously instead: ```sql -- Use the pg_background_list view (no column-definition list required). SELECT (pg_background_launch( 'VACUUM (VERBOSE, ANALYZE) large_table', 0, 'maintenance-vacuum')).* \gset h_ -- Check progress. SELECT state, sql_preview, launched_at FROM pg_background_list WHERE pid = :h_pid AND cookie = :h_cookie; -- Wait for completion (optional). SELECT pg_background_wait(:h_pid, :h_cookie); SELECT pg_background_detach(:h_pid, :h_cookie); ``` ### Autonomous audit logging Audit logs must persist even if the main transaction rolls back. Use the worker for an independent commit. > **⚠️ Worker exhaustion**: if `max_worker_processes` is exhausted, > `pg_background_launch()` raises `ERRCODE_INSUFFICIENT_RESOURCES`. > For audit logging, that means the message is **lost** unless your > caller handles the error. The robust template below retries with > exponential backoff and falls back to a synchronous insert. **Basic** (not fault-tolerant): ```sql CREATE FUNCTION log_audit_simple(event_type text, details jsonb) RETURNS void LANGUAGE plpgsql AS $$ DECLARE h pg_background_handle; BEGIN h := pg_background_submit( format( 'INSERT INTO audit_log (ts, event_type, details) VALUES (now(), %L, %L)', event_type, details::text)); PERFORM pg_background_detach(h.pid, h.cookie); END; $$; ``` **Robust** (handles worker exhaustion): ```sql CREATE FUNCTION log_audit(event_type text, details jsonb) RETURNS void LANGUAGE plpgsql AS $$ DECLARE h pg_background_handle; retries int := 3; backoff_ms int := 100; BEGIN FOR i IN 1..retries LOOP BEGIN h := pg_background_submit( format( 'INSERT INTO audit_log (ts, event_type, details) VALUES (now(), %L, %L)', event_type, details::text)); PERFORM pg_background_detach(h.pid, h.cookie); RETURN; EXCEPTION WHEN insufficient_resources THEN IF i = retries THEN -- Final fallback: synchronous insert (blocks but doesn't lose data). INSERT INTO audit_log (ts, event_type, details) VALUES (now(), event_type, details); RAISE WARNING 'pg_background exhausted, audit logged synchronously'; RETURN; END IF; PERFORM pg_sleep(backoff_ms / 1000.0); backoff_ms := backoff_ms * 2; END; END LOOP; END; $$; ``` **Usage in a transaction**: ```sql BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = 123; PERFORM log_audit('withdrawal', '{"account": 123, "amount": 100}'); ROLLBACK; -- audit row still exists SELECT * FROM audit_log ORDER BY ts DESC LIMIT 1; ``` ### Asynchronous notification delivery `pg_notify()` in the main transaction delays commit. Offload to a worker: ```sql CREATE FUNCTION notify_async(channel text, payload text) RETURNS void LANGUAGE plpgsql AS $$ DECLARE h pg_background_handle; BEGIN h := pg_background_submit( format('SELECT pg_notify(%L, %L)', channel, payload)); PERFORM pg_background_detach(h.pid, h.cookie); END; $$; -- Usage: SELECT notify_async('order_updates', '{"order_id": 456, "status": "shipped"}'); ``` > **NOTIFY caveat**: NOTIFY frames raised inside a `submit` worker > are *not* relayed back to the launcher's session. The notify row hits > `pg_listener` / the notify queue in the worker's own commit, so any > session that called `LISTEN ` separately *will* receive it — > but the launching session won't see the protocol-level message echoed. ### Long-running ETL pipeline ETL blocks a client connection for hours. Launch in the background and poll for completion via the `pg_background_list` view. ```sql SELECT (pg_background_launch($$ INSERT INTO fact_sales SELECT * FROM staging_sales WHERE processed = false; UPDATE staging_sales SET processed = true; $$, 0, 'etl-001')).* \gset etl_ -- Track in your own jobs table: INSERT INTO job_tracker (job_id, pid, cookie, started_at) VALUES ('etl-001', :etl_pid, :etl_cookie, now()); -- Later: status check using the convenience view. SELECT j.job_id, w.state, w.launched_at, (now() - w.launched_at) AS duration FROM job_tracker j CROSS JOIN LATERAL ( SELECT * FROM pg_background_list WHERE pid = j.pid AND cookie = j.cookie ) w WHERE j.job_id = 'etl-001'; ``` ### Parallel query simulation PostgreSQL doesn't natively parallelize across separate tables. Launch a worker per table and aggregate: ```sql DO $$ DECLARE hs pg_background_handle[]; total_rows bigint; BEGIN hs := ARRAY[ pg_background_launch('SELECT count(*) FROM sales', 0, 'pq-sales'), pg_background_launch('SELECT count(*) FROM orders', 0, 'pq-orders'), pg_background_launch('SELECT count(*) FROM customers', 0, 'pq-customers') ]; PERFORM pg_background_wait(h.pid, h.cookie) FROM unnest(hs) h; SELECT sum(cnt) INTO total_rows FROM ( SELECT * FROM pg_background_result((hs[1]).pid, (hs[1]).cookie) AS (cnt bigint) UNION ALL SELECT * FROM pg_background_result((hs[2]).pid, (hs[2]).cookie) AS (cnt bigint) UNION ALL SELECT * FROM pg_background_result((hs[3]).pid, (hs[3]).cookie) AS (cnt bigint) ) t; RAISE NOTICE 'Total rows: %', total_rows; END$$; ``` ### Timeout enforcement Cancel queries that exceed a time budget: ```sql CREATE FUNCTION run_with_timeout(sql text, timeout_sec int) RETURNS text LANGUAGE plpgsql AS $$ DECLARE h pg_background_handle; done bool; result_text text; BEGIN h := pg_background_launch(sql); done := pg_background_wait(h.pid, h.cookie, timeout_sec * 1000); IF NOT done THEN RAISE WARNING 'Query timed out after % seconds, cancelling', timeout_sec; PERFORM pg_background_cancel(h.pid, h.cookie, 1000); PERFORM pg_background_detach(h.pid, h.cookie); RETURN 'TIMEOUT'; END IF; SELECT * INTO result_text FROM pg_background_result(h.pid, h.cookie) AS (res text); RETURN result_text; END; $$; -- Usage: SELECT run_with_timeout('SELECT pg_sleep(10)', 5); -- returns 'TIMEOUT' ``` For one-shot uses where you only need metadata, prefer `pg_background_run(sql, queue_size, timeout_ms, label)` — it does launch + wait-with-timeout + outcome + detach in a single call and records the timeout in `pg_background_stats().workers_timed_out`. pg_background-2.0.2/docs/MIGRATION.md000066400000000000000000000461331521265151100171310ustar00rootroot00000000000000# pg_background — Migration Guide This guide is the authoritative reference for moving between pg_background versions. Read the section for the version you're upgrading **from**; each section is self-contained. > **Supported upgrade source for 2.0 is 1.8 or newer.** If you are running > a pre-1.8 version, first upgrade to 1.8 against the 1.10 release line, then > follow the 1.10 → 2.0 instructions below. The pre-1.8 upgrade scripts that > previously lived in `extension/legacy/` were removed. --- ## 1.10 → 2.0 (major release) 2.0 is a deliberate cleanup release. Several APIs were renamed or removed, and several composite types gained forward-compatibility columns. Any migration that touches the surface needs a code change. The upgrade is invoked the usual way: ```sql ALTER EXTENSION pg_background UPDATE TO '2.0'; ``` The script in `extension/pg_background--1.10--2.0.sql` performs the changes described below atomically. After it runs, every grant the `pgbackground_role` previously held is reapplied; explicit grants you made to other roles for renamed objects need to be reissued (see *Privilege helpers* below). ### The `_v2` suffix is retired (canonical names + deprecated aliases) The `_v2` suffix existed only to distinguish the cookie-protected API from the original v1 functions. With v1 gone (below), the suffix no longer distinguishes anything, so **2.0 makes the unsuffixed names canonical**: | Through 1.10 | 2.0 canonical | 2.0 `_v2` alias | |---|---|---| | `pg_background_launch_v2(...)` | `pg_background_launch(...)` | kept, deprecated | | `pg_background_submit_v2(...)` | `pg_background_submit(...)` | kept, deprecated | | `pg_background_result_v2(...)` | `pg_background_result(...)` | kept, deprecated | | `pg_background_detach_v2(...)` | `pg_background_detach(...)` | kept, deprecated | | `pg_background_cancel_v2(...)` | `pg_background_cancel(...)` | kept, deprecated | | `pg_background_wait_v2(...)` | `pg_background_wait(...)` | kept, deprecated | | `pg_background_list_v2()` | `pg_background_list()` | kept, deprecated | | `pg_background_stats_v2()` | `pg_background_stats()` | kept, deprecated | | `pg_background_get_progress_v2(...)` | `pg_background_get_progress(...)` | kept, deprecated | | `pg_background_result_info_v2(...)` | `pg_background_result_info(...)` | kept, deprecated | | `pg_background_error_info_v2(...)` | `pg_background_error_info(...)` | kept, deprecated | | `pg_background_detach_all_v2()` | `pg_background_detach_all()` | kept, deprecated | | `pg_background_cancel_all_v2()` | `pg_background_cancel_all()` | kept, deprecated | | `pg_background_outcome_v2(...)` | `pg_background_outcome(...)` | kept, deprecated | | `pg_background_run_v2(...)` | `pg_background_run(...)` | kept, deprecated | | `pg_background_run_query_v2(...)` | `pg_background_run_query(...)` | kept, deprecated | | `pg_background_drain_v2(...)` | `pg_background_drain(...)` | kept, deprecated | | `pg_background_wait_any_v2(...)` | `pg_background_wait_any(...)` | kept, deprecated | | `pg_background_cancel_by_label_v2(...)` | `pg_background_cancel_by_label(...)` | kept, deprecated | | `pg_background_purge_v2()` | `pg_background_purge()` | kept, deprecated | | `pg_background_full_sql_v2(...)` | `pg_background_full_sql(...)` | kept, deprecated | **No code change is required for the rename.** Every `_v2` name that shipped through 1.10 is kept as a thin deprecated alias that forwards to the canonical function with identical behavior; the aliases are **slated for removal in 3.0**. Migrate at your own pace — new code should use the unsuffixed names. A `DEPRECATED ... Removed in 3.0` comment is attached to each alias (`\df+`). The canonical names coexist with same-named objects of a different kind, resolved by call syntax: - `pg_background_list` is the **view** (preferred for monitoring); `pg_background_list()` is the raw set-returning function. - `pg_background_stats` / `pg_background_outcome` are **composite types**; `pg_background_stats()` / `pg_background_outcome(...)` are functions. Names that are **new in 2.0** have no `_v2` alias because no released `_v2` name ever existed for them: `pg_background_report_progress` (the worker-side writer that shipped through 1.10 was `pg_background_progress`, renamed below), the internal `pg_background_record_timeout`, and the privilege helpers (renamed from `grant_pg_background_privileges` / `revoke_pg_background_privileges`, below). ### Removed: the v1 API; the unsuffixed names now mean the v2 API The original v1 functions — `pg_background_launch(sql, queue_size)` returning `int4`, `pg_background_result(pid)`, `pg_background_detach(pid)` — are gone. They were thin shims over the v2 path that existed for backward compatibility with releases before 1.6. Note that the unsuffixed names are **back, but with the cookie-protected v2 semantics** (see the rename table above): `pg_background_launch` now returns a `pg_background_handle` (`pid` + `cookie`), not a bare `int4`. If you have v1 muscle memory, the name is the same but the shape changed. | Before (1.x, v1) | After (2.0, canonical) | |---|---| | `pg_background_launch(sql, queue_size)` returns `int4` | `pg_background_launch(sql, queue_size)` returns `pg_background_handle` (pid + cookie) | | `pg_background_result(pid)` | `pg_background_result(pid, cookie)` | | `pg_background_detach(pid)` | `pg_background_detach(pid, cookie)` | The rewrite is mechanical: capture both `pid` and `cookie` from the handle returned by `launch`/`submit`, and pass the cookie to every later operation on that worker. (The `_v2`-suffixed names work identically if you prefer to defer the rename — they are deprecated aliases removed in 3.0.) ```sql -- Before (v1) SELECT pg_background_launch('SELECT 1') AS pid \gset SELECT * FROM pg_background_result(:pid) AS (x int); -- After (2.0 canonical) SELECT (h).pid AS pid, (h).cookie AS cookie FROM (SELECT pg_background_launch('SELECT 1') AS h) s \gset SELECT * FROM pg_background_result(:pid, :cookie) AS (x int); ``` The cookie protects against PID-reuse hits — the worker you launched cannot be confused with an unrelated worker that happens to acquire the same PID later. ### Collapsed: `cancel_v2` / `wait_v2` overloads The `_grace` and `_timeout` suffix variants were merged into the base function with an extra parameter that defaults to a sensible value. | Before (1.10) | After (2.0) | |---|---| | `pg_background_cancel_v2(pid, cookie)` | `pg_background_cancel_v2(pid, cookie)` (unchanged — uses `grace_ms = 0`) | | `pg_background_cancel_v2_grace(pid, cookie, grace_ms)` | `pg_background_cancel_v2(pid, cookie, grace_ms)` | | `pg_background_wait_v2(pid, cookie)` returns `void` | `pg_background_wait_v2(pid, cookie)` returns `bool` (true) — uses `timeout_ms = 0`, blocks indefinitely | | `pg_background_wait_v2_timeout(pid, cookie, timeout_ms)` returns `bool` | `pg_background_wait_v2(pid, cookie, timeout_ms)` returns `bool` | Two semantic notes worth surfacing: - `wait_v2` now **always returns `bool`**. Old callers that ignored the void return value continue to work; old callers that assigned the result to a variable see no change either way. - For `wait_v2`, `timeout_ms <= 0` blocks indefinitely (matches the 1.x default of `wait_v2(pid, cookie)`) while `timeout_ms > 0` waits up to that many milliseconds. **`timeout_ms = 0` does not poll.** If you want a poll ("is it done yet?"), pass `timeout_ms = 1`. The legacy `wait_v2_timeout(pid, cookie, 0)` polling pattern needs to be rewritten to `wait_v2(pid, cookie, 1)`. ### Removed: `pg_background_status_v2` The jsonb wrapper around `pg_background_outcome_v2` was a thin one-liner. Drivers that decode JSON natively can call it themselves: ```sql -- Before SELECT pg_background_status_v2(pid, cookie); -- After SELECT to_jsonb(pg_background_outcome_v2(pid, cookie)); ``` ### Renamed: progress reporting The worker-side write call had no `_v2` suffix in 1.x and clashed with the type of the same name. In 2.0 both were renamed for coherence. | Before (1.x) | After (2.0) | |---|---| | `pg_background_progress(pct, msg)` (function) | `pg_background_report_progress(pct, msg)` | | `pg_background_progress` (type) | `pg_background_progress_info` | | `pg_background_get_progress_v2(pid, cookie)` returns `pg_background_progress` | `pg_background_get_progress(pid, cookie)` returns `pg_background_progress_info` | If your worker SQL calls `pg_background_progress(50, 'halfway')`, change the call to `pg_background_report_progress(50, 'halfway')`. This is a hard rename with **no alias** — the old `pg_background_progress` name is gone. The type rename is only visible if you explicitly reference the type by name (most callers don't). ### Renamed: privilege helpers | Before (1.x) | After (2.0) | |---|---| | `grant_pg_background_privileges(role)` | `pg_background_grant_privileges(role)` | | `revoke_pg_background_privileges(role)` | `pg_background_revoke_privileges(role)` | The unprefixed names were polluting the install schema. These are hard renames with **no `_v2` alias** (the `_v2`-suffixed privilege-helper names never shipped in a released version). After the upgrade script runs, the helper is reapplied to `pgbackground_role` automatically; if you've granted to other roles, reissue the grant by calling the new helper: ```sql SELECT pg_background_grant_privileges('app_executor', false); ``` ### Forward-compatible additions to composite types These are not breaking *if* you are using positional decoding, but they *are* breaking if you SELECT specific columns into a row variable that has the old shape. Most code uses named-field access, in which case nothing changes. | Type | New columns | Notes | |---|---|---| | `pg_background_stats` | `workers_timed_out int8` | Separate counter from `workers_canceled`, bumped by `pg_background_run_v2` on timeout | | `pg_background_result_info` | `started_at`, `finished_at` (timestamptz, nullable) | Worker writes these around the SPI loop | | `pg_background_error` | `schema_name`, `table_name`, `column_name`, `constraint_name` (text, nullable) | Sourced from PG's `edata`; populated for heap/access-layer errors | | `pg_background_run_result` | now extends `pg_background_outcome` (gains `cookie`, `state`, `consumed`, `label`, `launched_at`) plus `timed_out`, `elapsed_ms` | Replaces 1.10's standalone shape | If you do `SELECT * INTO some_row FROM pg_background_run_v2(...)`, `some_row` must have the new wider shape. PL/pgSQL will give you a clear "column count mismatch" error if not — easy to spot. ### What didn't change - `pg_background_handle` type — same shape. - `pg_background_launch_v2`, `pg_background_submit_v2`, `pg_background_result_v2`, `pg_background_detach_v2` — unchanged signatures. - `pg_background_outcome_v2`, `pg_background_outcome` type — unchanged. - `pg_background_list`, `pg_background_activity` views — unchanged. - `pg_background_full_sql_v2` — unchanged. - `pg_background_detach_all_v2`, `pg_background_cancel_all_v2` — unchanged. - The Tier A helpers (`run_query_v2`, `drain_v2`, `wait_any_v2`, `cancel_by_label_v2`, `purge_v2`) — same SQL signatures; their bodies were rewired to use the new `cancel_v2` / `wait_v2`. ### Internal: the workers_timed_out counter `pg_background_run_v2` now records a session-local counter when it cancels a worker due to its own `timeout_ms` parameter. `pg_background_stats_v2()` exposes this counter as `workers_timed_out`. A separate timeout via `statement_timeout` inside the worker will continue to be classified as a worker error (`workers_failed`) rather than a timeout. ### Privilege model `pgbackground_role` is unchanged. The grant helper is invoked once at the end of the upgrade script with `pgbackground_role`, which restores every function/type/view grant the role used to hold against the new surface. If you depend on the previous unprefixed helpers (e.g. inside a deployment playbook that calls `grant_pg_background_privileges(...)`), update it to the new name. --- ## Older upgrade paths For pre-1.10 source versions, chain through the supported steps. 2.0 itself only ships the 1.10 → 2.0 upgrade script; if you're on 1.7 or older you must reach 1.10 against the 1.10 release line first. ```sql -- pre-1.8 → 1.8 (against 1.10 release line) ALTER EXTENSION pg_background UPDATE TO '1.8'; -- 1.8 → 1.9 ALTER EXTENSION pg_background UPDATE TO '1.9'; -- 1.9 → 1.10 ALTER EXTENSION pg_background UPDATE TO '1.10'; -- 1.10 → 2.0 (this guide, above) ALTER EXTENSION pg_background UPDATE TO '2.0'; ``` ### 1.9 → 1.10 ```sql ALTER EXTENSION pg_background UPDATE TO '1.10'; ``` What you get: - `pg_background_list` view (no column-definition list at the call site). - `pg_background_activity` view (joined with `pg_stat_activity`). - `pg_background_outcome_v2()` — never-raises status snapshot. - `pg_background_run_v2()` — synchronous one-shot. Action items: - No code change required. Existing v1 and v2 callers keep working. - Optional: replace ad-hoc column-def lists with the new `pg_background_list` view in monitoring queries. - Optional: replace bespoke `launch + wait + cleanup` wrappers with `pg_background_run_v2()`. ### 1.8 → 1.9 ```sql ALTER EXTENSION pg_background UPDATE TO '1.9'; ``` What you get: - `label` parameter on `launch_v2`/`submit_v2` for operational clarity. - `pg_background_error_info_v2()` returning structured errors with real `SQLSTATE`. - `pg_background_result_info_v2()` for row count / command tag / completion flags. - Batch helpers: `pg_background_detach_all_v2()`, `pg_background_cancel_all_v2()`. ### 1.7 → 1.8 ```sql ALTER EXTENSION pg_background UPDATE TO '1.8'; ``` What you get: - `pg_background_stats_v2()` — session statistics. - `pg_background_progress()` (renamed to `pg_background_report_progress` in 2.0) — worker progress reporting. - `pg_background_get_progress_v2()` — get worker progress. - GUCs: `max_workers`, `worker_timeout`, `default_queue_size`. - Built-in `max_workers` enforcement. - Robustness fixes: overflow protection, UTF-8-aware truncation. Action items: - Review the new GUC settings and configure as needed. - Consider using progress reporting for long-running workers. - Use `stats_v2()` for monitoring. ### 1.6 → 1.7 ```sql ALTER EXTENSION pg_background UPDATE TO '1.7'; ``` Changes: - Cryptographically secure cookie generation. - Dedicated memory context (prevents session bloat). - Exponential backoff polling (reduces CPU usage). - **Fix**: custom-schema installation (`CREATE EXTENSION ... WITH SCHEMA`). - No breaking changes. > **⚠️ Upgrade note**: custom-schema support is only available for > *fresh* installs of 1.7+. If you already have 1.4/1.5/1.6 installed, > the extension is in `public` because those versions did not support > custom schemas. The upgrade scripts contain hardcoded `public.` > references and cannot relocate the extension. To move an existing > install to a custom schema, drop and reinstall: > > ```sql > DROP EXTENSION pg_background; > CREATE SCHEMA IF NOT EXISTS myschema; > CREATE EXTENSION pg_background WITH SCHEMA myschema; > ``` ### 1.5 → 1.6 ```sql ALTER EXTENSION pg_background UPDATE TO '1.6'; ``` Changes: - v1 API unchanged (fully backward compatible). - New v2 API functions added. - `pgbackground_role` created automatically. - Hardened privilege helpers added. - No breaking changes. Action items: - Review privilege grants (1.6 revokes PUBLIC access). - Grant `pgbackground_role` to application users. - Migrate v1 API calls to v2 in new code. ### 1.0 – 1.4 → 1.6 ```sql ALTER EXTENSION pg_background UPDATE TO '1.4'; ALTER EXTENSION pg_background UPDATE TO '1.6'; ``` Breaking changes along the way: - 1.4 removed PostgreSQL 9.x support. - 1.5 changed DSM lifecycle (no functional API changes). - 1.6 revoked PUBLIC access (requires explicit grants). Action items: - Test on non-production first. - Audit existing privilege grants. - Update application code to use v2 API. ### Migrating v1 → v2 (API surface mapping) The v1 API was *removed* in 2.0 (see the top section of this document). For pre-2.0 callers still on v1, this table is the side-by-side mapping to use when porting. | v1 (removed in 2.0) | v2 (current) | Notes | |---|---|---| | `pg_background_launch(sql, queue_size)` → `int4` | `pg_background_launch_v2(sql, queue_size, label)` → `pg_background_handle` | v2 returns `(pid, cookie)` composite; cookie protects against PID reuse | | `pg_background_result(pid)` → `SETOF record` | `pg_background_result_v2(pid, cookie)` → `SETOF record` | Same one-time consumption rule. Avoid calling on errored workers — use `error_info_v2` instead | | `pg_background_detach(pid)` → `void` | `pg_background_detach_v2(pid, cookie)` → `void` | Detach removes tracking; the worker keeps running and commits | | _(no equivalent)_ | `pg_background_submit_v2(sql, queue_size, label)` | Dedicated fire-and-forget; clearer than `launch + detach` | | _(no equivalent)_ | `pg_background_cancel_v2(pid, cookie, grace_ms)` | Cooperative SIGTERM; optional grace window to wait for the worker to stop (never force-killed) | | _(no equivalent)_ | `pg_background_wait_v2(pid, cookie, timeout_ms)` | Block until exit, optionally bounded | | _(no equivalent)_ | `pg_background_list_v2()` / `pg_background_list` view | Per-session worker registry with state, label, last_error | | _(no equivalent)_ | `pg_background_stats_v2()` | Counters: launched, completed, failed, canceled, timed_out, active, avg_execution_ms | | _(no equivalent)_ | `pg_background_result_info_v2(pid, cookie)` | Row count, command tag, started_at/finished_at, completion/error flags | | _(no equivalent)_ | `pg_background_error_info_v2(pid, cookie)` | Structured error: SQLSTATE, message, detail, hint, context, schema/table/column/constraint | | _(no equivalent)_ | `pg_background_outcome_v2(pid, cookie)` | Combined snapshot — never raises | | _(no equivalent)_ | `pg_background_run_v2(sql, queue_size, timeout_ms, label)` | Synchronous one-shot: launch + wait + outcome + detach | #### Example port Before (v1): ```sql -- fire-and-forget VACUUM SELECT pg_background_launch('VACUUM my_table') AS pid \gset SELECT pg_background_detach(:pid); ``` After (v2, idiomatic): ```sql -- explicit fire-and-forget SELECT (h).pid AS pid, (h).cookie AS cookie FROM (SELECT pg_background_submit_v2('VACUUM my_table', 0, 'nightly-vacuum') AS h) s \gset SELECT pg_background_detach_v2(:pid, :cookie); ``` After (v2, simpler — synchronous one-shot): ```sql -- launch, wait, capture metadata, detach — one call SELECT * FROM pg_background_run_v2('VACUUM my_table', 0, 0, 'nightly-vacuum'); ``` ## Verifying the upgrade After upgrading, confirm: ```sql SELECT extversion FROM pg_extension WHERE extname = 'pg_background'; -- Expected: 2.0 \df pg_background_launch -- Expected: 0 rows (v1 dropped). \df pg_background_cancel_v2 -- Expected: one row, signature (int4, int8, int4) with default for the third arg. \df pg_background_wait_v2 -- Expected: one row, signature (int4, int8, int4) returning bool. ``` If `\df pg_background_launch` returns a row, the upgrade did not run; check `SELECT extversion ...` and rerun the `ALTER EXTENSION ... UPDATE TO '2.0'`. ## Reporting upgrade issues File an issue at with the source version, target version, and the full error output (including any `pg_background_grant_privileges RAISE NOTICE` lines). pg_background-2.0.2/docs/SECURITY.md000066400000000000000000000240671521265151100170510ustar00rootroot00000000000000# Security Policy ## Reporting Security Vulnerabilities The pg_background team takes security seriously. We appreciate your efforts to responsibly disclose your findings. ### How to Report **DO NOT** open a public GitHub issue for security vulnerabilities. Instead, please email: - **Primary Contact**: vibhor.aim@gmail.com - **Subject**: `[SECURITY] pg_background vulnerability report` Include in your report: 1. **Description** of the vulnerability 2. **Steps to reproduce** (proof-of-concept) 3. **Impact assessment** (what can be exploited?) 4. **Affected versions** (if known) 5. **Suggested fix** (if you have one) ### Response Timeline - **Initial response**: Within 48 hours - **Triage**: Within 1 week - **Fix development**: Depends on severity (see below) - **Public disclosure**: After fix is released and users have time to upgrade ### Severity Levels | Severity | Response Time | Examples | |----------|---------------|----------| | **Critical** | 1-2 days | Privilege escalation, remote code execution | | **High** | 1 week | Data corruption, denial of service | | **Medium** | 2 weeks | Information disclosure, local DoS | | **Low** | 1 month | Minor information leaks | --- ## Security Best Practices ### Privilege Model #### Creating Roles ```sql -- Create dedicated role for pg_background CREATE ROLE pgbackground_role NOLOGIN INHERIT; -- Grant to application users (NOT superusers for production) GRANT pgbackground_role TO app_user; ``` #### Revoking Access ```sql -- Revoke from specific user REVOKE pgbackground_role FROM app_user; -- Or use helper function SELECT revoke_pg_background_privileges('app_user', true); ``` #### Never Grant to PUBLIC ```sql -- ❌ DANGEROUS: Don't do this! GRANT pgbackground_role TO PUBLIC; -- ✅ SAFE: Grant only to specific roles GRANT pgbackground_role TO trusted_app_role; ``` ### SQL Injection Prevention #### Unsafe (Vulnerable) ```sql -- ❌ DANGEROUS: Untrusted input in dynamic SQL DO $$ DECLARE user_input text := get_user_input(); -- Could be malicious BEGIN PERFORM pg_background_launch( 'SELECT * FROM users WHERE id = ' || user_input -- SQL INJECTION! ); END; $$; ``` #### Safe (Parameterized) ```sql -- ✅ SAFE: Use format() with proper quoting DO $$ DECLARE user_input text := get_user_input(); BEGIN PERFORM pg_background_launch( format('SELECT * FROM users WHERE id = %L', user_input) -- Safely quoted ); END; $$; ``` ### Resource Limits #### Prevent Resource Exhaustion ```sql -- 1. Set max_worker_processes appropriately ALTER SYSTEM SET max_worker_processes = 32; -- Not unlimited! -- 2. Monitor active workers CREATE OR REPLACE FUNCTION check_worker_limit() RETURNS boolean AS $$ DECLARE active_count int; BEGIN SELECT count(*) INTO active_count FROM pg_background_list() AS ( pid int4, cookie int8, launched_at timestamptz, user_id oid, queue_size int4, state text, sql_preview text, last_error text, consumed bool ) WHERE state = 'running'; RETURN active_count < 20; -- Application-specific limit END; $$ LANGUAGE plpgsql SECURITY DEFINER; -- 3. Use before launching workers DO $$ BEGIN IF NOT check_worker_limit() THEN RAISE EXCEPTION 'Too many active workers'; END IF; PERFORM pg_background_launch('...'); END; $$; ``` #### Statement Timeout ```sql -- Set timeout to prevent runaway queries SET statement_timeout = '5min'; -- Worker inherits this setting SELECT pg_background_launch('slow_query()'); ``` ### Autonomous Transaction Risks #### Understanding Isolation ```sql -- ⚠️ IMPORTANT: Worker commits are independent BEGIN; -- Launch background worker SELECT * FROM pg_background_launch( 'INSERT INTO audit_log VALUES (now(), ''user_login'')' ) AS h \gset; -- Main transaction work UPDATE users SET last_login = now() WHERE id = 123; -- If we ROLLBACK here, audit_log INSERT still commits! ROLLBACK; ``` **Implication**: Use background workers for truly independent operations only (e.g., logging, notifications, async processing). #### Secure Patterns ```sql -- ✅ GOOD: Audit logging (should commit regardless) SELECT pg_background_submit( format('INSERT INTO audit_log VALUES (now(), %L)', action) ); -- ✅ GOOD: Async notification SELECT pg_background_submit( format('SELECT pg_notify(''channel'', %L)', message) ); -- ❌ BAD: Interdependent data modifications BEGIN; INSERT INTO orders VALUES (...); -- Don't do this! Order INSERT might rollback, but payment won't SELECT pg_background_submit('INSERT INTO payments VALUES (...)'); COMMIT; ``` --- ## Known Security Considerations ### 1. PID Reuse (Mitigated in v2 API) **Issue**: On systems with high process churn, PIDs can be reused quickly. **v1 API Vulnerability**: ```sql -- ❌ VULNERABLE: PID might be reused after hours/days SELECT pg_background_launch('...') AS pid \gset -- ... session lives for weeks ... SELECT pg_background_result(:pid); -- Might attach to WRONG worker! ``` **v2 API Fix**: ```sql -- ✅ SAFE: Cookie prevents PID reuse confusion SELECT * FROM pg_background_launch('...') AS h \gset -- ... time passes ... SELECT pg_background_result(:'h.pid', :'h.cookie'); -- Cookie validated ``` **Recommendation**: Always use v2 API in production. ### 2. Information Disclosure **Issue**: Error messages may leak sensitive data. **Mitigation**: - Error messages are truncated at 512 bytes (as of v1.6) - Use `pg_background_list()` judiciously (shows SQL previews) - Restrict `pgbackground_role` to trusted users only **Example**: ```sql -- last_error in list() might show: -- "duplicate key value violates unique constraint: value (secret_data)" -- Protect with VIEW + RLS CREATE VIEW safe_worker_list AS SELECT pid, cookie, state, consumed FROM pg_background_list() AS (...); -- Omit sql_preview and last_error from public view ``` ### 3. Denial of Service **Issue**: Malicious user could spawn many workers. **Mitigation**: ```sql -- Application-level rate limiting CREATE TABLE user_worker_quota ( user_id oid PRIMARY KEY, workers_launched int DEFAULT 0, last_reset timestamptz DEFAULT now() ); -- Check quota before launch CREATE OR REPLACE FUNCTION launch_with_quota(sql text) RETURNS pg_background_handle AS $$ DECLARE h pg_background_handle; BEGIN -- Rate limit: 10 workers per minute UPDATE user_worker_quota SET workers_launched = CASE WHEN now() - last_reset > interval '1 minute' THEN 1 ELSE workers_launched + 1 END, last_reset = CASE WHEN now() - last_reset > interval '1 minute' THEN now() ELSE last_reset END WHERE user_id = current_user::regrole::oid; IF (SELECT workers_launched FROM user_worker_quota WHERE user_id = current_user::regrole::oid) > 10 THEN RAISE EXCEPTION 'Worker quota exceeded'; END IF; SELECT * INTO h FROM pg_background_launch(sql); RETURN h; END; $$ LANGUAGE plpgsql SECURITY DEFINER; ``` ### 4. Windows Cancel Limitations **Issue**: On Windows, `cancel()` cannot interrupt running queries. **Security Impact**: Low, but could enable resource exhaustion. **Mitigation**: ```sql -- Always set statement_timeout on Windows ALTER DATABASE mydb SET statement_timeout = '10min'; -- Or per-session: SET statement_timeout = '5min'; SELECT pg_background_launch('potentially_slow_query()'); ``` **See also**: [README.md § Windows Limitations](README.md#windows-limitations) --- ## Hardening Checklist Production deployments should: - [ ] Use v2 API exclusively (avoid v1 PID reuse issues) - [ ] Grant `pgbackground_role` only to trusted users - [ ] Never grant to `PUBLIC` - [ ] Set `max_worker_processes` appropriately (not unlimited) - [ ] Implement application-level rate limiting - [ ] Use `statement_timeout` to bound query execution - [ ] Validate/sanitize all user input in dynamic SQL - [ ] Monitor `pg_background_list()` for suspicious activity - [ ] Review audit logs for privilege escalation attempts - [ ] Test disaster recovery with background workers running - [ ] Document autonomous transaction usage in app code --- ## Security Update Notifications Subscribe to security advisories: 1. **GitHub Watch**: Watch this repo for "Releases only" or "All activity" 2. **GitHub Security Advisories**: https://github.com/vibhorkum/pg_background/security/advisories 3. **Mailing List**: pgsql-general@postgresql.org (major PostgreSQL ecosystem issues) --- ## Vulnerability Disclosure Policy ### Our Commitments - **Acknowledge** your report within 48 hours - **Keep you informed** of progress on a fix - **Credit you** in release notes (unless you prefer anonymity) - **Coordinate disclosure** with you before making details public ### Your Responsibilities - **Act in good faith**: Don't exploit the vulnerability beyond proof-of-concept - **Minimize impact**: Don't exfiltrate data, disrupt services, or access accounts - **Maintain confidentiality**: Don't disclose until we've released a fix - **Give us reasonable time**: Allow 90 days for fix development before public disclosure ### Bug Bounty This project does not currently offer a bug bounty program. However, we greatly appreciate responsible disclosure and will publicly acknowledge contributors. --- ## Past Security Issues | CVE ID | Severity | Affected Versions | Fixed In | Description | |--------|----------|-------------------|----------|-------------| | N/A | N/A | N/A | N/A | No CVEs assigned to date | --- ## Security-Related Changes in v1.6 - ✅ Grace period overflow protection (cap at 1 hour) - ✅ Error message truncation (512 bytes max) - ✅ Bounds checking for column count in RowDescription - ✅ Explicit ResourceOwner cleanup before proc_exit - ✅ Volatile access for cancel flag (race mitigation) - ✅ Enhanced documentation on PID reuse edge cases - ✅ Windows cancel limitations documented --- ## References - [PostgreSQL Security](https://www.postgresql.org/support/security/) - [PostgreSQL CVE List](https://www.cvedetails.com/product/575/PostgreSQL-PostgreSQL.html) - [OWASP SQL Injection Prevention](https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet.html) --- ## Contact - **Security Issues**: vibhor.aim@gmail.com - **General Support**: https://github.com/vibhorkum/pg_background/issues --- Last updated: 2026-02-05 pg_background-2.0.2/expected/000077500000000000000000000000001521265151100161205ustar00rootroot00000000000000pg_background-2.0.2/expected/pg_background.out000066400000000000000000002355751521265151100214770ustar00rootroot00000000000000CREATE EXTENSION pg_background; DROP TABLE IF EXISTS t; NOTICE: table "t" does not exist, skipping CREATE TABLE t(id integer); -- ---------------------------------------------------------------------- -- v2: basic launch_v2 + result_v2 -- (v1 API was removed in 2.0; see docs/MIGRATION.md.) -- ---------------------------------------------------------------------- SELECT (h).pid AS basic_pid, (h).cookie AS basic_cookie FROM (SELECT pg_background_launch_v2('INSERT INTO t SELECT 1', 65536) AS h) s \gset SELECT * FROM pg_background_result_v2(:basic_pid, :basic_cookie) AS (result TEXT); result ------------ INSERT 0 1 (1 row) SELECT * FROM t ORDER BY id; id ---- 1 (1 row) -- ---------------------------------------------------------------------- -- v2: detach should not crash the session -- ---------------------------------------------------------------------- SELECT (h).pid AS d_pid, (h).cookie AS d_cookie FROM (SELECT pg_background_launch_v2('SELECT 1', 65536) AS h) s \gset SELECT pg_background_detach_v2(:d_pid, :d_cookie); pg_background_detach_v2 ------------------------- (1 row) -- ---------------------------------------------------------------------- -- v2: launch + detach, worker should still commit its work -- ---------------------------------------------------------------------- SELECT (h).pid AS v2_pid, (h).cookie AS v2_cookie FROM (SELECT pg_background_launch_v2('INSERT INTO t SELECT 2', 65536) AS h) s \gset SELECT pg_sleep(0.2); pg_sleep ---------- (1 row) SELECT pg_background_detach_v2(:v2_pid, :v2_cookie); pg_background_detach_v2 ------------------------- (1 row) -- give worker a moment to finish and commit SELECT pg_sleep(0.5); pg_sleep ---------- (1 row) SELECT * FROM t ORDER BY id; id ---- 1 2 (2 rows) -- ---------------------------------------------------------------------- -- v2: cancel should prevent later statements from committing -- We run: sleep then insert, cancel during sleep, verify 99 not inserted. -- ---------------------------------------------------------------------- SELECT (h).pid AS c_pid, (h).cookie AS c_cookie FROM (SELECT pg_background_launch_v2('SELECT pg_sleep(10); INSERT INTO t SELECT 99', 65536) AS h) s \gset SELECT pg_sleep(0.2); pg_sleep ---------- (1 row) SELECT pg_background_cancel_v2(:c_pid, :c_cookie); pg_background_cancel_v2 ------------------------- (1 row) -- allow time for termination/cleanup SELECT pg_sleep(0.5); pg_sleep ---------- (1 row) SELECT count(*) AS canceled_insert_count FROM t WHERE id = 99; canceled_insert_count ----------------------- 0 (1 row) -- ------------------------------------------------------------------------- -- v2 detach is fire-and-forget (no cancel): inserts should happen -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS t_detach_v2; NOTICE: table "t_detach_v2" does not exist, skipping CREATE TABLE t_detach_v2(id int); DO $$ DECLARE h pg_background_handle; BEGIN SELECT * INTO h FROM pg_background_launch_v2('INSERT INTO t_detach_v2 SELECT 1', 65536); PERFORM pg_background_detach_v2(h.pid, h.cookie); END; $$; SELECT pg_sleep(1.0); pg_sleep ---------- (1 row) SELECT count(*) FROM t_detach_v2; count ------- 1 (1 row) -- ------------------------------------------------------------------------- -- wait_v2 (1.6 API): timeout + then success -- - pg_background_wait_v2(pid,cookie,timeout_ms) -> bool -- - pg_background_wait_v2(pid,cookie) -> void (blocking) -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS t_wait; NOTICE: table "t_wait" does not exist, skipping CREATE TABLE t_wait(id int); DO $$ DECLARE h pg_background_handle; DECLARE ok bool; BEGIN SELECT * INTO h FROM pg_background_launch_v2('SELECT pg_sleep(2); INSERT INTO t_wait VALUES (1)', 65536); -- Short wait should time out (false) ok := pg_background_wait_v2(h.pid, h.cookie, 200); RAISE NOTICE 'wait_short=%', ok; -- Long wait should succeed (true) ok := pg_background_wait_v2(h.pid, h.cookie, 5000); RAISE NOTICE 'wait_long=%', ok; -- cleanup bookkeeping (worker is already finished, but we detach handle) PERFORM pg_background_detach_v2(h.pid, h.cookie); END; $$; NOTICE: wait_short=f NOTICE: wait_long=t SELECT count(*) FROM t_wait; count ------- 1 (1 row) -- ------------------------------------------------------------------------- -- cancel_v2 (1.6 API): should prevent the INSERT -- - pg_background_cancel_v2(pid,cookie,grace_ms) is available too -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS t_cancel; NOTICE: table "t_cancel" does not exist, skipping CREATE TABLE t_cancel(id int); DO $$ DECLARE h pg_background_handle; BEGIN SELECT * INTO h FROM pg_background_launch_v2('SELECT pg_sleep(5); INSERT INTO t_cancel VALUES (1)', 65536); -- Explicit cancel; detach is not cancel. PERFORM pg_background_cancel_v2(h.pid, h.cookie, 500); -- Give server time to process cancel/terminate PERFORM pg_sleep(0.5); -- Detach handle bookkeeping PERFORM pg_background_detach_v2(h.pid, h.cookie); END; $$; SELECT count(*) FROM t_cancel; count ------- 0 (1 row) -- ------------------------------------------------------------------------- -- v2: list_v2 should show running job, then disappear after cleanup -- ------------------------------------------------------------------------- -- create a long-running job so list_v2 can observe it SELECT (h).pid AS l_pid, (h).cookie AS l_cookie FROM (SELECT pg_background_launch_v2('SELECT pg_sleep(2)', 65536) AS h) s \gset -- give it a moment to enter running state SELECT pg_sleep(0.1); pg_sleep ---------- (1 row) -- list should include our pid/cookie (at least once) SELECT COUNT(*) AS list_contains_launched_job FROM pg_background_list_v2() AS (pid int4, cookie int8, launched_at timestamptz, user_id oid, queue_size int4, state text, sql_preview text, last_error text, consumed bool) WHERE pid = :l_pid AND cookie = :l_cookie AND queue_size = 65536 AND user_id IS NOT NULL AND launched_at IS NOT NULL AND state IN ('starting', 'running', 'stopped', 'canceled'); list_contains_launched_job ---------------------------- 1 (1 row) -- cleanup explicitly (even if it already stopped) SELECT pg_background_cancel_v2(:l_pid, :l_cookie); pg_background_cancel_v2 ------------------------- (1 row) SELECT pg_background_detach_v2(:l_pid, :l_cookie); pg_background_detach_v2 ------------------------- (1 row) -- should be gone from list after detach SELECT COUNT(*) AS list_contains_after_detach FROM pg_background_list_v2() AS (pid int4, cookie int8, launched_at timestamptz, user_id oid, queue_size int4, state text, sql_preview text, last_error text, consumed bool) WHERE pid = :l_pid AND cookie = :l_cookie; list_contains_after_detach ---------------------------- 0 (1 row) -- ------------------------------------------------------------------------- -- v2: submit_v2 is fire-and-forget and should commit -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS t_submit; NOTICE: table "t_submit" does not exist, skipping CREATE TABLE t_submit(id int); SELECT (h).pid AS s_pid, (h).cookie AS s_cookie FROM (SELECT pg_background_submit_v2('INSERT INTO t_submit VALUES (1)', 65536) AS h) s \gset -- submit may detach internally; still allow time to commit SELECT pg_sleep(0.2); pg_sleep ---------- (1 row) SELECT count(*) AS submit_count FROM t_submit; submit_count -------------- 1 (1 row) -- ------------------------------------------------------------------------- -- v2: wait_v2_timeout times out, then succeeds; wait_v2 blocks to completion -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS t_wait2; NOTICE: table "t_wait2" does not exist, skipping CREATE TABLE t_wait2(id int); SELECT (h).pid AS w_pid, (h).cookie AS w_cookie FROM (SELECT pg_background_launch_v2('SELECT pg_sleep(1); INSERT INTO t_wait2 VALUES (1)', 65536) AS h) s \gset -- should time out quickly SELECT pg_background_wait_v2(:w_pid, :w_cookie, 50) AS wait_short; wait_short ------------ f (1 row) -- should succeed with longer timeout SELECT pg_background_wait_v2(:w_pid, :w_cookie, 5000) AS wait_long; wait_long ----------- t (1 row) -- wait_v2 should now return immediately (already done), but it must work SELECT pg_background_wait_v2(:w_pid, :w_cookie); pg_background_wait_v2 ----------------------- t (1 row) -- detach bookkeeping SELECT pg_background_detach_v2(:w_pid, :w_cookie); pg_background_detach_v2 ------------------------- (1 row) SELECT count(*) AS wait2_count FROM t_wait2; wait2_count ------------- 1 (1 row) -- ------------------------------------------------------------------------- -- v2: cancel_v2_grace should prevent later statements from committing -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS t_cancel2; NOTICE: table "t_cancel2" does not exist, skipping CREATE TABLE t_cancel2(id int); SELECT (h).pid AS cx_pid, (h).cookie AS cx_cookie FROM (SELECT pg_background_launch_v2('SELECT pg_sleep(10); INSERT INTO t_cancel2 VALUES (1)', 65536) AS h) s \gset SELECT pg_sleep(0.2); pg_sleep ---------- (1 row) SELECT pg_background_cancel_v2(:cx_pid, :cx_cookie, 500); pg_background_cancel_v2 ------------------------- (1 row) -- allow termination SELECT pg_sleep(0.5); pg_sleep ---------- (1 row) -- detach handle bookkeeping SELECT pg_background_detach_v2(:cx_pid, :cx_cookie); pg_background_detach_v2 ------------------------- (1 row) SELECT count(*) AS cancel2_count FROM t_cancel2; cancel2_count --------------- 0 (1 row) -- ------------------------------------------------------------------------- -- ops: detach all stopped workers returned by list -- ------------------------------------------------------------------------- DO $$ DECLARE r record; BEGIN FOR r IN SELECT * FROM pg_background_list_v2() AS (pid int4, cookie int8, launched_at timestamptz, user_id oid, queue_size int4, state text, sql_preview text, last_error text, consumed bool) WHERE state IN ('stopped', 'canceled') LOOP PERFORM pg_background_detach_v2(r.pid, r.cookie); END LOOP; END $$; SELECT * FROM pg_background_list_v2() AS (pid int4, cookie int8, launched_at timestamptz, user_id oid, queue_size int4, state text, sql_preview text, last_error text, consumed bool); pid | cookie | launched_at | user_id | queue_size | state | sql_preview | last_error | consumed -----+--------+-------------+---------+------------+-------+-------------+------------+---------- (0 rows) -- ------------------------------------------------------------------------- -- v1.8: GUC settings -- ------------------------------------------------------------------------- -- Show default GUC values SHOW pg_background.max_workers; pg_background.max_workers --------------------------- 16 (1 row) SHOW pg_background.default_queue_size; pg_background.default_queue_size ---------------------------------- 64kB (1 row) SHOW pg_background.worker_timeout; pg_background.worker_timeout ------------------------------ 0 (1 row) -- Test max_workers limit SET pg_background.max_workers = 2; SHOW pg_background.max_workers; pg_background.max_workers --------------------------- 2 (1 row) -- Reset to default RESET pg_background.max_workers; -- ------------------------------------------------------------------------- -- v1.8: stats_v2 - session statistics -- ------------------------------------------------------------------------- -- Stats should show some activity from previous tests SELECT workers_launched > 0 AS has_launched, workers_completed >= 0 AS has_completed_field, workers_failed >= 0 AS has_failed_field, workers_canceled >= 0 AS has_canceled_field, workers_active >= 0 AS has_active_field, avg_execution_ms >= 0 AS has_avg_time, max_workers > 0 AS has_max_workers FROM pg_background_stats_v2(); has_launched | has_completed_field | has_failed_field | has_canceled_field | has_active_field | has_avg_time | has_max_workers --------------+---------------------+------------------+--------------------+------------------+--------------+----------------- t | t | t | t | t | t | t (1 row) -- ------------------------------------------------------------------------- -- v1.8: progress reporting -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS t_progress; NOTICE: table "t_progress" does not exist, skipping CREATE TABLE t_progress(id int); -- Launch worker that reports progress SELECT (h).pid AS p_pid, (h).cookie AS p_cookie FROM (SELECT pg_background_launch_v2($$ SELECT pg_background_report_progress(0, 'Starting'); SELECT pg_sleep(0.1); SELECT pg_background_report_progress(50, 'Halfway'); SELECT pg_sleep(0.1); SELECT pg_background_report_progress(100, 'Done'); INSERT INTO t_progress VALUES (1); $$, 65536) AS h) s \gset -- Give worker time to start and report progress SELECT pg_sleep(0.15); pg_sleep ---------- (1 row) -- Check progress (should be >= 0 if reported) SELECT CASE WHEN progress_pct >= 0 THEN 'progress_reported' ELSE 'no_progress' END AS progress_status FROM pg_background_get_progress_v2(:p_pid, :p_cookie); progress_status ------------------- progress_reported (1 row) -- Wait for completion SELECT pg_background_wait_v2(:p_pid, :p_cookie, 5000) AS progress_worker_done; progress_worker_done ---------------------- t (1 row) -- Verify work was done SELECT count(*) AS progress_insert_count FROM t_progress; progress_insert_count ----------------------- 1 (1 row) -- Cleanup SELECT pg_background_detach_v2(:p_pid, :p_cookie); pg_background_detach_v2 ------------------------- (1 row) -- ------------------------------------------------------------------------- -- v1.8: max_workers enforcement -- ------------------------------------------------------------------------- -- Set very low limit SET pg_background.max_workers = 1; -- Launch one worker (should succeed) SELECT (h).pid AS mw_pid, (h).cookie AS mw_cookie FROM (SELECT pg_background_launch_v2('SELECT pg_sleep(2)', 65536) AS h) s \gset SELECT pg_sleep(0.1); pg_sleep ---------- (1 row) -- Try to launch another (should fail due to limit) DO $$ BEGIN PERFORM pg_background_launch_v2('SELECT 1', 65536); RAISE NOTICE 'max_workers_test=should_have_failed'; EXCEPTION WHEN OTHERS THEN RAISE NOTICE 'max_workers_test=correctly_limited'; END; $$; NOTICE: max_workers_test=correctly_limited -- Cancel and cleanup the first worker SELECT pg_background_cancel_v2(:mw_pid, :mw_cookie); pg_background_cancel_v2 ------------------------- (1 row) SELECT pg_sleep(0.2); pg_sleep ---------- (1 row) SELECT pg_background_detach_v2(:mw_pid, :mw_cookie); pg_background_detach_v2 ------------------------- (1 row) -- Reset limit RESET pg_background.max_workers; -- ------------------------------------------------------------------------- -- v1.8: worker_timeout GUC -- ------------------------------------------------------------------------- -- Test that worker_timeout can be set SET pg_background.worker_timeout = '1s'; SHOW pg_background.worker_timeout; pg_background.worker_timeout ------------------------------ 1s (1 row) RESET pg_background.worker_timeout; -- ------------------------------------------------------------------------- -- v1.9: Worker labels -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS t_label; NOTICE: table "t_label" does not exist, skipping CREATE TABLE t_label(id int); -- Launch worker with label SELECT (h).pid AS lbl_pid, (h).cookie AS lbl_cookie FROM (SELECT pg_background_launch_v2('INSERT INTO t_label VALUES (1)', 0, 'test-label') AS h) s \gset -- Wait for worker to complete (deterministic, not timing-dependent) SELECT pg_background_wait_v2(:lbl_pid, :lbl_cookie); pg_background_wait_v2 ----------------------- t (1 row) -- Verify worker with label completed successfully SELECT completed AS label_worker_completed, has_error AS label_worker_has_error FROM pg_background_result_info_v2(:lbl_pid, :lbl_cookie); label_worker_completed | label_worker_has_error ------------------------+------------------------ t | f (1 row) -- Verify label is exposed through list_v2 SELECT label AS visible_label FROM pg_background_list_v2() AS (pid int4, cookie int8, launched_at timestamptz, user_id oid, queue_size int4, state text, sql_preview text, last_error text, consumed bool, label text) WHERE pid = :lbl_pid AND cookie = :lbl_cookie; visible_label --------------- test-label (1 row) -- Cleanup SELECT pg_background_detach_v2(:lbl_pid, :lbl_cookie); pg_background_detach_v2 ------------------------- (1 row) SELECT count(*) AS label_insert_count FROM t_label; label_insert_count -------------------- 1 (1 row) -- ------------------------------------------------------------------------- -- v1.9: Submit with label -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS t_submit_label; NOTICE: table "t_submit_label" does not exist, skipping CREATE TABLE t_submit_label(id int); SELECT (h).pid AS slbl_pid, (h).cookie AS slbl_cookie FROM (SELECT pg_background_submit_v2('INSERT INTO t_submit_label VALUES (1)', 0, 'submit-label') AS h) s \gset -- Wait for worker to complete (deterministic) SELECT pg_background_wait_v2(:slbl_pid, :slbl_cookie); pg_background_wait_v2 ----------------------- t (1 row) SELECT count(*) AS submit_label_count FROM t_submit_label; submit_label_count -------------------- 1 (1 row) -- Cleanup: explicitly detach to avoid affecting later batch tests SELECT pg_background_detach_v2(:slbl_pid, :slbl_cookie); pg_background_detach_v2 ------------------------- (1 row) -- ------------------------------------------------------------------------- -- v1.9: Result info without consuming results -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS t_result_info; NOTICE: table "t_result_info" does not exist, skipping CREATE TABLE t_result_info(id int); SELECT (h).pid AS ri_pid, (h).cookie AS ri_cookie FROM (SELECT pg_background_launch_v2('INSERT INTO t_result_info SELECT generate_series(1,5)') AS h) s \gset -- Wait for worker to complete (deterministic) SELECT pg_background_wait_v2(:ri_pid, :ri_cookie); pg_background_wait_v2 ----------------------- t (1 row) -- Check result info (should show completed with row_count=5, command_tag='INSERT') SELECT row_count AS ri_row_count, command_tag AS ri_command_tag, completed AS ri_completed, has_error AS ri_has_error FROM pg_background_result_info_v2(:ri_pid, :ri_cookie); ri_row_count | ri_command_tag | ri_completed | ri_has_error --------------+----------------+--------------+-------------- 5 | INSERT | t | f (1 row) SELECT pg_background_detach_v2(:ri_pid, :ri_cookie); pg_background_detach_v2 ------------------------- (1 row) SELECT count(*) AS result_info_count FROM t_result_info; result_info_count ------------------- 5 (1 row) -- ------------------------------------------------------------------------- -- v1.9: Structured error info -- ------------------------------------------------------------------------- -- Launch worker that will fail SELECT (h).pid AS err_pid, (h).cookie AS err_cookie FROM (SELECT pg_background_launch_v2('SELECT 1/0') AS h) s \gset -- Wait for worker to complete (will error, but still completes) SELECT pg_background_wait_v2(:err_pid, :err_cookie); pg_background_wait_v2 ----------------------- t (1 row) -- Check result info shows error SELECT completed AS err_completed, has_error AS err_has_error FROM pg_background_result_info_v2(:err_pid, :err_cookie); err_completed | err_has_error ---------------+--------------- t | t (1 row) -- Get structured error (should have sqlstate for division by zero) SELECT CASE WHEN sqlstate IS NOT NULL THEN 'has_sqlstate' ELSE 'no_sqlstate' END AS error_sqlstate_check, CASE WHEN message IS NOT NULL THEN 'has_message' ELSE 'no_message' END AS error_message_check FROM pg_background_error_info_v2(:err_pid, :err_cookie); error_sqlstate_check | error_message_check ----------------------+--------------------- has_sqlstate | has_message (1 row) SELECT pg_background_detach_v2(:err_pid, :err_cookie); pg_background_detach_v2 ------------------------- (1 row) -- ------------------------------------------------------------------------- -- v1.9: Batch detach -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS t_batch; NOTICE: table "t_batch" does not exist, skipping CREATE TABLE t_batch(id int); -- Launch multiple workers (use launch_v2 so we can wait deterministically) SELECT (h).pid AS b1_pid, (h).cookie AS b1_cookie FROM (SELECT pg_background_launch_v2('INSERT INTO t_batch VALUES (1)', NULL, 'batch-1') AS h) s \gset SELECT (h).pid AS b2_pid, (h).cookie AS b2_cookie FROM (SELECT pg_background_launch_v2('INSERT INTO t_batch VALUES (2)', NULL, 'batch-2') AS h) s \gset -- Wait for both workers to complete (deterministic, avoids timing-based flakiness) SELECT pg_background_wait_v2(:b1_pid, :b1_cookie); pg_background_wait_v2 ----------------------- t (1 row) SELECT pg_background_wait_v2(:b2_pid, :b2_cookie); pg_background_wait_v2 ----------------------- t (1 row) -- Detach all at once SELECT pg_background_detach_all_v2() AS batch_detach_count; batch_detach_count -------------------- 2 (1 row) SELECT count(*) AS batch_insert_count FROM t_batch; batch_insert_count -------------------- 2 (1 row) -- ------------------------------------------------------------------------- -- v1.9: Batch cancel -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS t_batch_cancel; NOTICE: table "t_batch_cancel" does not exist, skipping CREATE TABLE t_batch_cancel(id int); -- Launch multiple long-running workers SELECT (h).pid AS bc1_pid, (h).cookie AS bc1_cookie FROM (SELECT pg_background_launch_v2('SELECT pg_sleep(10); INSERT INTO t_batch_cancel VALUES (1)') AS h) s \gset SELECT (h).pid AS bc2_pid, (h).cookie AS bc2_cookie FROM (SELECT pg_background_launch_v2('SELECT pg_sleep(10); INSERT INTO t_batch_cancel VALUES (2)') AS h) s \gset -- Brief pause to ensure workers have started their pg_sleep SELECT pg_sleep(0.2); pg_sleep ---------- (1 row) -- Cancel all at once SELECT pg_background_cancel_all_v2() AS batch_cancel_count; batch_cancel_count -------------------- 2 (1 row) -- Wait for each worker to stop (deterministic, with timeout) SELECT pg_background_wait_v2(:bc1_pid, :bc1_cookie, 5000) AS bc1_stopped; bc1_stopped ------------- t (1 row) SELECT pg_background_wait_v2(:bc2_pid, :bc2_cookie, 5000) AS bc2_stopped; bc2_stopped ------------- t (1 row) -- Detach remaining SELECT pg_background_detach_all_v2() AS batch_cancel_detach_count; batch_cancel_detach_count --------------------------- 2 (1 row) -- Inserts should not have happened SELECT count(*) AS batch_cancel_insert_count FROM t_batch_cancel; batch_cancel_insert_count --------------------------- 0 (1 row) -- ------------------------------------------------------------------------- -- Error Path: Cookie mismatch detection -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS t_cookie_mismatch; NOTICE: table "t_cookie_mismatch" does not exist, skipping CREATE TABLE t_cookie_mismatch(id int); -- Store handle in temp table for DO block access DROP TABLE IF EXISTS _test_cm_handle; NOTICE: table "_test_cm_handle" does not exist, skipping CREATE TEMP TABLE _test_cm_handle AS SELECT (h).pid, (h).cookie FROM (SELECT pg_background_launch_v2('INSERT INTO t_cookie_mismatch VALUES (1)') AS h) s; DO $$ DECLARE v_pid int; v_cookie bigint; BEGIN SELECT pid, cookie INTO v_pid, v_cookie FROM _test_cm_handle; PERFORM pg_background_wait_v2(v_pid, v_cookie); END; $$; -- Try operation with wrong cookie (should error) DO $$ DECLARE v_pid int; BEGIN SELECT pid INTO v_pid FROM _test_cm_handle; -- Use deliberately wrong cookie PERFORM pg_background_wait_v2(v_pid, 9999999999999999); RAISE NOTICE 'cookie_mismatch_test=should_have_failed'; EXCEPTION WHEN OTHERS THEN RAISE NOTICE 'cookie_mismatch_test=correctly_errored'; END; $$; NOTICE: cookie_mismatch_test=correctly_errored DO $$ DECLARE v_pid int; v_cookie bigint; BEGIN SELECT pid, cookie INTO v_pid, v_cookie FROM _test_cm_handle; PERFORM pg_background_detach_v2(v_pid, v_cookie); END; $$; SELECT count(*) AS cookie_mismatch_insert_count FROM t_cookie_mismatch; cookie_mismatch_insert_count ------------------------------ 1 (1 row) -- ------------------------------------------------------------------------- -- v2.0 (E6): Cookie-mismatch coverage for EVERY v2 function. -- -- CLAUDE.md §11 requires every cookie-protected v2 entrypoint to use the -- same error pattern: SQLSTATE 55000 (object_not_in_prerequisite_state) -- with errmsg "cookie mismatch for PID %d" and a "stale handle" errhint. -- This block launches one worker, then calls each protected function with -- a deliberately wrong cookie and asserts both the SQLSTATE and that the -- errmsg starts with "cookie mismatch". -- ------------------------------------------------------------------------- DO $$ DECLARE h pg_background_handle; bad_cookie int8 := 1; /* deliberately not the real cookie */ saw_state text; saw_msg text; BEGIN /* * v2.0: launch a fast-finishing worker and wait for it to complete BEFORE * the wrong-cookie sweep. The cookie-validation paths are identical * whether the worker is alive or already exited (the check fires before * any signal/queue work), and using a finished worker means cleanup is a * single detach_v2 instead of cancel→wait→detach. That avoids a flaky * worker-exit-then-launcher-followup race we hit earlier. */ h := pg_background_launch_v2('SELECT 1', 65536, 'cookie-mismatch-sweep'); PERFORM pg_background_wait_v2(h.pid, h.cookie); /* * Iterate every v2 function that validates cookie. We can't easily put * them in an array of function pointers, so we open-code each call. * Each block must produce SQLSTATE 55000 and a "cookie mismatch" message. */ /* result_v2 */ BEGIN PERFORM * FROM pg_background_result_v2(h.pid, bad_cookie) AS x(c text); RAISE EXCEPTION 'result_v2: should have raised on wrong cookie'; EXCEPTION WHEN OTHERS THEN GET STACKED DIAGNOSTICS saw_state = RETURNED_SQLSTATE, saw_msg = MESSAGE_TEXT; IF saw_state <> '55000' OR saw_msg NOT LIKE 'cookie mismatch%' THEN RAISE EXCEPTION 'result_v2: SQLSTATE=% msg=%', saw_state, saw_msg; END IF; END; /* detach_v2 */ BEGIN PERFORM pg_background_detach_v2(h.pid, bad_cookie); RAISE EXCEPTION 'detach_v2: should have raised on wrong cookie'; EXCEPTION WHEN OTHERS THEN GET STACKED DIAGNOSTICS saw_state = RETURNED_SQLSTATE, saw_msg = MESSAGE_TEXT; IF saw_state <> '55000' OR saw_msg NOT LIKE 'cookie mismatch%' THEN RAISE EXCEPTION 'detach_v2: SQLSTATE=% msg=%', saw_state, saw_msg; END IF; END; /* cancel_v2 (3-arg, default grace_ms) */ BEGIN PERFORM pg_background_cancel_v2(h.pid, bad_cookie); RAISE EXCEPTION 'cancel_v2: should have raised on wrong cookie'; EXCEPTION WHEN OTHERS THEN GET STACKED DIAGNOSTICS saw_state = RETURNED_SQLSTATE, saw_msg = MESSAGE_TEXT; IF saw_state <> '55000' OR saw_msg NOT LIKE 'cookie mismatch%' THEN RAISE EXCEPTION 'cancel_v2: SQLSTATE=% msg=%', saw_state, saw_msg; END IF; END; /* cancel_v2 with explicit grace_ms */ BEGIN PERFORM pg_background_cancel_v2(h.pid, bad_cookie, 100); RAISE EXCEPTION 'cancel_v2(grace): should have raised on wrong cookie'; EXCEPTION WHEN OTHERS THEN GET STACKED DIAGNOSTICS saw_state = RETURNED_SQLSTATE, saw_msg = MESSAGE_TEXT; IF saw_state <> '55000' OR saw_msg NOT LIKE 'cookie mismatch%' THEN RAISE EXCEPTION 'cancel_v2(grace): SQLSTATE=% msg=%', saw_state, saw_msg; END IF; END; /* wait_v2 (3-arg, default timeout_ms) */ BEGIN PERFORM pg_background_wait_v2(h.pid, bad_cookie); RAISE EXCEPTION 'wait_v2: should have raised on wrong cookie'; EXCEPTION WHEN OTHERS THEN GET STACKED DIAGNOSTICS saw_state = RETURNED_SQLSTATE, saw_msg = MESSAGE_TEXT; IF saw_state <> '55000' OR saw_msg NOT LIKE 'cookie mismatch%' THEN RAISE EXCEPTION 'wait_v2: SQLSTATE=% msg=%', saw_state, saw_msg; END IF; END; /* wait_v2 with explicit timeout_ms */ BEGIN PERFORM pg_background_wait_v2(h.pid, bad_cookie, 100); RAISE EXCEPTION 'wait_v2(timeout): should have raised on wrong cookie'; EXCEPTION WHEN OTHERS THEN GET STACKED DIAGNOSTICS saw_state = RETURNED_SQLSTATE, saw_msg = MESSAGE_TEXT; IF saw_state <> '55000' OR saw_msg NOT LIKE 'cookie mismatch%' THEN RAISE EXCEPTION 'wait_v2(timeout): SQLSTATE=% msg=%', saw_state, saw_msg; END IF; END; /* result_info_v2 */ BEGIN PERFORM pg_background_result_info_v2(h.pid, bad_cookie); RAISE EXCEPTION 'result_info_v2: should have raised on wrong cookie'; EXCEPTION WHEN OTHERS THEN GET STACKED DIAGNOSTICS saw_state = RETURNED_SQLSTATE, saw_msg = MESSAGE_TEXT; IF saw_state <> '55000' OR saw_msg NOT LIKE 'cookie mismatch%' THEN RAISE EXCEPTION 'result_info_v2: SQLSTATE=% msg=%', saw_state, saw_msg; END IF; END; /* error_info_v2 */ BEGIN PERFORM pg_background_error_info_v2(h.pid, bad_cookie); RAISE EXCEPTION 'error_info_v2: should have raised on wrong cookie'; EXCEPTION WHEN OTHERS THEN GET STACKED DIAGNOSTICS saw_state = RETURNED_SQLSTATE, saw_msg = MESSAGE_TEXT; IF saw_state <> '55000' OR saw_msg NOT LIKE 'cookie mismatch%' THEN RAISE EXCEPTION 'error_info_v2: SQLSTATE=% msg=%', saw_state, saw_msg; END IF; END; /* full_sql_v2 raises like the rest (it's a debugging accessor with the * same cookie validation as result_v2) */ BEGIN PERFORM pg_background_full_sql_v2(h.pid, bad_cookie); RAISE EXCEPTION 'full_sql_v2: should have raised on wrong cookie'; EXCEPTION WHEN OTHERS THEN GET STACKED DIAGNOSTICS saw_state = RETURNED_SQLSTATE, saw_msg = MESSAGE_TEXT; IF saw_state <> '55000' OR saw_msg NOT LIKE 'cookie mismatch%' THEN RAISE EXCEPTION 'full_sql_v2: SQLSTATE=% msg=%', saw_state, saw_msg; END IF; END; /* * get_progress_v2 is the lone exception: it returns NULL on cookie * mismatch instead of raising, since it's an informational accessor * meant to be polled by the launcher without exception-handling * boilerplate. Document the contract rather than test it as raising. */ IF pg_background_get_progress_v2(h.pid, bad_cookie) IS NOT NULL THEN RAISE EXCEPTION 'get_progress_v2: expected NULL on wrong cookie'; END IF; /* outcome_v2 swallows errors and returns NULL fields per its contract */ IF (pg_background_outcome_v2(h.pid, bad_cookie)).completed IS NOT NULL THEN RAISE EXCEPTION 'outcome_v2: expected completed=NULL on wrong cookie'; END IF; /* Real cleanup with the real cookie. Worker already exited above, so * detach_v2 is sufficient — no cancel + wait dance. */ PERFORM pg_background_detach_v2(h.pid, h.cookie); RAISE NOTICE 'v2.0 cookie-mismatch sweep (every protected entrypoint) OK'; END$$; NOTICE: v2.0 cookie-mismatch sweep (every protected entrypoint) OK -- ------------------------------------------------------------------------- -- Error Path: Result consumption guard (double-consume) -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS _test_rc_handle; NOTICE: table "_test_rc_handle" does not exist, skipping CREATE TEMP TABLE _test_rc_handle AS SELECT (h).pid, (h).cookie FROM (SELECT pg_background_launch_v2('SELECT 42 AS answer') AS h) s; DO $$ DECLARE v_pid int; v_cookie bigint; BEGIN SELECT pid, cookie INTO v_pid, v_cookie FROM _test_rc_handle; PERFORM pg_background_wait_v2(v_pid, v_cookie); END; $$; -- First consumption should succeed (use direct query, not DO block) SELECT answer FROM ( SELECT * FROM pg_background_result_v2( (SELECT pid FROM _test_rc_handle), (SELECT cookie FROM _test_rc_handle) ) AS (answer int) ) sub; answer -------- 42 (1 row) -- Second consumption should error (worker auto-detached after result consumed) -- Expect SQLSTATE 42704 (UNDEFINED_OBJECT) for "PID not attached to this session" DO $$ DECLARE v_pid int; v_cookie bigint; BEGIN SELECT pid, cookie INTO v_pid, v_cookie FROM _test_rc_handle; PERFORM * FROM pg_background_result_v2(v_pid, v_cookie) AS (answer int); RAISE NOTICE 'result_consumed_test=should_have_failed'; EXCEPTION WHEN undefined_object THEN -- Expected: "PID %d is not attached to this session" RAISE NOTICE 'result_consumed_test=correctly_errored (SQLSTATE=42704)'; WHEN OTHERS THEN -- Unexpected error - report it for debugging RAISE NOTICE 'result_consumed_test=unexpected_error (SQLSTATE=%, %)', SQLSTATE, SQLERRM; END; $$; NOTICE: result_consumed_test=correctly_errored (SQLSTATE=42704) -- No explicit detach needed - worker is auto-detached after result consumption -- ------------------------------------------------------------------------- -- Error Path: Invalid handle (detached worker) -- ------------------------------------------------------------------------- DROP TABLE IF EXISTS _test_dh_handle; NOTICE: table "_test_dh_handle" does not exist, skipping CREATE TEMP TABLE _test_dh_handle AS SELECT (h).pid, (h).cookie FROM (SELECT pg_background_launch_v2('SELECT 1') AS h) s; SELECT pg_sleep(0.2); pg_sleep ---------- (1 row) DO $$ DECLARE v_pid int; v_cookie bigint; BEGIN SELECT pid, cookie INTO v_pid, v_cookie FROM _test_dh_handle; PERFORM pg_background_detach_v2(v_pid, v_cookie); END; $$; -- Operations on detached handle should error DO $$ DECLARE v_pid int; v_cookie bigint; BEGIN SELECT pid, cookie INTO v_pid, v_cookie FROM _test_dh_handle; PERFORM pg_background_wait_v2(v_pid, v_cookie); RAISE NOTICE 'detached_handle_test=should_have_failed'; EXCEPTION WHEN OTHERS THEN RAISE NOTICE 'detached_handle_test=correctly_errored'; END; $$; NOTICE: detached_handle_test=correctly_errored -- ------------------------------------------------------------------------- -- Privilege Model: pgbackground_role exists -- ------------------------------------------------------------------------- SELECT CASE WHEN EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'pgbackground_role') THEN 'PASS' ELSE 'FAIL' END AS pgbackground_role_exists; pgbackground_role_exists -------------------------- PASS (1 row) -- ------------------------------------------------------------------------- -- Privilege Model: PUBLIC has no direct access -- ------------------------------------------------------------------------- SELECT CASE WHEN NOT has_function_privilege('public', 'pg_background_launch_v2(text, int4, text)', 'EXECUTE') THEN 'PASS' ELSE 'FAIL' END AS public_no_launch_access; public_no_launch_access ------------------------- PASS (1 row) SELECT CASE WHEN NOT has_function_privilege('public', 'pg_background_result_v2(int4, int8)', 'EXECUTE') THEN 'PASS' ELSE 'FAIL' END AS public_no_result_access; public_no_result_access ------------------------- PASS (1 row) -- ------------------------------------------------------------------------- -- Privilege Model: pgbackground_role has access -- ------------------------------------------------------------------------- SELECT CASE WHEN has_function_privilege('pgbackground_role', 'pg_background_launch_v2(text, int4, text)', 'EXECUTE') THEN 'PASS' ELSE 'FAIL' END AS role_has_launch_access; role_has_launch_access ------------------------ PASS (1 row) SELECT CASE WHEN has_function_privilege('pgbackground_role', 'pg_background_result_v2(int4, int8)', 'EXECUTE') THEN 'PASS' ELSE 'FAIL' END AS role_has_result_access; role_has_result_access ------------------------ PASS (1 row) -- ------------------------------------------------------------------------- -- Privilege Model: Grant/Revoke helpers work -- ------------------------------------------------------------------------- DROP ROLE IF EXISTS test_priv_user; NOTICE: role "test_priv_user" does not exist, skipping CREATE ROLE test_priv_user NOLOGIN; -- Should not have access initially SELECT CASE WHEN NOT has_function_privilege('test_priv_user', 'pg_background_launch_v2(text, int4, text)', 'EXECUTE') THEN 'PASS' ELSE 'FAIL' END AS user_no_initial_access; user_no_initial_access ------------------------ PASS (1 row) -- Grant privileges SELECT pg_background_grant_privileges('test_priv_user', false) AS grant_result; grant_result -------------- t (1 row) -- Should have access after grant SELECT CASE WHEN has_function_privilege('test_priv_user', 'pg_background_launch_v2(text, int4, text)', 'EXECUTE') THEN 'PASS' ELSE 'FAIL' END AS user_has_access_after_grant; user_has_access_after_grant ----------------------------- PASS (1 row) -- Revoke privileges SELECT pg_background_revoke_privileges('test_priv_user', false) AS revoke_result; revoke_result --------------- t (1 row) -- Should not have access after revoke SELECT CASE WHEN NOT has_function_privilege('test_priv_user', 'pg_background_launch_v2(text, int4, text)', 'EXECUTE') THEN 'PASS' ELSE 'FAIL' END AS user_no_access_after_revoke; user_no_access_after_revoke ----------------------------- PASS (1 row) DROP ROLE test_priv_user; -- ------------------------------------------------------------------------- -- Bounds Validation: cancel_v2_grace grace_ms bounds -- ------------------------------------------------------------------------- -- Test that very large grace_ms is handled (capped at 1 hour = 3600000ms) SELECT (h).pid AS bg_pid, (h).cookie AS bg_cookie FROM (SELECT pg_background_launch_v2('SELECT pg_sleep(10)') AS h) s \gset SELECT pg_sleep(0.1); pg_sleep ---------- (1 row) -- This should work without error (grace_ms capped internally) SELECT pg_background_cancel_v2(:bg_pid, :bg_cookie, 999999999); pg_background_cancel_v2 ------------------------- (1 row) SELECT pg_sleep(0.3); pg_sleep ---------- (1 row) SELECT pg_background_detach_v2(:bg_pid, :bg_cookie); pg_background_detach_v2 ------------------------- (1 row) SELECT 'PASS' AS grace_bounds_test; grace_bounds_test ------------------- PASS (1 row) -- ------------------------------------------------------------------------- -- Semantic test: detach allows worker to complete (does NOT cancel) -- This is already tested by t_detach_v1/t_detach_v2 above, but we add -- an explicit comment for clarity. The earlier tests verify that after -- detach, workers still commit their transactions. -- ------------------------------------------------------------------------- SELECT 'PASS' AS detach_semantic_verified; detach_semantic_verified -------------------------- PASS (1 row) -- ------------------------------------------------------------------------- -- v2.0 (E7): cancel idempotency -- -- Two race-flavoured guarantees: -- (a) Double-cancel is harmless: a second pg_background_cancel_v2 on a -- worker that's already been canceled does not raise, and the -- workers_canceled counter only increments by one (not two). -- (b) Cancel-after-exit is harmless: pg_background_cancel_v2 on a worker -- that has already finished does not raise. -- -- These are the deterministic cousins of the flaky cancel-vs-completion -- race; we keep them small and explicit rather than chasing scheduler -- order with sleeps. -- ------------------------------------------------------------------------- -- (a) Double cancel. -- -- KNOWN ISSUE: an immediate consecutive cancel against a worker mid- -- execute_sql_string sometimes triggers a worker SIGSEGV in error_exit. -- See README "Known Limitations" §10. We sidestep by waiting for the -- worker to finish exiting after the first cancel and asserting that the -- second cancel against an already-stopped worker is harmless. That still -- exercises the no-error-on-redundant-cancel contract that callers rely -- on; the "two cancels race" mid-execution variant is covered by the -- E7 (b) sub-test below as cancel-after-exit. DO $$ DECLARE h pg_background_handle; before_canceled int8; after_canceled int8; BEGIN SELECT workers_canceled INTO before_canceled FROM pg_background_stats_v2(); h := pg_background_launch_v2('SELECT pg_sleep(60)', 65536, 'e7-double-cancel'); PERFORM pg_background_cancel_v2(h.pid, h.cookie); PERFORM pg_background_wait_v2(h.pid, h.cookie); /* * Worker is now stopped. Second cancel on the same handle must be a * no-op, not an error. */ PERFORM pg_background_cancel_v2(h.pid, h.cookie); PERFORM pg_background_detach_v2(h.pid, h.cookie); SELECT workers_canceled INTO after_canceled FROM pg_background_stats_v2(); IF after_canceled - before_canceled <> 1 THEN RAISE EXCEPTION 'E7 double-cancel: workers_canceled went % -> % (want +1)', before_canceled, after_canceled; END IF; RAISE NOTICE 'E7 double-cancel idempotent OK'; END$$; NOTICE: E7 double-cancel idempotent OK -- (b) Cancel after exit DO $$ DECLARE h pg_background_handle; BEGIN h := pg_background_launch_v2('SELECT 1', 65536, 'e7-cancel-after-exit'); /* Wait until the worker has fully exited */ PERFORM pg_background_wait_v2(h.pid, h.cookie); /* The worker is BGWH_STOPPED. cancel_v2 should be a no-op (no error) */ BEGIN PERFORM pg_background_cancel_v2(h.pid, h.cookie); EXCEPTION WHEN OTHERS THEN RAISE EXCEPTION 'E7 cancel-after-exit: unexpected SQLSTATE % (%)', SQLSTATE, SQLERRM; END; PERFORM pg_background_detach_v2(h.pid, h.cookie); RAISE NOTICE 'E7 cancel-after-exit harmless OK'; END$$; NOTICE: E7 cancel-after-exit harmless OK -- ------------------------------------------------------------------------- -- Error propagation SQLSTATE tests (v1.9 bugfix) -- -- Pattern: launch_v2 -> wait_v2 -> error_info_v2 -> detach_v2 -- Do NOT call result_v2 for error cases: it ereport(ERROR)s in launcher, -- which aborts the transaction and destroys error_info_v2 data. -- ------------------------------------------------------------------------- -- Test 3.1: Division by zero — SQLSTATE 22012 (execute path) DO $$ DECLARE h pg_background_handle; s text; BEGIN h := pg_background_launch_v2('SELECT 1/0'); PERFORM pg_background_wait_v2(h.pid, h.cookie); SELECT sqlstate INTO s FROM pg_background_error_info_v2(h.pid, h.cookie); IF s IS DISTINCT FROM '22012' THEN RAISE EXCEPTION 'test 3.1: expected sqlstate 22012, got %', s; END IF; PERFORM pg_background_detach_v2(h.pid, h.cookie); RAISE NOTICE 'test 3.1 (22012 division_by_zero) OK'; END$$; NOTICE: test 3.1 (22012 division_by_zero) OK -- Test 3.2: RAISE EXCEPTION — SQLSTATE P0001 (execute path) -- Use a helper function instead of nested dollar-quoting to avoid psql parsing issues. CREATE OR REPLACE FUNCTION pgbg_test_raise_p0001() RETURNS void LANGUAGE plpgsql AS 'BEGIN RAISE EXCEPTION ''custom error''; END'; DO $$ DECLARE h pg_background_handle; s text; BEGIN h := pg_background_launch_v2('SELECT pgbg_test_raise_p0001()'); PERFORM pg_background_wait_v2(h.pid, h.cookie); SELECT sqlstate INTO s FROM pg_background_error_info_v2(h.pid, h.cookie); IF s IS DISTINCT FROM 'P0001' THEN RAISE EXCEPTION 'test 3.2: expected sqlstate P0001, got %', s; END IF; PERFORM pg_background_detach_v2(h.pid, h.cookie); RAISE NOTICE 'test 3.2 (P0001 raise_exception) OK'; END$$; NOTICE: test 3.2 (P0001 raise_exception) OK DROP FUNCTION pgbg_test_raise_p0001(); -- Test 3.3: NOT NULL violation — SQLSTATE 23502 (execute path) -- Note: temp tables are NOT visible to background workers (separate backend). -- Use a regular table with a unique prefix. DROP TABLE IF EXISTS pgbg_test_nn_23502; NOTICE: table "pgbg_test_nn_23502" does not exist, skipping CREATE TABLE pgbg_test_nn_23502(c int NOT NULL); DO $$ DECLARE h pg_background_handle; s text; BEGIN h := pg_background_launch_v2('INSERT INTO pgbg_test_nn_23502 VALUES (NULL)'); PERFORM pg_background_wait_v2(h.pid, h.cookie); SELECT sqlstate INTO s FROM pg_background_error_info_v2(h.pid, h.cookie); IF s IS DISTINCT FROM '23502' THEN RAISE EXCEPTION 'test 3.3: expected sqlstate 23502, got %', s; END IF; PERFORM pg_background_detach_v2(h.pid, h.cookie); RAISE NOTICE 'test 3.3 (23502 not_null_violation) OK'; END$$; NOTICE: test 3.3 (23502 not_null_violation) OK DROP TABLE pgbg_test_nn_23502; -- Test 3.4: Deferred FK violation — SQLSTATE 23503 (commit path) -- FK is INITIALLY DEFERRED, so INSERT succeeds but COMMIT fails. -- The error fires in the commit-wrapper PG_CATCH (Phase 2 path). DROP TABLE IF EXISTS pgbg_test_fk_child_23503; NOTICE: table "pgbg_test_fk_child_23503" does not exist, skipping DROP TABLE IF EXISTS pgbg_test_fk_parent_23503; NOTICE: table "pgbg_test_fk_parent_23503" does not exist, skipping CREATE TABLE pgbg_test_fk_parent_23503(id int PRIMARY KEY); CREATE TABLE pgbg_test_fk_child_23503( id int PRIMARY KEY, parent_id int, CONSTRAINT fk_23503_parent FOREIGN KEY (parent_id) REFERENCES pgbg_test_fk_parent_23503(id) DEFERRABLE INITIALLY DEFERRED ); DO $$ DECLARE h pg_background_handle; s text; BEGIN -- Worker runs INSERT in its own auto-transaction. -- INITIALLY DEFERRED FK fires at CommitTransactionCommand (commit-wrapper PG_CATCH). -- Do NOT pass BEGIN/COMMIT: worker already runs in its own transaction. h := pg_background_launch_v2( 'INSERT INTO pgbg_test_fk_child_23503 VALUES (1, 999)' ); PERFORM pg_background_wait_v2(h.pid, h.cookie); SELECT sqlstate INTO s FROM pg_background_error_info_v2(h.pid, h.cookie); IF s IS DISTINCT FROM '23503' THEN RAISE EXCEPTION 'test 3.4: expected sqlstate 23503, got %', s; END IF; PERFORM pg_background_detach_v2(h.pid, h.cookie); RAISE NOTICE 'test 3.4 (23503 foreign_key_violation deferred) OK'; END$$; NOTICE: test 3.4 (23503 foreign_key_violation deferred) OK DROP TABLE pgbg_test_fk_child_23503; DROP TABLE pgbg_test_fk_parent_23503; -- Test 3.5: Cancel — SQLSTATE 57014 (execute path, query_canceled) -- Uses pg_sleep(30) to ensure worker is truly sleeping before cancel. -- Positive sync via pg_stat_activity: cancel is only sent after the worker -- is confirmed to be actively executing pg_sleep. Without this sync, cancel -- could land before SPI_execute starts (early-failure path) and the -- launcher would see 08006 instead of 57014. DO $$ DECLARE h pg_background_handle; w_pid int4; s text; done boolean; cnt int; BEGIN h := pg_background_launch_v2('SELECT pg_sleep(30)'); w_pid := h.pid; /* Positive sync: let the worker fork and reach SPI_execute, then confirm it is actively running via pg_stat_activity. An initial sleep gives the OS scheduler time to start the child process; afterwards we poll at 100 ms intervals for up to ~5 s total. */ PERFORM pg_sleep(0.1); FOR i IN 1..55 LOOP SELECT count(*) INTO cnt FROM pg_stat_activity WHERE pid = w_pid AND state = 'active'; EXIT WHEN cnt > 0; PERFORM pg_sleep(0.1); END LOOP; IF cnt = 0 THEN RAISE EXCEPTION 'test 3.5: worker not active within 5.6s — test not valid'; END IF; /* Worker is past pq_redirect_to_shm_mq and executing SQL; cancel will be caught by the execute-phase PG_CATCH and produce 57014. */ PERFORM pg_background_cancel_v2(h.pid, h.cookie, 100); -- Wait for worker to stop after cancel (should be fast) done := pg_background_wait_v2(h.pid, h.cookie, 5000); IF NOT done THEN RAISE EXCEPTION 'test 3.5: worker did not stop within 5s after cancel'; END IF; SELECT sqlstate INTO s FROM pg_background_error_info_v2(h.pid, h.cookie); IF s IS DISTINCT FROM '57014' THEN RAISE EXCEPTION 'test 3.5: expected sqlstate 57014, got %', s; END IF; PERFORM pg_background_detach_v2(h.pid, h.cookie); RAISE NOTICE 'test 3.5 (57014 query_canceled) OK'; END$$; NOTICE: test 3.5 (57014 query_canceled) OK -- ------------------------------------------------------------------------- -- Stats sanity check (mid-suite snapshot). -- -- We don't pin exact counts here — the v1.10 / v2.0 sections below launch -- many additional workers and the totals shift every time a new test is -- added. Instead we assert structural invariants: no workers active right -- now, every launched worker has finished one way or another, and the -- canceled bucket is non-empty (proves the cancel tests actually fired). -- ------------------------------------------------------------------------- DO $$ DECLARE s pg_background_stats; BEGIN s := pg_background_stats_v2(); IF s.workers_active <> 0 THEN RAISE EXCEPTION 'mid-suite stats: expected 0 active workers, got %', s.workers_active; END IF; IF (s.workers_completed + s.workers_failed + s.workers_canceled + s.workers_timed_out) <> s.workers_launched THEN RAISE EXCEPTION 'mid-suite stats: launched (%) <> completed+failed+canceled+timed_out (% + % + % + %)', s.workers_launched, s.workers_completed, s.workers_failed, s.workers_canceled, s.workers_timed_out; END IF; IF s.workers_canceled = 0 THEN RAISE EXCEPTION 'mid-suite stats: expected at least one canceled worker, got 0'; END IF; RAISE NOTICE 'mid-suite stats invariants OK'; END$$; NOTICE: mid-suite stats invariants OK -- ========================================================================= -- v1.10: ergonomics (pg_background_list view, outcome_v2, run_v2) -- ========================================================================= -- pg_background_list view: every column should be populated with a sane -- value (not just a non-zero row count). v2.0 (E4) replaces the previous -- "did one row come back?" test with explicit per-column assertions. DO $$ DECLARE h pg_background_handle; r_state text; r_qsize int; r_preview text; r_consumed bool; r_label text; r_user oid; r_launched timestamptz; BEGIN h := pg_background_launch_v2('SELECT pg_sleep(0.5)', 65536, 'v1_10_view'); SELECT state, queue_size, sql_preview, consumed, label, user_id, launched_at INTO r_state, r_qsize, r_preview, r_consumed, r_label, r_user, r_launched FROM pg_background_list WHERE pid = h.pid AND cookie = h.cookie; IF r_state NOT IN ('starting', 'running', 'stopped') THEN RAISE EXCEPTION 'list view state: unexpected value %', r_state; END IF; IF r_qsize <> 65536 THEN RAISE EXCEPTION 'list view queue_size: expected 65536, got %', r_qsize; END IF; IF r_preview NOT LIKE 'SELECT pg_sleep%' THEN RAISE EXCEPTION 'list view sql_preview: did not match expected prefix, got %', r_preview; END IF; IF r_consumed IS DISTINCT FROM false THEN RAISE EXCEPTION 'list view consumed: expected false, got %', r_consumed; END IF; IF r_label IS DISTINCT FROM 'v1_10_view' THEN RAISE EXCEPTION 'list view label: expected v1_10_view, got %', r_label; END IF; IF r_user <> (SELECT oid FROM pg_roles WHERE rolname = current_user) THEN RAISE EXCEPTION 'list view user_id: did not match current_user oid'; END IF; IF r_launched IS NULL OR r_launched > clock_timestamp() THEN RAISE EXCEPTION 'list view launched_at: NULL or in the future (%)', r_launched; END IF; PERFORM pg_background_wait_v2(h.pid, h.cookie); PERFORM pg_background_detach_v2(h.pid, h.cookie); RAISE NOTICE 'v1.10 list view (per-column assertions) OK'; END$$; NOTICE: v1.10 list view (per-column assertions) OK -- pg_background_activity view: assert the join with pg_stat_activity actually -- produces a row for our worker, and that the joined backend_state column -- carries through (v2.0 (E4): replaces the previous RAISE NOTICE 'OK'). DO $$ DECLARE h pg_background_handle; found_pgbg text; found_backend text; BEGIN h := pg_background_launch_v2('SELECT pg_sleep(2)', 65536, 'v1_10_activity'); /* worker needs a moment to register in pg_stat_activity */ PERFORM pg_sleep(0.3); SELECT pgbg_state, backend_state INTO found_pgbg, found_backend FROM pg_background_activity WHERE pid = h.pid AND cookie = h.cookie; IF found_pgbg NOT IN ('starting', 'running') THEN RAISE EXCEPTION 'activity view pgbg_state: expected starting/running, got %', found_pgbg; END IF; /* backend_state can be 'active' (executing) or NULL (joined too early); accept either */ IF found_backend IS NOT NULL AND found_backend NOT IN ('active', 'idle', 'idle in transaction') THEN RAISE EXCEPTION 'activity view backend_state: unexpected value %', found_backend; END IF; PERFORM pg_background_cancel_v2(h.pid, h.cookie); PERFORM pg_background_wait_v2(h.pid, h.cookie); PERFORM pg_background_detach_v2(h.pid, h.cookie); RAISE NOTICE 'v1.10 activity view (joined columns) OK'; END$$; NOTICE: v1.10 activity view (joined columns) OK -- pg_background_outcome_v2: success path DO $$ DECLARE h pg_background_handle; o pg_background_outcome; BEGIN h := pg_background_launch_v2('SELECT 1', 65536, 'v1_10_outcome_ok'); PERFORM pg_background_wait_v2(h.pid, h.cookie); o := pg_background_outcome_v2(h.pid, h.cookie); IF o.completed IS DISTINCT FROM true THEN RAISE EXCEPTION 'v1.10 outcome ok: expected completed=true, got %', o.completed; END IF; IF o.has_error IS DISTINCT FROM false THEN RAISE EXCEPTION 'v1.10 outcome ok: expected has_error=false, got %', o.has_error; END IF; IF o.label IS DISTINCT FROM 'v1_10_outcome_ok' THEN RAISE EXCEPTION 'v1.10 outcome ok: expected label v1_10_outcome_ok, got %', o.label; END IF; PERFORM pg_background_detach_v2(h.pid, h.cookie); RAISE NOTICE 'v1.10 outcome ok OK'; END$$; NOTICE: v1.10 outcome ok OK -- pg_background_outcome_v2: missing handle returns NULL fields, never raises DO $$ DECLARE o pg_background_outcome; BEGIN o := pg_background_outcome_v2(0, 0); IF o.pid <> 0 OR o.cookie <> 0 THEN RAISE EXCEPTION 'v1.10 outcome missing: pid/cookie not echoed'; END IF; IF o.state IS NOT NULL OR o.completed IS NOT NULL OR o.has_error IS NOT NULL THEN RAISE EXCEPTION 'v1.10 outcome missing: expected all NULL fields'; END IF; RAISE NOTICE 'v1.10 outcome missing OK'; END$$; NOTICE: v1.10 outcome missing OK -- pg_background_run_v2: success path. v2.0 (E5) widens the assertions to -- cover every column of the now-extended pg_background_run_result. DROP TABLE IF EXISTS t_v1_10_run; NOTICE: table "t_v1_10_run" does not exist, skipping CREATE TABLE t_v1_10_run(id int); DO $$ DECLARE r pg_background_run_result; BEGIN r := pg_background_run_v2('INSERT INTO t_v1_10_run VALUES (1), (2), (3)', 65536, 0, 'v1_10_run_ok'); /* core completion / error fields */ IF r.completed IS DISTINCT FROM true THEN RAISE EXCEPTION 'run ok: completed=% (want true)', r.completed; END IF; IF r.has_error IS DISTINCT FROM false THEN RAISE EXCEPTION 'run ok: has_error=% (want false)', r.has_error; END IF; IF r.timed_out IS DISTINCT FROM false THEN RAISE EXCEPTION 'run ok: timed_out=% (want false)', r.timed_out; END IF; IF r.sqlstate IS NOT NULL THEN RAISE EXCEPTION 'run ok: sqlstate=% (want NULL)', r.sqlstate; END IF; IF r.error_message IS NOT NULL THEN RAISE EXCEPTION 'run ok: error_message=% (want NULL)', r.error_message; END IF; /* result metadata */ IF r.row_count IS DISTINCT FROM 3 THEN RAISE EXCEPTION 'run ok: row_count=% (want 3)', r.row_count; END IF; IF r.command_tag IS NULL OR r.command_tag NOT LIKE 'INSERT%' THEN RAISE EXCEPTION 'run ok: command_tag=% (want INSERT*)', r.command_tag; END IF; /* extended outcome fields gained in v2.0 */ IF r.pid IS NULL OR r.pid <= 0 THEN RAISE EXCEPTION 'run ok: pid=% (want > 0)', r.pid; END IF; IF r.cookie IS NULL OR r.cookie = 0 THEN RAISE EXCEPTION 'run ok: cookie=% (want non-zero)', r.cookie; END IF; IF r.label IS DISTINCT FROM 'v1_10_run_ok' THEN RAISE EXCEPTION 'run ok: label=% (want v1_10_run_ok)', r.label; END IF; /* * After detach the worker is gone from pg_background_list, so state and * consumed come back NULL from outcome_v2 — that's the point of the * extended run_result: the caller sees the snapshot, not a live cursor. */ /* elapsed_ms invariants — non-negative, bounded above by something sane */ IF r.elapsed_ms IS NULL OR r.elapsed_ms < 0 THEN RAISE EXCEPTION 'run ok: elapsed_ms=% (want >= 0)', r.elapsed_ms; END IF; IF r.elapsed_ms > 30000 THEN RAISE EXCEPTION 'run ok: elapsed_ms=% (suspiciously slow, > 30s)', r.elapsed_ms; END IF; RAISE NOTICE 'v1.10 run ok (extended assertions) OK'; END$$; NOTICE: v1.10 run ok (extended assertions) OK -- The launched worker actually inserted rows (committed via worker exit). SELECT count(*) AS v1_10_run_inserted FROM t_v1_10_run; v1_10_run_inserted -------------------- 3 (1 row) DROP TABLE t_v1_10_run; -- pg_background_run_v2: error case (1/0 -> sqlstate 22012) DO $$ DECLARE r pg_background_run_result; BEGIN r := pg_background_run_v2('SELECT 1/0', 65536, 0, 'v1_10_run_err'); IF r.has_error IS DISTINCT FROM true THEN RAISE EXCEPTION 'v1.10 run err: expected has_error=true, got %', r.has_error; END IF; IF r.sqlstate IS DISTINCT FROM '22012' THEN RAISE EXCEPTION 'v1.10 run err: expected sqlstate=22012, got %', r.sqlstate; END IF; IF r.timed_out IS DISTINCT FROM false THEN RAISE EXCEPTION 'v1.10 run err: expected timed_out=false, got %', r.timed_out; END IF; RAISE NOTICE 'v1.10 run err OK'; END$$; NOTICE: v1.10 run err OK -- pg_background_run_v2: timeout case. v2.0 (E5) verifies the actual -- timeout-driven invariants: timed_out=true, elapsed_ms >= the timeout we -- supplied (we asked for 200ms; with 1s grace the wait can run a bit -- longer), and the workers_timed_out stats counter incremented. DO $$ DECLARE r pg_background_run_result; before_timeout int8; after_timeout int8; BEGIN SELECT workers_timed_out INTO before_timeout FROM pg_background_stats_v2(); r := pg_background_run_v2('SELECT pg_sleep(5)', 65536, 200, 'v1_10_run_timeout'); IF r.timed_out IS DISTINCT FROM true THEN RAISE EXCEPTION 'run timeout: timed_out=% (want true)', r.timed_out; END IF; IF r.completed IS DISTINCT FROM true THEN /* * The worker WAS canceled and DID exit cleanly within grace, so * GetBackgroundWorkerPid returns BGWH_STOPPED → completed=true even * though timed_out=true. This pair is the documented contract. */ RAISE EXCEPTION 'run timeout: completed=% (want true after grace cancel)', r.completed; END IF; IF r.elapsed_ms IS NULL OR r.elapsed_ms < 200 THEN RAISE EXCEPTION 'run timeout: elapsed_ms=% (want >= 200, the requested timeout)', r.elapsed_ms; END IF; IF r.label IS DISTINCT FROM 'v1_10_run_timeout' THEN RAISE EXCEPTION 'run timeout: label=% (want v1_10_run_timeout)', r.label; END IF; SELECT workers_timed_out INTO after_timeout FROM pg_background_stats_v2(); IF after_timeout <> before_timeout + 1 THEN RAISE EXCEPTION 'run timeout: workers_timed_out went % -> % (want +1)', before_timeout, after_timeout; END IF; RAISE NOTICE 'v1.10 run timeout (timed_out + elapsed_ms + stats counter) OK'; END$$; NOTICE: v1.10 run timeout (timed_out + elapsed_ms + stats counter) OK -- ========================================================================= -- Refactor: metadata-driven grant/revoke helpers -- -- Contract: pg_background_grant_privileges() must cover every -- extension-owned function (EXCEPT the SECURITY DEFINER privilege helpers, -- which must NOT be granted to the executor role -- privilege-escalation -- guard), type, and view, without an explicit list. This test pins the -- contract by round-tripping a temp role and asserting that EXECUTE/USAGE/ -- SELECT privileges flip on grant and off on revoke, and that the -- SECURITY DEFINER helpers stay unreachable by the role. -- ========================================================================= DO $$ DECLARE n_funcs_granted int; n_funcs_total int; n_views_granted int; n_views_total int; BEGIN CREATE ROLE pgbg_meta_test_role NOLOGIN; PERFORM pg_background_grant_privileges('pgbg_meta_test_role'); -- Every extension-owned function EXCEPT the SECURITY DEFINER privilege -- helpers must now be EXECUTE-able by the role. SELECT count(*) INTO n_funcs_total FROM pg_depend d JOIN pg_proc p ON p.oid = d.objid WHERE d.classid = 'pg_proc'::regclass AND d.refclassid = 'pg_extension'::regclass AND d.refobjid = (SELECT oid FROM pg_extension WHERE extname='pg_background') AND d.deptype = 'e' AND NOT p.prosecdef; SELECT count(*) INTO n_funcs_granted FROM pg_depend d JOIN pg_proc p ON p.oid = d.objid WHERE d.classid = 'pg_proc'::regclass AND d.refclassid = 'pg_extension'::regclass AND d.refobjid = (SELECT oid FROM pg_extension WHERE extname='pg_background') AND d.deptype = 'e' AND NOT p.prosecdef AND has_function_privilege('pgbg_meta_test_role', p.oid, 'EXECUTE'); IF n_funcs_granted <> n_funcs_total THEN RAISE EXCEPTION 'metadata grant: % of % functions reachable by role', n_funcs_granted, n_funcs_total; END IF; -- Privilege-escalation guard: the SECURITY DEFINER helpers must NOT be -- reachable by the executor role. IF has_function_privilege('pgbg_meta_test_role', 'pg_background_grant_privileges(text, boolean)', 'EXECUTE') OR has_function_privilege('pgbg_meta_test_role', 'pg_background_revoke_privileges(text, boolean)', 'EXECUTE') THEN RAISE EXCEPTION 'security: SECURITY DEFINER privilege helpers reachable by role'; END IF; -- Every extension-owned view must now be SELECT-able by the role. SELECT count(*) INTO n_views_total FROM pg_depend d JOIN pg_class c ON c.oid = d.objid WHERE d.classid = 'pg_class'::regclass AND d.refclassid = 'pg_extension'::regclass AND d.refobjid = (SELECT oid FROM pg_extension WHERE extname='pg_background') AND d.deptype = 'e' AND c.relkind IN ('v','r','m'); SELECT count(*) INTO n_views_granted FROM pg_depend d JOIN pg_class c ON c.oid = d.objid WHERE d.classid = 'pg_class'::regclass AND d.refclassid = 'pg_extension'::regclass AND d.refobjid = (SELECT oid FROM pg_extension WHERE extname='pg_background') AND d.deptype = 'e' AND c.relkind IN ('v','r','m') AND has_table_privilege('pgbg_meta_test_role', c.oid, 'SELECT'); IF n_views_granted <> n_views_total THEN RAISE EXCEPTION 'metadata grant: % of % views reachable by role', n_views_granted, n_views_total; END IF; -- Symmetric revoke: nothing should remain reachable. PERFORM pg_background_revoke_privileges('pgbg_meta_test_role'); SELECT count(*) INTO n_funcs_granted FROM pg_depend d JOIN pg_proc p ON p.oid = d.objid WHERE d.classid = 'pg_proc'::regclass AND d.refclassid = 'pg_extension'::regclass AND d.refobjid = (SELECT oid FROM pg_extension WHERE extname='pg_background') AND d.deptype = 'e' AND has_function_privilege('pgbg_meta_test_role', p.oid, 'EXECUTE'); IF n_funcs_granted <> 0 THEN RAISE EXCEPTION 'metadata revoke: % functions still reachable', n_funcs_granted; END IF; DROP ROLE pgbg_meta_test_role; RAISE NOTICE 'metadata-driven grant/revoke OK (% funcs, % views round-tripped)', n_funcs_total, n_views_total; END$$; NOTICE: metadata-driven grant/revoke OK (48 funcs, 2 views round-tripped) -- ========================================================================= -- v1.10 Tier A: loop-killer helpers -- ========================================================================= -- A1: pg_background_run_query_v2 success + error path SELECT * FROM pg_background_run_query_v2('SELECT 42 AS n', col_def => 'n int') AS r(n int); n ---- 42 (1 row) DO $$ BEGIN BEGIN PERFORM * FROM pg_background_run_query_v2('SELECT 1/0', col_def => 'x int') AS r(x int); RAISE EXCEPTION 'Tier A run_query_v2: error not propagated'; EXCEPTION WHEN division_by_zero THEN RAISE NOTICE 'Tier A run_query_v2 SQLSTATE 22012 OK'; END; END$$; NOTICE: Tier A run_query_v2 SQLSTATE 22012 OK -- A2: drain_v2 with N=3 — v2.0 (E5) asserts row order matches input order -- and every per-row outcome is well-formed. DO $$ DECLARE hs pg_background_handle[]; rows pg_background_outcome[]; i int; expected_lbl text; BEGIN hs := ARRAY( SELECT pg_background_launch_v2('SELECT pg_sleep(0.05)', 65536, 'tier-a-drain-' || g) FROM generate_series(1,3) g ); rows := ARRAY( SELECT pg_background_drain_v2(hs, 5000) ); IF array_length(rows, 1) <> 3 THEN RAISE EXCEPTION 'drain_v2: drained %d rows, want 3', array_length(rows, 1); END IF; FOR i IN 1..3 LOOP expected_lbl := 'tier-a-drain-' || i; IF rows[i].pid IS DISTINCT FROM hs[i].pid THEN RAISE EXCEPTION 'drain_v2 row % pid mismatch: % vs %', i, rows[i].pid, hs[i].pid; END IF; IF rows[i].cookie IS DISTINCT FROM hs[i].cookie THEN RAISE EXCEPTION 'drain_v2 row % cookie mismatch', i; END IF; IF rows[i].label IS DISTINCT FROM expected_lbl THEN RAISE EXCEPTION 'drain_v2 row % label=% want %', i, rows[i].label, expected_lbl; END IF; IF rows[i].completed IS DISTINCT FROM true THEN RAISE EXCEPTION 'drain_v2 row % completed=% want true', i, rows[i].completed; END IF; IF rows[i].has_error IS DISTINCT FROM false THEN RAISE EXCEPTION 'drain_v2 row % has_error=% want false', i, rows[i].has_error; END IF; END LOOP; RAISE NOTICE 'Tier A drain_v2 (per-row assertions) OK'; END$$; NOTICE: Tier A drain_v2 (per-row assertions) OK -- A3: wait_any_v2 returns one winner. v2.0 (E5) asserts the winner's pid -- belongs to the input array and its outcome reflects a finished worker. DO $$ DECLARE hs pg_background_handle[]; winner pg_background_handle; winner_in_set bool; winner_pids int[]; BEGIN hs := ARRAY( SELECT pg_background_launch_v2(format('SELECT pg_sleep(%s)', g*0.03), 65536, 'tier-a-any-' || g) FROM generate_series(1,3) g ); winner := pg_background_wait_any_v2(hs, 5000); IF winner IS NULL THEN RAISE EXCEPTION 'wait_any_v2: timed out without a winner'; END IF; /* the winner must be one of the handles we passed in */ winner_pids := ARRAY(SELECT (h).pid FROM unnest(hs) h); winner_in_set := winner.pid = ANY (winner_pids); IF NOT winner_in_set THEN RAISE EXCEPTION 'wait_any_v2: winner pid % not in input array %', winner.pid, winner_pids; END IF; /* and pg_background_wait_v2(winner, 1) should report it finished */ IF NOT pg_background_wait_v2(winner.pid, winner.cookie, 1) THEN RAISE EXCEPTION 'wait_any_v2: winner % is not actually stopped', winner.pid; END IF; /* Cleanup via drain — handles wait + detach for every handle */ PERFORM count(*) FROM pg_background_drain_v2(hs, 5000); RAISE NOTICE 'Tier A wait_any_v2 (winner-pid + finished assertion) OK'; END$$; NOTICE: Tier A wait_any_v2 (winner-pid + finished assertion) OK -- A4: cancel_by_label_v2 with a LIKE pattern. v2.0 (E5) asserts that the -- canceled worker actually appears in stats.workers_canceled. -- -- KNOWN ISSUE (pre-existing, see README "Known Limitations" §10): launching -- multiple workers in close succession then cancelling them concurrently -- triggers a SIGSEGV in PostgreSQL's background-worker startup machinery — -- the segfaulting worker never reaches our `pg_background_worker_main`, so -- the issue is upstream of pg_background. To keep the suite deterministic, -- this test exercises the function against a single worker. Multi-worker -- coverage is provided by drain_v2 / wait_any_v2 elsewhere (which also use -- multiple concurrent workers but don't cancel them mid-flight). DO $$ DECLARE cnt int; before_canceled int8; after_canceled int8; h pg_background_handle; BEGIN SELECT workers_canceled INTO before_canceled FROM pg_background_stats_v2(); h := pg_background_launch_v2('SELECT pg_sleep(60)', 65536, 'tier-a-cancel-1'); cnt := pg_background_cancel_by_label_v2('tier-a-cancel-%'); IF cnt <> 1 THEN RAISE EXCEPTION 'cancel_by_label_v2: cnt=% want 1', cnt; END IF; PERFORM pg_background_wait_v2(h.pid, h.cookie); PERFORM pg_background_detach_v2(h.pid, h.cookie); SELECT workers_canceled INTO after_canceled FROM pg_background_stats_v2(); IF after_canceled - before_canceled < 1 THEN RAISE EXCEPTION 'cancel_by_label_v2: workers_canceled +%, want >= 1', after_canceled - before_canceled; END IF; RAISE NOTICE 'Tier A cancel_by_label_v2 (single-worker grace=0 path) OK'; END$$; NOTICE: Tier A cancel_by_label_v2 (single-worker grace=0 path) OK -- v2.0: status_v2 was dropped; drivers can call to_jsonb(outcome_v2(...)) directly. DO $$ DECLARE r pg_background_run_result; j jsonb; BEGIN r := pg_background_run_v2('SELECT 1', label => 'tier-a-status'); j := to_jsonb(pg_background_outcome_v2(r.pid, r.cookie)); IF NOT (j ? 'pid' AND j ? 'completed' AND j ? 'has_error' AND j ? 'sqlstate') THEN RAISE EXCEPTION 'outcome_v2 jsonb: missing expected keys: %', j; END IF; RAISE NOTICE 'outcome_v2 jsonb OK'; END$$; NOTICE: outcome_v2 jsonb OK -- A6: purge_v2 detaches only stopped workers DO $$ DECLARE h_done pg_background_handle; h_running pg_background_handle; purged int; BEGIN h_done := pg_background_launch_v2('SELECT 1', 65536, 'tier-a-purge-done'); PERFORM pg_background_wait_v2(h_done.pid, h_done.cookie); h_running := pg_background_launch_v2('SELECT pg_sleep(60)', 65536, 'tier-a-purge-keep'); /* Give the running worker a moment to attach */ PERFORM pg_sleep(0.1); purged := pg_background_purge_v2(); IF purged < 1 THEN RAISE EXCEPTION 'Tier A purge_v2: expected at least 1, got %', purged; END IF; /* The still-running worker should remain in the list */ IF (SELECT count(*) FROM pg_background_list WHERE pid = h_running.pid AND cookie = h_running.cookie) <> 1 THEN RAISE EXCEPTION 'Tier A purge_v2: running worker incorrectly purged'; END IF; /* * v2.0 (E8): explicit per-handle cleanup instead of detach_all_v2(), * which CLAUDE.md §7 calls out as a banned cleanup pattern. h_done was * already detached by purge_v2 above; only h_running remains. */ PERFORM pg_background_cancel_v2(h_running.pid, h_running.cookie); PERFORM pg_background_wait_v2(h_running.pid, h_running.cookie); PERFORM pg_background_detach_v2(h_running.pid, h_running.cookie); RAISE NOTICE 'Tier A purge_v2 OK'; END$$; NOTICE: Tier A purge_v2 OK -- ========================================================================= -- v1.10 Tier B (small): full_sql_v2 (B3), application_name (B6) -- ========================================================================= -- B3: pg_background_full_sql_v2 returns the original SQL DO $$ DECLARE h pg_background_handle; full_sql text; BEGIN h := pg_background_launch_v2('SELECT pg_sleep(60), 1 AS phase2_marker', 65536, 'tier-b-fullsql'); full_sql := pg_background_full_sql_v2(h.pid, h.cookie); IF full_sql IS NULL OR full_sql NOT LIKE '%phase2_marker%' THEN RAISE EXCEPTION 'Tier B full_sql_v2: expected SQL containing phase2_marker, got %', full_sql; END IF; PERFORM pg_background_cancel_v2(h.pid, h.cookie); PERFORM pg_background_detach_v2(h.pid, h.cookie); RAISE NOTICE 'Tier B full_sql_v2 OK'; END$$; NOTICE: Tier B full_sql_v2 OK -- B6: application_name should be 'pg_background: