././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7783737 pgbackrest-2.60.0/0000755000175100017510000000000015265102133013370 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/LICENSE0000664000175100017510000000222015265101745014403 0ustar00runnerrunnerThe MIT License (MIT) Portions Copyright (c) 2015-2026, The PostgreSQL Global Development Group Portions Copyright (c) 2013-2026, David Steele Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7783737 pgbackrest-2.60.0/README.md0000644000175100017510000000667115265102133014661 0ustar00runnerrunner# pgBackRest Distribution ## Overview This is a pgBackRest build distribution. Unlike a checkout of the [git repository](https://github.com/pgbackrest/pgbackrest), it ships the generated source and the rendered documentation pre-built, so pgBackRest can be built and installed without the code generation or the documentation tooling. See [https://pgbackrest.org](https://pgbackrest.org) for complete documentation. ## Contents - `src` -- pgBackRest source, including the pre-generated `*.auto.c.inc` files - `doc/man` -- command reference - `doc/html` -- HTML documentation - `test` -- smoke test to verify the build - `meson.build`, `meson_options.txt` -- build configuration - `LICENSE` ## Build pgBackRest builds with meson and ninja. The required libraries are libpq, OpenSSL (>= 1.1.1), libxml2, lz4, bz2, and zlib; zstd, libssh2 (for SFTP), and libsystemd are optional. For example, on Debian/Ubuntu: ``` apt-get install meson gcc libpq-dev libssl-dev libxml2-dev pkg-config \ liblz4-dev libzstd-dev libbz2-dev libz-dev libssh2-1-dev libsystemd-dev ``` Then configure, build, and install: ``` meson setup build . ninja -C build meson install -C build ``` The `pgbackrest` binary is built at `build/src/pgbackrest`. See the [user guide](https://pgbackrest.org/user-guide.html#build) for detailed build instructions, including dependencies for other operating systems. ## Test A smoke test is included to verify that the build works. It finds each PostgreSQL installation on the system and runs the newly-built `pgbackrest` through a backup and restore cycle against every supported version found: create a cluster, configure archiving, create the stanza, check, back up, restore, and verify that the restored data matches. Versions that pgBackRest does not support are reported and skipped. The test requires nothing beyond the build environment and PostgreSQL -- no additional Python packages. Run it with meson after building: ``` meson test -C build --suite smoke ``` Add `-v` to stream the output of each step as it runs: ``` meson test -C build --suite smoke -v ``` PostgreSQL is searched for in the standard packaging paths: - `/usr/lib/postgresql//bin` (Debian/Ubuntu) - `/usr/pgsql-/bin` (RHEL) - `/usr/libexec/postgresql` (Alpine) - `/usr/bin` Each version is tested in a temporary directory with its own repository and a private unix socket, so a running PostgreSQL cluster is not disturbed. The test fails when there is no PostgreSQL to test against, since a test that silently does nothing does not verify the build. Since PostgreSQL will not run as root, running the test as root drops to the `postgres` user and fails if that user does not exist. The test can also be run directly, which allows the defaults to be overridden: ``` python3 test/smoke.py --pgbackrest build/src/pgbackrest ``` Useful options are `--pg-bin-path` to search an additional path for PostgreSQL, `--version` to test only a specific version, and `--user` to select the user to drop to when running as root. Use `--help` for the complete list. ## Documentation The rendered documentation is in `doc`: the command reference in `doc/man` and the HTML documentation in `doc/html`. Two user guides are provided, Debian/Ubuntu and RHEL/Rocky/Alma. If only one user guide is required, replace `user-guide-index.html` with the desired user guide. For example, to keep only the Debian/Ubuntu user guide: ``` mv user-guide.html user-guide-index.html rm user-guide-rhel.html ``` ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7783737 pgbackrest-2.60.0/dist0000644000175100017510000000000715265102133014253 0ustar00runnerrunner2.60.0 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7761402 pgbackrest-2.60.0/doc/0000755000175100017510000000000015265102133014135 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7774975 pgbackrest-2.60.0/doc/html/0000750000175100017510000000000015265102133015074 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7770674 pgbackrest-2.60.0/doc/html/command.html0000644000175100017510000126044015265102133017414 0ustar00runnerrunner pgBackRest - Command Reference
pgBackRest
Command Reference
Home
User Guides
Releases
Configuration
Commands
FAQ
Table of Contents
Introduction
General Options
Annotate Backup
Archive Get
Archive Push
Backup
Check Configuration
Expire Backups
Help
Repository Info
Repository Get
Repository List
Restore Backup
TLS Server
TLS Server Ping
Stanza Create
Stanza Delete
Stanza Upgrade
Stanza Start
Stanza Stop
Repository Verify
Version
Commands are used to execute the various pgBackRest functions. Every option applicable to a command is listed with that command, even if it applies to one or more other commands. An option that can only be set on the command-line is described in General Options below and an option that may also be configured in pgbackrest.conf is described in the Configuration Reference.
Non-boolean options configured in pgbackrest.conf can be reset to default on the command-line by using the reset- prefix. This feature may be used to restore a backup directly on a repository host. Normally, pgBackRest will error because it can see that the database host is remote and restores cannot be done remotely. By adding --reset-pg1-host on the command-line, pgBackRest will ignore the remote database host and restore locally. It may be necessary to pass a new --pg1-path to force the restore to happen in a specific path, i.e. not the path used on the database host.
The no- prefix may be used to set a boolean option to false on the command-line.
Any option may be set in an environment variable using the PGBACKREST_ prefix and the option name in all caps replacing - with _, e.g. pg1-path becomes PGBACKREST_PG1_PATH and stanza becomes PGBACKREST_STANZA. Boolean options are represented as they would be in a configuration file, e.g. PGBACKREST_COMPRESS="n", and reset-* variants are not allowed. Options that can be specified multiple times on the command-line or in a config file can be represented by separating the values with colons, e.g. PGBACKREST_DB_INCLUDE="db1:db2".
Command-line options override environment options which override config file options.
See Configuration Introduction for information on option types
General Options
Config (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
Dry Run (--dry-run)
Execute a dry-run for the command.
The --dry-run option is a command-line only option and can be passed when it is desirable to determine what modifications will be made by the command without the command actually making any modifications. 
default: n
example: --dry-run
Raw Data (--raw)
Do not transform data.
Do not transform (i.e, encrypt, decompress, etc.) data for the current command. 
default: n
example: --raw
Set Repository (--repo)
Set repository.
Set the repository for a command to operate on.
For example, this option may be used to perform a restore from a specific repository, rather than letting pgBackRest choose. 
allowed: [1, 256]
example: --repo=1
Stanza (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Annotate Backup (annotate)
Annotations included with the backup command can be added, modified, or removed afterwards using the annotate command.
Command Options
Backup Annotation (--annotation)
Annotate backup with user-defined key/value pairs.
Users can attach informative key/value pairs to the backup. This option may be used multiple times to attach multiple annotations.
Annotations are output by the info command text output when a backup is specified with --set and always appear in the JSON output. 
example: --annotation=source="Sunday backup for website database"
Set (--set)
Backup set to annotate.
The backup set to annotate. 
example: --set=20150131-153358F_20150131-153401I
General Options
Log Options
Repository Options
Archive Get (archive-get)
This command is used by PostgreSQL to restore a backup, perform PITR, or as an alternative to streaming for keeping a replica up to date. WAL segments are required for PostgreSQL recovery or to maintain a replica.
When multiple repositories are configured, WAL will be fetched from the repositories in priority order (e.g. repo1, repo2, etc.). In general it is better if faster/cheaper storage has higher priority. If a repository is specified with the --repo option then only that repository will be searched.
The archive-get command is configured and generated by pgBackRest during a restore for use by PostgreSQL. See Point-in-Time Recovery for an example.
Command Options
Asynchronous Archiving (--archive-async)
Push/get WAL segments asynchronously.
Enables asynchronous operation for the archive-push and archive-get commands.
Asynchronous operation is more efficient because it can reuse connections and take advantage of parallelism. See the spool-path, archive-get-queue-max, and archive-push-queue-max options for more information. 
default: n
example: --archive-async
Maximum Archive Get Queue Size (--archive-get-queue-max)
Maximum size of the pgBackRest archive-get queue.
Specifies the maximum size of the archive-get queue when archive-async is enabled. The queue is stored in the spool-path and is used to speed providing WAL to PostgreSQL. 
default: 128MiB
allowed: [0B, 4PiB]
example: --archive-get-queue-max=1GiB
Retry Missing WAL Segment (--archive-missing-retry)
Retry missing WAL segment
Retry a WAL segment that was previously reported as missing by the archive-get command when in asynchronous mode. This prevents notifications in the spool path from a prior restore from being used and possibly causing a recovery failure if consistency has not been reached.
Disabling this option allows PostgreSQL to more reliably recognize when the end of the WAL in the archive has been reached, which permits it to switch over to streaming from the primary. With retries enabled, a steady stream of WAL being archived will cause PostgreSQL to continue getting WAL from the archive rather than switch to streaming.
When disabling this option it is important to ensure that the spool path for the stanza is empty. The restore command does this automatically if the spool path is configured at restore time. Otherwise, it is up to the user to ensure the spool path is empty. 
default: y
example: --no-archive-missing-retry
Archive Timeout (--archive-timeout)
Archive timeout.
Set maximum time, in seconds, to wait for each WAL segment to reach the pgBackRest archive repository. The timeout applies to the check and backup commands when waiting for WAL segments required for backup consistency to be archived. 
default: 1m
allowed: [100ms, 1d]
example: --archive-timeout=30
General Options
Log Options
Maintainer Options
Repository Options
Stanza Options
Archive Push (archive-push)
Accepts a WAL segment from PostgreSQL and archives it in each repository defined by the indexed repo-path option (see the Repository section for information on configuring repositories). The WAL segment may be pushed immediately to the archive or stored locally depending on the value of archive-async. With multiple repositories configured, archive-push will attempt to push to as many repositories as possible.
The archive-push is intended to be configured and called by PostgreSQL. See Configure Archiving for an example.
Command Options
Asynchronous Archiving (--archive-async)
Push/get WAL segments asynchronously.
Enables asynchronous operation for the archive-push and archive-get commands.
Asynchronous operation is more efficient because it can reuse connections and take advantage of parallelism. See the spool-path, archive-get-queue-max, and archive-push-queue-max options for more information. 
default: n
example: --archive-async
Check Archive (--archive-check)
Check that WAL segments are in the archive before backup completes.
Checks that all WAL segments required to make the backup consistent are present in the WAL archive. It's a good idea to leave this as the default unless you are using another method for archiving.
This option must be enabled if archive-copy is enabled. 
default: y
example: --no-archive-check
Check Archive Mode (--archive-mode-check)
Check the PostgreSQL archive_mode setting.
Enabled by default, this option disallows PostgreSQL archive_mode=always.
WAL segments pushed from a standby server might be logically the same as WAL segments pushed from the primary but have different checksums. Disabling archiving from multiple sources is recommended to avoid conflicts.
CAUTION:
If this option is disabled then it is critical to ensure that only one archiver is writing to the repository via the archive-push command. 
default: y
example: --no-archive-mode-check
Archive Push Batch Size (--archive-push-batch-size)
Maximum amount of WAL to push per asynchronous run.
In asynchronous mode the archive-push process pushes all the WAL segments that are ready in a single run. Since archive-push-queue-max is only checked at the start of each run, a run that processes a very large number of segments can let the queue grow well beyond the limit before it is rechecked.
This option limits the amount of WAL processed per run so the process exits and is spawned again by the next archive-push, which rechecks the queue. Lower values recheck the queue more often at the cost of spawning the asynchronous process more frequently. The value is rounded down to a whole number of WAL segments but at least one segment is always processed. 
default: 16GiB
allowed: [1MiB, 4PiB]
example: --archive-push-batch-size=1GiB
Maximum Archive Push Queue Size (--archive-push-queue-max)
Maximum size of the PostgreSQL archive queue.
After the limit is reached, the following will happen:
  • pgBackRest will notify PostgreSQL that the WAL was successfully archived, then DROP IT.
  • A warning will be output to the PostgreSQL log.
If this occurs then the archive log stream will be interrupted and PITR will not be possible past that point. A new backup will be required to regain full restore capability.
In asynchronous mode the entire queue will be dropped to prevent spurts of WAL getting through before the queue limit is exceeded again.
In asynchronous mode this limit is only checked at the start of each archive-push run, so the queue can grow beyond it within a single run. Reduce archive-push-batch-size to check the queue more frequently.
The purpose of this feature is to prevent the log volume from filling up at which point PostgreSQL will stop completely. Better to lose the backup than have PostgreSQL go down. 
allowed: [0B, 4PiB]
example: --archive-push-queue-max=1TiB
Deprecated Name: archive-queue-max
Archive Timeout (--archive-timeout)
Archive timeout.
Set maximum time, in seconds, to wait for each WAL segment to reach the pgBackRest archive repository. The timeout applies to the check and backup commands when waiting for WAL segments required for backup consistency to be archived. 
default: 1m
allowed: [100ms, 1d]
example: --archive-timeout=30
General Options
Log Options
Maintainer Options
Repository Options
Stanza Options
Backup (backup)
When multiple repositories are configured, pgBackRest will backup to the highest priority repository (e.g. repo1) unless the --repo option is specified.
pgBackRest does not have a built-in scheduler so it's best to run it from cron or some other scheduling mechanism.
See Perform a Backup for more details and examples.
Command Options
Backup Annotation (--annotation)
Annotate backup with user-defined key/value pairs.
Users can attach informative key/value pairs to the backup. This option may be used multiple times to attach multiple annotations.
Annotations are output by the info command text output when a backup is specified with --set and always appear in the JSON output. 
example: --annotation=source="Sunday backup for website database"
Check Archive (--archive-check)
Check that WAL segments are in the archive before backup completes.
Checks that all WAL segments required to make the backup consistent are present in the WAL archive. It's a good idea to leave this as the default unless you are using another method for archiving.
This option must be enabled if archive-copy is enabled. 
default: y
example: --no-archive-check
Copy Archive (--archive-copy)
Copy WAL segments needed for consistency to the backup.
This slightly paranoid option protects against corruption in the WAL segment archive by storing the WAL segments required for consistency directly in the backup. WAL segments are still stored in the archive so this option will use additional space.
It is best if the archive-push and backup commands have the same compress-type (e.g. lz4) when using this option. Otherwise, the WAL segments will need to be recompressed with the compress-type used by the backup, which can be fairly expensive depending on how much WAL was generated during the backup.
On restore, the WAL segments will be present in pg_xlog/pg_wal and PostgreSQL will use them in preference to calling the restore_command.
The archive-check option must be enabled if archive-copy is enabled. 
default: n
example: --archive-copy
Check Archive Mode (--archive-mode-check)
Check the PostgreSQL archive_mode setting.
Enabled by default, this option disallows PostgreSQL archive_mode=always.
WAL segments pushed from a standby server might be logically the same as WAL segments pushed from the primary but have different checksums. Disabling archiving from multiple sources is recommended to avoid conflicts.
CAUTION:
If this option is disabled then it is critical to ensure that only one archiver is writing to the repository via the archive-push command. 
default: y
example: --no-archive-mode-check
Archive Timeout (--archive-timeout)
Archive timeout.
Set maximum time, in seconds, to wait for each WAL segment to reach the pgBackRest archive repository. The timeout applies to the check and backup commands when waiting for WAL segments required for backup consistency to be archived. 
default: 1m
allowed: [100ms, 1d]
example: --archive-timeout=30
Backup from Standby (--backup-standby)
Backup from the standby cluster.
Enable backup from standby to reduce load on the primary cluster. This option requires that both the primary and standby hosts be configured.
The following modes are supported:
  • y - Standby is required for backup.
  • prefer - Backup from standby if available otherwise backup from primary.
  • n - Backup from primary only. 
default: n
example: --backup-standby=y
Page Checksums (--checksum-page)
Validate data page checksums.
Directs pgBackRest to validate all data page checksums while backing up a cluster. This option is automatically enabled when data page checksums are enabled on the cluster.
Failures in checksum validation will not abort a backup. Rather, warnings will be emitted in the log (and to the console with default settings) and the list of invalid pages will be stored in the backup manifest. 
example: --no-checksum-page
Error on Invalid Page Checksums (--checksum-page-error)
Error on invalid page checksums.
When enabled with the checksum-page option, a failure in page checksum validation will terminate the backup. 
default: n
example: --no-checksum-page-error
Path/File Exclusions (--exclude)
Exclude paths/files from the backup.
All exclusions are relative to $PGDATA. If the exclusion ends with / then only files in the specified directory will be excluded, e.g. --exclude=junk/ will exclude all files in the $PGDATA/junk directory but include the directory itself. If the exclusion does not end with / then the file may match the exclusion exactly or match with / appended to the exclusion, e.g. --exclude=junk will exclude the $PGDATA/junk directory and all the files it contains.
Be careful using this feature -- it is very easy to exclude something critical that will make the backup inconsistent. Be sure to test your restores!
All excluded files will be logged at info level along with the exclusion rule. Be sure to audit the list of excluded files to ensure nothing unexpected is being excluded.
NOTE:
Exclusions are not honored on delta restores. Any files/directories that were excluded by the backup will be removed on delta restore.
This option should not be used to exclude PostgreSQL logs from a backup. Logs can be moved out of the PGDATA directory using the PostgreSQL log_directory setting, which has the benefit of allowing logs to be preserved after a restore.
Multiple exclusions may be specified on the command-line or in a configuration file. 
example: --exclude=junk/
Expire Auto (--expire-auto)
Automatically run the expire command after a successful backup.
The setting is enabled by default. Use caution when disabling this option as doing so will result in retaining all backups and archives indefinitely, which could cause your repository to run out of space. The expire command will need to be run regularly to prevent this from happening.
When expire is run automatically after a successful backup it uses the configuration of the backup command, so options set only in an expire command section (e.g. [global:expire]) are not applied. To apply expire-specific configuration, disable this option and run the expire command separately. 
default: y
example: --expire-auto
Force (--force)
Force an offline backup.
When used with --no-start-stop a backup will be run even if pgBackRest thinks that PostgreSQL is running. This option should be used with extreme care as it will likely result in a bad backup.
There are some scenarios where a backup might still be desirable under these conditions. For example, if a server crashes and the database cluster volume can only be mounted read-only, it would be a good idea to take a backup even if postmaster.pid is present. In this case it would be better to revert to the prior backup and replay WAL, but possibly there is a very important transaction in a WAL segment that did not get archived. 
default: n
example: --force
Manifest Save Threshold (--manifest-save-threshold)
Manifest save threshold during backup.
Defines how often the manifest will be saved during a backup. Saving the manifest is important because it stores the checksums and allows the resume function to work efficiently. The actual threshold used is 1% of the backup size or manifest-save-threshold, whichever is greater. 
default: 1GiB
allowed: [1B, 1TiB]
example: --manifest-save-threshold=8GiB
Online (--online)
Perform an online backup.
Specifying --no-online prevents pgBackRest from running the backup start/stop functions on the database cluster. In order for this to work PostgreSQL should be shut down and pgBackRest will generate an error if it is not.
The purpose of this option is to allow offline backups. The pg_xlog/pg_wal directory is copied as-is and archive-check is automatically disabled for the backup. 
default: y
example: --no-online
Resume (--resume)
Allow resume of failed backup.
Defines whether the resume feature is enabled. Resume can greatly reduce the amount of time required to run a backup after a previous backup of the same type has failed. It adds complexity, however, so it may be desirable to disable in environments that do not require the feature. 
default: y
example: --no-resume
Start Fast (--start-fast)
Force a checkpoint to start backup quickly.
Forces a checkpoint (by passing y to the fast parameter of the backup start function) so the backup begins immediately. Otherwise the backup will start after the next regular checkpoint. 
default: n
example: --start-fast
Type (--type)
Backup type.
The following backup types are supported:
  • full - all database cluster files will be copied and there will be no dependencies on previous backups.
  • incr - incremental from the last successful backup.
  • diff - like an incremental backup but always based on the last full backup. 
default: incr
example: --type=full
General Options
Log Options
Maintainer Options
Repository Options
Stanza Options
Check Configuration (check)
The check command validates that pgBackRest and the archive_command setting are configured correctly for archiving and backups for the specified stanza. It will attempt to check all repositories and databases that are configured for the host on which the command is run. It detects misconfigurations, particularly in archiving, that result in incomplete backups because required WAL segments did not reach the archive. The command can be run on the PostgreSQL or repository host. The command may also be run on the standby host, however, since pg_switch_xlog()/pg_switch_wal() cannot be performed on the standby, the command will only test the repository configuration.
Note that pg_create_restore_point('pgBackRest Archive Check') and pg_switch_xlog()/pg_switch_wal() are called to force PostgreSQL to archive a WAL segment.
Command Options
Check Archive (--archive-check)
Check that WAL segments are in the archive before backup completes.
Checks that all WAL segments required to make the backup consistent are present in the WAL archive. It's a good idea to leave this as the default unless you are using another method for archiving.
This option must be enabled if archive-copy is enabled. 
default: y
example: --no-archive-check
Check Archive Mode (--archive-mode-check)
Check the PostgreSQL archive_mode setting.
Enabled by default, this option disallows PostgreSQL archive_mode=always.
WAL segments pushed from a standby server might be logically the same as WAL segments pushed from the primary but have different checksums. Disabling archiving from multiple sources is recommended to avoid conflicts.
CAUTION:
If this option is disabled then it is critical to ensure that only one archiver is writing to the repository via the archive-push command. 
default: y
example: --no-archive-mode-check
Archive Timeout (--archive-timeout)
Archive timeout.
Set maximum time, in seconds, to wait for each WAL segment to reach the pgBackRest archive repository. The timeout applies to the check and backup commands when waiting for WAL segments required for backup consistency to be archived. 
default: 1m
allowed: [100ms, 1d]
example: --archive-timeout=30
Backup from Standby (--backup-standby)
Backup from the standby cluster.
Enable backup from standby to reduce load on the primary cluster. This option requires that both the primary and standby hosts be configured.
The following modes are supported:
  • y - Standby is required for backup.
  • prefer - Backup from standby if available otherwise backup from primary.
  • n - Backup from primary only. 
default: n
example: --backup-standby=y
General Options
Log Options
Maintainer Options
Repository Options
Stanza Options
Expire Backups (expire)
pgBackRest does full backup rotation based on the retention type which can be a count or a time period. When a count is specified, then expiration is not concerned with when the backups were created but with how many must be retained. Differential backups are count-based but will always be expired when the full backup they depend on is expired. Incremental backups are not expired by retention independently — they are always expired with their related full or differential backup. See sections Full Backup Retention and Differential Backup Retention for details and examples.
Archived WAL is retained by default for backups that have not expired, however, although not recommended, this schedule can be modified per repository with the retention-archive options. See section Archive Retention for details and examples.
The expire command is run automatically after each successful backup and can also be run by the user. When run by the user, expiration will occur as defined by the retention settings for each configured repository. If the --repo option is provided, expiration will occur only on the specified repository. Expiration can also be limited by the user to a specific backup set with the --set option and, unless the --repo option is specified, all repositories will be searched and any matching the set criteria will be expired. It should be noted that the archive retention schedule will be checked and performed any time the expire command is run.
Command Options
Expire Archive Before (--archive-expire-before)
Remove WAL archive earlier than the specified WAL segment.
Remove WAL segments earlier than the specified segment, but only those that are not required by a retained backup, that is, segments older than the earliest backup set to retain, or any segment when the repository contains no backup. The specified segment and everything after it are kept. This mirrors the value passed by PostgreSQL to archive_cleanup_command as %r.
This option is primarily intended for archive-only repositories where no backups are stored. It can also be used to reclaim WAL archive space before the first backup when full retention has not yet been met to trigger archive expiration. 
example: --archive-expire-before=000000010000000000000010
Error on Invalid Page Checksums (--checksum-page-error)
Error on invalid page checksums.
If enabled, the expire command will terminate with an error before any changes if the oldest retained backup would contain checksum errors. If this option is disabled, the expire will proceed and issue a warning if there are checksum errors in the oldest retained backup.
If the --set is used, the expire will proceed even if this option is enabled. It will emit a warning if the oldest retained backup contains checksum errors. 
default: y
example: --no-checksum-page-error
Oldest (--oldest)
Expire the oldest eligible backup set.
Expire the oldest full backup set that can be removed (meaning at least one newer full backup remains). This is equivalent to manually decrementing retention by one, but computed automatically. All backups related to the expired full backup set (differential and incremental) are also expired.
When used, archive retention is also temporarily adjusted so WAL for the expired backups can be removed in the same run.
If time-based full retention is configured (using --repo-retention-full-type=time) then --oldest uses count-based expiration for this execution.
WARNING:
This option cannot be combined with --set. 
default: n
example: --oldest
Set (--set)
Backup set to expire.
The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
WARNING:
Use this option with extreme caution — it will permanently remove all backups and archives not required to make a backup consistent from the pgBackRest repository for the specified backup set. This process may negate the ability to perform PITR. If --repo-retention-full and/or --repo-retention-archive options are configured, then it is recommended that you override these options by setting their values to the maximum while performing ad hoc expiration in order to prevent an unintended expiration of archives. 
example: --set=20150131-153358F_20150131-153401I
General Options
Log Options
Repository Options
Help (help)
Three levels of help are provided. If no command is specified then general help will be displayed. If a command is specified (e.g. pgbackrest help backup) then a full description of the command will be displayed along with a list of valid options. If an option is specified in addition to a command (e.g. pgbackrest help backup type) then a full description of the option as it applies to the command will be displayed.
Command Options
Display Help (--help)
Display help.
Displays help even if the help command is not specified and overrides the --version option. 
default: n
example: --help
Display Version (--version)
Display version.
Displays version even if the version or help command is not specified. 
default: n
example: --version
Repository Info (info)
The info command operates on a single stanza or all stanzas. Text output is the default and gives a human-readable summary of backups for the stanza(s) requested. This format is subject to change with any release.
For machine-readable output use --output=json. The JSON output contains far more information than the text output and is kept stable unless a bug is found.
To speed up execution, limit the output to only progress information by specifying --detail-level=progress. Note that this skips all checks except for availability of the stanza.
Each stanza has a separate section and it is possible to limit output to a single stanza with the --stanza option. The stanza 'status' gives a brief indication of the stanza's health. If this is 'ok' then pgBackRest is functioning normally. If there are multiple repositories, then a status of 'mixed' indicates that the stanza is not in a healthy state on one or more of the repositories; in this case the state of the stanza will be detailed per repository. For cases in which an error on a repository occurred that is not one of the known error codes, then an error code of 'other' will be used and the full error details will be provided. The 'wal archive min/max' shows the minimum and maximum WAL currently stored in the archive and, in the case of multiple repositories, will be reported across all repositories unless the --repo option is set. Note that there may be gaps due to archive retention policies or other reasons.
The 'backup/expire running' and/or 'restore running' messages will appear beside the 'status' information if any of those commands are currently running on the host. Per-repo progress will be also reported in text output and a 'repo' array will be included in JSON output.
The backups are displayed oldest to newest. The oldest backup will always be a full backup (indicated by an F at the end of the label) but the newest backup can be full, differential (ends with D), or incremental (ends with I).
The 'timestamp start/stop' defines the time period when the backup ran. The 'timestamp stop' can be used to determine the backup to use when performing Point-In-Time Recovery. More information about Point-In-Time Recovery can be found in the Point-In-Time Recovery section.
The 'wal start/stop' defines the WAL range that is required to make the database consistent when restoring. The backup command will ensure that this WAL range is in the archive before completing.
The 'database size' is the full uncompressed size of the database while 'database backup size' is the amount of data in the database to actually back up (these will be the same for full backups).
The 'repo' indicates in which repository this backup resides. The 'backup set size' includes all the files from this backup and any referenced backups in the repository that are required to restore the database from this backup while 'backup size' includes only the files in this backup (these will also be the same for full backups). Repository sizes reflect compressed file sizes if compression is enabled in pgBackRest.
The 'backup reference total' summarizes the list of additional backups that are required to restore this backup. Use the --set option to display the complete reference list.
Command Options
Detail level (--detail-level)
Output detail level.
The following levels are supported:
  • progress - Output only the current backup/expire progress. This level cannot be used with the --set option.
  • full - Output full info. 
default: full
example: --detail-level=progress
Output (--output)
Output format.
The following output types are supported:
  • text - Human-readable summary of backup information.
  • json - Exhaustive machine-readable backup information in JSON format. 
default: text
example: --output=json
Set (--set)
Backup set to detail.
Details include a complete list of additional backups that are required to restore this backup, a list of databases (with OIDs) in the backup set (excluding template databases), tablespaces (with OIDs) with the destination where they will be restored by default, and symlinks with the destination where they will be restored when --link-all is specified. 
example: --set=20150131-153358F_20150131-153401I
Type (--type)
Filter on backup type.
The following backup types are supported:
  • full - Output only full backups.
  • diff - Output only differential backups.
  • incr - Output only incremental backups. 
example: --type=full
General Options
Log Options
Repository Options
Repository Get (repo-get)
Similar to the unix cat command but works on any supported repository type. This command requires a fully qualified file name and is primarily for administration, investigation, and testing. It is not a required part of a normal pgBackRest setup.
If the repository is encrypted then repo-get will automatically decrypt the file. Files are not automatically decompressed but the output can be piped through the appropriate decompression command, e.g. gzip -d.
If more than one repository is configured, the command will default to the highest priority repository (e.g. repo1) unless the --repo option is specified.
Command Options
Ignore Missing (--ignore-missing)
Ignore missing source file.
Exit with 1 if the source file is missing but don't throw an error. 
default: n
example: --ignore-missing
General Options
Log Options
Repository Options
Repository List (repo-ls)
Similar to the unix ls command but works on any supported repository type. This command accepts a path, absolute or relative to the repository path defined by the --repo-path option, and is primarily for administration, investigation, and testing. It is not a required part of a normal pgBackRest setup.
The default text output prints one file name per line. JSON output is available by specifying --output=json.
If more than one repository is configured, the command will default to the highest priority repository (e.g. repo1) unless the --repo option is specified.
Command Options
Filter Output (--filter)
Filter output with a regular expression.
The filter is applied against the file/path names before they are output. 
example: --filter="(F|D|I)$"
Output (--output)
Output format.
The following output types are supported:
  • text - Simple list with one file/link/path name on each line.
  • json - Detailed file/link/path information in JSON format.
In JSON format the available fields are:
  • name - file/link/path name (and partial path when recursing).
  • type - file, path, or link.
  • size - size in bytes (files only).
  • time - time last modified (files only).
  • destination - link destination (links only). 
default: text
example: --output=json
Recurse Subpaths (--recurse)
Include all subpaths in output.
All subpaths and their files will be included in the output. 
default: n
example: --recurse
Sort Output (--sort)
Sort output ascending, descending, or none.
The following sort types are supported:
  • asc - sort ascending.
  • desc - sort descending.
  • none - no sorting. 
default: asc
example: --sort=desc
General Options
Log Options
Repository Options
Restore Backup (restore)
The restore command automatically defaults to selecting the latest backup from the first repository where backups exist (see Quick Start - Restore a Backup). The order in which the repositories are checked is dictated by the pgbackrest.conf (e.g. repo1 will be checked before repo2). To select from a specific repository, the --repo option can be passed (e.g. --repo=1). The --set option can be passed if a backup other than the latest is desired.
When PITR of --type=time or --type=lsn is specified, then the target time or target lsn must be specified with the --target option. If a backup is not specified via the --set option, then the configured repositories will be checked, in order, for a backup that contains the requested time or lsn. If no matching backup is found, the latest backup from the first repository containing backups will be used for --type=time while no backup will be selected for --type=lsn. For other types of PITR, e.g. xid, the --set option must be provided if the target is prior to the latest backup. See Point-in-Time Recovery for more details and examples.
Replication slots are not included per recommendation of PostgreSQL. See Backing Up The Data Directory in the PostgreSQL documentation for more information.
Command Options
Archive Mode (--archive-mode)
Preserve or disable archiving on restored cluster.
This option allows archiving to be preserved or disabled on a restored cluster. This is useful when the cluster must be promoted to do some work but is not intended to become the new primary. In this case it is not a good idea to push WAL from the cluster into the repository.
The following modes are supported:
  • off - disable archiving by setting archive_mode=off.
  • preserve - preserve current archive_mode setting.
NOTE: This option is not available on PostgreSQL < 12. 
default: preserve
example: --archive-mode=off
Error on Invalid Page Checksums (--checksum-page-error)
Error on invalid page checksums.
If enabled, the restore command will terminate with an error if it encounters a file containing checksum errors.
A warning will be emitted for each file with checksum errors if the setting is disabled. 
default: y
example: --no-checksum-page-error
Exclude Database (--db-exclude)
Restore excluding the specified databases.
Databases excluded will be restored as sparse, zeroed files to save space but still allow PostgreSQL to perform recovery. After recovery, those databases will not be accessible but can be removed with the drop database command. The --db-exclude option can be passed multiple times to specify more than one database to exclude.
When used in combination with the --db-include option, --db-exclude will only apply to standard system databases (template0, template1, and postgres). 
example: --db-exclude=db_main
Include Database (--db-include)
Restore only specified databases.
This feature allows only selected databases to be restored. Databases not specifically included will be restored as sparse, zeroed files to save space but still allow PostgreSQL to perform recovery. After recovery, the databases that were not included will not be accessible but can be removed with the drop database command.
NOTE:
built-in databases (template0, template1, and postgres) are always restored unless specifically excluded.
The --db-include option can be passed multiple times to specify more than one database to include.
See Restore Selected Databases for additional information and caveats. 
example: --db-include=db_main
Force (--force)
Force a restore.
By itself this option forces the PostgreSQL data and tablespace paths to be completely overwritten. In combination with --delta a timestamp/size delta will be performed instead of using checksums. 
default: n
example: --force
Link All (--link-all)
Restore all symlinks.
By default symlinked directories and files are restored as normal directories and files in $PGDATA. This is because it may not be safe to restore symlinks to their original destinations on a system other than where the original backup was performed. This option restores all the symlinks just as they were on the original system where the backup was performed. 
default: n
example: --link-all
Link Map (--link-map)
Modify the destination of a symlink.
Allows the destination file or path of a symlink to be changed on restore. This is useful for restoring to systems that have a different storage layout than the original system where the backup was generated. 
example: --link-map=pg_xlog=/data/xlog
Recovery Option (--recovery-option)
Set an option in postgresql.auto.conf or recovery.conf.
See Server Configuration for details on postgresql.auto.conf or recovery.conf options (be sure to select your PostgreSQL version). This option can be used multiple times.
For PostgreSQL >= 12, options will be written into postgresql.auto.conf. For all other versions, options will be written into recovery.conf.
NOTE:
The restore_command option will be automatically generated but can be overridden with this option. Be careful about specifying your own restore_command as pgBackRest is designed to handle this for you. Target Recovery options (recovery_target_name, recovery_target_time, etc.) are generated automatically by pgBackRest and should not be set with this option.
Since pgBackRest does not start PostgreSQL after writing the postgresql.auto.conf or recovery.conf file, it is always possible to edit/check postgresql.auto.conf or recovery.conf before manually restarting. 
example: --recovery-option=primary_conninfo=db.mydomain.com
Set (--set)
Backup set to restore.
The backup set to be restored. latest will restore the latest backup, otherwise provide the name of the backup to restore. 
default: latest
example: --set=20150131-153358F_20150131-153401I
Tablespace Map (--tablespace-map)
Restore a tablespace into the specified directory.
Moves a tablespace to a new location during the restore. This is useful when tablespace locations are not the same on a replica, or an upgraded system has different mount points.
Tablespace locations are not stored in pg_tablespace so moving tablespaces can be done with impunity. However, moving a tablespace to the data_directory is not recommended and may cause problems. For more information on moving tablespaces http://www.databasesoup.com/2013/11/moving-tablespaces.html is a good resource. 
example: --tablespace-map=ts_01=/db/ts_01
Map All Tablespaces (--tablespace-map-all)
Restore all tablespaces into the specified directory.
Tablespaces are restored into their original locations by default. This behavior can be modified for each tablespace with the tablespace-map option, but it is sometimes preferable to remap all tablespaces to a new directory all at once. This is particularly useful for development or staging systems that may not have the same storage layout as the original system where the backup was generated.
The path specified will be the parent path used to create all the tablespaces in the backup.
CAUTION:
Tablespaces created after the backup started will not be mapped. Make a new backup after a tablespace is created if tablespace mapping is required. 
example: --tablespace-map-all=/data/tablespace
Target (--target)
Recovery target.
Defines the recovery target when --type is lsn, name, xid, or time. If the target is prior to the latest backup and --type is not time or lsn, then use the --set option to specify the backup set. 
example: --target=2015-01-30 14:15:11 EST
Target Action (--target-action)
Action to take when recovery target is reached.
When hot_standby=on, the default since PostgreSQL 10, this option consistently controls what the cluster does when the target is reached or there is no more WAL in the archive.
When hot_standby=off in PostgreSQL >= 12, pause acts like shutdown. When hot_standby=off in PostgreSQL < 12, pause acts like promote.
The following actions are supported:
  • pause - pause when recovery target is reached.
  • promote - promote and switch timeline when recovery target is reached.
  • shutdown - shutdown server when recovery target is reached. (PostgreSQL >= 9.5) 
default: pause
example: --target-action=promote
Target Exclusive (--target-exclusive)
Stop just before the recovery target is reached.
Defines whether recovery to the target would be exclusive (the default is inclusive) and is only valid when --type is lsn, time or xid. For example, using --target-exclusive would exclude the contents of transaction 1007 when --type=xid and --target=1007. See the recovery_target_inclusive option in the PostgreSQL docs for more information. 
default: n
example: --no-target-exclusive
Target Timeline (--target-timeline)
Recover along a timeline.
See recovery_target_timeline in the PostgreSQL docs for more information. 
example: --target-timeline=3
Type (--type)
Recovery type.
The following recovery types are supported:
  • default - recover to the end of the archive stream.
  • immediate - recover only until the database becomes consistent.
  • lsn - recover to the LSN (Log Sequence Number) specified in --target. This option is only supported on PostgreSQL >= 10.
  • name - recover the restore point specified in --target.
  • xid - recover to the transaction id specified in --target.
  • time - recover to the time specified in --target.
  • preserve - preserve the existing postgresql.auto.conf or recovery.conf file.
  • standby - add standby_mode=on to the postgresql.auto.conf or recovery.conf file so cluster will start in standby mode.
  • none - no postgresql.auto.conf or recovery.conf file is written so PostgreSQL will attempt to achieve consistency using WAL segments present in pg_xlog/pg_wal. Provide the required WAL segments or use the archive-copy setting to include them with the backup.
WARNING:
Recovery type=none should be avoided because the timeline will not be incremented at the end of recovery. This can lead to, for example, PostgreSQL attempting to archive duplicate WAL, which will be rejected, and may cause the disk to fill up and result in a PostgreSQL panic. In addition, tools like pg_rewind may not work correctly or may cause corruption.
Note that the default restore type for offline backups is none since Point-in-Time-Recovery is not possible if wal_level=minimal. If type is set explicitly then it will be honored since Point-in-Time-Recovery is possible from offline backups as long as wal_level > minimal. 
default: default
example: --type=xid
General Options
Log Options
Maintainer Options
Repository Options
Stanza Options
TLS Server (server)
The pgBackRest server allows access to remote hosts without using the SSH protocol.
Command Options
TLS Server Address (--tls-server-address)
TLS server address.
IP address the server will listen on for client requests. 
default: localhost
example: --tls-server-address=*
TLS Server Authorized Clients (--tls-server-auth)
TLS server authorized clients.
Clients are authorized on the server by verifying their certificate and checking their certificate CN (Common Name) against a list on the server configured with the tls-server-auth option.
A client CN can be authorized for as many stanzas as needed by providing a comma-separated list to the tls-server-auth option or for all stanzas by specifying tls-server-auth=client-cn=*. Wildcards may not be specified for the client CN. 
example: --tls-server-auth=client-cn=stanza1,stanza2
TLS Server Certificate Authorities (--tls-server-ca-file)
TLS server certificate authorities.
Checks that client certificates are signed by a trusted certificate authority. 
example: --tls-server-ca-file=/path/to/server.ca
TLS Server Certificate (--tls-server-cert-file)
TLS server certificate file.
Sent to the client to show the server identity. 
example: --tls-server-cert-file=/path/to/server.crt
TLS Server Key (--tls-server-key-file)
TLS server key file.
Proves server certificate was sent by the owner. 
example: --tls-server-key-file=/path/to/server.key
TLS Server Port (--tls-server-port)
TLS server port.
Port the server will listen on for client requests. 
default: 8432
allowed: [1, 65535]
example: --tls-server-port=8000
General Options
Log Options
TLS Server Ping (server-ping)
Ping a pgBackRest TLS server to ensure it is accepting connections. This serves as an aliveness check only since no authentication is attempted.
If no host is specified on the command-line then the tls-server-host option will be used.
Command Options
TLS Server Address (--tls-server-address)
TLS server address.
IP address the server will listen on for client requests. 
default: localhost
example: --tls-server-address=*
TLS Server Port (--tls-server-port)
TLS server port.
Port the server will listen on for client requests. 
default: 8432
allowed: [1, 65535]
example: --tls-server-port=8000
General Options
Log Options
Stanza Create (stanza-create)
The stanza-create command must be run after the stanza has been configured in pgbackrest.conf. If there is more than one repository configured, the stanza will be created on each. Stanzas that have already been created will be skipped so it is always safe to run stanza-create, even when a new repository has been configured.
See Create the Stanza for more information and an example.
Command Options
Online (--online)
Create on an online cluster.
Specifying --no-online prevents pgBackRest from connecting to PostgreSQL when creating the stanza. 
default: y
example: --no-online
General Options
Log Options
Maintainer Options
Repository Options
Stanza Options
Stanza Delete (stanza-delete)
The stanza-delete command removes data in the repository associated with a stanza.
WARNING:
Use this command with caution — it will permanently remove all backups and archives from the pgBackRest repository for the specified stanza.
To delete a stanza:
  • Shut down the PostgreSQL cluster associated with the stanza (or use --force to override).
  • Run the stop command on the host where the stanza-delete command will be run.
  • Run the stanza-delete command.
Once the command successfully completes, it is the responsibility of the user to remove the stanza from all pgBackRest configuration files and/or environment variables.
A stanza may only be deleted from one repository at a time. To delete the stanza from multiple repositories, repeat the stanza-delete command for each repository while specifying the --repo option.
Command Options
Force (--force)
Force stanza delete.
If PostgreSQL is still running for the stanza, then this option can be used to force the stanza to be deleted from the repository. 
default: n
example: --no-force
General Options
Log Options
Maintainer Options
Repository Options
Stanza Options
Stanza Upgrade (stanza-upgrade)
Immediately after upgrading PostgreSQL to a newer major version, the pg-path for all pgBackRest configurations must be set to the new database location and the stanza-upgrade command run. If there is more than one repository configured on the host, the stanza will be upgraded on each. If the database is offline use the --no-online option.
Command Options
Online (--online)
Update an online cluster.
Specifying --no-online prevents pgBackRest from connecting to PostgreSQL when upgrading the stanza. 
default: y
example: --no-online
General Options
Log Options
Maintainer Options
Repository Options
Stanza Options
Stanza Start (start)
If the pgBackRest processes were previously stopped using the stop command then they can be started again using the start command. Note that this will not immediately start up any pgBackRest processes but they are allowed to run. See Starting and Stopping for more information and examples.
General Options
Log Options
Stanza Stop (stop)
Does not allow any new pgBackRest processes to run. By default running processes will be allowed to complete successfully. Use the --force option to terminate running processes.
pgBackRest processes will return an error if they are run after the stop command completes. See Starting and Stopping for more information and examples.
Command Options
Force (--force)
Force all pgBackRest processes to stop.
This option will send TERM signals to all running pgBackRest processes to effect a graceful but immediate shutdown. Note that this will also shutdown processes that were initiated on another system but have remotes running on the current system. For instance, if a backup was started on the backup server then running stop --force on the database server will shutdown the backup process on the backup server. 
default: n
example: --force
General Options
Log Options
Repository Verify (verify)
Verify determines if the backups and archives in a repository are valid.
Command Options
Output (--output)
Output type.
The following output types are supported:
  • none - No verify output.
  • text - Output verify information to stdout. 
default: none
example: --output=text
Set (--set)
Backup set to verify.
Verify all database and archive files associated with the specified backup set. 
example: --set=20150131-153358F_20150131-153401I
Verbose (--verbose)
Verbose output.
Verbose defaults to false, providing a minimal response with important information about errors in the repository. Specifying true provides more information about what was successfully verified. 
default: n
example: --verbose
General Options
Log Options
Maintainer Options
Repository Options
Version (version)
Displays installed pgBackRest version.
Command Options
Output (--output)
Output type.
The following output types are supported:
  • text - Display the installed pgBackRest version as text.
  • num - Display the installed pgBackRest version as an integer. 
default: text
example: --output=num
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated October 18, 2026
././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7768915 pgbackrest-2.60.0/doc/html/configuration.html0000644000175100017510000052527615265102133020657 0ustar00runnerrunner pgBackRest - Configuration Reference
pgBackRest
Configuration Reference
Home
User Guides
Releases
Configuration
Commands
FAQ
Table of Contents
Introduction
Archive Options
Asynchronous Archiving
Maximum Archive Get Queue Size
Retry Missing WAL Segment
Archive Push Batch Size
Maximum Archive Push Queue Size
Archive Timeout
Backup Options
Backup Annotation
Check Archive
Copy Archive
Check Archive Mode
Backup from Standby
Page Checksums
Path/File Exclusions
Expire Auto
Manifest Save Threshold
Resume
Start Fast
General Options
Allow Run as Root
Buffer Size
pgBackRest Command
SSH Client Command
Compress
Compress Level
Network Compress Level
Compress Type
Database Timeout
Delta
I/O Timeout
Lock Path
Neutral Umask
Set Process Priority
Process Maximum
Protocol Timeout
Socket Keep Alive
Spool Path
TCP Keep Alive Count
TCP Keep Alive Idle
TCP Keep Alive Interval
TLSv1.2 Cipher Suites
TLSv1.3 Cipher Suites
Log Options
Console Log Level
File Log Level
Std Error Log Level
Log Path
Log Subprocesses
Log Timestamp
Maintainer Options
WAL Header Check
Page Header Check
Force PostgreSQL Version
Repository Options
Azure Repository Account
Azure Repository Container
Azure Repository Endpoint
Azure Repository Key
Azure Repository Key Type
Azure Repository URI Style
Block Incremental Backup
Repository Bundles
Repository Bundle Limit
Repository Bundle Size
Repository Cipher Passphrase
Repository Cipher Type
GCS Repository Bucket
GCS Repository Endpoint
GCS Repository Key
GCS Repository Key Type
GCS Repository Project ID
Repository Hardlink
Repository Host
Repository Host Certificate Authority File
Repository Host Certificate Authority Path
Repository Host Certificate File
Repository Host Command
Repository Host Configuration
Repository Host Configuration Include Path
Repository Host Configuration Path
Repository Host Key File
Repository Host Port
Repository Host Protocol Type
Repository Host User
Repository Path
Archive Retention
Archive Retention Type
Differential Retention
Full Retention
Full Retention Type
Backup History Retention
S3 Repository Bucket
S3 Repository Endpoint
S3 Repository Access Key
S3 Repository Secret Access Key
S3 Repository Key Type
S3 Repository KMS Key ID
S3 Authentication Process Command
S3 Repository Region
S3 Repository Requestor Pays
S3 Repository Role
S3 Repository Service
S3 Repository SSE Customer Key
S3 Repository STS Endpoint
S3 Repository Security Token
S3 Repository URI Style
SFTP Repository Host
SFTP Repository Host Fingerprint
SFTP Host Key Check Type
SFTP Repository Host Key Hash Type
SFTP Repository Host Port
SFTP Repository Host User
SFTP Known Hosts File
SFTP Repository Private Key File
SFTP Repository Private Key Passphrase
SFTP Repository Public Key File
Repository Storage CA File
Repository Storage TLS CA Path
Repository Storage Host
Repository Storage Port
Repository Storage Prefetch
Repository Storage Read Over
Repository Storage Tag
Repository Storage Upload Chunk Size
Repository Storage Certificate Verify
Repository Symlink
Target Time for Repository
Repository Type
Restore Options
Archive Mode
Exclude Database
Include Database
Link All
Link Map
Recovery Option
Tablespace Map
Map All Tablespaces
Server Options
TLS Server Address
TLS Server Authorized Clients
TLS Server Certificate Authorities
TLS Server Certificate
TLS Server Key
TLS Server Port
Stanza Options
PostgreSQL Database
PostgreSQL Host
PostgreSQL Host Certificate Authority File
PostgreSQL Host Certificate Authority Path
PostgreSQL Host Certificate File
PostgreSQL Host Command
PostgreSQL Host Configuration
PostgreSQL Host Configuration Include Path
PostgreSQL Host Configuration Path
PostgreSQL Host Key File
PostgreSQL Host Port
PostgreSQL Host Protocol Type
PostgreSQL Host User
PostgreSQL Path
PostgreSQL Port
PostgreSQL Socket Path
PostgreSQL Database User
pgBackRest can be used entirely with command-line parameters but a configuration file is more practical for installations that are complex or set a lot of options. The default location for the configuration file is /etc/pgbackrest/pgbackrest.conf. If no file exists in that location then the old default of /etc/pgbackrest.conf will be checked.
The following option types are used:
String: A text string, commonly an identifier, password, etc. 
Command line example: --stanza=demo
Configuration file example: repo1-cipher-pass=zWaf6XtpjIVZC5444yXB...
Path: Used to uniquely identify a location in a directory structure. Paths must begin with /, double // is not allowed, and no ending / is expected. 
Command line example: --repo1-path=/var/lib/pgbackrest
Configuration file example: repo1-path=/var/lib/pgbackrest
Boolean: Enables or disables the option. Only y/n are valid argument values. 
Command line examples: --start-fast, --no-start-fast, --start-fast=y, --start-fast=n
Configuration file examples: start-fast=y, start-fast=n
Integer: Used for ports, retention/retry counts, parallel processes allowed, etc. 
Command line example: --compress-level=3
Configuration file example: pg1-port=5432
Size: Used for buffer sizes, disk usage, etc. Size can be specified in bytes (default) or KiB, MiB, GiB, TiB, or PiB where the multiplier is a power of 1024. For example, the case-insensitive value 5GiB (or 5GB, 5g) can be used instead of 5368709120. Fractional values such as 2.5GiB are not allowed, use 2560MiB instead. 
Command line example: --archive-get-queue-max=1GiB
Configuration file example: buffer-size=2MiB
Time: Time in seconds. 
Command line example: --io-timeout=90
Configuration file example: db-timeout=600
List: Option may be provided multiple times. 
Command line example: --db-exclude=db1 --db-exclude=db2 --db-exclude=db5
Configuration file example, each on its own line:
    db-exclude=db1
    db-exclude=db2
    db-exclude=db5
Key/Value: Option may be provided multiple times in the form key=value. 
Command line example: --tablespace-map=ts_01=/db/ts_01 --tablespace-map=ts_02=/db/ts_02
Configuration file example, each on its own line:
    tablespace-map=ts_01=/db/ts_01
    tablespace-map=ts_02=/db/ts_02
Archive Options
The archive section defines options for the archive-push and archive-get commands.
Asynchronous Archiving (--archive-async)
Push/get WAL segments asynchronously.
Enables asynchronous operation for the archive-push and archive-get commands.
Asynchronous operation is more efficient because it can reuse connections and take advantage of parallelism. See the spool-path, archive-get-queue-max, and archive-push-queue-max options for more information. 
default: n
example: archive-async=y
Maximum Archive Get Queue Size (--archive-get-queue-max)
Maximum size of the pgBackRest archive-get queue.
Specifies the maximum size of the archive-get queue when archive-async is enabled. The queue is stored in the spool-path and is used to speed providing WAL to PostgreSQL. 
default: 128MiB
allowed: [0B, 4PiB]
example: archive-get-queue-max=1GiB
Retry Missing WAL Segment (--archive-missing-retry)
Retry missing WAL segment
Retry a WAL segment that was previously reported as missing by the archive-get command when in asynchronous mode. This prevents notifications in the spool path from a prior restore from being used and possibly causing a recovery failure if consistency has not been reached.
Disabling this option allows PostgreSQL to more reliably recognize when the end of the WAL in the archive has been reached, which permits it to switch over to streaming from the primary. With retries enabled, a steady stream of WAL being archived will cause PostgreSQL to continue getting WAL from the archive rather than switch to streaming.
When disabling this option it is important to ensure that the spool path for the stanza is empty. The restore command does this automatically if the spool path is configured at restore time. Otherwise, it is up to the user to ensure the spool path is empty. 
default: y
example: archive-missing-retry=n
Archive Push Batch Size (--archive-push-batch-size)
Maximum amount of WAL to push per asynchronous run.
In asynchronous mode the archive-push process pushes all the WAL segments that are ready in a single run. Since archive-push-queue-max is only checked at the start of each run, a run that processes a very large number of segments can let the queue grow well beyond the limit before it is rechecked.
This option limits the amount of WAL processed per run so the process exits and is spawned again by the next archive-push, which rechecks the queue. Lower values recheck the queue more often at the cost of spawning the asynchronous process more frequently. The value is rounded down to a whole number of WAL segments but at least one segment is always processed. 
default: 16GiB
allowed: [1MiB, 4PiB]
example: archive-push-batch-size=1GiB
Maximum Archive Push Queue Size (--archive-push-queue-max)
Maximum size of the PostgreSQL archive queue.
After the limit is reached, the following will happen:
  • pgBackRest will notify PostgreSQL that the WAL was successfully archived, then DROP IT.
  • A warning will be output to the PostgreSQL log.
If this occurs then the archive log stream will be interrupted and PITR will not be possible past that point. A new backup will be required to regain full restore capability.
In asynchronous mode the entire queue will be dropped to prevent spurts of WAL getting through before the queue limit is exceeded again.
In asynchronous mode this limit is only checked at the start of each archive-push run, so the queue can grow beyond it within a single run. Reduce archive-push-batch-size to check the queue more frequently.
The purpose of this feature is to prevent the log volume from filling up at which point PostgreSQL will stop completely. Better to lose the backup than have PostgreSQL go down. 
allowed: [0B, 4PiB]
example: archive-push-queue-max=1TiB
Deprecated Name: archive-queue-max
Archive Timeout (--archive-timeout)
Archive timeout.
Set maximum time, in seconds, to wait for each WAL segment to reach the pgBackRest archive repository. The timeout applies to the check and backup commands when waiting for WAL segments required for backup consistency to be archived. 
default: 1m
allowed: [100ms, 1d]
example: archive-timeout=30
Backup Options
The backup section defines settings related to backup.
Backup Annotation (--annotation)
Annotate backup with user-defined key/value pairs.
Users can attach informative key/value pairs to the backup. This option may be used multiple times to attach multiple annotations.
Annotations are output by the info command text output when a backup is specified with --set and always appear in the JSON output. 
example: annotation=source="Sunday backup for website database"
Check Archive (--archive-check)
Check that WAL segments are in the archive before backup completes.
Checks that all WAL segments required to make the backup consistent are present in the WAL archive. It's a good idea to leave this as the default unless you are using another method for archiving.
This option must be enabled if archive-copy is enabled. 
default: y
example: archive-check=n
Copy Archive (--archive-copy)
Copy WAL segments needed for consistency to the backup.
This slightly paranoid option protects against corruption in the WAL segment archive by storing the WAL segments required for consistency directly in the backup. WAL segments are still stored in the archive so this option will use additional space.
It is best if the archive-push and backup commands have the same compress-type (e.g. lz4) when using this option. Otherwise, the WAL segments will need to be recompressed with the compress-type used by the backup, which can be fairly expensive depending on how much WAL was generated during the backup.
On restore, the WAL segments will be present in pg_xlog/pg_wal and PostgreSQL will use them in preference to calling the restore_command.
The archive-check option must be enabled if archive-copy is enabled. 
default: n
example: archive-copy=y
Check Archive Mode (--archive-mode-check)
Check the PostgreSQL archive_mode setting.
Enabled by default, this option disallows PostgreSQL archive_mode=always.
WAL segments pushed from a standby server might be logically the same as WAL segments pushed from the primary but have different checksums. Disabling archiving from multiple sources is recommended to avoid conflicts.
CAUTION:
If this option is disabled then it is critical to ensure that only one archiver is writing to the repository via the archive-push command. 
default: y
example: archive-mode-check=n
Backup from Standby (--backup-standby)
Backup from the standby cluster.
Enable backup from standby to reduce load on the primary cluster. This option requires that both the primary and standby hosts be configured.
The following modes are supported:
  • y - Standby is required for backup.
  • prefer - Backup from standby if available otherwise backup from primary.
  • n - Backup from primary only. 
default: n
example: backup-standby=y
Page Checksums (--checksum-page)
Validate data page checksums.
Directs pgBackRest to validate all data page checksums while backing up a cluster. This option is automatically enabled when data page checksums are enabled on the cluster.
Failures in checksum validation will not abort a backup. Rather, warnings will be emitted in the log (and to the console with default settings) and the list of invalid pages will be stored in the backup manifest. 
example: checksum-page=n
Path/File Exclusions (--exclude)
Exclude paths/files from the backup.
All exclusions are relative to $PGDATA. If the exclusion ends with / then only files in the specified directory will be excluded, e.g. --exclude=junk/ will exclude all files in the $PGDATA/junk directory but include the directory itself. If the exclusion does not end with / then the file may match the exclusion exactly or match with / appended to the exclusion, e.g. --exclude=junk will exclude the $PGDATA/junk directory and all the files it contains.
Be careful using this feature -- it is very easy to exclude something critical that will make the backup inconsistent. Be sure to test your restores!
All excluded files will be logged at info level along with the exclusion rule. Be sure to audit the list of excluded files to ensure nothing unexpected is being excluded.
NOTE:
Exclusions are not honored on delta restores. Any files/directories that were excluded by the backup will be removed on delta restore.
This option should not be used to exclude PostgreSQL logs from a backup. Logs can be moved out of the PGDATA directory using the PostgreSQL log_directory setting, which has the benefit of allowing logs to be preserved after a restore.
Multiple exclusions may be specified on the command-line or in a configuration file. 
example: exclude=junk/
Expire Auto (--expire-auto)
Automatically run the expire command after a successful backup.
The setting is enabled by default. Use caution when disabling this option as doing so will result in retaining all backups and archives indefinitely, which could cause your repository to run out of space. The expire command will need to be run regularly to prevent this from happening.
When expire is run automatically after a successful backup it uses the configuration of the backup command, so options set only in an expire command section (e.g. [global:expire]) are not applied. To apply expire-specific configuration, disable this option and run the expire command separately. 
default: y
example: expire-auto=y
Manifest Save Threshold (--manifest-save-threshold)
Manifest save threshold during backup.
Defines how often the manifest will be saved during a backup. Saving the manifest is important because it stores the checksums and allows the resume function to work efficiently. The actual threshold used is 1% of the backup size or manifest-save-threshold, whichever is greater. 
default: 1GiB
allowed: [1B, 1TiB]
example: manifest-save-threshold=8GiB
Resume (--resume)
Allow resume of failed backup.
Defines whether the resume feature is enabled. Resume can greatly reduce the amount of time required to run a backup after a previous backup of the same type has failed. It adds complexity, however, so it may be desirable to disable in environments that do not require the feature. 
default: y
example: resume=n
Start Fast (--start-fast)
Force a checkpoint to start backup quickly.
Forces a checkpoint (by passing y to the fast parameter of the backup start function) so the backup begins immediately. Otherwise the backup will start after the next regular checkpoint. 
default: n
example: start-fast=y
General Options
The general section defines options that are common for many commands.
Allow Run as Root (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: allow-root=y
Buffer Size (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: buffer-size=2MiB
pgBackRest Command (--cmd)
pgBackRest command.
pgBackRest may generate a command string, e.g. when the restore command generates the restore_command setting. The command used to run the pgBackRest process will be used in this case unless the cmd option is provided.
CAUTION:
Wrapping the pgBackRest command may cause unpredictable behavior and is not recommended. 
default: [path of executed pgbackrest binary]
example: cmd=/var/lib/pgsql/bin/pgbackrest_wrapper.sh
SSH Client Command (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: cmd-ssh=/usr/bin/ssh
Compress (--compress)
Use file compression.
Backup files are compatible with command-line compression tools.
This option is now deprecated. The compress-type option should be used instead. 
default: y
example: compress=n
Compress Level (--compress-level)
File compression level.
Sets the level to be used for file compression when compress-type does not equal none or compress=y (deprecated). 
default (depending on compress-type):
    bz2 - 9
    gz - 6
    lz4 - 1
    zst - 3

allow range (depending on compress-type):
    bz2 - [1, 9]
    gz - [-1, 9]
    lz4 - [-5, 12]
    zst - [-7, 22]

example: compress-level=9
Network Compress Level (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: compress-level-network=1
Compress Type (--compress-type)
File compression type.
The following compression types are supported:
  • none - no compression
  • bz2 - bzip2 compression format
  • gz - gzip compression format
  • lz4 - lz4 compression format (not available on all platforms)
  • zst - Zstandard compression format (not available on all platforms) 
default: gz
example: compress-type=none
Database Timeout (--db-timeout)
Database query timeout.
Sets the timeout, in seconds, for queries against the database. This includes the backup start/stop functions which can each take a substantial amount of time. Because of this the timeout should be kept high unless you know that these functions will return quickly (i.e. if you have set start-fast=y and you know that the database cluster will not generate many WAL segments during the backup).
NOTE:
The db-timeout option must be less than the protocol-timeout option. 
default: 30m
allowed: [100ms, 7d]
example: db-timeout=600
Delta (--delta)
Restore or backup using checksums.
During a restore, by default the PostgreSQL data and tablespace directories are expected to be present but empty. This option performs a delta restore using checksums.
During a backup, this option will use checksums instead of the timestamps to determine if files will be copied. 
default: n
example: delta=y
I/O Timeout (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: io-timeout=120
Lock Path (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: lock-path=/backup/db/lock
Neutral Umask (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: neutral-umask=n
Set Process Priority (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: priority=19
Process Maximum (--process-max)
Max processes to use for compress/transfer.
Each process will perform compression and transfer to make the command run faster, but don't set process-max so high that it impacts database performance. 
default: 1
allowed: [1, 999]
example: process-max=4
Protocol Timeout (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: protocol-timeout=630
Socket Keep Alive (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: sck-keep-alive=n
Spool Path (--spool-path)
Path where transient data is stored.
This path is used to store data for the asynchronous archive-push and archive-get command.
The asynchronous archive-push command writes acknowledgements into the spool path when it has successfully stored WAL in the archive (and errors on failure) so the foreground process can quickly notify PostgreSQL. Acknowledgement files are very small (zero on success and a few hundred bytes on error).
The asynchronous archive-get command queues WAL in the spool path so it can be provided very quickly when PostgreSQL requests it. Moving files to PostgreSQL is most efficient when the spool path is on the same filesystem as pg_xlog/pg_wal. However, it is not recommended to place the spool path within the pg_xlog/pg_wal directory as this may cause issues for PostgreSQL utilities such as pg_rewind.
The data stored in the spool path is not strictly temporary since it can and should survive a reboot. However, loss of the data in the spool path is not a problem. pgBackRest will simply recheck each WAL segment to ensure it is safely archived for archive-push and rebuild the queue for archive-get.
The spool path is intended to be located on a local Posix-compatible filesystem, not a remote filesystem such as NFS or CIFS. 
default: /var/spool/pgbackrest
example: spool-path=/backup/db/spool
TCP Keep Alive Count (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: tcp-keep-alive-count=3
TCP Keep Alive Idle (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: tcp-keep-alive-idle=60
TCP Keep Alive Interval (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: tcp-keep-alive-interval=30
TLSv1.2 Cipher Suites (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 Cipher Suites (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
The log section defines logging-related settings.
CAUTION:
Trace-level logging may expose secrets such as keys and passwords. Use with caution!
Console Log Level (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: log-level-console=error
File Log Level (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: log-level-file=debug
Std Error Log Level (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: log-level-stderr=error
Log Path (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: log-path=/backup/db/log
Log Subprocesses (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: log-subprocess=y
Log Timestamp (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: log-timestamp=n
Maintainer Options
Maintainer options are intended to support PostgreSQL forks. The proper settings should be determined by the fork maintainer and then communicated to users of the fork.
WARNING:
Improper use of these options may lead to unexpected behavior or data corruption.
It is the responsibility of the fork maintainer to test pgBackRest with the required options. pgBackRest does not guarantee compatibility with any fork.
WAL Header Check (--archive-header-check)
Check PostgreSQL version/id in WAL headers.
Enabled by default, this option checks the WAL header against the PostgreSQL version and system identifier to ensure that the WAL is being copied to the correct stanza. This is in addition to checking pg_control against the stanza and verifying that WAL is being copied from the same PostgreSQL data directory where pg_control is located.
Therefore, disabling this check is fairly safe but should only be done when needed, e.g. if the WAL is encrypted. 
default: y
example: archive-header-check=n
Page Header Check (--page-header-check)
Check PostgreSQL page headers.
Enabled by default, this option adds page header checks.
Disabling this option should be avoided except when necessary, e.g. if pages are encrypted. 
default: y
example: page-header-check=n
Force PostgreSQL Version (--pg-version-force)
Force PostgreSQL version.
The specified PostgreSQL version will be used instead of the version automatically detected by reading pg_control or WAL headers. This is mainly useful for PostgreSQL forks or development versions where those values are different from the release version. The version reported by PostgreSQL via `server_version_num` must match the forced version.
WARNING:
Be cautious when using this option because pg_control and WAL headers will still be read with the expected format for the specified version, i.e. the format from the official open-source version of PostgreSQL. If the fork or development version changes the format of the fields that pgBackRest depends on it will lead to unexpected behavior. In general, this option will only work as expected if the fork adds all custom struct members after the standard PostgreSQL members. 
example: pg-version-force=15
Repository Options
The repository section defines options used to configure the repository.
Indexing: All repo- options are indexed to allow for configuring multiple repositories. For example, a single repository is configured with the repo1-path, repo1-host, etc. options. If there is more than one repository configured and the --repo option is not specified for a command, the repositories will be acted upon in highest priority order (e.g. repo1 then repo2).
The repo-retention-* options define how long backups will be retained. Expiration only occurs when the count of complete backups exceeds the allowed retention. In other words, if repo1-retention-full-type is set to count (default) and repo1-retention-full is set to 2, then there must be 3 complete backups before the oldest will be expired. If repo1-retention-full-type is set to time then repo1-retention-full represents days so there must be at least that many days worth of full backups before expiration can occur. Make sure you always have enough space for retention + 1 backups.
Azure Repository Account (--repo-azure-account)
Azure repository account.
Azure account used to store the repository. 
example: repo1-azure-account=pg-backup
Azure Repository Container (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: repo1-azure-container=pg-backup
Azure Repository Endpoint (--repo-azure-endpoint)
Azure repository endpoint.
Endpoint used to connect to the blob service. The default is generally correct unless using Azure Government.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
default: blob.core.windows.net
example: repo1-azure-endpoint=blob.core.usgovcloudapi.net
Azure Repository Key (--repo-azure-key)
Azure repository key.
A shared key or shared access signature depending on the repo-azure-key-type option. 
example: repo1-azure-key=T+9+aov82qNhrcXSNGZCzm9mjd4d75/oxxOr6r1JVpgTLA==
Azure Repository Key Type (--repo-azure-key-type)
Azure repository key type.
The following authorization types are supported:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: repo1-azure-key-type=sas
Azure Repository URI Style (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: repo1-azure-uri-style=path
Block Incremental Backup (--repo-block)
Enable block incremental backup.
Block incremental allows for more granular backups by splitting files into blocks that can be backed up independently. This saves space in the repository and can improve delta restore performance because individual blocks can be fetched without reading the entire file from the repository.
NOTE:
The repo-bundle option must be enabled before repo-block can be enabled.
The block size for a file is determined based on the file size and age. Generally, older/larger files will get larger block sizes. If a file is old enough, it will not be backed up using block incremental.
Block incremental is most efficient when enabled for all backup types, including full. This makes the full a bit larger but subsequent differential and incremental backups can make use of the block maps generated by the full backup to save space. 
default: n
example: repo1-block=y
Repository Bundles (--repo-bundle)
Bundle files in repository.
Bundle (combine) smaller files to reduce the total number of files written to the repository. Writing fewer files is generally more efficient, especially on object stores such as S3. In addition, zero-length files are not stored (except in the manifest), which saves time and space. 
default: n
example: repo1-bundle=y
Repository Bundle Limit (--repo-bundle-limit)
Limit for file bundles.
Size limit for files that will be included in bundles. Files larger than this size will be stored separately.
Bundled files cannot be reused when a backup is resumed, so this option controls the files that can be resumed, i.e. higher values result in fewer resumable files. 
default: 2MiB
allowed: [8KiB, 1PiB]
example: repo1-bundle-limit=10MiB
Repository Bundle Size (--repo-bundle-size)
Target size for file bundles.
Defines the target size for files that will be added to a single bundle. The uncompressed bundle size may be as large as repo-bundle-size + repo-bundle-limit, so do not set this option to the maximum size that your file system allows.
In general, it is not a good idea to set this option too high because retries will need to redo the entire bundle. 
default: 20MiB
allowed: [1MiB, 1PiB]
example: repo1-bundle-size=10MiB
Repository Cipher Passphrase (--repo-cipher-pass)
Repository cipher passphrase.
Passphrase used to encrypt/decrypt files of the repository.
NOTE:
When run without the stanza option the info command reads encryption settings only from the global section. If encryption settings are configured per stanza, run the info command with the stanza option to read an encrypted stanza. 
example: repo1-cipher-pass=zWaf6XtpjIVZC5444yXB+cgFDFl7MxGlgkZSaoPvTGirhPygu4jOKOXf9LO4vjfO
Repository Cipher Type (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: repo1-cipher-type=aes-256-cbc
GCS Repository Bucket (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: repo1-gcs-endpoint=localhost
GCS Repository Key (--repo-gcs-key)
GCS repository key.
A token, service key file, or external account credential file depending on the repo-gcs-key-type option. 
example: repo1-gcs-key=/etc/pgbackrest/gcs-key.json
GCS Repository Key Type (--repo-gcs-key-type)
GCS repository key type.
The following authorization types are supported:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
  • web-id - Authorize using Workload Identity Federation.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed.
When repo-gcs-key-type=web-id the repo-gcs-key option must contain an external account credential file and the web identity token will be reloaded when the authentication token is renewed.
service_account_impersonation_url is not supported. Only credential_source.file is supported and when credential_source.format is specified the type must be text. 
default: service
example: repo1-gcs-key-type=auto
GCS Repository Project ID (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: repo1-gcs-user-project=my-project
Repository Hardlink (--repo-hardlink)
Hardlink files between backups in the repository.
Enable hard-linking of files in differential and incremental backups to their full backups. This gives the appearance that each backup is a full backup at the file-system level. Be careful, though, because modifying files that are hard-linked can affect all the backups in the set. 
default: n
example: repo1-hardlink=y
Deprecated Name: hardlink
Repository Host (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: repo1-host-cert-file=/path/to/client.crt
Repository Host Command (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: repo1-host-config-path=/conf/pgbackrest
Repository Host Key File (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: repo1-host-key-file=/path/to/client.key
Repository Host Port (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: repo1-host-type=tls
Repository Host User (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: repo1-path=/backup/db/backrest
Archive Retention (--repo-retention-archive)
Number of backups worth of continuous WAL to retain.
NOTE:
WAL segments required to make a backup consistent are always retained until the backup is expired regardless of how this option is configured.
If this value is not set and repo-retention-full-type is count (default), then the archive to expire will default to the repo-retention-full (or repo-retention-diff) value corresponding to the repo-retention-archive-type if set to full (or diff). This will ensure that WAL is only expired for backups that are already expired. If repo-retention-full-type is time, then this value will default to removing archives that are earlier than the oldest full backup retained after satisfying the repo-retention-full setting.
This option must be set if repo-retention-archive-type is set to incr. If disk space is at a premium, then this setting, in conjunction with repo-retention-archive-type, can be used to aggressively expire WAL segments. However, doing so negates the ability to perform PITR from the backups with expired WAL and is therefore not recommended. 
allowed: [1, 9999999]
example: repo1-retention-archive=2
Deprecated Name: retention-archive
Archive Retention Type (--repo-retention-archive-type)
Backup type for WAL retention.
The following backup types are supported:
  • full - pgBackRest will keep archive logs for the number of full backups defined by repo-retention-archive.
  • diff - (differential) pgBackRest will keep archive logs for the number of full and differential backups defined by repo-retention-archive, meaning if the last backup taken was a full backup, it will be counted as a differential for the purpose of repo-retention.
  • incr - (incremental) pgBackRest will keep archive logs for the number of full, differential, and incremental backups defined by repo-retention-archive.
It is recommended that this setting not be changed from the default which will only expire WAL in conjunction with expiring full backups. 
default: full
example: repo1-retention-archive-type=diff
Deprecated Name: retention-archive-type
Differential Retention (--repo-retention-diff)
Number of differential backups to retain.
When a differential backup expires, all incremental backups associated with the differential backup will also expire. When not defined all differential backups will be kept until the full backups they depend on expire.
Note that full backups are included in the count of differential backups for the purpose of expiration. This slightly reduces the number of differential backups that need to be retained in most cases. 
allowed: [1, 9999999]
example: repo1-retention-diff=3
Deprecated Name: retention-diff
Full Retention (--repo-retention-full)
Full backup retention count/time.
When a full backup expires, all differential and incremental backups associated with the full backup will also expire. When the option is not defined a warning will be issued. If indefinite retention is desired then set the option to the max value. 
allowed: [1, 9999999]
example: repo1-retention-full=2
Deprecated Name: retention-full
Full Retention Type (--repo-retention-full-type)
Retention type for full backups.
Determines whether the repo-retention-full setting represents a time period (days) or count of full backups to keep.
The following retention types are supported:
  • time - full backups older than repo-retention-full will be removed from the repository if there is at least one other backup that is equal to or greater than the repo-retention-full setting. For example, if repo-retention-full is 30 (days) and there are 2 full backups: one 25 days old and one 35 days old, no full backups will be expired because expiring the 35 day old backup would leave only the 25 day old backup, which would violate the 30 day retention policy of having at least one backup 30 days old before an older one can be expired. Archived WAL older than the oldest full backup remaining will be automatically expired unless repo-retention-archive-type and repo-retention-archive are explicitly set.
  • count - full backups that exceed repo-retention-full will be expired. For example, if repo-retention-full is 4 and a fifth full backup is completed, then the oldest full backup will be expired to keep the count at 4.
Note that a backup must be successfully completed before it will be considered for retention. For example, if repo-retention-full-type is count and repo-retention-full is 2, then there must be 3 complete full backups before the oldest will be expired. 
default: count
example: repo1-retention-full-type=time
Backup History Retention (--repo-retention-history)
Days of backup history manifests to retain.
A copy of the backup manifest is stored in the backup.history path when a backup completes. By default these files are never expired since they are useful for data mining, e.g. measuring backup and WAL growth over time.
Set repo-retention-history to define the number of days of backup history manifests to retain. Unexpired backups are always kept in the backup history. Specify repo-retention-history=0 to retain the backup history only for unexpired backups.
When a full backup history manifest is expired, all differential and incremental backup history manifests associated with the full backup also expire. 
allowed: [0, 9999999]
example: repo1-retention-history=365
S3 Repository Bucket (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: repo1-s3-bucket=pg-backup
S3 Repository Endpoint (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Access Key (--repo-s3-key)
S3 repository access key.
AWS key used to access this bucket. 
example: repo1-s3-key=AKIAIOSFODNN7EXAMPLE
S3 Repository Secret Access Key (--repo-s3-key-secret)
S3 repository secret access key.
AWS secret key used to access this bucket. 
example: repo1-s3-key-secret=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
S3 Repository Key Type (--repo-s3-key-type)
S3 repository key type.
The following authorization types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: repo1-s3-key-type=auto
S3 Repository KMS Key ID (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: repo1-s3-process-cmd=/usr/local/bin/get-credentials
example: repo1-s3-process-cmd=--role
example: repo1-s3-process-cmd=my-role
S3 Repository Region (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: repo1-s3-region=us-east-1
S3 Repository Requestor Pays (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: repo1-s3-requester-pays=n
S3 Repository Role (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: repo1-s3-role=authrole
S3 Repository Service (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: repo1-s3-service=s3-outposts
S3 Repository SSE Customer Key (--repo-s3-sse-customer-key)
S3 repository SSE customer key.
Enables S3 server-side encryption using the specified customer key. 
example: repo1-s3-sse-customer-key=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Repository STS Endpoint (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository Security Token (--repo-s3-token)
S3 repository security token.
AWS security token used with temporary credentials. 
example: repo1-s3-token=AQoDYXdzEPT//////////wEXAMPLEtc764bNrC9SAPBSM22 ...
S3 Repository URI Style (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: repo1-s3-uri-style=path
SFTP Repository Host (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Allowed values are md5, sha1, and sha256, though sha256 is only supported by newer versions of libssh2. 
example: repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: repo1-sftp-host-port=22
SFTP Repository Host User (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: repo1-sftp-host-user=pg-backup
SFTP Known Hosts File (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Private Key Passphrase (--repo-sftp-private-key-passphrase)
SFTP private key passphrase.
Passphrase used to access the private key. This is an optional feature when creating an SSH public/private key pair. 
example: repo1-sftp-private-key-passphrase=BeSureToGenerateAndUseASecurePassphrase
SFTP Repository Public Key File (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Prefetch (--repo-storage-prefetch)
Repository storage prefetch.
Determines the number of requests to prefetch when reading from object stores, e.g. S3. This is not true concurrency but it reduces the latency involved in submitting requests.
Note that the number of open requests per read will be one greater than prefetch, since the initial read cannot be prefetched. 
default: 3
allowed: [0, 1024]
example: repo1-storage-prefetch=7
Repository Storage Read Over (--repo-storage-read-over)
Repository storage read over.
Bundles may contain data that is not needed during restore, for example files that already exist. This option skips over unneeded data without starting a new read request on object stores. This is generally faster since the latency on new read requests is higher than continuing a read already in progress. It is also less expensive for most storage classes. 
default: 64KiB
allowed: [0B, 1MiB]
example: repo1-storage-read-over=128KiB
Repository Storage Tag (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: repo1-storage-verify-tls=n
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Repository Symlink (--repo-symlink)
Create symlinks within the repository.
Enable creation of the lastest and tablespace symlinks. These symlinks are most useful when using snapshots to do in-place recovery in the repository, which is an uncommon use case.
While this feature is likely not useful for the vast majority of users it remains on by default for legacy purposes. However, it may be useful to disable symlinks for Posix-like storage that does not support them. 
default: y
example: repo1-symlink=n
Target Time for Repository (--repo-target-time)
Target time for repository.
The target time defines the time that commands use to read a repository on versioned storage. This allows the command to read the repository as it was at a point-in-time in order to recover data that has been deleted or corrupted by user accident or malware.
Versioned storage is supported by S3, GCS, and Azure but is generally not enabled by default. In addition to enabling versioning, it may be useful to enable object locking for S3 and soft delete for GCS or Azure.
When the repo-target-time option is specified then the repo option must also be provided. It is likely that not all repository types will support versioning and in general it makes sense to target a single repository for recovery.
Note that comparisons to the storage timestamp are <= the timestamp provided and milliseconds are truncated from the timestamp when provided. 
example: repo-target-time=2024-08-08 12:12:12+00
Repository Type (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: repo1-type=cifs
Restore Options
The restore section defines settings used for restoring backups.
Archive Mode (--archive-mode)
Preserve or disable archiving on restored cluster.
This option allows archiving to be preserved or disabled on a restored cluster. This is useful when the cluster must be promoted to do some work but is not intended to become the new primary. In this case it is not a good idea to push WAL from the cluster into the repository.
The following modes are supported:
  • off - disable archiving by setting archive_mode=off.
  • preserve - preserve current archive_mode setting.
NOTE: This option is not available on PostgreSQL < 12. 
default: preserve
example: archive-mode=off
Exclude Database (--db-exclude)
Restore excluding the specified databases.
Databases excluded will be restored as sparse, zeroed files to save space but still allow PostgreSQL to perform recovery. After recovery, those databases will not be accessible but can be removed with the drop database command. The --db-exclude option can be passed multiple times to specify more than one database to exclude.
When used in combination with the --db-include option, --db-exclude will only apply to standard system databases (template0, template1, and postgres). 
example: db-exclude=db_main
Include Database (--db-include)
Restore only specified databases.
This feature allows only selected databases to be restored. Databases not specifically included will be restored as sparse, zeroed files to save space but still allow PostgreSQL to perform recovery. After recovery, the databases that were not included will not be accessible but can be removed with the drop database command.
NOTE:
built-in databases (template0, template1, and postgres) are always restored unless specifically excluded.
The --db-include option can be passed multiple times to specify more than one database to include.
See Restore Selected Databases for additional information and caveats. 
example: db-include=db_main
Link All (--link-all)
Restore all symlinks.
By default symlinked directories and files are restored as normal directories and files in $PGDATA. This is because it may not be safe to restore symlinks to their original destinations on a system other than where the original backup was performed. This option restores all the symlinks just as they were on the original system where the backup was performed. 
default: n
example: link-all=y
Link Map (--link-map)
Modify the destination of a symlink.
Allows the destination file or path of a symlink to be changed on restore. This is useful for restoring to systems that have a different storage layout than the original system where the backup was generated. 
example: link-map=pg_xlog=/data/xlog
Recovery Option (--recovery-option)
Set an option in postgresql.auto.conf or recovery.conf.
See Server Configuration for details on postgresql.auto.conf or recovery.conf options (be sure to select your PostgreSQL version). This option can be used multiple times.
For PostgreSQL >= 12, options will be written into postgresql.auto.conf. For all other versions, options will be written into recovery.conf.
NOTE:
The restore_command option will be automatically generated but can be overridden with this option. Be careful about specifying your own restore_command as pgBackRest is designed to handle this for you. Target Recovery options (recovery_target_name, recovery_target_time, etc.) are generated automatically by pgBackRest and should not be set with this option.
Since pgBackRest does not start PostgreSQL after writing the postgresql.auto.conf or recovery.conf file, it is always possible to edit/check postgresql.auto.conf or recovery.conf before manually restarting. 
example: recovery-option=primary_conninfo=db.mydomain.com
Tablespace Map (--tablespace-map)
Restore a tablespace into the specified directory.
Moves a tablespace to a new location during the restore. This is useful when tablespace locations are not the same on a replica, or an upgraded system has different mount points.
Tablespace locations are not stored in pg_tablespace so moving tablespaces can be done with impunity. However, moving a tablespace to the data_directory is not recommended and may cause problems. For more information on moving tablespaces http://www.databasesoup.com/2013/11/moving-tablespaces.html is a good resource. 
example: tablespace-map=ts_01=/db/ts_01
Map All Tablespaces (--tablespace-map-all)
Restore all tablespaces into the specified directory.
Tablespaces are restored into their original locations by default. This behavior can be modified for each tablespace with the tablespace-map option, but it is sometimes preferable to remap all tablespaces to a new directory all at once. This is particularly useful for development or staging systems that may not have the same storage layout as the original system where the backup was generated.
The path specified will be the parent path used to create all the tablespaces in the backup.
CAUTION:
Tablespaces created after the backup started will not be mapped. Make a new backup after a tablespace is created if tablespace mapping is required. 
example: tablespace-map-all=/data/tablespace
Server Options
The server section defines options used for configuring the TLS server.
TLS Server Address (--tls-server-address)
TLS server address.
IP address the server will listen on for client requests. 
default: localhost
example: tls-server-address=*
TLS Server Authorized Clients (--tls-server-auth)
TLS server authorized clients.
Clients are authorized on the server by verifying their certificate and checking their certificate CN (Common Name) against a list on the server configured with the tls-server-auth option.
A client CN can be authorized for as many stanzas as needed by providing a comma-separated list to the tls-server-auth option or for all stanzas by specifying tls-server-auth=client-cn=*. Wildcards may not be specified for the client CN. 
example: tls-server-auth=client-cn=stanza1,stanza2
TLS Server Certificate Authorities (--tls-server-ca-file)
TLS server certificate authorities.
Checks that client certificates are signed by a trusted certificate authority. 
example: tls-server-ca-file=/path/to/server.ca
TLS Server Certificate (--tls-server-cert-file)
TLS server certificate file.
Sent to the client to show the server identity. 
example: tls-server-cert-file=/path/to/server.crt
TLS Server Key (--tls-server-key-file)
TLS server key file.
Proves server certificate was sent by the owner. 
example: tls-server-key-file=/path/to/server.key
TLS Server Port (--tls-server-port)
TLS server port.
Port the server will listen on for client requests. 
default: 8432
allowed: [1, 65535]
example: tls-server-port=8000
Stanza Options
A stanza defines the backup configuration for a specific PostgreSQL database cluster. The stanza section must define the database cluster path and host/user if the database cluster is remote. Also, any global configuration sections can be overridden to define stanza-specific settings.
Indexing: All pg- options are indexed to allow for configuring multiple PostgreSQL hosts. For example, a single primary is configured with the pg1-path, pg1-port, etc. options. If a standby is configured then index the pg- options on the repository host as pg2- (e.g. pg2-host, pg2-path, etc).
PostgreSQL Database (--pg-database)
PostgreSQL database.
The database name used when connecting to PostgreSQL. The default is usually best but some installations may not contain this database.
Note that for legacy reasons the setting of the PGDATABASE environment variable will be ignored. 
default: postgres
example: pg1-database=backupdb
PostgreSQL Host (--pg-host)
PostgreSQL host for operating remotely.
Used for backups where the PostgreSQL host is different from the repository host. 
example: pg1-host=db.domain.com
Deprecated Name: db-host
PostgreSQL Host Certificate Authority File (--pg-host-ca-file)
PostgreSQL host certificate authority file.
Use a CA file other than the system default for connecting to the PostgreSQL host. 
example: pg1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
PostgreSQL Host Certificate Authority Path (--pg-host-ca-path)
PostgreSQL host certificate authority path.
Use a CA path other than the system default for connecting to the PostgreSQL host. 
example: pg1-host-ca-path=/etc/pki/tls/certs
PostgreSQL Host Certificate File (--pg-host-cert-file)
PostgreSQL host certificate file.
Sent to PostgreSQL host to prove client identity. 
example: pg1-host-cert-file=/path/to/client.crt
PostgreSQL Host Command (--pg-host-cmd)
PostgreSQL host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and PostgreSQL hosts. If not defined, the PostgreSQL host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: pg1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: db-cmd
PostgreSQL Host Configuration (--pg-host-config)
pgBackRest database host configuration file.
Sets the location of the configuration file on the PostgreSQL host. This is only required if the PostgreSQL host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: pg1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: db-config
PostgreSQL Host Configuration Include Path (--pg-host-config-include-path)
pgBackRest database host configuration include path.
Sets the location of the configuration include path on the PostgreSQL host. This is only required if the PostgreSQL host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: pg1-host-config-include-path=/conf/pgbackrest/conf.d
PostgreSQL Host Configuration Path (--pg-host-config-path)
pgBackRest database host configuration path.
Sets the location of the configuration path on the PostgreSQL host. This is only required if the PostgreSQL host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: pg1-host-config-path=/conf/pgbackrest
PostgreSQL Host Key File (--pg-host-key-file)
PostgreSQL host key file.
Proves client certificate was sent by owner. 
example: pg1-host-key-file=/path/to/client.key
PostgreSQL Host Port (--pg-host-port)
PostgreSQL host port when pg-host is set.
Use this option to specify a non-default port for the PostgreSQL host protocol.
NOTE:
When pg-host-type=ssh there is no default for pg-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on pg-host-type):
    tls - 8432

allowed: [0, 65535]
example: pg1-host-port=25
Deprecated Name: db-ssh-port
PostgreSQL Host Protocol Type (--pg-host-type)
PostgreSQL host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: pg1-host-type=tls
PostgreSQL Host User (--pg-host-user)
PostgreSQL host logon user when pg-host is set.
This user will also own the remote pgBackRest process and will initiate connections to PostgreSQL. For this to work correctly the user should be the PostgreSQL database cluster owner which is generally postgres, the default. 
default: postgres
example: pg1-host-user=db_owner
Deprecated Name: db-user
PostgreSQL Path (--pg-path)
PostgreSQL data directory.
This should be the same as the data_directory reported by PostgreSQL. Even though this value can be read from various places, it is prudent to set it in case those resources are not available during a restore or offline backup scenario.
The pg-path option is tested against the value reported by PostgreSQL on every online backup so it should always be current. 
example: pg1-path=/data/db
Deprecated Name: db-path
PostgreSQL Port (--pg-port)
PostgreSQL port.
Port that PostgreSQL is running on. This usually does not need to be specified as most PostgreSQL clusters run on the default port. 
default: 5432
allowed: [0, 65535]
example: pg1-port=6543
Deprecated Name: db-port
PostgreSQL Socket Path (--pg-socket-path)
PostgreSQL unix socket path.
The unix socket directory that was specified when PostgreSQL was started. pgBackRest will automatically look in the standard location for your OS so there is usually no need to specify this setting unless the socket directory was explicitly modified with the unix_socket_directories setting in postgresql.conf. 
example: pg1-socket-path=/var/run/postgresql
Deprecated Name: db-socket-path
PostgreSQL Database User (--pg-user)
PostgreSQL database user.
The database user name used when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER. 
example: pg1-user=backupuser
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated October 18, 2026
././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7769954 pgbackrest-2.60.0/doc/html/default.css0000644000175100017510000005624615265102133017254 0ustar00runnerrunner:root { --font-body: system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif; --font-mono: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; --page-width: 69rem; --gutter: max(1.25rem, calc((100% - var(--page-width)) / 2)); --radius: 6px; --radius-lg: 10px; --bar-height: 3rem; --sidebar-width: clamp(13rem, 20vw, 17rem); --sidebar-gap: 1.5rem; --icon-copy: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24' fill='none'\ stroke='black' stroke-width='2'%3E%3Crect x='9' y='9' width='12' height='12' rx='2'/%3E%3Cpath d='M6 15H5a2 2 0 0 1-2-2V5a2\ 2 0 0 1 2-2h8a2 2 0 0 1 2 2v1'/%3E%3C/svg%3E"); --icon-tick: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 24 24' fill='none'\ stroke='black' stroke-width='2' stroke-linecap='round' stroke-linejoin='round'%3E%3Cpath d='M4 12.5l5.5 5.5L20\ 7'/%3E%3C/svg%3E"); --bg: #ffffff; --bg-surface: #f7f9fb; --bg-surface-2: #eef2f6; --bg-bar: #ffffff; --text: #0f172a; --text-muted: #55637a; --border: #e3e8ef; --border-strong: #cbd4e1; --keyword-weight: 500; --keyword-bg: var(--bg-surface); --keyword-border: #e9edf2; --menu-edge: var(--border); --accent: #0d5f9e; --accent-hover: #0a4a7c; --accent-soft: #e8f1f9; --code-bg: #1b2330; --code-text: #e2e8f0; --code-border: #2a3441; --code-output-bg: #263141; --code-prompt: #6aa9e0; --code-highlight-bg: #1d6b33; --code-error-bg: #9c3030; --note-accent: #2f74b5; --note-bg: #eef5fb; --tip-accent: #2f8f4e; --tip-bg: #edf7f0; --important-accent: #7c56b8; --important-bg: #f3effa; --caution-accent: #b07d16; --caution-bg: #fbf5e6; --warning-accent: #bf3d3d; --warning-bg: #fdeeee; --shadow: 0 1px 2px rgba(15, 23, 42, .06); } @media (prefers-color-scheme: dark) { :root { --bg: #0b0f15; --bg-surface: #121923; --bg-surface-2: #18212e; --bg-bar: #0e141c; --text: #e2e8f0; --text-muted: #94a3b8; --border: #1f2937; --border-strong: #334155; --keyword-weight: 600; --keyword-bg: var(--bg-surface-2); --keyword-border: var(--border); --menu-edge: #1d2634; --accent: #64a8ef; --accent-hover: #8fc3f5; --accent-soft: #12233a; --code-bg: #060a0f; --code-text: #e2e8f0; --code-border: #1c2531; --code-output-bg: #121a24; --code-prompt: #64a8ef; --code-highlight-bg: #1c5c2c; --code-error-bg: #8c2f2f; --note-accent: #64a8ef; --note-bg: #101c2b; --tip-accent: #4fb974; --tip-bg: #0f2216; --important-accent: #a98be0; --important-bg: #1c172c; --caution-accent: #d3a13c; --caution-bg: #241c10; --warning-accent: #e57676; --warning-bg: #291616; --shadow: 0 1px 2px rgba(0, 0, 0, .4); } } html { font-family: var(--font-body); font-size: 16px; -webkit-text-size-adjust: 100%; overflow-anchor: none; } body { margin: 0; background: var(--bg); color: var(--text); font-size: 1rem; line-height: 1.7; font-weight: 400; overflow-wrap: break-word; } .page-header, .page-toc, .page-body, .page-footer, .menu-body { padding-left: var(--gutter); padding-right: var(--gutter); } a:link, a:visited { color: var(--accent); text-decoration: none; } a:hover, a:active { color: var(--accent-hover); text-decoration: underline; text-underline-offset: 2px; } .section-body-text a:link, .section-body-text a:visited, .section-intro a:link, .section-intro a:visited, li.list-unordered a:link, li.list-unordered a:visited { text-decoration: underline; text-underline-offset: 2px; text-decoration-thickness: 1px; text-decoration-color: var(--border-strong); } .section-body-text a:hover, .section-intro a:hover, li.list-unordered a:hover { text-decoration-color: var(--accent); } a[id] { scroll-margin-top: calc(var(--bar-height) + 1rem); } .page-header { display: flex; flex-wrap: wrap; align-items: center; row-gap: .425rem; padding-top: 1rem; padding-bottom: 1rem; } .page-header-logo { flex: none; height: 3.5rem; aspect-ratio: 383.2 / 101.3; background-image: url("slogo.svg"); background-repeat: no-repeat; background-position: left center; background-size: contain; } .page-header-title, .page-header-logo-name { position: absolute; width: 1px; height: 1px; margin: -1px; overflow: hidden; clip-path: inset(50%); white-space: nowrap; } .page-header::before, .page-header::after { content: ""; flex: 1 0 0; } .page-header::before { order: 1; flex-basis: .85rem; } .page-header-subtitle { order: 2; color: var(--text-muted); font-size: 1.25rem; } .page-header::after { order: 3; } .page-menu { position: sticky; top: 0; z-index: 20; border-top: 1px solid var(--border); border-bottom: 1px solid var(--border); background: var(--bg-bar); box-shadow: var(--shadow); } .menu-body { display: flex; align-items: center; gap: 1.75rem; height: var(--bar-height); overflow-x: auto; scrollbar-width: thin; scrollbar-color: var(--border-strong) transparent; } .page-menu::before, .page-menu::after { content: ""; position: absolute; top: 0; bottom: 0; width: 2rem; opacity: 0; pointer-events: none; } .page-menu::before { left: 0; background: linear-gradient(to right, var(--menu-edge) 20%, transparent); } .page-menu::after { right: 0; background: linear-gradient(to left, var(--menu-edge) 20%, transparent); } .page-menu-fade-left::before, .page-menu-fade-right::after { opacity: 1; } .menu:first-child { margin-left: auto; } .menu:last-child { margin-right: auto; } .menu { white-space: nowrap; } a.menu-link:link, a.menu-link:visited, .menu-current { display: block; color: var(--text-muted); font-size: .9375rem; font-weight: 550; letter-spacing: -.005em; text-decoration: none; } a.menu-link:hover, a.menu-link:active { color: var(--accent); text-decoration: none; } .menu-current { color: var(--text); text-decoration: underline; text-decoration-color: var(--accent); text-decoration-thickness: 2px; text-underline-offset: .5em; } .page-toc { margin-top: 1.5rem; margin-bottom: 2rem; } .page-toc-header { margin-bottom: .75rem; color: var(--text-muted); font-size: .8125rem; font-weight: 700; letter-spacing: .09em; text-transform: uppercase; } .page-toc-body { border: 1px solid var(--border); border-radius: var(--radius-lg); background: var(--bg-surface); padding: 1.25rem 1.5rem; } .section1-toc { margin-top: .9rem; } .section1-toc:first-of-type { margin-top: 0; } .section1-toc-title { font-size: 1.0625rem; font-weight: 650; } .section2-toc { margin-top: .1rem; margin-left: 1.1rem; } .section3-toc { margin-left: 1.1rem; } .page-toc a:link, .page-toc a:visited { color: var(--text); text-decoration: none; } .page-toc a.toc-active:link, .page-toc a.toc-active:visited { color: var(--accent); font-weight: 700; } .page-toc a.toc-active-parent:link, .page-toc a.toc-active-parent:visited { color: var(--text); font-weight: 700; } .page-toc a:hover, .page-toc a:active { color: var(--accent); text-decoration: underline; text-underline-offset: 3px; } @media (min-width: 56rem) { body.page-sidebar { display: grid; grid-template-columns: var(--sidebar-width) minmax(0, 1fr); column-gap: var(--sidebar-gap); padding-left: var(--gutter); padding-right: var(--gutter); } body.page-sidebar > .page-header, body.page-sidebar > .page-menu, body.page-sidebar > .page-footer { grid-column: 1 / -1; } body.page-sidebar > .page-menu { margin-left: calc(var(--gutter) * -1); margin-right: calc(var(--gutter) * -1); } body.page-sidebar > .page-header, body.page-sidebar > .page-toc, body.page-sidebar > .page-body, body.page-sidebar > .page-footer { padding-left: 0; padding-right: 0; } body.page-sidebar > .page-toc { grid-column: 1; margin-top: 1rem; margin-bottom: 0; font-size: .875rem; } .page-toc-header { display: none; } body.page-sidebar > .page-toc > .page-toc-inner { position: sticky; top: var(--bar-height); max-height: calc(100vh - var(--bar-height)); padding-right: .5rem; overflow-y: auto; overscroll-behavior: contain; } body.page-sidebar > .page-body { grid-column: 2; } body.page-sidebar .page-toc-body { border: none; border-radius: 0; background: none; padding: 0; } .page-toc .section1-toc, .page-toc .section2-toc, .page-toc .section3-toc { line-height: 1.45; } .page-toc .section1-toc { margin-top: .7rem; } .page-toc .section1-toc:first-of-type { margin-top: 0; } .page-toc .section2-toc, .page-toc .section3-toc { margin-top: .25rem; margin-left: .9rem; } .page-toc .section1-toc-title { font-size: .9375rem; } .page-toc .path, .page-toc .br-option, .page-toc .br-setting, .page-toc .pg-option, .page-toc .pg-setting, .page-toc .id, .page-toc .user, .page-toc .file { border: none; background: none; padding: 0; white-space: normal; } } .section1 { margin-top: 1.75rem; } .section1:first-of-type { margin-top: 1rem; } .section1-header { border-left: 4px solid var(--accent); border-radius: 0 var(--radius) var(--radius) 0; background: var(--bg-surface-2); padding: .45rem .85rem; margin-bottom: .9rem; font-size: 1.6rem; font-weight: 700; letter-spacing: -.022em; line-height: 1.25; } .section2 { margin-top: 1.25rem; } .section2-header { width: fit-content; border-bottom: 1.5px solid var(--accent); padding: 0 0 .35rem 0; margin-bottom: .7rem; font-size: 1.3125rem; font-weight: 650; letter-spacing: -.015em; line-height: 1.35; } .section3 { margin-top: 1rem; } .section3-header { border-left: 2px solid var(--border-strong); padding: 0 0 0 .7rem; margin-bottom: .35rem; font-size: 1.0625rem; font-weight: 650; } .section1-subtitle, .section2-subtitle, .section3-subtitle { margin-top: 1.5rem; font-size: 1.0625rem; font-weight: 650; } .section1-subsubtitle, .section2-subsubtitle, .section3-subsubtitle { color: var(--text-muted); font-size: 1rem; font-style: italic; } .section-intro { margin-top: .85rem; } .page-body:has(> .section-aside) { display: flex; flex-direction: column; } .section-aside { order: 1; } .section-after-aside { order: 2; } @media (min-width: 56rem) { .section-aside { float: right; width: 17rem; margin-left: 2.5rem; margin-bottom: 1.5rem; } .section-aside .section1-header { border-left-width: 3px; padding: .3rem .7rem; margin-bottom: .5rem; font-size: 1.1875rem; } .section-aside .section-body-text { margin-top: .6rem; font-size: .875rem; line-height: 1.55; } .section-aside + .section1 { margin-top: 1rem; } .section-aside .sponsor-list { grid-template-columns: minmax(0, 1fr); gap: 1.25rem; padding: 1.25rem 1rem; margin: .75rem 0; } .section1-header, .section2-header, .section3-header, .execute-title, .execute-body, .config-title, .config-body { display: flow-root; } .page-body:has(> .section-aside) { display: flow-root; } } .section-body-text { margin-top: 1rem; } .section-body-text:first-of-type { margin-top: .85rem; } ul.list-unordered { margin: .85rem 0 0 0; padding-left: 1.4rem; } li.list-unordered { margin-top: .45rem; padding-left: .25rem; } li.list-unordered::marker { color: var(--accent); } li.list-unordered:has(> .list-detail) { position: relative; list-style-type: none; } .list-detail-toggle { position: absolute; top: 0; left: -1.4rem; display: flex; align-items: center; justify-content: center; width: 1.3rem; height: 1.7em; border-radius: var(--radius); color: var(--text-muted); cursor: pointer; list-style: none; } .list-detail-toggle::-webkit-details-marker { display: none; } .list-detail-toggle:hover { background-color: var(--accent-soft); color: var(--accent-hover); } .list-detail-toggle:focus-visible { outline: 2px solid var(--accent); outline-offset: -2px; } .list-detail-toggle::before { content: ""; width: 6px; height: 6px; border-right: 1.5px solid currentColor; border-bottom: 1.5px solid currentColor; transform: translateX(-1px) rotate(-45deg); } .list-detail[open] > .list-detail-toggle::before { transform: translateY(-2px) rotate(45deg); } .list-detail:has(> .list-detail-text) > .list-detail-toggle { color: var(--accent); } .list-detail:has(> .list-detail-text) > .list-detail-toggle::before { width: 10px; height: 10px; border: none; background: linear-gradient(currentColor, currentColor) center / 100% 2px no-repeat, linear-gradient(currentColor, currentColor) center / 2px 100% no-repeat; transform: none; } .list-detail[open]:has(> .list-detail-text) > .list-detail-toggle::before { background: linear-gradient(currentColor, currentColor) center / 100% 2px no-repeat; } .list-detail[open] { margin: .35rem 0 .25rem 0; border-left: 2px solid var(--border); padding-left: .75rem; } .list-detail-text + .list-detail-text { margin-top: .5rem; } .list-detail-text + .list-detail-ref { margin-top: .4rem; } .list-detail-ref { color: var(--text-muted); font-size: .875rem; line-height: 1.6; } .list-detail-ref-item + .list-detail-ref-item::before { content: "\00b7"; margin: 0 .45em; } .list-detail-ref .id { border: none; background: none; padding: 0; font-size: .9375em; } .list-detail-all { display: block; margin-top: .5rem; border: 1px solid var(--border); border-radius: var(--radius); background-color: var(--bg-surface); padding: .1rem .65rem; color: var(--text-muted); font-family: inherit; font-size: .8125rem; line-height: 1.6; cursor: pointer; } .list-detail-all:hover { border-color: var(--border-strong); color: var(--text); } .list-detail-all:focus-visible { outline: 2px solid var(--accent); outline-offset: 1px; } .table { width: 100%; margin-top: 1.25rem; border: 1px solid var(--border); border-radius: var(--radius-lg); border-spacing: 0; overflow: hidden; font-size: .9375rem; } .table-caption { margin-bottom: .5rem; color: var(--text-muted); text-align: left; font-weight: 650; } .table-header-row { background: var(--bg-surface-2); } .table-header-left, .table-header-center, .table-header-right { border-bottom: 1px solid var(--border); padding: .5rem .85rem; color: var(--text-muted); white-space: nowrap; font-size: .75rem; font-weight: 700; letter-spacing: .07em; text-transform: uppercase; } .table-header-left { text-align: left; } .table-header-center { text-align: center; } .table-header-right { text-align: right; } .table-header-fill { width: 100%; } .table-row:nth-child(even) { background: var(--bg-surface); } .table-data-left, .table-data-center, .table-data-right { padding: .4rem .85rem; white-space: nowrap; } .table-data-left { text-align: left; } .table-data-center { text-align: center; } .table-data-right { text-align: right; font-variant-numeric: tabular-nums; } .execute, .config, .code-block { margin: 1.25rem 0 1.5rem 0; } .execute-title, .config-title { border: 1px solid var(--code-border); border-bottom: none; border-radius: var(--radius-lg) var(--radius-lg) 0 0; background: var(--code-output-bg); padding: .4rem .9rem; color: var(--code-text); font-size: .8125rem; } .execute-title .host, .config-title .host, .config-title .file { color: var(--code-prompt); } .code-copy { display: none; flex: none; margin-left: .6rem; border: 1px solid var(--code-border); border-radius: var(--radius); background-color: var(--code-bg); padding: .15rem .3rem; color: var(--code-text); cursor: pointer; } .code-copy-ready { display: block; opacity: 0; transition: opacity .15s ease; } .execute-body-cmd:hover .code-copy-ready, .config:hover .code-copy-ready, .code-copy-ready:focus-visible { opacity: 1; } @media (hover: none) { .code-copy-ready { opacity: 1; } } .code-copy:hover { border-color: var(--border-strong); background-color: var(--code-border); } .code-copy:before { content: ""; display: block; width: .9rem; height: .9rem; background-color: currentColor; -webkit-mask: var(--icon-copy) no-repeat center / contain; mask: var(--icon-copy) no-repeat center / contain; } .code-copy-done { opacity: 1; color: var(--tip-accent); } .code-copy-done:before { -webkit-mask-image: var(--icon-tick); mask-image: var(--icon-tick); } .code-copy-fail { opacity: 1; border-color: var(--warning-accent); color: var(--warning-accent); } .config-title { display: flow-root; } .config-title .code-copy { float: right; } .execute-body, .config-body, .code-block { border: 1px solid var(--code-border); border-radius: 0 0 var(--radius-lg) var(--radius-lg); background-color: var(--code-bg); padding: .7rem 0; color: var(--code-text); font-family: var(--font-mono); font-size: .8125rem; line-height: 1.6; unicode-bidi: embed; } .code-block { border-radius: var(--radius-lg); padding: .7rem .9rem; } pre, .execute-body-output, .config-body-output { overflow-x: auto; scrollbar-width: thin; scrollbar-color: var(--code-border) transparent; } pre { margin: 0; padding: 0; white-space: pre; } .execute-body-cmd { display: flex; align-items: flex-start; padding-right: .9rem; } .execute-cmd { flex: 1; min-width: 0; padding: .1rem .9rem; } .execute-cmd:before { content: "$ "; color: var(--code-prompt); } .execute-body-output, .config-body-output { margin: .4rem .9rem; border-radius: var(--radius); background-color: var(--code-output-bg); padding: .35rem .7rem; } .execute-line-list, .config-line-list { width: max-content; min-width: 100%; } .execute-line, .execute-line-highlight, .execute-line-highlight-error, .config-line, .config-line-add, .config-line-remove { overflow-x: visible; margin: 0 -.7rem; padding: 0 .7rem; } .execute-line-highlight { background-color: var(--code-highlight-bg); } .execute-line-highlight-error { background-color: var(--code-error-bg); } .config-line:before { content: " "; } .config-line-add { background-color: var(--code-highlight-bg); } .config-line-add:before { content: "+ "; } .config-line-remove { background-color: var(--code-error-bg); } .config-line-remove:before { content: "- "; } .admonition { display: flex; flex-direction: column; gap: .15rem; border: 1px solid var(--note-accent); border-left-width: 4px; border-radius: var(--radius); background-color: var(--note-bg); padding: .75rem 1rem; margin: 1.25rem 0; } .admonition > .note, .admonition > .warning, .admonition > .caution, .admonition > .important, .admonition > .tip { float: none; color: var(--note-accent); font-size: .75rem; font-weight: 700; letter-spacing: .07em; line-height: 1.4; } .admonition > .note-text, .admonition > .warning-text, .admonition > .caution-text, .admonition > .important-text, .admonition > .tip-text { overflow: visible; min-width: 0; } .admonition:has(> .tip) { border-color: var(--tip-accent); background-color: var(--tip-bg); } .admonition:has(> .tip) > .tip { color: var(--tip-accent); } .admonition:has(> .important) { border-color: var(--important-accent); background-color: var(--important-bg); } .admonition:has(> .important) > .important { color: var(--important-accent); } .admonition:has(> .caution) { border-color: var(--caution-accent); background-color: var(--caution-bg); } .admonition:has(> .caution) > .caution { color: var(--caution-accent); } .admonition:has(> .warning) { border-color: var(--warning-accent); background-color: var(--warning-bg); } .admonition:has(> .warning) > .warning { color: var(--warning-accent); } .sponsor-list { display: grid; grid-template-columns: repeat(auto-fit, minmax(12rem, 1fr)); justify-items: center; align-items: center; gap: 1.75rem 2rem; border: 1px solid var(--border); border-radius: var(--radius-lg); background: var(--bg-surface); padding: 2rem 1.5rem; margin: 1.75rem 0; } .sponsor { text-align: center; } .sponsor a { display: inline-block; opacity: .78; transition: opacity .15s ease; } .sponsor a:hover { opacity: 1; } .sponsor-img { max-width: 100%; height: auto; } .sponsor-img-dark { display: none; } @media (prefers-color-scheme: dark) { .sponsor-img-light { display: none; } .sponsor-img-dark { display: inline; } } .backrest, .cmd, .postgres, .host, .proper { font-weight: var(--keyword-weight); } .path, .br-option, .br-setting, .pg-option, .pg-setting, .id, .user, .file { border: 1px solid var(--keyword-border); border-radius: 4px; background-color: var(--keyword-bg); padding: .05em .35em; font-family: var(--font-mono); font-size: .9em; unicode-bidi: embed; } .path, .br-option, .br-setting, .pg-option, .pg-setting, .user, .file { white-space: nowrap; } .execute-title .path, .execute-title .file, .execute-title .id, .execute-title .br-option, .config-title .path, .config-title .file, .config-title .id, .config-title .br-option { border: none; background: none; padding: 0; font-size: 1em; } .page-footer { border-top: 1px solid var(--border); margin-top: 1.5rem; padding-top: .75rem; padding-bottom: 1.125rem; color: var(--text-muted); text-align: center; font-size: .8125rem; } @media (max-width: 40rem) { body { font-size: .9375rem; line-height: 1.65; } .page-header { padding-top: 1.5rem; padding-bottom: .5rem; } .page-header-title { font-size: 1.625rem; } .page-header-subtitle { flex-basis: 100%; } .section2-header { font-size: 1.1875rem; } .page-toc-body { padding: 1rem; } .table { display: block; overflow-x: auto; } } ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792312410.777019 pgbackrest-2.60.0/doc/html/default.js0000644000175100017510000002026415265102133017067 0ustar00runnerrunner"use strict"; (function() { var TOC_LEVEL_LIST = [".section1-toc", ".section2-toc", ".section3-toc"]; var column = document.querySelector(".page-toc"); var toc = document.querySelector(".page-toc-inner") || column; var bar = document.querySelector(".page-menu"); if (toc === null) return; var linkMap = Object.create(null); var linkList = toc.querySelectorAll("a[href^='#']"); for (var linkIdx = 0; linkIdx < linkList.length; linkIdx++) { var id = decodeURIComponent(linkList[linkIdx].getAttribute("href").substring(1)); if (id !== "") linkMap[id] = linkList[linkIdx]; } var anchorList = []; var anchorAll = document.querySelectorAll(".page-body a[id]"); for (var anchorIdx = 0; anchorIdx < anchorAll.length; anchorIdx++) { if (anchorAll[anchorIdx].id in linkMap) anchorList.push(anchorAll[anchorIdx]); } if (anchorList.length === 0) return; function markAdd(link) { link.classList.add("toc-active"); var node = link.closest(TOC_LEVEL_LIST.join(", ")); while (node !== null && node.parentElement !== null) { node = node.parentElement.closest(TOC_LEVEL_LIST.join(", ")); if (node !== null) { var parent = node.querySelector(":scope > [class$='-toc-title'] > a"); if (parent !== null) parent.classList.add("toc-active-parent"); } } } function fit() { if (window.getComputedStyle(toc).position !== "sticky") { toc.style.maxHeight = ""; return; } var barBox = bar === null ? null : bar.getBoundingClientRect(); var top = Math.max(barBox === null ? 0 : barBox.height, column.getBoundingClientRect().top); toc.style.maxHeight = Math.max(0, window.innerHeight - top) + "px"; } function markShow(link) { if (toc.scrollHeight <= toc.clientHeight) return; var linkBox = link.getBoundingClientRect(); var tocBox = toc.getBoundingClientRect(); if (linkBox.top < tocBox.top || linkBox.bottom > tocBox.bottom) toc.scrollTop += linkBox.top - tocBox.top - toc.clientHeight / 3; } var active = null; var pending = false; var edge = null; function mark() { pending = false; fit(); if (edge === null) edge = parseFloat(window.getComputedStyle(anchorList[0]).scrollMarginTop) + 1; var found = anchorList[0]; for (var idx = 0; idx < anchorList.length; idx++) { if (anchorList[idx].getBoundingClientRect().top > edge) break; found = anchorList[idx]; } if (window.innerHeight + window.scrollY >= document.documentElement.scrollHeight - 2) found = anchorList[anchorList.length - 1]; if (found === active) return; active = found; var markList = toc.querySelectorAll(".toc-active, .toc-active-parent"); for (var markIdx = 0; markIdx < markList.length; markIdx++) markList[markIdx].classList.remove("toc-active", "toc-active-parent"); markAdd(linkMap[found.id]); markShow(linkMap[found.id]); } function schedule() { if (pending) return; pending = true; window.requestAnimationFrame(mark); } function resize() { edge = null; schedule(); } window.addEventListener("scroll", schedule, {passive: true}); window.addEventListener("resize", resize, {passive: true}); mark(); })(); (function() { var blockList = document.querySelectorAll("pre[tabindex], .execute-body-output[tabindex], .config-body-output[tabindex]"); if (blockList.length === 0) return; var pending = false; function reach() { pending = false; for (var idx = 0; idx < blockList.length; idx++) { if (blockList[idx].scrollWidth > blockList[idx].clientWidth + 1) blockList[idx].setAttribute("tabindex", "0"); else blockList[idx].removeAttribute("tabindex"); } } function schedule() { if (pending) return; pending = true; window.requestAnimationFrame(reach); } window.addEventListener("resize", schedule, {passive: true}); reach(); })(); (function() { var bar = document.querySelector(".page-menu"); var menu = bar === null ? null : bar.querySelector(".menu-body"); if (menu === null) return; var pending = false; function fade() { pending = false; bar.classList.toggle("page-menu-fade-left", menu.scrollLeft > 1); bar.classList.toggle("page-menu-fade-right", menu.scrollLeft + menu.clientWidth < menu.scrollWidth - 1); } function schedule() { if (pending) return; pending = true; window.requestAnimationFrame(fade); } menu.addEventListener("scroll", schedule, {passive: true}); window.addEventListener("resize", schedule, {passive: true}); fade(); })(); (function() { var SAID_TIME = 2000; var buttonList = document.querySelectorAll(".code-copy"); if (buttonList.length === 0 || navigator.clipboard === undefined) return; function copyText(button) { var block = button.closest(".execute-body-cmd, .config"); var lineList = block.querySelectorAll(".execute-cmd, .config-line, .config-line-add"); var result = ""; for (var idx = 0; idx < lineList.length; idx++) result += lineList[idx].textContent + "\n"; return result; } function ready(button) { var timeout = null; function reset() { button.classList.remove("code-copy-done", "code-copy-fail"); timeout = null; } function said(state) { if (timeout !== null) window.clearTimeout(timeout); reset(); button.classList.add(state); timeout = window.setTimeout(reset, SAID_TIME); } function done() { said("code-copy-done"); } function fail() { said("code-copy-fail"); } function copy() { navigator.clipboard.writeText(copyText(button)).then(done, fail); } button.addEventListener("click", copy); button.classList.add("code-copy-ready"); } for (var buttonIdx = 0; buttonIdx < buttonList.length; buttonIdx++) ready(buttonList[buttonIdx]); })(); (function() { var SECTION = ".section1, .section2, .section3"; var sectionList = document.querySelectorAll(SECTION); function ready(section, detailList) { var button = document.createElement("button"); var open = false; button.type = "button"; button.className = "list-detail-all"; function label() { button.textContent = open ? "Hide details" : "Show details"; } function toggle() { open = !open; for (var idx = 0; idx < detailList.length; idx++) detailList[idx].open = open; label(); } button.addEventListener("click", toggle); label(); var body = section.querySelector(":scope > .section-body"); var subtitleList = body.querySelectorAll(":scope > [class$='subtitle']"); if (subtitleList.length > 0) subtitleList[subtitleList.length - 1].after(button); else body.prepend(button); } for (var sectionIdx = 0; sectionIdx < sectionList.length; sectionIdx++) { var section = sectionList[sectionIdx]; var detailAll = section.querySelectorAll(".list-detail"); var detailList = []; for (var detailIdx = 0; detailIdx < detailAll.length; detailIdx++) { if (detailAll[detailIdx].closest(SECTION) === section) detailList.push(detailAll[detailIdx]); } if (detailList.length > 0) ready(section, detailList); } })(); ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7771838 pgbackrest-2.60.0/doc/html/faq.html0000644000175100017510000003646715265102133016556 0ustar00runnerrunner pgBackRest - Frequently Asked Questions
pgBackRest
Frequently Asked Questions
Home
User Guides
Releases
Configuration
Commands
FAQ
Table of Contents
Introduction
What if I get the could not find WAL segment error?
How do I manually purge a backup set?
How can I configure options independently for each command?
Can I use dots (periods) in my S3 bucket name?
Where can I find packages for older versions of pgBackRest?
Why does a backup attempt fail when backup-standby=y and the standby database is down?
Should I setup my repository on a standby host?
Time-based Point-in-Time Recovery does not appear to work, why?
What does the WAL archive suffix mean?
Does it take longer to restore specific backup types (full, differential, incremental)?
How can I export a backup for use in a network-isolated environment?
Frequently Asked Questions are intended to provide details for specific questions that may or may not be covered in the User Guide, Configuration, or Command reference. If you are unable to find details for your specific issue here, remember that the pgBackRest Issues List in GitHub is also a valuable resource.
What if I get the could not find WAL segment error?
The cause of this error can be a result of many different issues, some of which may be:
  • misconfigured archive_command
  • misconfigured pgBackRest configuration files
  • network or permissions issue
  • third party product (e.g. S3, Swift or Minio) configuration issue
  • large amount of WAL queueing to be archived
It is advisable to:
  • check the archive_command in PostgreSQL
  • check the pgBackRest configuration settings on each host (e.g. pg* settings are set on the repository host and repo* settings on the pg host)
  • run the check command with --archive-timeout set to a higher value than in the pgBackRest configuration file (or default) to see if the WAL queue needs more time to clear. If the system is generating a lot of WAL, then consider configuring asynchronous archiving
How do I manually purge a backup set?
A full backup set can be expired using the --set option as explained in Command Reference: Expire.
How can I configure options independently for each command?
pgBackRest has the ability to set options independently in the configuration file for each command. Configure Cluster Stanza details this feature as well as option precedence.
For example, the process-max option can be optimized for each command: 
[global]
# used where not overridden
process-max=2

[global:backup]
# more cores for backup
process-max=4

[global:restore]
# all the cores for restore
process-max=8

[global:archive-push]
# more cores for archive-push
process-max=3

[global:archive-get]
# fewer cores for archive-get
process-max=1
Can I use dots (periods) in my S3 bucket name?
RFC-2818 does not allow wildcards to match on a dot (.) so s3 bucket names must not contain dots. If there are dots in the S3 bucket name then an error such as unable to find hostname 'my.backup.bucket.s3.amazonaws.com' in certificate common name or subject alternative names will occur.
The exception is repo-s3-uri-style=path, which prepends the bucket to the URI instead of the host. Since the bucket is not part of the host name, the certificate is matched against the endpoint alone and dots in the bucket name are not a problem. Note that path-style URIs are not supported by all S3-compatible object stores.
Where can I find packages for older versions of pgBackRest?
The apt.postgresql.org repository maintains an archive of older versions. Debian also maintains snapshots of all test builds.
Why does a backup attempt fail when backup-standby=y and the standby database is down?
Configuring backup from standby is generally intended to reduce load on the primary, so switching backups to the primary when the standby is down often defeats the point. Putting more load on the primary in a situation where there are already failures in the system is not recommended. Backups are not critical as long as you have one that is fairly recent -- the important thing is to keep up with WAL archiving. There is plenty of time to get a backup when the system is stable again.
If you really need a backup, the solution is to have more standbys or remove backup-standby. This can be overridden on the command line with --no-backup-standby, so there is no need to reconfigure for a one-off backup.
Should I setup my repository on a standby host?
No. When primary and standby databases are configured, the pgBackRest configuration files should be symmetric in order to seamlessly handle failovers. If they are not, the configurations will need to be changed on failover or further problems may result.
See the Dedicated Repository Host section of the User Guide for more information.
Time-based Point-in-Time Recovery does not appear to work, why?
The most common mistake when using time-based Point-in-Time Recovery is forgetting to choose a backup set that is before the target time. pgBackRest will attempt to discover a backup to play forward from the time specified by the --target= if the --set option is not specified. If a backup set cannot be found, then restore will default to the latest backup. However, if the latest backup is after the target time, then --target= is not considered valid by PostgreSQL and is therefore ignored, resulting in WAL recovery to the latest time available.
To use the --set option, choose a backup set by running the info command and finding the backup with a timestamp stop that is before the target time. Then when running the restore, specify the option --set=BACKUP_LABEL where BACKUP_LABEL is the chosen backup set.
See the Point-in-Time Recovery section of the User Guide for more information.
What does the WAL archive suffix mean?
The suffix is the SHA1 checksum used to verify file integrity. There is no way to omit it.
Does it take longer to restore specific backup types (full, differential, incremental)?
The various backup types require the same amount of time to restore. Restore retrieves files based on the backup manifest, which may reference files from a previous backup in the case of incremental or differential backups. While there could be differences in time spent making a given backup (depending on backup type), database size determines restore time (disk I/O, network I/O, etc. being equal).
How can I export a backup for use in a network-isolated environment?
pgBackRest uses the repository not only to store backups and WAL archives but also to maintain essential metadata required for features such as compression, encryption, and file bundling. Because of this, simply copying a backup along with a subset of WAL files usually will not work unless very specific and restrictive conditions are met.
However, there is a workaround if your goal is to create a self-contained export of a database that you can transfer (e.g., via USB). You can make a backup with the --archive-copy option enabled to ensure that the necessary WAL segments are stored along with the backup. Then, restore it using --type=none --pg1-path=/your/target/path. This produces a restored PostgreSQL data directory with all required WAL files already placed in pg_wal, similar to what pg_basebackup would create.
You can then copy this directory to another system, and PostgreSQL should be able to recover from it without needing access to the pgBackRest repository.
Please note that recovering this backup will not result in a timeline switch, which means that this cluster should not push WAL to the original repository that it was exported from. If the new cluster is in a network-isolated environment this should not be a problem.
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated October 18, 2026
././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7771995 pgbackrest-2.60.0/doc/html/index.html0000644000175100017510000004255715265102133017113 0ustar00runnerrunner pgBackRest - Reliable PostgreSQL Backup & Restore
pgBackRest
Reliable PostgreSQL Backup & Restore
Home
User Guides
Releases
Configuration
Commands
FAQ
pgBackRest is a reliable backup and restore solution for PostgreSQL that seamlessly scales up to the largest databases and workloads.
pgBackRest v2.60.0 is the current stable release. Release notes are on the Releases page.
Please give us a star on GitHub if you like pgBackRest!
Features
Parallel Backup & Restore
Compression is usually the bottleneck during backup operations so pgBackRest solves this problem with parallel processing and more efficient compression algorithms such as lz4 and zstd.
Local or Remote Operation
A custom protocol allows pgBackRest to backup, restore, and archive locally or remotely via TLS/SSH with minimal configuration. An interface to query PostgreSQL is also provided via the protocol layer so that remote access to PostgreSQL is never required, which enhances security.
Multiple Repositories
Multiple repositories allow, for example, a local repository with minimal retention for fast restores and a remote repository with a longer retention for redundancy and access across the enterprise.
Backup from Standby
Backups can be made from a standby to reduce load on the primary. Only the few files that are not replicated are copied from the primary.
Full, Differential, & Incremental Backups (at File or Block Level)
Full, differential, and incremental backups are supported. pgBackRest is not susceptible to the time resolution issues of rsync, making differential and incremental backups safe without the requirement to checksum each file. Block-level backups save space by only copying the parts of files that have changed.
Backup Rotation & Archive Expiration
Retention policies can be set for full and differential backups to create coverage for any time frame. The WAL archive can be maintained for all backups or strictly for the most recent backups. In the latter case WAL required to make older backups consistent will be maintained in the archive.
Backup Integrity
Checksums are calculated for every file in the backup and rechecked during a restore or verify. After a backup finishes copying files, it waits until every WAL segment required to make the backup consistent reaches the repository.
Backups in the repository may be stored in the same format as a standard PostgreSQL cluster (including tablespaces). If compression is disabled and hard links are enabled it is possible to snapshot a backup in the repository and bring up a PostgreSQL cluster directly on the snapshot. This is advantageous for terabyte-scale databases that are time consuming to restore in the traditional way.
All operations utilize file and directory level fsync to ensure durability.
Page Checksums
If page checksums are enabled pgBackRest will validate the checksums for every file that is copied during a backup. All page checksums are validated during a full backup and checksums in files that have changed are validated during differential and incremental backups.
Validation failures do not stop the backup process, but warnings with details of exactly which pages have failed validation are output to the console and file log. By default restore errors when the backup has invalid page checksums and expire will not leave a backup with invalid page checksums as the oldest backup.
This feature allows page-level corruption to be detected early, before backups that contain valid copies of the data have expired.
Backup Resume
An interrupted backup can be resumed from the point where it was stopped. Files that were already copied are compared with the checksums in the manifest to ensure integrity. Since this operation can take place entirely on the repository host, it reduces load on the PostgreSQL host and saves time since checksum calculation is faster than compressing and retransmitting data.
Streaming Compression & Checksums
Compression and checksum calculations are performed in stream while files are being copied to the repository, whether the repository is located locally or remotely.
If the repository is on a repository host, compression is performed on the PostgreSQL host and files are transmitted in a compressed format and simply stored on the repository host. When compression is disabled a lower level of compression is utilized to make efficient use of available bandwidth while keeping CPU cost to a minimum.
File Bundling
Small files are combined into bundles in the repository so fewer files are created during backup. This is especially effective on object stores.
Delta Restore
The manifest contains checksums for every file in the backup so that during a restore it is possible to use these checksums to speed processing enormously. On a delta restore any files not present in the backup are first removed and then checksums are generated for the remaining files. Files that match the backup are left in place and the rest of the files are restored as usual. Parallel processing can lead to a dramatic reduction in restore times.
Parallel, Asynchronous WAL Push & Get
Dedicated commands are included for pushing WAL to the archive and getting WAL from the archive. Both commands support parallelism to accelerate processing and run asynchronously to provide the fastest possible response time to PostgreSQL.
WAL push automatically detects WAL segments that are pushed multiple times and de-duplicates when the segment is identical, otherwise an error is raised. Asynchronous WAL push allows transfer to be offloaded to another process which compresses WAL segments in parallel for maximum throughput. This can be a critical feature for databases with extremely high write volume.
Asynchronous WAL get maintains a local queue of WAL segments that are decompressed and ready for replay. This reduces the time needed to provide WAL to PostgreSQL which maximizes replay speed. Higher-latency connections and storage (such as S3) benefit the most.
The push and get commands both ensure that the database and repository match by comparing PostgreSQL versions and system identifiers. This virtually eliminates the possibility of misconfiguring the WAL archive location.
Tablespace & Link Support
Tablespaces are fully supported and on restore tablespaces can be remapped to any location. It is also possible to remap all tablespaces to one location with a single command which is useful for development restores.
File and directory links are supported for any file or directory in the PostgreSQL cluster. When restoring it is possible to restore all links to their original locations, remap some or all links, or restore some or all links as normal files or directories within the cluster directory.
S3, Azure, GCS, and SFTP Support
pgBackRest repositories can be located in S3, Azure, and GCS compatible object stores to allow for virtually unlimited capacity and retention. Repositories can also be stored on an SFTP server. Object store credentials can be retrieved automatically in cloud environments using a variety of authentication methods.
Encryption
pgBackRest can encrypt the repository to secure backups wherever they are stored.
Ransomware & Malware Protection
When the repository is stored on versioned object storage, pgBackRest can read the repository as it was at a point-in-time. If backups are deleted or corrupted by accident, malware, or ransomware, a target time can be used to recover data from before the damage occurred.
Versioning is supported by S3, Azure, and GCS compatible object stores. Object locking for S3 and soft delete for GCS or Azure can provide additional protection against tampering.
Compatibility with ten versions of PostgreSQL
pgBackRest includes support for ten versions of PostgreSQL, the five supported versions and the last five EOL versions. This allows ample time to upgrade to a supported version.
Getting Started
pgBackRest strives to be easy to configure and operate:
Contributions
Contributions to pgBackRest are always welcome! Please see our Contributing Guidelines for details on how to contribute features, improvements, or issues.
Support
pgBackRest is completely free and open source under the MIT license. You may use it for personal or commercial purposes without any restrictions whatsoever. Bug reports are taken very seriously and will be addressed as quickly as possible. Please report bugs here.
Creating a robust disaster recovery policy with proper replication and backup strategies can be a very complex and daunting task. You may find that you need help during the architecture phase and ongoing support to ensure that your enterprise continues running smoothly.
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated October 18, 2026
././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792312410.777054 pgbackrest-2.60.0/doc/html/logo.svg0000644000175100017510000000200715265102133016561 0ustar00runnerrunner ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7772174 pgbackrest-2.60.0/doc/html/release.html0000644000175100017510000234455215265102133017426 0ustar00runnerrunner pgBackRest - Releases
pgBackRest
Releases
Home
User Guides
Releases
Configuration
Commands
FAQ
pgBackRest release numbers consist of three parts: major, minor, and patch. A major release may break compatibility with the prior major release, but v2 releases are fully compatible with v1 repositories and will accept all v1 options. Minor releases can include bug fixes and features but do not change the repository format and strive to avoid changing options and naming. Patch releases contain bug fixes for the most recent minor release.

Feature releases are scheduled for the third Monday of January, April, July, and October. Patch releases are scheduled for the third Monday of any other month as required.

Documentation for the v1 release can be found here.

The notes for a release may also contain Additional Notes but changes in this section are only to documentation or the test suite and have no direct impact on the pgBackRest codebase.
Current Stable Release
v2.60.0 Release Notes
Page Checksum Errors and Object Store Prefetch
Released October 18, 2026
IMPORTANT NOTE: The restore command will now error and stop if any restored file contains an invalid page checksum, which leaves the cluster incomplete and unable to start, so the restore must be rerun. The expire command will now refuse to expire if the oldest retained backup would contain checksum errors, which also prevents archive from being expired in that repository. Set checksum-page-error=n to warn instead of erroring.
Features:
  • Add checksum-page-error option to error on invalid page checksums. (Contributed by Andrew Pogrebnoi. Reviewed by Douglas J Hunley, David Steele, Greg Sabino Mullane, Stefan Fercot. Suggested by Greg Sabino Mullane.)
    This option applies to the backup, restore, and expire commands. The backup command defaults to a warning for each file (as before) and now also reports the total number of checksum warnings at the end of the backup. The restore and expire commands default to an error. Previously the restore command did not report invalid page checksums at all.
    Issue #2687Pull request #2833Commit d7f54b2
  • Support for GCS Workload Identity Federation. (Contributed by Adol Rodriguez. Reviewed by Douglas J Hunley, David Steele.)
    Set repo-gcs-key-type=web-id and set repo-gcs-key to an external account credential file, e.g. as generated by gcloud iam workload-identity-pools create-cred-config. The web identity token is exchanged for an access token without a service account key. The token file is reloaded on each exchange to pick up rotated tokens, e.g. Kubernetes projected service account tokens. Service account impersonation is not supported.
    Issue #2838Pull request #2839Commit 1d217e6
  • Prefetch multi-file/range reads from object stores. (Reviewed by Andrew Pogrebnoi, Douglas J Hunley.)
    Requests are queued ahead of the current read so requests proceed while reading, and small gaps between files are read over instead of starting a new request. Restore uses this to read bundled files and block incremental maps, and backup uses it to read block incremental maps from the prior backup. repo-storage-prefetch (default 3) sets how many requests are queued and repo-storage-read-over (default 64KiB) sets the largest gap that is read over.
    e63dec3 Prefetch multi-file/range reads from object stores.Pull request #2783
    5ae4015 Combine prior block incremental map reads during backup.Pull request #2816
Improvements:
  • Reduce keep-alive requests to remote processes. (Reviewed by Douglas J Hunley.)
    A keep-alive was sent to every remote each time a job completed in archive-push and backup, even when the remotes had just been sent a request. A keep-alive is now sent only when no request has been sent to the remote for 30 seconds.
    Commit f652261
  • Retry Azure batch sub-requests missing from the response. (Reviewed by Douglas J Hunley.)
    When Azure fails a batch delete it may return a single response for the entire batch. The sub-requests with no response were treated as complete, so only one file per batch was removed while the command reported success. Sub-requests with no response are now retried.
    Issue #2840Pull request #2849Commit 567a245
  • Reduce the manifest checks made when reconstructing backup.info. (Reviewed by Douglas J Hunley. Suggested by todeb.)
    The backup, expire, and annotate commands no longer check for a backup.manifest when the repository listing shows the backup path does not exist, or when a diff or incr backup belongs to a full backup set that is not in backup.info. Each check is a request so on an object store the check for a missing manifest is logged as a 404.
    Issue #2769Pull request #2825Commit 0d3c340
  • Use original size to determine if a file should be bundled. (Reviewed by Douglas J Hunley.)
    The size of a block incremental file in the prior backup was used to decide whether to bundle it, so a file that had grown past repo-bundle-limit could still be stored in a bundle. The current size of the file is now used.
    Pull request #2843Commit 340d58b
  • Add context to block map read errors during backup. (Reviewed by Stefan Fercot, Douglas J Hunley.)
    An error reading the block map of a prior backup now reports the file, repository path, offset, and map size. A read that returns fewer bytes than expected is reported as such instead of as a parse error.
    bc91688 Add context to block map read errors during backup.Issue #2489Issue #2829Pull request #2830
    f46bfaf Fix option name in block incremental map errors.
  • Cast current_timestamp to text in the backup start label. (Contributed by Tiago Corcelli de Oliveira. Reviewed by David Steele.)
    A user-defined implicit cast from timestamptz to text caused the backup to fail with operator is not unique: unknown || timestamp with time zone. The label text is unchanged.
    Issue #2873Commit 86936e1
  • Support building on illumos. (Contributed by Kalevi Kolttonen. Reviewed by Douglas J Hunley, David Steele.)
    An OmniOS job has been added to CI. Solaris may also work but has not been tested.
    Pull request #2870Commit ad72e58
  • Support OpenSSL 4. (Reviewed by Douglas J Hunley.)
    Building against OpenSSL 4 failed because X509_get_subject_name() now returns a const name.
    Commit a729ac8
  • Fix clang builds. (Reviewed by Douglas J Hunley.)
    On Linux, gcc installs libbacktrace.a where clang also looks for it but installs backtrace.h where only gcc can find it. A clang build found the library and then failed to compile. The library is now only used when its header is found. Return types that clang reports as narrowing conversions in debug builds are also fixed.
    84b031a Require the header when checking for libbacktrace.
    195e66b Return the declared type from functions that return a StringId-based enum.
    d9e5e14 Return bool from a function declared to return bool.
  • Migrate code generation from C to Python.
    Code is generated during the build from the option, error, help, and PostgreSQL declarations by a Python tool instead of a C tool that had to be built first. Building from a git checkout requires PyYAML instead of libyaml. The release tarball includes the generated code and requires neither.
    Commit 59bd2cd
Documentation Improvements:
  • Modernize the website. Special thanks to Jan for the insightful (and hilarious) logo. (Contributed by David Steele, Jan Wieremjewicz. Reviewed by Steve Atkins, Kai Wagner, Douglas J Hunley, Nikolaos Stamatopoulos, Matt Hudson, Stefan Fercot.)
    The website has a new design with light and dark themes and a layout that works on phones. Pages with a table of contents show it beside the text and mark the section being read. Commands and config files can be copied with a button, config files mark the lines each section changes, and release notes can be expanded to show more detail.
    5bc14b5 Shorten and disambiguate the documentation titles.
    b668d71 Move the sponsors down the index and shorten the appeal.
    60b85c5 Modernize the website.
    ab8df23 Show a card rather than the logo where a link is posted.
    76ae494 Remove the contents from the releases page and close up the date.
    3e86c71 Link rather than repeat the options in the command reference.
    dc3af7b Clarify the command titles in the command reference.
    078d52c Mark the lines a section changes in a config file.
    bd33623 Mark the output of a command the way a config file is marked.
    578fc2d Copy a command as well as a config file.
    1fb93f9 Error when a config section changes nothing the reader can see.
    2f3b2f2 Make the site logo a link home.
    8f902c6 Speed up the documentation build.
    b56b6ab List the most recent news items on the index automatically.
    5b8987d Render option type examples as code blocks in the reference.
    b5d5274 Scroll the menu sideways rather than wrapping it out of view.
    9ae2c32 Highlight the current page in the menu rather than leaving it out.
    b8da5b0 Version the style and script links so stale caches are refetched.
    ec57431 Jump straight to a section rather than gliding to it.
    46babed Fade an end the menu carries on past rather than cutting it flat.
    17de501 Put the bar of a section2 header under the title rather than beside it.
    940f431 Center the page title beside the logo and left align it when wrapped.
    050ca9e Add expandable details to the release notes.
    bd8f99b Add release note details for v2.33 through v2.60.
    28a1e3d Adjust keyword styling in light mode.
    1911f5f End the release note history at upstream integration in CI.
  • Document the release schedule.
    Feature releases are scheduled for the third Monday of January, April, July, and October. Patch releases are scheduled for the third Monday of any other month as required. The release notes introduction also describes the major, minor, and patch parts of a release number.
    9bca69a Document the release schedule.
    c179f3f Update release numbering in the release notes intro.
  • Migrate the documentation tools from C and Perl to Python.
    The documentation is built by doc/doc.py instead of a C tool run by doc/doc.pl, and building the documentation no longer requires Perl.
    59bd2cd Migrate the build and documentation code from C to Python.
    fa3d197 Migrate the remaining documentation code from Perl to Python.
Test Suite Improvements:
  • Migrate the test harness from C and Perl to Python.
    Tests are run with test/test.py instead of test/test.pl. The test containers and CI runners no longer require Perl and the harness is covered by tests.
    163e5ba Migrate the test harness from C to Python.
    e25c662 Migrate the rest of the test harness from Perl to Python.
  • Add a Fil-C job to CI.
    Fil-C is a memory-safe implementation of C that checks every pointer access. It found two out-of-bounds reads that gcc, clang, and Valgrind missed. A new workflow also builds and tests on the CI runner with gcc and clang, with and without the optional libraries.
    f32a5d5 Add a workflow to compile and test on the runner without a container.
    12396fb Add a Fil-C job to the compile workflow.
    908a11a Use the Fil-C distribution as-is now that it ships every library needed.
  • Add fuzz testing for the JSON, Ini, and Pack readers.
    The readers are fuzzed with libFuzzer, AddressSanitizer, and UndefinedBehaviorSanitizer in CI. Fuzzing found several bugs in how invalid input is read, e.g. deeply nested JSON arrays in an info file or an object store response could overflow the memory context stack.
    Commit 4c65010
  • Allow test user names that adduser on Debian rejects. (Contributed by Stefan Fercot. Reviewed by David Steele.)
    The test container is built with the same user name as the host user. Debian's adduser rejects names that contain a dot, e.g. first.last, so the container build failed on Debian-based images for these users.
    Issue #2858Pull request #2859Commit ce19dae
Stable Releases
v2.59.3 Release Notes
Weak Encryption Subkey Bug
Released October 4, 2026
IMPORTANT NOTE: In prior versions it was possible for weak encryption subkeys and salts to be generated if there was an error accessing system random data. See Weak Encryption Subkeys and Salts for more information.
Bug Fixes:
  • Fix possible checksum error loading encrypted info and manifest files. (Fixed by j0uinim. Reviewed by David Steele.)
    A large encrypted info or manifest file could lose its last lines when loaded, which failed the checksum. The lost lines had to fit in the final cipher block, so this is unlikely in practice.
    Issue #2866Pull request #2868Commit 2def063
  • Fix possible weak encryption subkeys/salts. (Fixed by j0uinim. Reviewed by David Steele, Stefan Fercot, Douglas J Hunley.)
    When OpenSSL could not get random data from the operating system the failure was ignored, and subkeys and salts were generated from memory that was not random. An error is now reported.
    Issue #2866Pull request #2867Commit 47ed78f
v2.59.2 Release Notes
PostgreSQL 19beta4 Support
Released September 27, 2026
Bug Fixes:
  • Fix empty block map when file is truncated to zero during backup. (Reviewed by Douglas J Hunley. Reported by JPCOSTA78.)
    A block incremental file larger than repo-bundle-limit and truncated to zero by PostgreSQL during a backup was stored as a block map with no blocks. Every later backup that referenced the file failed with unexpected eof until the next full backup.
    Issue #2829Pull request #2842Commit 572d163
  • Disable read retry when filters are passed to the driver. (Reviewed by Douglas J Hunley. Reported by megamaced.)
    A retried read from a repository on an object store, accessed through a repository host, crashed. These reads are no longer retried by the client. The repository host still retries its read from the object store.
    Issue #2845Pull request #2846Commit c6eca28
  • Fix pagination of S3 version listing. (Reviewed by Douglas J Hunley, guruguruguru. Reported by guruguruguru.)
    Reading a repository at a target time with repo-target-time failed with unable to find child 'NextContinuationToken' when more than a thousand versions were listed under a prefix.
    Issue #2854Pull request #2855Commit 6f0490d
  • Set the Azure version header for SAS authentication. (Reviewed by Douglas J Hunley. Reported by Florian Helmberger.)
    Batch deletes require the version header, so with SAS authentication every batch was rejected. The rejection looked like a single failed file, and the rest of each batch was not deleted while expire reported success, leaving expired backups and WAL in the repository.
    Issue #2840Pull request #2848Commit ea3293b
  • Fix async archive-push stalling on a zero-size backup history file. (Fixed by Andrew Kroh. Reviewed by David Steele, Douglas J Hunley.)
    A zero-size backup history file, e.g. in a cluster copied from a storage snapshot, failed every asynchronous archive-push run, so no WAL was archived. archive-push-batch-size is now converted to a segment count using the size of the first ready WAL segment.
    Issue #2856Pull request #2857Commit 12b1b3c
  • Fix annotate command on a remote encrypted repository. (Fixed by Douglas J Hunley. Reviewed by David Steele.)
    When the repository was encrypted and the cipher was configured only on the repository host, annotate failed to load backup.info.
    Commit 497ad5a
Features:
  • PostgreSQL 19beta4/rc1 support.
    PostgreSQL 19beta4 reverted online page checksum changes, so the page checksum version is again limited to 0 and 1. It also changed the pg_control version and the WAL page magic number.
    Commit 89eae83
v2.59.1 Release Notes
PostgreSQL 19beta3 Support
Released August 17, 2026
Bug Fixes:
  • Fix hang when the chunk buffer is smaller than the input buffer. (Reviewed by Andrew Pogrebnoi, Douglas J Hunley. Reported by crajac66.)
    A multi-part upload to S3, GCS, or Azure could loop forever when the chunk buffer was smaller than the input buffer, e.g. after the chunk size grew during the upload. The chunk buffer is now always large enough to hold the input buffer.
    Issue #2818Pull request #2819Commit a53e470
  • Compare a new backup label only against its own full backup set. (Reviewed by Douglas J Hunley. Reported by Anton Glushakov.)
    Some object stores keep listing a prefix after everything under it has been deleted, e.g. MinIO with object locking or SeaweedFS, so a full backup that was removed could still appear in the repository. Every diff and incr backup after it then failed with new backup label ... is not later than latest backup label .... A new label is now compared only against the backups in its own full backup set.
    Issue #2792Pull request #2824Commit d01f77e
  • Fix source archives generated by GitHub missing files required to build. (Reported by aardvarkzed.)
    The source archives GitHub generates for a release were missing the code generation tooling and the files it reads, so they could not be built. Only the distribution tarball is trimmed now.
    Issue #2821Commit 163670a
Features:
  • PostgreSQL 19beta3 support. (Contributed by Lardière Sébastien. Reviewed by David Steele.)
    PostgreSQL 19beta3 changed the WAL page magic number.
    Commit 5cbc256
Documentation Improvements:
  • Document exception for dots in S3 bucket names when repo-s3-uri-style=path.
    With repo-s3-uri-style=path the bucket is part of the path instead of the host name, so dots in the bucket name do not cause a certificate host name mismatch.
    Commit bbdc4b9
  • Remove explicit hot_standby configuration from user guide.
    hot_standby has been enabled by default since PostgreSQL 10. The quickstart now notes that a standby does not accept read-only connections without it.
    Commit cda454d
v2.59.0 Release Notes
PostgreSQL 19beta2 Support
Released July 20, 2026
NOTE TO PACKAGERS: A new distribution tarball is available which simplifies the build process by providing pregenerated HTML documentation, man page, and code. The tarball is attached to each release as an asset and is named pgbackrest-{version}.tar.gz. See the README.md in the tarball for details. Please use the new distribution tarball to avoid the additional build tooling required to generate code in future releases. See New Distribution Tarball for more information.

NOTE TO PACKAGERS: There is a new optional dependency on libsystemd.

IMPORTANT NOTE: Starting with this release, only the restore command may be run as root by default. Use allow-root to run other commands as root (though this is not recommended).
Bug Fixes:
  • Fix expire removing a backup in progress on another host. (Reviewed by Douglas J Hunley. Reported by Dmitrii.)
    A backup is not added to backup.info until it completes, so when expire ran on one host while a backup was running on another, e.g. expire on a standby while the primary is backed up, the backup in progress was removed. Expiration no longer removes the latest backup when it is in progress or resumable.
    Issue #2622Pull request #2797Commit aa0e215
  • Fix archive-push-queue-max not enforced when a WAL file errors. (Reviewed by Stefan Fercot. Reported by vut12.)
    In asynchronous mode the queue was measured against the WAL yet to be pushed. When a WAL file errored repeatedly the WAL files after it were pushed and no longer counted, although they stayed in pg_wal, so the queue limit was never reached and pg_wal could fill the disk. The queue is now measured against all the WAL PostgreSQL is waiting to archive, as in synchronous mode.
    Issue #2629Pull request #2799Commit 7fe3636
  • Fix async archive-get failure during recovery across a timeline switch. (Reviewed by Douglas J Hunley. Reported by Jimmy Yih.)
    Two async processes could race for the archive lock, and the one that lost exited before fetching the segment it was started for. The archive-get that started it then waited for archive-timeout and returned an error for a segment that exists, which could end recovery early, especially across a timeline switch. The segment is now fetched directly when the async process that should fetch it is not running.
    Issue #2423Pull request #2806Commit baed942
  • Fix potential buffer overrun in error module. (Fixed by Christophe Pettus. Reviewed by David Steele.)
    A stack trace too long for its buffer was not terminated and could be read past the end of the buffer, and an error message too long for its buffer could write past the end of it when a system error was appended. Both require an unusually long message.
    7fd8bc8 Fix potential buffer overrun in error module.
    5ca98ab Use the full buffer when formatting error messages.
    3a71dc8 Prevent an out-of-bounds error buffer write on an oversized message.
  • Fix use-after-free of parallel protocol client. (Reviewed by Douglas J Hunley. Reported by Georgy Shelkovy.)
    A client that ran out of work was freed but left in the client list and freed again on a later call. With debug or trace logging this read freed memory and could crash, e.g. a parallel backup with log-level-file=trace.
    Issue #2754Pull request #2804Commit 31432ed
  • Fix heap overflow when a pg option is set without pg1. (Reviewed by Douglas J Hunley. Reported by Naveen.)
    When pg options were set for other indexes without pg1-path, e.g. pg2-path and pg3-path, the index map was overwritten. This corrupted the heap on musl, e.g. Alpine, and caused a segmentation fault. The highest numbered pg-path was also dropped.
    Issue #1768Pull request #2796Commit 2d767de
  • Fix crash when a signal arrives during exit on musl 1.2.6.
    musl 1.2.6, e.g. on Alpine 3.24, crashes when exit() is called while an exit is already in progress. This happened when a signal arrived during exit, e.g. when the server command terminated a child process that was already exiting. A signal that arrives during exit is now ignored.
    Commit e166a33
Features:
  • PostgreSQL 19beta1/beta2 support. (Reviewed by Stefan Fercot.)
    Adds support for PostgreSQL 19beta1/beta2. PostgreSQL 19 can enable and disable page checksums online, and page checksums are not validated while either is in progress.
    bc6d399 PostgreSQL 19beta1 support.Pull request #2779
    2777395 PostgreSQL 19beta2 support.
  • Add archive-expire-before option to clean up WAL archive. (Contributed by Stefan Fercot. Reviewed by David Steele.)
    The expire option archive-expire-before removes WAL before a given segment from the archive but never WAL that a retained backup requires. It can clean up the archive from the restartpoint of a standby with archive_cleanup_command instead of pg_archivecleanup, which does not know about retention.
    6cd692b Extract WAL expiration into expireArchiveId().
    4945139 Add archive-expire-before option to clean up WAL archive.Pull request #2801
  • Add batch delete for Azure storage. (Reviewed by Douglas J Hunley, Crispy.)
    Files are removed with the Azure Blob Batch API, up to 256 per request, instead of one request per file. This speeds up expire and other commands that remove files from the repository.
    Pull request #2786Commit ffa325b
  • Add support for S3 Outposts. (Contributed by Shiva Kumar Ambigi. Reviewed by David Steele, Roberto Mello.)
    Set repo-s3-service=s3-outposts when using an S3 Outposts endpoint. The option sets the service name used for request signing and defaults to s3.
    Pull request #2762Commit 6c51800
  • Add S3 process authentication. (Reviewed by Andrew Charlton. Suggested by Andrew Charlton.)
    Set repo-s3-key-type=process and set repo-s3-process-cmd to a command that outputs temporary credentials in the JSON format of the AWS process credential provider. The credentials are refreshed before they expire.
    Pull request #2778Commit b5a578a
  • Provide a distribution tarball with generated code and documentation.
    The tarball includes the generated code and the rendered man page and HTML documentation, so it builds and installs without the code generation and documentation tooling, including libyaml.
    Commit 8fcfef5
Improvements:
  • Reconnect SFTP storage after the server drops an idle connection. (Reviewed by Stefan Fercot. Suggested by mustafa0x, tisserat.)
    SFTP servers and firewalls may drop a connection that has been idle for a while, e.g. the repository read connection during the copy phase of a backup, and the next use failed with a socket error. A dropped connection is now reopened and the operation is retried once.
    d72bea9 Fetch the libssh2 session from storage in SFTP read/write objects.
    a31febf Reconnect SFTP storage after the server drops an idle connection.Issue #2739Issue #2772Pull request #2793
  • Add user/group caching for faster manifest build. (Contributed by Gunnar Lindholm. Reviewed by David Steele.)
    When user and group names are looked up through a remote name service, e.g. sssd or LDAP, each lookup is a round trip, which dominated the manifest build for clusters with millions of files. User and group names are now cached.
    Pull request #2773Commit 1e6d23f
  • Add per-repo backup progress to info command output. (Contributed by Will Morland. Reviewed by David Steele, Stefan Fercot.)
    When backups run on more than one repository at once, the info command now reports progress for each repository as well as overall progress. The JSON output has a repo array for each backup lock.
    Issue #2736Pull request #2740Commit 77312b3
  • Add backup.info checks to verify command. (Contributed by Denis Garsh. Reviewed by David Steele, Douglas J Hunley.)
    The verify command now warns when a backup in the repository is not in backup.info and when a backup in backup.info is missing from the repository. A missing backup is reported as having a missing manifest.
    Issue #2723Pull request #2752Commit 7f82fb6
  • Allow the S3 STS endpoint to be configured. (Contributed by Simon Gratton. Reviewed by David Steele.)
    Set repo-s3-sts-host to use a regional STS endpoint with repo-s3-key-type=web-id, e.g. sts.eu-central-2.amazonaws.com. AWS opt-in regions do not accept credentials from the global endpoint, which is still the default.
    Issue #2791Pull request #2798Commit 11f5582
  • Add archive-push-batch-size to limit WAL pushed per asynchronous run. (Reviewed by Stefan Fercot.)
    In asynchronous mode archive-push-queue-max is only checked at the start of a run, so a run working through a large backlog could let the queue grow well past the limit. archive-push-batch-size (default 16GiB) limits the WAL pushed per run, so the queue is checked again sooner.
    Pull request #2803Commit 34a035f
  • Exit async archive-push on first error. (Reviewed by Stefan Fercot. Suggested by Lardière Sébastien.)
    When the repository was unavailable the async process kept trying every ready WAL segment, so a run could take long enough for pg_wal to fill before archive-push-queue-max was checked again. The run now exits on the first error, after the jobs in progress finish.
    Issue #1848Pull request #2802Commit a0c1435
  • Harden HTTP chunked response parsing. (Contributed by Shubham. Reviewed by David Steele.)
    Chunk extensions and trailers are now accepted, transfer-encoding is compared without regard to case, and a response with both transfer-encoding and content-length is rejected.
    Pull request #2785Commit 34bba82
  • Error when running as root unless allow-root is enabled. (Reviewed by Douglas J Hunley.)
    Running a command other than restore as root could create files in the repository, e.g. backup.info, that the PostgreSQL user cannot access. allow-root is disabled by default for every command except restore, and the check also applies on remote hosts.
    Issue #1693Pull request #2812Commit 0daf53d
  • Fewer binary searches when building manifest. (Contributed by Gunnar Lindholm. Reviewed by David Steele.)
    The manifest build for a diff or incr backup searched the prior manifest three times for each file. It now searches once, which saves millions of searches for clusters with millions of files.
    Pull request #2774Commit f1ef629
  • Improve seek performance during block incremental delta restore. (Reviewed by David Christensen, Douglas J Hunley.)
    A seek and a flush of the write buffer were done for every block written during a block incremental delta restore, and blocks are often 8KiB. The file position is now tracked and the seek and flush are only done when needed.
    Pull request #2732Commit b908d5e
  • Bundle backup files in order rather than scanning for a better fit. (Reviewed by Stefan Fercot.)
    A file that would push a bundle over repo-bundle-size was skipped for a smaller file later in the queue. Files are now added in order and a bundle is closed once it reaches repo-bundle-size, so a bundle may exceed it by up to repo-bundle-limit.
    Pull request #2794Commit 253dbc3
  • Report the underlying error when a query fails to complete. (Reviewed by Andrew Pogrebnoi. Suggested by Marco Fontana.)
    When the connection was lost after the first result of a query was delivered, e.g. under load, the error was the generic NULL result required to complete request. The error from PostgreSQL is now reported, with hints about a lost connection and about aggressive idle_session_timeout and idle_in_transaction_session_timeout settings.
    Issue #2789Pull request #2790Commit 23e966c
  • Report the actual error when an S3 credential request fails. (Reviewed by Douglas J Hunley. Suggested by Mitchell Grice.)
    An error from AWS for a web-id or pod-id credential request was reported as a failure to parse the response. The error returned by AWS is now reported, and the tokens are redacted when the request is logged.
    Issue #1997Pull request #2808Commit b0c57c2
  • Report archive-push spool path errors to the PostgreSQL log. (Reviewed by Douglas J Hunley. Suggested by Don Seiler.)
    When the async spool path could not be created, e.g. permission was denied, the error could not be written to the spool and archive-push failed with a generic timeout that pointed at the async log. The spool path is now created before the async process starts, so the error appears in the PostgreSQL log.
    Issue #1408Pull request #2810Commit 7b0159c
  • Hint that pgBackRest may be out of date when a version is unsupported.
    When pgBackRest does not recognize a PostgreSQL version, e.g. an older pgBackRest on the restore host reading a backup of a newer cluster, the error now hints that pgBackRest may need to be updated on all hosts. The version is reported in user-facing form, e.g. 13 rather than 130000.
    Commit d4f697e
  • Add systemd notify integration. (Contributed by Andrew Jackson. Reviewed by David Steele.)
    When built with libsystemd, the server command notifies systemd when it is ready to accept connections and when it is stopping, as PostgreSQL and PgBouncer do.
    Issue #2757Pull request #2760Commit 742fff1
  • Suppress unused parameter errors in meson compiler probes. (Contributed by Jörg Plate. Reviewed by David Steele.)
    The meson compiler probes declared main() parameters they did not use, which failed the probes when unused parameter warnings were errors.
    Issue #2724Commit d4465bd
  • C11 is now the minimum C standard. (Reviewed by Mohammad Ali Nazir Kosar.)
    Pull request #2738Commit 3a6c671
Documentation Improvements:
  • Make stanza option internal for the repo-* commands. (Reviewed by Stefan Fercot.)
    The stanza option only has an effect on the repo-* commands with undocumented storage path constants, so the documentation implied it did something it does not. The option is no longer documented for these commands and works as before.
    Issue #2735Commit 3e9eabd
  • Document that info reads encryption settings from global section. (Reviewed by Stefan Fercot. Suggested by Ron Johnson.)
    When run without stanza the info command reads encryption settings only from the global section, so encryption settings configured for a stanza require stanza.
    Issue #1407Pull request #2805Commit 2137bda
  • Document that expire-auto uses the backup command configuration. (Reviewed by Stefan Fercot. Suggested by Lardière Sébastien.)
    Automatic expiration after a backup uses the configuration of the backup command, so options set only for the expire command are not applied. Disable expire-auto and run expire separately when it needs its own configuration.
    Issue #1874Pull request #2807Commit cf847f9
  • Document adjusting the logrotate su directive for a dedicated user. (Reviewed by Douglas J Hunley. Suggested by lkanbus.)
    Packages may install a logrotate configuration that rotates the logs as the postgres user. When a dedicated user owns the repository and logs, the su directive must be changed to that user or rotation fails with a permission error.
    Issue #2737Pull request #2809Commit bde2c50
  • Document that end-of-line comments are not supported in config files. (Reviewed by Douglas J Hunley. Suggested by Alex Richman.)
    End-of-line comments have never been supported, although they may appear to work for some options. Only comments on a line of their own are supported.
    Issue #1633Pull request #2811Commit 8571799
Test Suite Improvements:
  • Fix Alpine group conflicts in CI containers. (Contributed by Artur Zakirov. Reviewed by David Steele.)
    Group entries with members are now handled when the test group is renamed, and the Alpine group is only created when it does not already exist.
    Pull request #2784Commit 760dd8d
v2.58.0 Release Notes
Object Storage Improvements
Released January 19, 2026
IMPORTANT NOTE: The minimum values for the repo-storage-upload-chunk-size option have increased. They now represent the minimum allowed by the vendors.
Bug Fixes:
  • Fix deadlock due to logging in signal handler. (Fixed by Maxim Michkov. Reviewed by David Steele.)
    A signal that arrived while a lock was held by gmtime_r(), e.g. the SIGTERM sent by stop with --force, could deadlock when the signal handler logged. The signal handler now logs with only async-signal-safe functions.
    Issue #2710Pull request #2715Commit e13c462
Features:
  • HTTP support for S3, GCS, and Azure. (Contributed by Will Morland. Reviewed by David Steele.)
    An object store endpoint can now begin with http:// to connect without TLS. HTTPS is still used when the endpoint does not specify a protocol.
    Issue #2340Pull request #2703Commit 2398dd4
  • Allow expiration of oldest full backup regardless of current retention. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Ron Johnson.)
    The expire option --oldest expires the oldest full backup and the backups that depend on it regardless of the retention settings, e.g. when the repository is running out of space. WAL before the oldest remaining full backup is expired as well.
    Issue #2666Pull request #2709Commit bf2b276
  • Support for Azure managed identities. (Contributed by Moiz Ibrar, Matthew Mols. Reviewed by David Steele.)
    Set repo-azure-key-type=auto to fetch credentials automatically using managed identities instead of configuring them. The credentials are refreshed before they expire.
    Issue #2023Pull request #2712Commit 38f14d7
  • Experimental support for S3 EKS pod identity. (Contributed by Pierre BOUTELOUP. Reviewed by David Steele.)
    Set repo-s3-key-type=pod-id to fetch credentials automatically using EKS pod identity instead of configuring them. The credentials are refreshed before they expire.
    Issue #2718Pull request #2719Commit 79544f6
  • Allow configuration of TLS cipher suites. (Contributed by Gunnar "Nick" Bluth. Reviewed by David Steele.)
    Set tls-cipher-12 for TLS 1.2 and tls-cipher-13 for TLS 1.3, e.g. for compliance. The cipher suites apply to every connection a process makes.
    fbc607a Allow configuration of TLS cipher suites.Issue #2697Pull request #2700
    91ad655 Fix flapping common/io-tls unit test added in fbc607a4.
  • Allow process priority to be set. (Reviewed by Douglas J Hunley.)
    priority sets the scheduling priority of pgBackRest processes from -20 to 19, as with nice. A higher value makes pgBackRest less likely to interfere with PostgreSQL.
    Pull request #2693Commit 0fc7e96
Improvements:
  • Allow dots in S3 bucket names when using path-style URIs. (Contributed by Joakim Hindersson. Reviewed by David Steele.)
    Dots in a bucket name only cause a certificate host name mismatch with host-style URIs, so they are now allowed with repo-s3-uri-style=path, which most self-hosted S3 services use.
    Pull request #2713Commit 3fdb4a1
  • Require TLS >= 1.2 unless verification is disabled. (Reviewed by Douglas J Hunley, Gunnar "Nick" Bluth.)
    TLS 1.0 and 1.1 have been deprecated since 2021. Verification is always enabled for connections between pgBackRest hosts. When verification is disabled for an object store with repo-storage-verify-tls, e.g. for a self-signed certificate, every protocol version is allowed.
    Pull request #2720Commit 5b4b20a
  • Dynamically size S3/GCS/Azure chunks for large uploads. (Reviewed by Douglas J Hunley. Suggested by Timothée Peignier.)
    The number of chunks in an upload is limited, so with the default repo-storage-upload-chunk-size the largest file that could be uploaded was 50GiB to S3, 40GiB to GCS, and 200GiB to Azure. The chunk size now grows during an upload, so files up to 5TiB can be uploaded to S3 and GCS and up to 45TiB to Azure.
    Issue #2679Pull request #2701Commit 86554fa
  • Optimize S3/GCS/Azure chunk size for small files. (Reviewed by Douglas J Hunley.)
    The whole chunk buffer was allocated when a file was opened, although many files are smaller than a chunk. The buffer now grows as data arrives, which saves memory when small files are uploaded.
    Pull request #2705Commit 28bfa20
  • Remove support for PostgreSQL 9.5. (Reviewed by Douglas J Hunley.)
    PostgreSQL 9.5 is no longer supported, per the policy of supporting five end-of-life versions of PostgreSQL.
    Pull request #2692Commit 5171e8b
  • Improve logging of default for options with an unresolved dependency. (Reviewed by Stefan Fercot.)
    An option set in the configuration file was logged at the start of a command when its dependency was not resolved, although its default value was used. The option is no longer logged in this case.
    Issue #2707Pull request #2708Commit 38ac638
Documentation Improvements:
  • Remove explicit max_wal_senders/wal_level configuration from user guide. (Suggested by Jamie Nguyen.)
    max_wal_senders and wal_level have defaulted to the values the user guide requires since PostgreSQL 10.
    Issue #2699Commit bce97d3
  • Clarify that bundling is useful for filesystems with large block sizes. (Suggested by Ron Johnson.)
    Bundling saves space on filesystems with large block sizes, where each small file takes a whole block.
    Issue #2706Commit a7e5f28
v2.57.0 Release Notes
Suppress Repository Symlinks
Released October 18, 2025
Bug Fixes:
  • Unnest HTTP/TLS/socket timeouts. (Reviewed by David Christensen.)
    A failed socket connection was also retried by the TLS and HTTP layers, so with the default io-timeout of 60 seconds a connection took nine minutes to time out. Each layer now retries only its own failures.
    Issue #2654Pull request #2662Commit 9c7f064
  • Fix possible segfault in page checksum error message. (Fixed by Zsolt Parragi. Reviewed by David Steele.)
    The error message for invalid page checksums used a string that was out of scope, which could log garbage or crash. This was most likely with pg_tde, since every encrypted file fails the page checksum.
    Pull request #2675Commit adc5e5b
  • Fix possible segfault in backup resume messages.
    The message that reports why a backup cannot be resumed used a string that was out of scope, which could log garbage or crash.
    Commit c1c40a4
Features:
  • Add repo-symlink option to suppress creation of repository symlinks. (Reviewed by Douglas J Hunley. Suggested by Ron Johnson.)
    Some filesystems that are otherwise Posix-compatible do not support symlinks. Set repo-symlink=n to keep the latest and tablespace symlinks from being created in the repository.
    Issue #2673Pull request #2674Commit bfd9790
Improvements:
  • Add HTTP retries for 408 and 429 errors. (Reviewed by David Christensen.)
    HTTP 408 (Request Timeout) and 429 (Too Many Requests) errors are now retried. 429 errors are common on Cloudflare R2.
    Pull request #2661Commit 8733900
v2.56.0 Release Notes
Progress Info Improvements
Released July 21, 2025
Bug Fixes:
  • Fix issue with adhoc expiration when no backups in a repository. (Reviewed by Stefan Fercot. Reported by Anup Gupta.)
    Adhoc expiration failed with cannot get index 0 from list with 0 value(s) when a repository had no backups.
    Issue #2645Pull request #2646Commit 638815b
Features:
  • Add restore progress to info command output. (Contributed by Denis Garsh, Maxim Michkov. Reviewed by David Steele.)
    The restore command now records its progress, which the info command reports alongside backup progress. A restore is also no longer terminated by the stop command.
    Issue #2546Pull request #2652Commit 8cdd9ce
  • Add progress-only detail level for info command output. (Contributed by Denis Garsh. Reviewed by David Steele, Stefan Fercot.)
    Set --detail-level=progress to report only progress, which is faster because the repository is not read. The default is full.
    Pull request #2550Commit 2706168
Improvements:
  • Retry failed reads on object stores. (Reviewed by David Christensen.)
    A read from an object store that failed after it started was fatal, which could restart an expensive job, e.g. during restore. A failed read is now retried from where it left off.
    Pull request #2642Commit 5e14692
  • Fix defaults in command-line help. (Reviewed by David Christensen, Chris Bandy.)
    When an option has more than one default, e.g. repo-storage-upload-chunk-size with an S3 and an Azure repository, the option list shows and the help for the option lists each default. Defaults are not shown for options that are not valid in the current configuration.
    Pull request #2625Commit 8737cad
Documentation Improvements:
  • Describe discrete option values in a list where appropriate. (Contributed by Anton Kurochkin. Reviewed by David Steele.)
    The values of options such as output for the verify and version commands are now described as a list, as for other options.
    Pull request #2638Commit 2e76e9f
  • Fix less than in help output for archive-mode option. (Contributed by Anton Kurochkin. Reviewed by David Steele.)
    Pull request #2633Commit cb67322
v2.55.1 Release Notes
Bug Fixes
Released May 5, 2025
Bug Fixes:
  • Revert calculate content-md5 on S3 only when required. (Reviewed by David Christensen. Reported by Frank Brendel.)
    S3 requires either content-md5 or an x-amz-content-* checksum for uploads when object lock is enabled, and Scality did not accept the alternate checksums, so uploads failed. content-md5 is calculated for all requests with content again.
    Issue #2613Pull request #2615Commit 8bbb982
  • Fix lower bounds checking for option keys. (Reviewed by David Christensen, Wolfgang Walther. Reported by Wolfgang Walther.)
    An indexed option with a key of 0, e.g. pg0-path, caused a segfault. It is now an error.
    Issue #2610Pull request #2611Commit 5e01098
v2.55.0 Release Notes
Verification Improvements and PostgreSQL 18 Support
Released April 21, 2025
Bug Fixes:
  • Fix block incremental restore issue on non-default repository. (Reviewed by David Christensen. Tested by Aleksander Łukasz. Reported by Aleksander Łukasz.)
    When the backup being restored used block incremental and was not in the lowest numbered repository, the restore read part of the block incremental data from the lowest numbered repository. Setting repo worked around the problem.
    Issue #2577Pull request #2579Commit 5d2c67c
  • Do not set recovery_target_timeline=current for PostgreSQL < 12. (Reviewed by Stefan Fercot.)
    Before PostgreSQL 12, explicitly setting recovery_target_timeline to current behaves as latest, although current is the default. The setting is now left out of recovery.conf when it is set to current for these versions.
    Pull request #2533Commit 6fbb28f
  • Fix expire archive range logging. (Reviewed by Stefan Fercot. Reported by Aleš Zelený.)
    Only the last archive path removed was logged as the start of the range, so the log suggested that far fewer files were expired than were.
    Issue #2543Pull request #2554Commit 47f050f
  • Fix error reporting for queries with no results. (Reviewed by Stefan Fercot. Reported by Susantha Bathige.)
    An error from a query that expected no results was reported as an unexpected result instead of as the error.
    Pull request #2539Commit d5cefb7
Features:
  • Verify recovery target timeline. (Reviewed by Stefan Fercot.)
    A backup that cannot follow the target timeline, e.g. one made after a standby was promoted and the new timeline forked off, failed only when recovery started, with a PostgreSQL error that is hard to interpret. The timeline is now checked against the history files before the restore starts.
    Pull request #2534Commit 922e9f0
  • Allow verification of a specified backup. (Contributed by Maxim Michkov. Reviewed by David Steele.)
    The verify option --set verifies only the specified backup and the WAL required to make it consistent.
    Issue #2419Pull request #2544Commit e4def88
  • Add support for S3/GCS requester pays. (Contributed by Timothée Peignier. Reviewed by David Steele.)
    Set repo-s3-requester-pays or repo-gcs-user-project to bill the requester instead of the bucket owner, e.g. so a hosted service can give customers access to their backups.
    4db0263 Add support for S3 requester pays.Pull request #2560
    814bf48 Add support for GCS requester pays.Pull request #2557
  • PostgreSQL 18 support. (Reviewed by Stefan Fercot.)
    This release was made before PostgreSQL 18 had a beta, so support was initially experimental. Nothing that affects pgBackRest changed before PostgreSQL 18.0.
    Pull request #2603Commit 565d2e0
  • Allow connections to PostgreSQL on abstract domain sockets. (Tested by Chris Bandy. Suggested by Chris Bandy.)
    pg-socket-path no longer has to be an absolute path, so it can name an abstract domain socket. libpq reports a path that is not valid.
    Issue #2529Pull request #2535Commit 68f22ae
  • Add numeric output to version command. (Contributed by Stefan Fercot. Reviewed by David Steele.)
    Set --output=num to output the version as a number, e.g. 2055000, which is easier to compare.
    Issue #2410Pull request #2526Commit 750a051
Improvements:
  • Allow backup command to operate on remote repositories. (Reviewed by Stefan Fercot.)
    The backup command can now run on a host where the repository is remote, which allows more flexibility in where backups are run.
    4a94b6b Refactor protocol helper.
    931435c Allow backup command to operate on remote repositories.Pull request #2512
  • Use lz4 for protocol compression. (Reviewed by Stefan Fercot.)
    lz4 compresses much faster than gz and gives similar ratios at the low levels used for protocol compression. Setting compress-level-network=0 no longer disables protocol compression.
    0577b03 Use lz4 for protocol compression.Pull request #2488
    5c8296d Remove reference to disabling network compression in the documentation.
  • Calculate content-md5 on S3 only when required. (Reviewed by David Christensen.)
    The content-md5 header is only calculated for batch delete requests, which require it, to save CPU when files are uploaded. This change was reverted in v2.55.1.
    Pull request #2592Commit 20bfd14
  • Warn when a value for a multi-key option is overwritten. (Reviewed by David Christensen, Stefan Fercot.)
    Options with key/value pairs, e.g. recovery-option, silently overwrote a value when its key was specified again. A warning is now logged.
    Issue #2591Pull request #2594Commit cab4d6f
  • Add detail logging for expired archive path. (Contributed by Stefan Fercot. Reviewed by David Steele.)
    Removing a large number of archive files could look like a hang. A message at detail level now shows progress.
    4dcfa9e Add detail logging to expire test.
    2155a12 Add detail logging for expired archive path.Issue #2543Pull request #2559
  • Remove support for PostgreSQL 9.4. (Reviewed by Stefan Fercot.)
    PostgreSQL 9.4 is no longer supported, per the policy of supporting five end-of-life versions of PostgreSQL. Backups of 9.4 clusters can still be expired.
    Pull request #2472Commit 48f511d
  • Remove autoconf/make build. (Reviewed by David Christensen.)
    v2.54.0 was the last release to support the autoconf/make build. Use meson to build.
    Pull request #2470Commit 89c9bab
Documentation Improvements:
  • Fix documentation for specifying multiple stanzas with tls-server-auth. (Reviewed by David Christensen, Stefan Fercot. Suggested by Terry MacAndrew.)
    Multiple stanzas are specified for tls-server-auth as a comma-separated list. The documentation said to repeat the option.
    Issue #2591Pull request #2595Commit 53a45d2
  • Clarify incremental backup expiration. (Reviewed by Stefan Fercot.)
    The documentation implied that incremental backups are expired by count. It now explains how incremental backups are expired and that full backups count as differential backups for differential retention.
    Pull request #2601Commit 8bdba74
  • Clarify requirement for local/remote pgBackRest versions to match. (Contributed by Greg Clough. Reviewed by David Steele.)
    Issue #2540Pull request #2541Commit 82299eb
  • Add FAQ about exporting self-contained cluster. (Contributed by Stefan Fercot. Reviewed by David Steele.)
    The FAQ explains how to create a self-contained export of a cluster with a backup made with --archive-copy and restored with --type=none.
    Pull request #2605Commit 9386920
  • Caveat --tablespace-map-all regarding tablespace creation. (Reviewed by Stefan Fercot, Christophe Courtois. Suggested by Christophe Courtois.)
    A tablespace created after the backup starts is not in the manifest, so tablespace-map-all cannot map it.
    Pull request #2538Commit e625ed8
  • Clarify behavior of --repo-retention-full-type. (Reviewed by Antoine Beaupré. Suggested by Antoine Beaupré.)
    Issue #2562Pull request #2583Commit 84eb9a7
  • Change --process-max recommendation for object stores to --repo-bundle. (Reviewed by Stefan Fercot.)
    On object stores repo-bundle reduces the time spent creating files far more than process-max does, because fewer files are created.
    Pull request #2542Commit aa7210a
  • Update unix_socket_directory to unix_socket_directories. (Contributed by hyunkyu han. Reviewed by David Steele.)
    unix_socket_directory has not been valid since PostgreSQL 9.2, so only unix_socket_directories is documented.
    Issue #2589Commit b28ae98
  • Recommend not placing spool-path within pg_xlog/pg_wal. (Reviewed by Martín Marqués, Don Seiler. Suggested by Martín Marqués.)
    A spool path inside pg_xlog or pg_wal can cause problems for PostgreSQL tools such as pg_rewind.
    Pull request #2600Commit dd3a7d9
v2.54.2 Release Notes
Bug Fix
Released January 20, 2025
Bug Fixes:
  • Fix issue after disabling bundling with block incremental enabled. (Reviewed by David Christensen.)
    When bundling was disabled after backups were made with bundling and block incremental enabled, the prior backups could not be read. The setting needed to read them is now preserved.
    Pull request #2511Commit dbc6f28
Documentation Improvements:
  • Clarify behavior of multiple configuration files. (Reviewed by Paul Bierly. Suggested by Paul Bierly.)
    Each configuration file must be valid on its own, so a section must be specified in each file that sets an option in it. The documentation also notes that quoting is not supported, whitespace is trimmed from keys and values, and repeated sections are merged.
    Commit 5da2df4
v2.54.1 Release Notes
Bug Fix
Released December 16, 2024
Bug Fixes:
  • Fix issue with version/help commands attempting to load pgbackrest.conf. (Reviewed by Stefan Fercot. Reported by Bradford Boyle, Julian.)
    In v2.54.0 the version and help commands loaded pgbackrest.conf and failed when it could not be read or parsed.
    Issue #2476Pull request #2477Commit 8d69118
Test Suite Improvements:
  • Stabilize async archiving in integration tests. (Contributed by Viktor Kurilko. Reviewed by David Steele.)
    An async archive-get left running by one integration test could write an error to the queue of the next test and fail its recovery.
    Issue #2463Pull request #2481Commit b90ce9e
v2.54.0 Release Notes
Target Time for Versioned Storage
Released October 21, 2024
NOTE TO PACKAGERS: This is last feature release to support the autoconf/make build. Please migrate to meson if you have not already done so. 2.54.X patch releases (if any) will continue to support autoconf/make.
Bug Fixes:
  • Fix PostgreSQL query performance for large datasets. (Fixed by Thibault Vincent, David Steele. Reviewed by David Christensen. Tested by Antoine Millet. Reported by Antoine Millet.)
    The query timeout logic delayed reading query results, which could make a query that returned a large result take up to db-timeout. Results are now read as soon as they are available.
    6ac4e8d Move fd module shim to io module test.
    77ae753 Allow fdReady() shim to be run only once with a specified return value.
    33fa396 Refactor Wait object to expose remaining wait as waitRemains().
    c8ccaaa Fix PostgreSQL query performance for large datasets.Issue #2458Pull request #2459
Features:
  • Allow repositories on versioned storage to be read at a target time. (Reviewed by Stefan Fercot, David Christensen.)
    The archive-get, info, repo-get, repo-ls, restore, and verify commands can read a repository on versioned Azure, GCS, or S3 storage as it was at repo-target-time. This allows recovery even when the repository has been deleted or corrupted, accidentally or maliciously.
    Pull request #2404Commit b19134d
  • Allow requested standby backup to proceed with no standby. (Reviewed by Stefan Fercot.)
    Set backup-standby=prefer to back up from a standby when one is available and from the primary otherwise. This does not help when the standby responds but fails to sync with the primary after the backup starts.
    Pull request #2378Commit a42629f
Improvements:
  • Summarize backup reference list for info command text output. (Contributed by Stefan Fercot. Reviewed by David Steele.)
    The backup reference list can be very long, so the text output of the info command now summarizes it. The full list is shown when --set is specified.
    Pull request #2399Commit b306f83
  • Refresh web-id token for each S3 authentication. (Contributed by Brent Graveland. Reviewed by David Steele.)
    The token file named by AWS_WEB_IDENTITY_TOKEN_FILE was read once at startup, so the token could expire during a long operation. The token is now read for each authentication.
    Issue #2428Pull request #2429Commit 95a4b9e
  • Correctly display current values for indexed options in help. (Reviewed by David Christensen.)
    The current value shown for an indexed option was always the value for the first index, e.g. pg1-path. The option help now lists the value for each index, and the option list shows when there is more than one.
    Pull request #2453Commit b3ca2e3
  • Save backup.info only when contents have changed. (Reviewed by Stefan Fercot.)
    expire saved backup.info even when it had not changed. Fewer saves also make it easier to choose a target time on versioned storage.
    Pull request #2403Commit b454b9e
  • Remove limitation on reading files in parallel during restore. (Reviewed by David Christensen.)
    The protocol could only read or write one file at a time on a remote, so a restore read one file at a time from the repository. Protocol commands can now be multiplexed, so a restore can read more than one file at a time, e.g. a bundle and the block incremental data for a file in it.
    df8cbc9 Protocol command multiplexing.Pull request #2108
    46c1297 Remove limitation on reading files in parallel during restore.Pull request #2436
  • Improve SFTP error messages. (Contributed by Reid Thompson. Reviewed by David Steele.)
    SFTP errors were reported as numbers only. They now include a description of the error.
    Pull request #2439Commit 70bda2c
Documentation Features:
  • Add performance tuning section to user guide. (Reviewed by Stefan Fercot.)
    The options that commonly improve performance were spread through the documentation. The quickstart now has a section that lists them.
    Pull request #2446Commit 278319e
Documentation Improvements:
  • Clarify source for data_directory. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Matthias.)
    pg-path should match the data_directory reported by PostgreSQL, which may differ from the value in postgresql.conf.
    Issue #2464Pull request #2467Commit 9ba2db0
  • Better logic for deciding when a summary should be lower-cased. (Suggested by Daniel Westermann.)
    Summaries that start with a proper name or an acronym, e.g. PostgreSQL, are no longer lower-cased in the documentation.
    Issue #2462Commit eb2f279
v2.53.1 Release Notes
PostgreSQL 17 Support
Released August 19, 2024
Bug Fixes:
  • Fix permissions when restore run as root user. (Reviewed by Stefan Fercot. Reported by Will M.)
    When restore was run as root, pg_control was owned by root. A restore as root also failed when the base path did not exist.
    Issue #2407Pull request #2416Commit 7bd8da4
  • Fix segfault on delayed connection errors. (Reviewed by David Christensen. Reported by Anton Glushakov.)
    A connection error that was delayed long enough to be reported on a later wait could cause a segfault.
    Issue #2420Pull request #2422Commit 377ba06
  • Skip local repository duplicate check for SFTP. (Fixed by Reid Thompson. Reviewed by David Steele. Reported by Anton Kurochkin.)
    The check for local repositories that are configured more than once was also applied to SFTP repositories, which are never local.
    Issue #2418Pull request #2421Commit 9840304
Improvements:
  • PostgreSQL 17 support.
    PostgreSQL 17beta3 changed the control and WAL formats, which is unusual for a beta.
    Commit 189d15d
v2.53 Release Notes
Concurrent Backups
Released July 22, 2024
IMPORTANT NOTE: The log-level-stderr option default has been changed from warn to off. This makes it easier to capture errors when only redirecting stdout. To preserve the prior behavior set log-level-stderr=warn.

NOTE TO PACKAGERS: The lz4 library is now required by the meson build.

NOTE TO PACKAGERS: Compiler support for __builtin_clzl() and __builtin_bswap64() is now required by the meson build.
Bug Fixes:
  • Fix SFTP renaming failure when file already exists. (Fixed by Reid Thompson. Reviewed by David Steele. Reported by ahmed112212.)
    Some SFTP servers report that a file already exists with an error that was not handled, so renaming a file over an existing file failed.
    Issue #2376Pull request #2377Commit d6f0bf8
Features:
  • Allow backups to run concurrently on different repositories. (Reviewed by Reid Thompson, Stefan Fercot.)
    Backups of the same stanza can now run at the same time on different repositories, since locks are now per stanza and repository instead of per stanza. This is not possible for PostgreSQL <= 9.5.
    270dce4 Refactor lock module.Pull request #2371
    cfb8aa2 Add remote locks for stanza commands missed in 31c7824a.
    edd6163 Allow backups to run concurrently on different repositories.Pull request #2389
  • Support IP-based SANs for TLS certificate validation. (Contributed by David Christensen. Reviewed by David Steele.)
    Certificates with an IP address as the subject alternative name, e.g. IP:1.2.3.4, are now validated. IPv4 and IPv6 addresses must match exactly.
    Issue #1977Pull request #2047Commit dfb620b
Improvements:
  • Default log-level-stderr option to off. (Reviewed by Greg Sabino Mullane, Stefan Fercot.)
    Warnings and errors written to stderr were often lost when only stdout was redirected while collecting logs. An explicitly set log-level-stderr is not affected.
    Pull request #2379Commit 7d54073
  • Allow alternative WAL segment sizes for PostgreSQL ≤ 10. (Contributed by Viktor Kurilko. Reviewed by David Steele.)
    PostgreSQL <= 10 can be built with a different WAL segment size, and forks such as Greenplum set it with initdb.
    Pull request #2303Commit 4ac3b82
  • Add hint to check SFTP authorization log. (Contributed by Vitalii Zurian. Reviewed by Reid Thompson, David Steele.)
    When SFTP authentication fails the error now hints that the authorization log on the server may have more detail.
    Issue #2381Pull request #2385Commit ccae609
  • Display time option defaults and allowed values with appropriate units.
    Time option defaults and allowed values were shown in seconds, which was hard to read for large values. Fractional seconds are no longer allowed, so a value such as archive-timeout=0.5 must be written as archive-timeout=500ms.
    Commit 3a2266f
  • Require OpenSSL >= 1.1.1.
    Commit cf8625d
Documentation Improvements:
  • Clarify archive-push multi-repo behavior. (Reviewed by Stefan Fercot.)
    When one repository is failing, archive-push cannot push ahead to the other repositories unless archive-async is enabled.
    Pull request #2398Commit ec3e387
v2.52.1 Release Notes
Bug Fix
Released June 25, 2024
Bug Fixes:
  • Fix issue with files larger on the replica than on the primary. (Reviewed by Stefan Fercot. Reported by Nicolas Lassimonne.)
    With block incremental, when a file was a different size on the primary than on the standby during a backup from standby, the next diff or incr backup stored the size from the primary. The backup after that failed with repo size must be > 0.
    83aaa86 Add unit tests for backup from standby with block incremental.
    aabc122 Fix issue with files larger on the replica than on the primary.Issue #2358Pull request #2367
v2.52 Release Notes
PostgreSQL 17beta1 Support
Released May 27, 2024
NOTE TO PACKAGERS: The build system for pgBackRest is now meson. The autoconf/make build will not receive any new features and will be removed after a few releases.
Features:
  • Add GCS batch delete support. (Reviewed by Reid Thompson.)
    Files are removed with batch requests instead of one request per file, which speeds up expire and other commands that remove files from the repository.
    Pull request #2326Commit 76bcb74
  • S3 SSE-C encryption support. (Reviewed by Tim Jones. Suggested by Tim Jones.)
    Set repo-s3-sse-customer-key to have S3 encrypt the data with the provided key after it is transmitted. This is not as secure as encrypting the data before it is transmitted with repo-cipher-type.
    Issue #2279Pull request #2282Commit 57731b6
  • PostgreSQL 17beta1 support. (Reviewed by Stefan Fercot.)
    Adds the catalog version and WAL magic number for PostgreSQL 17.
    Pull request #2328Commit fb22f04
Improvements:
  • Allow explicit disabling of optional dependencies in meson builds. (Contributed by Michael Schout. Reviewed by David Steele.)
    An optional feature can be disabled in the meson build even when its library is installed, as some platforms require, e.g. FreeBSD. By default a feature is enabled when its library is found.
    Pull request #2321Commit 7b95fd3
  • Dynamically find python in meson build. (Contributed by Michael Schout. Reviewed by David Steele.)
    The meson build now finds the python installation instead of requiring a python3 binary on the path, e.g. when only python3.9 is installed. The minimum meson version is now 0.47.
    55e9969 Update meson minimum version to 0.47.
    e00e33b Dynamically find python in meson build.Pull request #2338
  • Tag pgbackrest build target in meson as installable. (Contributed by Bradford Boyle. Reviewed by David Steele.)
    Pull request #2312Commit 7877983
Documentation Improvements:
  • Update start/stop documentation to reflect actual functionality. (Reviewed by Stefan Fercot.)
    Issue #2325Issue #2334Pull request #2352Commit b6ac11b
v2.51 Release Notes
Meson Build System
Released March 25, 2024
Bug Fixes:
  • Skip zero-length files for block incremental delta restore. (Reviewed by Sebastian Krause, René Højbjerg Larsen. Reported by Sebastian Krause.)
    An existing zero-length file caused a block incremental delta restore to fail in the block checksum filter. These files are now skipped, since there is nothing in them to reuse.
    Issue #2271Pull request #2272Commit f7a7ab1
  • Fix performance regression in storage list. (Reviewed by Stephen Frost. Reported by Maksym Boguk.)
    Listing a path requested detailed information for each file, which could be slow, e.g. on object stores. Only the information the list needs is now requested.
    Issue #2275Pull request #2276Commit e00bfe2
  • Fix progress logging when file size changes during backup. (Reviewed by Stephen Frost. Reported by samkingno.)
    Issue #2298Pull request #2304Commit 76956e7
Improvements:
  • Improved support for dual stack connections. (Reviewed by Stephen Frost. Suggested by Timothée Peignier.)
    When a host name resolves to more than one address, e.g. IPv4 and IPv6, the next address is tried once the prior address has had reasonable time to connect, following the happy eyeballs approach of RFC 8305. A connection attempt that neither fails nor connects no longer uses up all the available time.
    Issue #2154Pull request #2235Commit 7448fde
  • Make meson the primary build system. (Reviewed by Stephen Frost.)
    meson is now the build system used for testing and for building the documentation. Functions not used outside their module are now hidden, which makes the binary smaller.
    e3d9df3 Make meson the primary build system.Pull request #2264
    f287178 Set function/variable visibility to hidden.Pull request #2265
  • Detect files that have not changed during non-delta incremental backup. (Reviewed by Stephen Frost.)
    During a non-delta diff or incr backup, a file that fits in a single buffer, or any block incremental file, is compared to the file in the prior backup and referenced instead of stored when it has not changed.
    Pull request #2257Commit 9d91d1b
  • Prevent invalid recovery when backup_label removed. (Reviewed by Stephen Frost.)
    When backup_label is removed from a restored cluster, PostgreSQL uses the checkpoint in pg_control for crash recovery, which nearly always corrupts the cluster. Restore now invalidates the checkpoint in pg_control, so recovery fails instead. pg_controldata shows the checkpoint as 0/DEAD.
    Pull request #2232Commit e634fd8
  • Improve archive-push WAL segment queue handling. (Reviewed by Stephen Frost.)
    The size of every WAL segment in the queue was read to measure the queue. The size of the first segment is now used for all of them.
    Pull request #2277Commit cf17515
  • Limit resume functionality to full backups. (Reviewed by Stephen Frost, Stefan Fercot.)
    Resume did not work correctly for delta diff and incr backups, and a file that changed since the prior backup is likely to change again before a resume. Only full backups are resumed now.
    Pull request #2268Commit dddcbcd
  • Update resume functionality for block incremental. (Reviewed by Stephen Frost.)
    Resume removed block incremental files, so it worked but copied all of them again. Block incremental files are now kept on resume.
    Pull request #2296Commit ff47450
  • Allow --version and --help for version and help. (Reviewed by Greg Sabino Mullane. Suggested by Greg Sabino Mullane.)
    --version and --help now work as they do for most programs. Use help for help on a specific command or option.
    Pull request #2263Commit 68db307
  • Add detailed backtrace to autoconf/make build. (Reviewed by Stephen Frost.)
    Error stack traces in the autoconf/make build now include detail from libbacktrace, which performance tests showed has no noticeable cost.
    Pull request #1970Commit 676700d
Documentation Improvements:
  • Update references to recovery.conf. (Reviewed by Stefan Fercot. Suggested by Stephen Frost.)
    References to recovery.conf now include postgresql.auto.conf, which newer versions of PostgreSQL use.
    Pull request #2281Commit 6835178
v2.50 Release Notes
Performance Improvements and Bug Fixes
Released January 22, 2024
Bug Fixes:
  • Fix short read in block incremental restore. (Reviewed by Stephen Frost. Tested by Brent Graveland. Reported by Adol Rodriguez, Brent Graveland.)
    A compressed super block can end with bytes that are not needed to decompress it but mark the end of the data. Depending on how a buffer lined up with the data, these bytes were not read and the restore failed.
    Issue #2215Issue #2238Pull request #2256Commit f4a4af2
  • Fix overflow suppressing backup progress in info output. (Fixed by Robert Donovan. Reviewed by Joe Wildish.)
    The sizes in the lock file were read as 32-bit values, so the info command did not report progress for a backup larger than 4GiB.
    Issue #2251Pull request #2252Commit 25f1489
Improvements:
  • Preserve partial files during block incremental delta restore. (Reviewed by Stephen Frost.)
    A file smaller than expected was not kept for a block incremental delta restore, although its blocks can be reused, e.g. when retrying a restore that failed or restoring over a smaller file.
    Pull request #2254Commit a42614e
  • Add support for alternate compile-time page sizes. (Contributed by Viktor Kurilko. Reviewed by David Steele.)
    PostgreSQL can be built with a different page size, e.g. 4096, and forks such as Greenplum do this.
    Pull request #2197Commit 89d5278
  • Skip files truncated during backup when bundling. (Contributed by Georgy Shelkovy. Reviewed by David Steele.)
    A file truncated to zero after the files were assigned to bundles was still stored in its bundle, and a debug build failed an assertion. These files are now skipped.
    02eea55 Skip files truncated during backup when bundling.Pull request #2227
    c879509 Allow const checksum buffers to be returned from backupFile().
    9049fec Refactor skip files truncated during backup when bundling.
    Development in v2.49:
    337da35 Add test to show behavior of bundled files truncated during backup.
  • Improve SFTP storage error messages. (Contributed by Reid Thompson. Reviewed by David Steele.)
    Pull request #2249Commit 7c17eec
v2.49 Release Notes
Remove PostgreSQL 9.3 Support
Released November 27, 2023
Bug Fixes:
  • Fix regression in retries. (Reviewed by Stephen Frost. Reported by Norman Adkins, Tanel Suurhans, Jordan English, Timothée Peignier.)
    v2.48 dropped the retry that was made when an operation used up its whole timeout, so a timeout became an error, e.g. waiting for a socket to be writable on an object store. Two retries are now made even when the timeout has expired.
    3c116e1 Remove unused header.
    fa5b2d4 Fix regression in retries.Issue #2200Issue #2214Pull request #2221
  • Fix recursive path remove in SFTP storage driver. (Fixed by Reid Thompson. Reviewed by Stephen Frost. Reported by Luc.)
    Some SFTP servers report permission denied instead of a failure when a directory is unlinked as a file, so removing a path recursively failed on those servers.
    Issue #2219Pull request #2226Commit c4dc466
Improvements:
  • Remove support for PostgreSQL 9.3. (Reviewed by Stephen Frost.)
    PostgreSQL 9.3 is no longer supported, per the policy of supporting five end-of-life versions of PostgreSQL.
    Pull request #2220Commit dcf0781
Documentation Features:
  • Document maintainer options. (Reviewed by Stefan Fercot.)
    Maintainer options are now documented in their own section with the caveats for using them, and pg-version-force is no longer internal.
    Pull request #2202Commit 21c8c8a
  • Update point-in-time recovery documentation for PostgreSQL >= 13.
    Since PostgreSQL 13 recovery errors when the recovery target is not reached. The point-in-time recovery section now works for PostgreSQL 13 and later as well as for earlier versions.
    Commit eb69e2e
Test Suite Improvements:
  • Allow config/load unit test to run without libssh2 installed. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Wu Ning.)
    Issue #2193Pull request #2194Commit e2b734e
v2.48 Release Notes
Repository Storage Tags
Released September 25, 2023
Bug Fixes:
  • Fix issue restoring block incremental without a block list. (Reviewed by Stephen Frost. Tested by Burak Yurdakul. Reported by Burak Yurdakul.)
    A block map with no blocks, which is written when a file timestamp changes but the file does not, was not decrypted when it followed a file stored without block incremental in the same bundle read. With enough affected files the retries were exhausted and the restore failed.
    Issue #2182Pull request #2184Commit 31de127
Features:
  • Add --repo-storage-tag option to create object tags. (Reviewed by Stephen Frost, Stefan Fercot, Timothée Peignier.)
    Set repo-storage-tag to add tags to the objects written to an S3, GCS, or Azure repository. Tags are set for each repository.
    Issue #2150Pull request #2159Commit 1b4e0cc
  • Add known hosts checking for SFTP storage driver. (Contributed by Reid Thompson. Reviewed by Stephen Frost, David Steele.)
    SFTP host keys are now checked against the known hosts files by default, i.e. repo-sftp-host-key-check-type=strict. Set repo-sftp-host-key-check-type=accept-new to add new hosts automatically, or repo-sftp-host-key-check-type=none to skip the check as before.
    Pull request #2125Commit ce9ba0f
  • Support for dual stack connections. (Reviewed by Stephen Frost.)
    Each address a host name resolves to is now tried in turn, e.g. an IPv6 address with no route to the host and then an IPv4 address, until one connects or the timeout expires. Repeated retry errors are also summarized in error messages.
    5314dbf Adjust Wait object to be more accurate when nested.
    c180513 Aggregate error retries in ErrorRetry output.
    edbd520 Refactor address list lookup to include all returned addresses.Pull request #2174
    39bb8a0 Support for dual stack connections.Issue #2154Pull request #2175
  • Add backup size completed/total to info command JSON output. (Contributed by Stefan Fercot. Reviewed by David Steele.)
    The JSON output of the info command now includes the completed and total size of a backup in progress, so progress can be monitored by other programs.
    Issue #2007Pull request #2141Commit e3fcd8c
Improvements:
  • Multi-stanza check command. (Reviewed by Stephen Frost.)
    When stanza is not specified, the check command now checks every stanza in the configuration.
    Pull request #2057Commit 1141dc2
  • Retry reads of pg_control until checksum is valid. (Reviewed by Stefan Fercot, Stephen Frost.)
    On some filesystems, e.g. ext4, a read of pg_control during a concurrent write can return torn data. The read is now retried until the checksum is valid.
    Pull request #2131Commit f42d927
  • Optimize WAL segment check after successful backup. (Reviewed by Stephen Frost.)
    The check for WAL segments after a backup listed the storage once for each segment. Each WAL directory is now listed once and the list is reused.
    Pull request #2169Commit 9d3a605
  • Improve GCS multi-part performance. (Reviewed by Reid Thompson.)
    Chunks of a multi-part upload are now uploaded as soon as they are available.
    Pull request #2162Commit 6cb9c40
  • Allow archive-get command to run when stanza is stopped. (Reviewed by Tom Swartz, David Christensen, Reid Thompson.)
    The archive-get command does not write to the repository, so it can now run while the stanza is stopped, as restore can.
    Commit fbd992a
  • Accept leading tilde in paths for SFTP public/private keys. (Contributed by Reid Thompson. Reviewed by David Steele.)
    Paths for repo-sftp-private-key-file and repo-sftp-public-key-file may now begin with ~, as documented.
    Pull request #2136Commit eb32d6d
  • Reload GCS credentials before renewing authentication token. (Reviewed by Stephen Frost. Suggested by Daniel Farina.)
    The service key file is now read again before the authentication token is renewed, so the key can be replaced while a command is running.
    Pull request #2072Commit 995a8e9
Documentation Bug Fixes:
  • Fix configuration reference example for the tls-server-address option. (Fixed by Hartmut Goebel. Reviewed by David Steele.)
    The example ::* was not valid. Use * for IPv4 or :: for IPv6.
    Pull request #2165Commit 43524a4
  • Fix command reference example for the filter option.
    Commit cb3ff6e
Test Suite Improvements:
  • Allow storage/sftp unit test to run without libssh2 installed. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Wu Ning.)
    Issue #2163Pull request #2177Commit c3c0834
v2.47 Release Notes
Performance Improvements and Bug Fixes
Released July 24, 2023
Bug Fixes:
  • Preserve block incremental info in manifest during delta backup. (Reviewed by Stephen Frost. Reported by Francisco Miguel Biete Banon.)
    When a file changed so that it would not use block incremental if it were new, e.g. it shrank below the block incremental limit or its timestamp moved far enough into the past, its block incremental info was lost from the manifest of a delta backup even though the file could still be referenced to the prior backup.
    1d42aed Improve validation of referenced files in command/backup unit test.
    4c27d74 Preserve block incremental info in manifest during delta backup.Issue #2112Pull request #2123
  • Fix block incremental file names in verify command. (Reviewed by Reid Thompson. Reported by Francisco Miguel Biete Banon.)
    The verify command used the compression extension instead of .pgbi for block incremental files stored outside a bundle, so it could not find them.
    Issue #2112Pull request #2120Commit cdb7e26
  • Fix spurious automatic delta backup on backup from standby. (Reviewed by Stephen Frost. Reported by krmozejko, Don Seiler.)
    File sizes on the standby can differ from the primary, e.g. for free space maps, so the next incremental backup could see a size change on the primary and switch to a checksum delta, which can take much longer. The original size is now kept in the manifest and used for the comparison.
    Issue #2020Pull request #2124Commit 5ed6f8d
  • Skip recovery.signal for PostgreSQL >= 12 when recovery type=none. (Reviewed by Stefan Fercot. Reported by T.Anastacio.)
    With type=none, recovery.signal is no longer written for PostgreSQL >= 12, which matches the behavior for earlier versions.
    Issue #2062Pull request #2070Commit 9cceed6
  • Fix unique label generation for diff/incr backup. (Fixed by Andrey Sokolov. Reviewed by David Steele.)
    With at least two full backups and the latest expired, a diff or incr backup failed as if the clock had gone backward, because its label was compared with the label of the expired full backup.
    Issue #1873Pull request #2104Commit 0ac0934
  • Fix time-based archive expiration when no backups are expired. (Reviewed by Stefan Fercot.)
    When time-based retention was met but no backups were expired, archive was not expired and time-based archive retention not met was logged. One more backup was then needed to expire the earlier WAL.
    Issue #2103Pull request #2109Commit 762498f
Improvements:
  • Improve performance of SFTP storage driver. (Contributed by Stephen Frost, Reid Thompson. Reviewed by David Steele.)
    The SFTP storage driver waited for a fixed time when libssh2 could not continue. It now waits until the socket can be read or written.
    e280ed9 Make result code handling in storage/sftp more consistent.
    125676a Cleanup of init/handshake in storageSftpNew().
    28b6b2d Improve performance of SFTP storage driver.Pull request #2089
  • Add timezone offset to info command date/time output. (Reviewed by Stefan Fercot, Philip Hurst. Suggested by Philip Hurst.)
    Timestamps in the info command output now include the timezone offset, so they can be used directly as a recovery target, e.g. with --target.
    9efd5cd Add timezone offset to info command date/time output.Issue #2093Pull request #2095
    aa229a1 Modify time_t casts based on complaints from Coverity.
  • Centralize error handling for unsupported features. (Reviewed by Stefan Fercot.)
    Errors for features that were not compiled in, e.g. lz4, are now more detailed and consistent. Specifying SFTP storage when it was not compiled in no longer fails an assertion.
    de46276 Refactor allow list processing in config/parse module.
    36ff81d Centralize error handling for unsupported features.Pull request #2075
Documentation Improvements:
  • Clarify preference to install from packages in the user guide. (Reviewed by Stefan Fercot. Suggested by dr-kd.)
    Pull request #2090Commit a36f480
v2.46 Release Notes
Block Incremental Backup and SFTP Storage
Released May 22, 2023
Features:
  • Block incremental backup. (Reviewed by John Morris, Stephen Frost, Stefan Fercot.)
    Block incremental stores only the parts of a file that changed since the prior backup, which saves space and time in diff and incr backups of large files. Enable it with repo-block, which requires repo-bundle.
    dd4e526 Removing chunking and block numbers from incremental block list.Pull request #2029
    822f2a5 Finalize block incremental feature.Pull request #2053
  • SFTP support for repository storage. (Contributed by Reid Thompson. Reviewed by Stephen Frost, David Steele.)
    Set repo-type=sftp and repo-sftp-host to store the repository on an SFTP server.
    Pull request #1959Commit 87087fa
    Development in v2.42:
    01b81f9 Move link creation to storage interface.Pull request #1879
  • PostgreSQL 16 support. (Reviewed by Stefan Fercot.)
    Adds the catalog version and WAL magic number for PostgreSQL 16.
    Pull request #2051Commit 3fc3690
Improvements:
  • Allow page header checks to be skipped. (Reviewed by David Christensen. Suggested by David Christensen.)
    The page header checks made while verifying page checksums can report false errors for encrypted pages. Set page-header-check=n to skip them. They are enabled by default.
    Pull request #2045Commit f5e6bc2
  • Avoid chown() on recovery files during restore. (Reviewed by Stefan Fercot, Marcelo Henrique Neppel. Suggested by Marcelo Henrique Neppel.)
    Restore skipped chown() for restored files that already had the right owner but not for the recovery files it writes. The Posix driver now also skips chown() when the owner is already correct.
    Issue #2036Pull request #2038Commit 3ff88ff
  • Add error retry detail for HTTP retries.
    An error after HTTP retries now reports how many retries were made and for how long.
    801e396 Move error modules to common/error directory.
    df419b3 Improve retry error messages.
    f33e1cf Add error retry detail for HTTP retries.
Documentation Improvements:
  • Add warning about using recovery type=none. (Reviewed by Stefan Fercot.)
    Recovery type=none does not increment the timeline at the end of recovery, which can cause PostgreSQL to archive duplicate WAL that is rejected and fills the disk, and can make tools like pg_rewind fail or corrupt the cluster. The documentation now warns against it.
    Issue #2062Pull request #2069Commit a28b72c
  • Add note about running stanza-create on already-created repositories.
    Stanzas that already exist are skipped, so stanza-create is safe to run again, e.g. after a new repository is configured.
    Commit 0c1f823
v2.45 Release Notes
Block Incremental Backup (BETA)
Released March 20, 2023
Bug Fixes:
  • Skip writing recovery.signal by default for restores of offline backups. (Reviewed by Stefan Fercot. Reported by Marcel Borger.)
    On PostgreSQL >= 12, recovery.signal makes PostgreSQL fail to start when the backup was made offline with wal_level=minimal. Restore now skips writing it for offline backups unless type is set explicitly, e.g. for Point-in-Time Recovery from an offline backup made with a higher wal_level.
    0818601 Set online flag in manifest in command/restore unit tests.
    2fa7e53 Skip writing recovery.signal by default for restores of offline backups.Issue #2018Pull request #2019
Features:
  • Block incremental backup (BETA). (Reviewed by John Morris, Stephen Frost, Stefan Fercot.)
    Block incremental stores only the parts of a file that changed since the prior backup instead of the entire file. Enable it with repo-block, which requires repo-bundle. The feature is in beta and also requires the beta option. Do not use it in production: the on-disk format will change before the GA release.
    089fae0 Add block incremental to real/all test output.
    3feed38 Improve IoChunkedRead end-of-file handling.
    d520816 Remove parameter list from deltaMapNew().
    779efe0 Consistently declare block incremental size as size_t.
    dffc933 Rename DeltaMap to BlockHash.
    a9867cb Add repo-block-age-map and repo-block-size-map options.
    16ac5ee Rename block incremental manifest keys.
    da91858 Add optional raw format for compression types.
    7e5adc0 Use raw compression/encryption to bundling and block incremental backup.
    8b5153a Block-level incremental backup super blocks.Pull request #1996
    210bed4 Use xxHash instead of SHA-1 for block incremental checksums.
    6252c0e Exclude backup set size from info for block incremental backups.
    1119a53 Rename BlockHash to BlockChecksum.
    6b409d0 Update default block size and super block values based on testing.
    1281a6e Ensure no continuations when block size equals super block size.
    5c1f78d Fix typo in blockIncrProcess().
    c30d3e4 Block incremental map fixes and improvements.Pull request #2026
    24f7252 Add beta feature infrastructure.
    04d224b Add beta feature and documentation for block incremental backup.
    Development in v2.44:
    912eec6 Block-level incremental backup.Issue #959Issue #1806Issue #1913Pull request #1916
    Development in v2.42:
    1ea6a41 Improve manifest file updates.Pull request #1888
    c647bcb Add manifest flags for file processing during backup.Pull request #1889
    102ce5d Add persistent reference list to manifest.Pull request #1890
    ac99201 Add bufferSize to cvtUInt64FromVarInt128() to further limit reads.
    5602f17 Add varint-128 encode/decode to IoRead/IoWrite.Pull request #1891
    352cbf1 Update ManifestFile booleans to bit fields.
    e7e106f Use read-only storage to calculate checksum in restoreFile().
    909be41 Swap command/backup and command/restore unit tests.
    1730ef4 Add noTruncate flag to storageNewWriteP().Pull request #1898
    fee38c2 Pass filters to remote storage as a handler array.
    5fceee8 Add backupFileRepoPathP().
    221db61 Shorten names in real/all integration test matrix.
    c9db7bc Update cipherBlockNew() to allow optional parameters.
    58b3c91 Add raw mode to CipherBlock to save space.
Improvements:
  • Keep only one all-default group index. (Reviewed by Stefan Fercot.)
    An option that does not apply, e.g. repo2-s3-key when repo2-type is not s3, could add a phantom repository that has only default values. Repository and PostgreSQL indexes with only default values are now removed unless no other index remains.
    Pull request #2011Commit 1648c13
Documentation Improvements:
  • Add explicit instructions for upgrading between 2.x versions. (Contributed by Christophe Courtois. Reviewed by David Steele.)
    A new user guide section explains that upgrading between 2.x versions requires only installing the new binaries. Downgrading is also possible when no features unsupported by the older version were used.
    Issue #1774Pull request #1993Commit 15d5dcd
  • Remove references to SSH made obsolete when TLS was introduced.
    The help for repo-host, pg-host, and their port options no longer refers to SSH as the only remote protocol. Details about SSH compression were also removed.
    Commit c5907a2
v2.44 Release Notes
Remove PostgreSQL 9.0/9.1/9.2 Support
Released January 30, 2023
Improvements:
  • Remove support for PostgreSQL 9.0/9.1/9.2. (Reviewed by Stefan Fercot.)
    pgBackRest now supports ten versions of PostgreSQL: the five supported versions and the last five EOL versions. As of PostgreSQL 15 that excludes 9.0, 9.1, and 9.2.
    Pull request #1941Commit 77c721e
  • Restore errors when no backup matches the current version of PostgreSQL. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Soulou.)
    After a stanza-upgrade with no new backup, restore used the latest backup, which was made from the prior PostgreSQL version. Replicas built this way could fail. Restore now errors in this case unless set or type=time/type=lsn is specified.
    Issue #1942Pull request #1967Commit b9be4fa
  • Add compress-level range checking for each compress-type. (Reviewed by Stefan Fercot. Suggested by gkleen, ViperRu.)
    The compress-level range was 0-9 for every compress-type. This is the range for gz but a subset of what lz4 and zst allow. Each type now has its own range, e.g. -7 to 22 for zst.
    Issue #1216Issue #1931Pull request #1962Commit 44da314
Documentation Improvements:
  • Add warning about enabling hierarchical namespace on Azure storage. (Reviewed by Stefan Fercot. Suggested by Vojtech Galda, Pluggi, asjonos.)
    With hierarchical namespace enabled, Azure requires directories to be deleted separately and expire fails. The user guide now warns not to enable it.
    Issue #1770Pull request #1980Commit 3aea997
  • Add replacement for linefeeds in monitoring example. (Reviewed by Stefan Fercot. Suggested by rudonx, gmustdie, Ivan Shelestov.)
    The monitoring example failed when info output contained a linefeed, e.g. in some error messages, because COPY converts \n to a linefeed that jsonb rejects. The example now replaces linefeeds before converting to jsonb.
    Issue #1910Pull request #1963Commit ae258f6
  • Clarify target-action behavior on various PostgreSQL versions. (Contributed by Chris Bandy. Reviewed by David Steele, Anton Kurochkin, Stefan Fercot. Suggested by Anton Kurochkin, Chris Bandy.)
    The effect of pause depends on hot_standby and the PostgreSQL version. With hot_standby=off, pause acts like shutdown on PostgreSQL >= 12 and like promote on earlier versions.
    Issue #987Issue #1314Pull request #1960Commit 84a3ff8
  • Updates and clarifications to index page. (Reviewed by Stefan Fercot.)
    The section about other backup solutions not supporting parallel processing was no longer accurate and has been reworded. Other sections were updated for current terminology and clarity.
    Pull request #1966Commit 36ee30d
  • Add dark mode to the website. (Suggested by Stephen Frost.)
    The website uses dark colors when the browser prefers a dark color scheme.
    Commit ed818b3
v2.43 Release Notes
Bug Fix
Released November 28, 2022
Bug Fixes:
  • Fix missing reference in diff/incr backup. (Reviewed by Stefan Fercot. Reported by Marcel Borger, ulfedf, jaymefSO.)
    In v2.42 a diff or incr backup failed with unable to find '...' in string list when its prior backup was made by an earlier version. The reference list built for a manifest from an earlier version now includes the backup's own label.
    28f6604 Move manifestBackupLabelSet() test in info/manifest unit.
    c4bf775 Fix missing reference in diff/incr backup.Issue #1933Pull request #1935
Improvements:
  • Add hint when an option is specified without an index. (Reviewed by Stefan Fercot.)
    An option that requires an index, e.g. repo-storage-ca-file, now errors with a hint that shows the indexed name, e.g. repo1-storage-ca-file. A deprecated option also gets a hint with its current name.
    Issue #1925Pull request #1929Commit 3f363cb
v2.42 Release Notes
Bug Fixes
Released November 22, 2022
Bug Fixes:
  • Fix memory leak in file bundle backup/restore. (Reviewed by John Morris. Tested by Oscar. Reported by Oscar.)
    Restore freed some I/O objects for bundled files only after every file in the bundle was restored. With many small files in a bundle, memory use grew until restore failed with unable to allocate. Backup had a similar, smaller leak when resuming.
    Issue #1877Pull request #1878Commit cd8db7d
  • Fix protocol error on short read of remote file. (Reviewed by Stephen Frost.)
    A remote file read that stopped early or failed partway left the protocol in a bad state. The next command failed with client state is 'data-get' but expected 'idle' and the original error was lost. The rest of the file is now read when the file is closed.
    ea162e8 Move storageReadRemoteOpen() in storage/remote/read module.
    7967c75 Fix protocol error on short read of remote file.Pull request #1900
Improvements:
  • Do not store references for zero-length files when bundling. (Reviewed by Stefan Fercot.)
    Bundled diff/incr backups stored a reference to a prior backup for zero-length files, which are not stored in any backup. The reference was ignored on restore but used space in the manifest.
    Pull request #1887Commit f981fb4
  • Use more generic descriptions for pg_start_backup()/pg_stop_backup(). (Reviewed by Greg Sabino Mullane, David Christensen. Suggested by Greg Sabino Mullane.)
    PostgreSQL 15 renamed these functions to pg_backup_start()/pg_backup_stop(). The logs and documentation now use descriptions that do not depend on the PostgreSQL version.
    Pull request #1901Commit ddd966c
Test Suite Improvements:
  • Update test.pl --psql-bin option to match command-line help. (Contributed by Koshi Shibagaki. Reviewed by David Steele.)
    The help showed --psql-bin but the option was named --pgsql-bin. The option now has the name shown in the help.
    Pull request #1920Commit 3ad5884
v2.41 Release Notes
Backup Annotations
Released September 19, 2022
Bug Fixes:
  • Fix incorrect time expiration being used for non-default repositories. (Reviewed by Stefan Fercot. Reported by Adam Brusselback.)
    When expire ran without repo, the time-based repo-retention-full for every repository was read from the lowest-numbered repository. Each repository now uses its own value.
    Issue #1852Pull request #1854Commit eda7f81
  • Fix issue when listing directories recursively with a filter. (Reviewed by Stephen Frost. Reported by Efremov Egor.)
    repo-ls with recurse and filter could fail with a double free or segfault when the last entry in a directory was a directory that had entries. A parent list was freed before the entries of its subdirectories were processed.
    Issue #1870Pull request #1871Commit c39c9f2
Features:
  • Backup key/value annotations. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Adam Berlin.)
    Add key/value annotations to a backup with the annotation option of the backup command, and add, modify, or remove them later with the new annotate command. The info command shows annotations in text output when set is specified and always includes them in JSON output.
    Issue #1709Pull request #1724Commit 381fd0a
Improvements:
  • Support --set in JSON output for info command. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Anton Kurochkin.)
    With set, JSON output from info now includes the details that text output already showed, e.g. the error and tablespace lists.
    Issue #1484Pull request #1501Commit db75ffd
  • Allow upload chunk size to be configured for object stores. (Reviewed by Stefan Fercot. Suggested by Anton Glushakov.)
    Set repo-storage-upload-chunk-size to change the size of each upload chunk for S3, GCS, or Azure. Larger chunks can improve performance but use more memory. The defaults are unchanged.
    ff1188f Allow quote tag in command-line help.
    b05d31f Allow upload chunk size to be configured for object stores.Issue #1843Pull request #1847
  • Update archive.info timestamps after a successful backup. (Reviewed by Stefan Fercot. Suggested by Alex Richman.)
    archive.info and its copy were written only by stanza-create and stanza-upgrade, and an object store lifecycle policy that expires old files could remove them. Their timestamps are now updated after each successful backup.
    Issue #1853Pull request #1855Commit 3b5df1e
  • Move standby timeline check after checkpoint. (Reviewed by Stefan Fercot. Tested by Keith Fiske. Suggested by Keith Fiske.)
    The check that the standby is on the expected timeline used pg_control as it was before the backup started. Right after a promotion the standby may not have written the new timeline to pg_control yet, and the backup failed with standby is on timeline ... but expected .... The check now runs after the checkpoint.
    Issue #1816Pull request #1819Commit c267ba5
  • Improve warning message on backup resume. (Suggested by Cynthia Shang.)
    The warning read as if the user needed to take action. It now states that invalid files will be removed and then the backup will resume.
    Commit cbbe93f
Documentation Improvements:
  • Add absolute path for kill in pgbackrest.service. (Suggested by Don Seiler.)
    The pgbackrest.service example now uses /bin/kill in ExecReload. Older versions of systemd require an absolute path.
    Commit e9ff524
v2.40 Release Notes
OpenSSL 3 Support
Released July 18, 2022
NOTE TO PACKAGERS: An experimental meson build has been added but packagers should continue to use the autoconf/make build for the foreseeable future.
Improvements:
  • OpenSSL 3 support. (Reviewed by Stephen Frost.)
    pgBackRest now builds with OpenSSL 3. Deprecation warnings are suppressed, and an unexpected EOF is detected with the error codes that OpenSSL 3 reports.
    Pull request #1767Commit 08242ee
  • Create snapshot when listing contents of a path. (Reviewed by John Morris, Stephen Frost.)
    A path listed without sorting was read while its entries were processed. When files were deleted during the listing, some filesystems skipped files that still existed and the path could not be removed. Listings now always use a snapshot of the path.
    61ca9b5 Replace hrnStorageInfoListCallback() with TEST_STORAGE_LIST() in tests.
    1c0bf0b Reorder null user/group checks in remote protocol and add tests.
    72960bb Rename strTrunc() to strTruncIdx() and add strTrunc().
    75623d4 Create snapshot when listing contents of a path.Issue #1754Pull request #1805
  • Force target-timeline=current when restore type=immediate. (Reviewed by Stephen Frost.)
    On PostgreSQL >= 12 the default target timeline is latest, and recovery could fail when the latest timeline in the repository cannot be reached from the backup. Restore with type=immediate does not change timelines, so it now sets the target timeline to current.
    Pull request #1814Commit 364af16
  • Truncate files during delta restore when they are larger than expected. (Reviewed by Stephen Frost.)
    Delta restore copied a file from the repository whenever its size differed from the manifest. A larger file is now truncated to the size in the manifest and copied only when its checksum does not match. Files that were only appended to, e.g. logs, are no longer copied.
    Pull request #1758Commit 7ec51e7
  • Disable incremental manifest save when resume=n. (Contributed by Reid Thompson. Reviewed by David Steele.)
    During a backup the manifest is saved periodically for use by resume. These saves are now skipped when resume=n.
    Pull request #1760Commit 60d70fa
  • Set backup percent complete to zero before copy start. (Contributed by Reid Thompson. Reviewed by David Steele.)
    The progress of a running backup was not reported until the first file was copied. It is now set to zero when copying starts.
    Pull request #1759Commit 38ad283
  • Use S3 IsTruncated flag to determine list continuation. (Reviewed by John Morris. Tested by Soulou. Suggested by Christian Montagne.)
    Recent versions of Scality send an empty NextContinuationToken when a list is not truncated, which caused an infinite loop. The IsTruncated flag now determines whether a list continues, and a missing or empty token for a truncated list is an error.
    Issue #1748Pull request #1783Commit 845d82f
Documentation Bug Fixes:
  • Skip internal options in the configuration reference. (Reported by Francisco Miguel Biete Banon.)
    Options that are internal only for some commands were listed in the configuration reference. The command-line help was not affected.
    Commit 2643050
Documentation Improvements:
  • Add link to PostgreSQL configuration in repository host section. (Reviewed by Stefan Fercot. Suggested by Julien Cigar.)
    Pull request #1809Commit 74a4ac8
Test Suite Improvements:
  • Add experimental Meson build. (Reviewed by Eli Schwartz, Sam Bassaly.)
    Meson has simpler syntax and better performance than autoconf/make. It is used for code generation and test builds, and errors on anything but a debug build. Packagers should continue to use autoconf/make.
    2feaaea Add .inc extension to C files included in other C files.
    b8fc20d Add experimental Meson build.Pull request #1765
    665da12 Update meson.build include comment to be more general.
    9fd85a1 Disable meson for all but debug builds.
    59f148b Generate -fmacro-prefix-map to strip relative path from __FILE__.
    0eccbc8 Meson builds work on version 0.45.
  • Allow any path to be passed to the --test-path option. (Contributed by Andrey Sokolov. Reviewed by David Steele.)
    Pull request #1764Commit 29b2a54
  • Fix compile error when DEBUG_EXEC_TIME is defined without DEBUG. (Contributed by Andrey Sokolov. Reviewed by David Steele.)
    Without DEBUG the ASSERT() that used the start time expanded to nothing, and the unused variable caused a compile error. The check now uses CHECK().
    Pull request #1775Commit 1a00ab1
v2.39 Release Notes
Verify and File Bundling
Released May 16, 2022
Bug Fixes:
  • Fix error thrown from FINALLY() causing an infinite loop. (Reviewed by Stephen Frost.)
    An error thrown in a FINALLY() block returned to the same block, where the error was likely thrown again and caused an infinite loop. FINALLY() now runs only once.
    9a271e9 Fix error thrown from FINALLY() causing an infinite loop.Pull request #1732
    ef4c4ab Use variable instead of function to track FINALLY() state.Pull request #1744
  • Error on all lock failures except another process holding the lock. (Reviewed by Reid Thompson. Tested by Geir Råness. Reported by Geir Råness.)
    archive-get/archive-push did not error when the lock could not be taken for another reason, e.g. permissions, and the async process was not launched. The user got a generic failure message and there was no async log. These failures are now errors.
    Issue #1722Pull request #1730Commit 9629908
Features:
  • Backup file bundling for improved small file support. (Reviewed by Reid Thompson, Stefan Fercot, Chris Bandy.)
    Bundling stores small files together in the repository to save time and space during backup, especially on object stores where creating each file is expensive. Enable it with repo-bundle and tune it with repo-bundle-size and repo-bundle-limit. Restore reads adjacent files in a bundle with a single read.
    4d2fef1 Remove redundant restoreFile() test and improve coverage.
    5141370 Add limit parameter to ioCopyP().
    dca6da8 Optimize restore command for file bundling.Issue #1149Pull request #1683
    0054677 Add bundle logging to backup command.
    3f66f42 Rename bundle-* options to repo-bundle-*.Pull request #1694
    fe9fd2f Disable repo-hardlink option when repo-bundle option is enabled.
    424008d Allow files that become zero-length after the backup manifest is built.
    f405fc6 Backup file bundling documentation.Pull request #1740
    Development in v2.38:
    a79034a Add read range to all storage drivers.Pull request #1619
    b791f1c Implement restore ownership without updating manifest internals.
    f84909f Fix issue with Posix read offset handling after an error.
    4ca0590 Refactor backupProcessQueue() to use BackupJobData struct.
    16559d9 Use the PG_FILE_POSTMTRPID constant where appropriate.
    8c062e1 Remove primary flag from manifest.Pull request #1630
    b0db4b8 Simplify base path mode in mock/all integration tests.
    4a73a02 Simplify manifest defaults.Pull request #1631
    575ae77 Convert varNewUInt64() to VARUINT64() where possible in manifest.
    61ce586 Pack manifest file structs to save memory.Pull request #1626
    ca13f11 Optimization for jsonFromStrInternal().
    e4df5b7 Simplify manifest file defaults.Pull request #1632
    7dd657b Simplify filename construction in command/verify module.
    34d6495 Bundle files in the repository during backup.Issue #1149Pull request #1655
    efc09db Limit files that can be bundled.Issue #1149Pull request #1662
    e6e1122 Pass file by reference in manifestFileAdd().
  • Verify command to validate the contents of a repository. (Contributed by Cynthia Shang, Reid Thompson. Reviewed by David Steele, Stefan Fercot.)
    The verify command checks that the backups and WAL in the repository are valid. Set output=text to print the results and verbose to include what was verified successfully along with errors.
    Pull request #1607Commit 65d22e4
  • PostgreSQL 15 support. (Reviewed by Stefan Fercot.)
    PostgreSQL 15 removes exclusive backup, renames the backup start/stop functions, and drops pg_database.datlastsysoid, which pgBackRest no longer uses.
    692fe49 Remove dependency on pg_database.datlastsysoid.Pull request #1735
    20782c8 PostgreSQL 15 support.Pull request #1736
  • Show backup percent complete in info output. (Contributed by Reid Thompson. Reviewed by David Steele.)
    info shows the percent complete of a running backup on the host where the backup is running. JSON output reports it as an integer, the percent times 100.
    d9088b2 Show backup percent complete in info output.Issue #1490Pull request #1621
    a6b1adb Remove extraneous linefeed when writing a lock file.
    Development in v2.38:
    22734eb Add ioBufferReadNewOpen() and ioBufferWriteNewOpen().
    9b2f10d Refactor lock code.Pull request #1643
  • Auto-select backup for restore command --type=lsn. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele.)
    With type=lsn, restore now selects the most recent backup that stopped at or before the target LSN.
    08d9e26 Refactor target type checking for clarity in restore module.
    d8d4132 Auto-select backup for restore command --type=lsn.Pull request #1706
    ea4d73f Fix ordering of backup-lsn-stop field in command/restore unit test.
  • Suppress existing WAL warning when archive-mode-check is disabled. (Contributed by Reid Thompson. Reviewed by David Steele.)
    With archive-mode-check=n and WAL pushed from multiple hosts, a WAL file already in the archive with the same checksum is expected and no longer logs a warning. A different checksum is still an error.
    Pull request #1711Commit aad7171
  • Add AWS IMDSv2 support. (Contributed by Nuno Pires. Reviewed by David Steele.)
    IMDSv2 protects the instance metadata that provides credentials for repo-s3-key-type=auto. pgBackRest falls back to IMDSv1 when the IMDSv2 token request fails.
    Issue #1685Pull request #1691Commit 21cef09
Improvements:
  • Allow repo-hardlink option to be changed after full backup. (Reviewed by Reid Thompson.)
    Changing repo-hardlink required a new full backup. It can now be changed at any time, e.g. to disable it for file bundling.
    Pull request #1699Commit 7afaac0
  • Increase precision of percent complete logging for backup and restore. (Contributed by Reid Thompson. Reviewed by David Steele.)
    Percent complete is now logged with hundredths, which gives more frequent updates for large backups and restores.
    Pull request #1679Commit 330e199
  • Improve path validation for repo-* commands. (Contributed by Reid Thompson. Reviewed by David Steele.)
    A path may not contain //, a trailing / is removed, and an absolute path must be in the repository path.
    5ae84d5 Improve path validation for repo-* commands.Pull request #1697
    19dd015 Fix issues in improved path validation for repo-* commands.Pull request #1751
  • Improve stop command to honor stanza option. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by ragaoua.)
    stop with force and stanza now terminates only the processes holding locks for that stanza. Before, it terminated processes for all stanzas.
    Issue #1608Pull request #1676Commit f7ab002
  • Improve error message for invalid repo-azure-key. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Seth Daniel.)
    With repo-azure-key-type=shared, repo-azure-key is now checked for valid base64.
    Issue #1686Pull request #1688Commit 7c9208b
  • Add hint to check the log on archive-get/archive-push async error. (Reviewed by Reid Thompson.)
    When the async process cannot write its status files, archive-get/archive-push times out with a generic error. The error now includes a hint with the path of the async log, which contains the actual error.
    Issue #1722Commit 03c71aa
  • Add ClockError for unexpected clock skew and timezone changes. (Reviewed by Greg Sabino Mullane, Stefan Fercot. Suggested by Greg Sabino Mullane.)
    Pull request #1739Commit 5f8c9cd
  • Strip extensions from history manifest before showing in error message. (Reviewed by Stefan Fercot.)
    When a new backup label was not later than the latest label from the backup history, the error showed the history manifest name, e.g. 20220504-222042F_20220504-222141I.manifest.gz. It now shows only the label.
    Pull request #1738Commit b6bfd9f
  • Add user:group to lock permission error. (Reviewed by Reid Thompson.)
    Issue #1722Commit 0055fa4
  • Add hint when unable to find the primary cluster.
    The unable to find primary cluster error now includes a hint asking whether all available clusters are in recovery.
    Commit 46b7b72
Documentation Bug Fixes:
  • Fix incorrect reference to stanza-update in the user guide. (Fixed by Abubakar Mohammed. Reviewed by David Steele.)
    Commit 3dd7960
  • Fix example for repo-gcs-key-type option in configuration reference. (Reviewed by Reid Thompson.)
    The example was none, which is not a valid value. It is now auto.
    Commit 166039c
  • Fix tls-server-auth example and add clarifications. (Reviewed by Reid Thompson.)
    The example was missing the client CN and is now client-cn=stanza1. The help also states that wildcards are not allowed for the client CN.
    Commit c222ce1
Documentation Improvements:
  • Simplify messaging around supported versions in the documentation. (Reviewed by Stefan Fercot, Reid Thompson, Greg Sabino Mullane.)
    The user guides listed PostgreSQL version ranges, which made it look like other versions were not supported, e.g. PostgreSQL 12 on RHEL. The ranges were removed and the introduction now explains which versions the user guides cover.
    Pull request #1742Commit eefa0b1
  • Add option type descriptions. (Contributed by Reid Thompson. Reviewed by David Steele.)
    The configuration reference now describes each option type, e.g. the units for size options, rather than repeating the description for each option.
    Pull request #1747Commit a913113
  • Add FAQ about backup types and restore speed. (Contributed by David Christensen. Reviewed by Reid Thompson.)
    A new FAQ explains that the backup type affects only the backup and not how fast a restore runs.
    Pull request #1731Commit 50d409a
  • Document required base branch for pull requests. (Contributed by David Christensen. Reviewed by Reid Thompson.)
    The contributing guide now states that pull requests must be based on the integration branch.
    Pull request #1733Commit cc5b061
v2.38 Release Notes
Minor Bug Fixes and Improvements
Released March 6, 2022
IMPORTANT NOTE: Repository size reported by the info command is now entirely based on what pgBackRest has written to storage. Previously, in certain cases, pgBackRest could detect if additional compression was being applied by the storage but this is no longer supported.
Bug Fixes:
  • Retry errors in S3 batch file delete. (Reviewed by Reid Thompson. Reported by Alex Richman.)
    A failed batch delete was retried, but errors for individual files in the batch were not, and pgBackRest could terminate during expire or when removing a backup that could not be resumed. Files that fail in a batch are now deleted individually with the usual HTTP retries.
    Issue #1650Pull request #1653Commit 551e5bc
  • Allow case-insensitive matching of HTTP connection header values. (Reviewed by Reid Thompson. Reported by Rémi Vidier.)
    The HTTP specification allows connection header values in any case, e.g. Close.
    Issue #1638Pull request #1675Commit b33cabe
Features:
  • Add support for AWS S3 server-side encryption using KMS. (Contributed by Christoph Berg. Reviewed by David Steele. Tested by Tharindu Amila.)
    Set repo-s3-kms-key-id to encrypt objects with a key managed by AWS KMS.
    Issue #1430Pull request #1567Commit 3097acd
  • Add archive-missing-retry option. (Reviewed by Stefan Fercot.)
    By default archive-get retries a WAL segment it already reported as missing, which ignores missing notifications left in the spool path by a prior restore. With archive-missing-retry=n, PostgreSQL can detect the end of the archive and switch to streaming even while WAL is being archived. The spool path must then be empty when recovery starts.
    Pull request #1642Commit 10038db
  • Add backup type filter to info command. (Contributed by Stefan Fercot. Reviewed by David Steele.)
    Set type to show only backups of that type, e.g. type=full.
    Issue #600Pull request #1651Commit b26097f
Improvements:
  • Retry on page validation failure during backup. (Reviewed by Stephen Frost, David Christensen.)
    A page that fails checksum validation is now read again. It is reported as an error only when it has not changed. A changed page is being written by PostgreSQL, and the WAL contains a full page image. Zero pages are now checked to be all zeroes rather than only by the pd_upper field.
    b489707 Move command/backup-common tests in the command/backup module.
    9eec98c Retry on page checksum validation failure during backup.Issue #952Pull request #1667
    53f1b25 Improve validation of zero pages.Issue #952Pull request #1669
  • Handle TLS servers that do not close connections gracefully. (Reviewed by David Christensen, Stephen Frost. Tested by Rémi Vidier.)
    Some servers close the socket rather than the TLS connection. When a response has no content length or chunked encoding and its content can be parsed, the closed socket is now treated as EOF. These servers stopped working in v2.35.
    Issue #1638Pull request #1677Commit 59a5373
  • Add backup LSNs to info command output. (Contributed by Stefan Fercot. Reviewed by David Steele.)
    The start and stop LSNs of a backup are shown with set and in JSON output, for use with LSN-based Point-in-Time Recovery.
    Pull request #1610Commit d866dd5
  • Automatically strip trailing slashes for repo-ls paths. (Contributed by David Christensen. Reviewed by David Steele.)
    A trailing slash on the path made repo-ls show no content on some repository storage types.
    Pull request #1673Commit 6320712
  • Do not retry fatal errors. (Reviewed by Reid Thompson.)
    Retrying fatal errors, especially out of memory errors, may cause lockups. Assertion and memory errors are now reported without retrying.
    8573a2d Improve protocol module error test for protocolClientFree().
    8d0cce6 Use normal error for protocol module error retry test.
    8046f06 Do not retry fatal errors.Pull request #1658
  • Remove support for PostgreSQL 8.3/8.4. (Reviewed by Reid Thompson, Stefan Fercot.)
    There is no evidence that these versions are still in use and supporting them costs development and test time. Use an older version of pgBackRest to back up or migrate from these versions.
    Pull request #1611Commit bb4b30d
  • Remove logic that tried to determine additional file system compression. (Reviewed by Reid Thompson, Stefan Fercot.)
    After a file was copied, pgBackRest checked its size on disk to detect compression by the file system, e.g. ZFS. This was untested, slow with many small files, and blocked file bundling. The repository size in info is now what pgBackRest wrote.
    Pull request #1652Commit cb630ff
Documentation Bug Fixes:
  • Move repo options in TLS documentation to the global section. (Reported by Anton Kurochkin.)
    The user guide set these options in the stanza section, which works in some cases but is not best practice.
    Commit 53de3e3
  • Remove unused backup-standby option from stanza commands. (Reported by Stefan Fercot.)
    stanza-create and stanza-upgrade listed backup-standby in the command reference and help, but did not use it.
    Commit 5cbb4a3
  • Fix typos in help and release notes. (Fixed by Daniel Gustafsson. Reviewed by David Steele.)
    cacfdd9 Fix typos in help and release notes.Pull request #1671
    5c332a0 Fix release note attribution in cacfdd94 and f798458e.
Documentation Improvements:
  • Add aliveness check to systemd service configuration. (Suggested by Yogesh Sharma.)
    The pgbackrest.service example now checks that the server is still running shortly after it starts. Before, a server that failed to start reported no error while systemd retried in the background.
    Commit f1bdf3e
  • Add FAQ explaining WAL archive suffix. (Contributed by Stefan Fercot. Reviewed by David Steele.)
    The suffix on a WAL file in the repository is its SHA1 checksum, which is used to verify the file. It cannot be omitted.
    Issue #1485Issue #1663Pull request #1664Commit 98d525d
  • Note that replications slots are not restored. (Contributed by Reid Thompson. Reviewed by David Steele, Stefan Fercot. Suggested by Christophe Courtois.)
    Issue #1585Pull request #1601Commit a82f017
v2.37 Release Notes
TLS Server
Released January 3, 2022
IMPORTANT NOTE: If the restore command is unable to find a backup that matches a specified time target then an error will be thrown, whereas before a warning was logged.
Bug Fixes:
  • Fix restore delta link mapping when path/file already exists. (Reviewed by Reid Thompson. Reported by Younes Alhroub.)
    When link-map or link-all mapped a path or file that already existed to a link, a delta restore left the existing path or file in place. It is now replaced by the link.
    Issue #1588Pull request #1589Commit a73fe4e
  • Fix socket leak on connection retries. (Reviewed by Reid Thompson. Reported by James Coleman.)
    In optimized builds the socket of a failed connection attempt was not closed, and each retry leaked a socket.
    Issue #1593Pull request #1594Commit 615bdff
Features:
  • Add TLS server. (Reviewed by Stephen Frost, Reid Thompson. Tested by Andrew L'Ecuyer.)
    The server command starts a TLS server that can be used instead of SSH to reach repository and PostgreSQL hosts. Set repo-host-type=tls or pg-host-type=tls on the client, and configure certificates and client authorization with the tls-server-* options. SIGHUP reloads the configuration and SIGTERM stops the server.
    2d963ce Rename server-start command to server.
    7b3ea88 Add SIGTERM and SIGHUP handling to TLS server.Pull request #1572
    f06101d Add TLS server documentation.Pull request #1597
    Development in v2.36:
    ccc255d Add TLS Server.Pull request #1504
    13d4559 Check return value of getsockopt().
    Development in v2.35:
    947c7a8 Refactor cmdLocal()/cmdRemote() to accept a ProtocolServer object.
    0999de0 Move noop from protocolClientNew() to local/remote initialization.
  • Add --cmd option. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele. Suggested by Virgile CREVON.)
    Set cmd to the pgBackRest executable used in commands that pgBackRest writes, e.g. restore_command in recovery.conf.
    Issue #1537Pull request #1540Commit 2a57647
Improvements:
  • Check archive immediately after backup start. (Reviewed by Reid Thompson, David Christensen.)
    A backup failed because WAL archiving was not working only after all files were copied. The archive is now checked right after the backup starts. On an idle system a restore point is created to switch the WAL segment.
    Issue #152Pull request #1570Commit 1f14f45
  • Add timeline and checkpoint checks to backup. (Reviewed by Stefan Fercot, Reid Thompson.)
    The backup now checks that pg_control has the checkpoint and timeline of the backup start, and that a standby is following the primary, i.e. it is on the same timeline and its checkpoint is not later than the backup checkpoint.
    Pull request #1581Commit 49145d7
  • Check that clusters are alive and correctly configured during a backup. (Reviewed by Stefan Fercot.)
    A backup now fails as soon as a cluster stops or the standby is promoted. Before, a stopped primary was detected only at the end of the backup and a promoted standby was not detected.
    Pull request #1584Commit bd2ba80
  • Error when restore is unable to find a backup to match the time target. (Reviewed by Reid Thompson, Douglas J Hunley. Suggested by Douglas J Hunley.)
    This was a warning that was easy to miss, and a lot of time could be lost restoring a backup that cannot reach the target.
    Issue #1582Pull request #1586Commit be4ac39
  • Parse protocol/port in S3/Azure endpoints. (Contributed by Reid Thompson. Reviewed by David Steele.)
    repo-s3-endpoint, repo-azure-endpoint, and repo-storage-host may now include a protocol and port, e.g. https://host:9000.
    Issue #1560Pull request #1591Commit 6a12458
  • Add warning when checkpoint_timeout exceeds db-timeout. (Contributed by Stefan Fercot. Reviewed by David Steele.)
    With start-fast=n, a backup can time out waiting for the checkpoint when checkpoint_timeout is greater than db-timeout. The backup now warns in this case.
    Pull request #1564Commit 6723305
  • Add verb to HTTP error output. (Contributed by Christoph Berg. Reviewed by David Steele.)
    Pull request #1566Commit c38e2d3
  • Allow y/n arguments for boolean command-line options. (Contributed by Reid Thompson. Reviewed by David Steele.)
    Pull request #1571Commit 1a0560d
  • Make backup size logging exactly match info command output. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Mahomed Hussein.)
    The logged backup size now includes only the files copied, plus backup_label and tablespace_map. Before, e.g. an incremental backup after a switchover logged the size of every file checked.
    6e63576 Match backup log size with size reported by info command.Issue #766Pull request #1562
    afe77e7 Update contributor for 6e635764.
    1e77fc3 Include backup_label and tablespace_map file sizes in log output.Pull request #1568
    a3d7a23 Use infoBackupDataByLabel() to log backup size.Pull request #1573
    dcb4f09 Revert changes to backupFilePut() made in 1e77fc3d.
Documentation Improvements:
  • Display size option default and allowed values with appropriate units. (Reviewed by Reid Thompson.)
    cff961e Centralize logic to build value lists during config rendering.
    038abaa Display size option default and allowed values with appropriate units.Pull request #1557
  • Fix typos and improve documentation for the tablespace-map-all option. (Reviewed by Reid Thompson. Suggested by Reid Thompson.)
    Commit df89eff
  • Remove obsolete statement about future multi-repository support. (Suggested by David Christensen.)
    The documentation stated that only one repository was allowed, but multiple repositories have been supported since v2.33.
    Commit dea7524
v2.36 Release Notes
Minor Bug Fixes and Improvements
Released November 1, 2021
Bug Fixes:
  • Allow global as a stanza prefix. (Reviewed by Stefan Fercot. Reported by Younes Alhroub.)
    A stanza name that begins with global, e.g. global_stanza, was rejected because the config parser treated it as a global section.
    Issue #1524Pull request #1525Commit 498902e
  • Fix segfault on invalid GCS key file. (Reviewed by Stephen Frost. Reported by Henrik Feldt.)
    A GCS key file without the client email, private key, or token URI caused a segfault. It now causes an error.
    Issue #1536Commit 2cea005
Improvements:
  • Allow link-map option to create new links. (Reviewed by Don Seiler, Stefan Fercot, Chris Bandy. Suggested by Don Seiler.)
    link-map could only remap links that were in the backup. It can now create a new link for any path or file in the backup.
    d89a677 Refactor restoreManifestMap() tests in the command/restore unit.
    cf1a575 Refactor restoreManifestMap() to be driven by link-map.
    68c5f3e Allow link-map option to create new links.Issue #1409Pull request #1520
  • Increase max index allowed for pg/repo options to 256. (Reviewed by Cynthia Shang.)
    The maximum index was 8 for pg options and 4 for repo options. Both are now 256, e.g. pg256-path.
    02b06aa Increase max index allowed for pg/repo options to 256.Pull request #1481
    71047a9 Use strncpy() to limit characters copied to optionName.
  • Add WebIdentity authentication for AWS S3. (Reviewed by James Callahan, Reid Thompson, Andrew L'Ecuyer. Tested by Benjamin Blattberg.)
    Set repo-s3-key-type=web-id to get credentials automatically in an EKS environment.
    cfd8233 Refactor S3 storage driver for additional auth methods.
    3879bc6 Add WebIdentity authentication for AWS S3.Issue #1203Pull request #1527
    1152f7a Fix mismatched parameters in tlsClientNew() call.
  • Add repo-azure-uri-style option.
    Set repo-azure-uri-style to host (default) or path to select the URI style for Azure. Azurite uses host-style URIs when the endpoint looks like a multipart host name.
    Commit 096829b
  • Report backup file validation errors in backup.info. (Contributed by Stefan Fercot. Reviewed by David Steele.)
    Errors found during a backup, e.g. invalid page checksums, were shown only by info with set. A flag in backup.info now shows in text and JSON output that a backup has errors. The details still require set.
    Issue #1484Pull request #1508Commit 34f7873
  • Add recovery start time to online backup restore log. (Reviewed by Tom Swartz, Stefan Fercot. Suggested by Tom Swartz.)
    Restore now logs the time that recovery will start from, which shows how much WAL must be replayed.
    Issue #1507Pull request #1518Commit 9e79f0e
  • Report original error and retries on local job failure. (Reviewed by Stefan Fercot.)
    When a local job fails after its retries, the error now reports the first error with its stack trace, followed by the retry errors. Before, only the last error was reported, which was often a cascade failure such as the protocol being out of sync.
    136d309 Allow stack trace to be specified for errorInternalThrow().
    ae40ed6 Add jobRetry parameter to HRN_CFG_LOAD().
    6af827c Report original error and retries on local job failure.Pull request #1521
  • Rename page checksum error to error list in info text output. (Reviewed by Stefan Fercot.)
    The label in info text output is now error list rather than page checksum error. A file may have an alignment error rather than a page checksum error.
    Pull request #1519Commit 9346895
  • Add hints to standby replay timeout message. (Reviewed by Cynthia Shang, Stefan Fercot. Suggested by Leigh Downs.)
    The timeout waiting for the standby to replay now has hints to check that replication is running and to disable backup-standby to back up from the primary.
    Issue #1434Commit b7ef12a
Documentation Bug Fixes:
  • Fix incorrect host name in user guide. (Fixed by Reid Thompson. Reviewed by David Steele.)
    The replication setup used the IP address of pg-primary in pg_hba.conf rather than the IP address of pg-standby.
    Pull request #1544Commit 1cb8ae1
Documentation Improvements:
  • Improve option documentation.
    The help for buffer-size now describes all the operations that use it, the descriptions of cmd-ssh, repo-host-cmd, and pg-host-cmd are consistent, and archive-header-check moved to the archive section of the configuration reference.
    3763624 Move archive-header-check option to the archive reference section.
    e1f6c06 Improve documentation for buffer-size option.
    fa564ee Improve documentation for cmd-ssh, repo-host-cmd, pg-host-cmd options.
v2.35 Release Notes
Binary Protocol
Released August 23, 2021
IMPORTANT NOTE: The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail
Bug Fixes:
  • Detect errors in S3 multi-part upload finalize. (Reviewed by Cynthia Shang. Tested by Marco Montagna. Reported by Marco Montagna, Lev Kokotov, Anderson A. Mallmann.)
    Completing a multi-part upload can fail even when S3 returns HTTP success. A response without an ETag is now treated as a failure, and the job is retried.
    8e1807c Detect errors in S3 multi-part upload finalize.Issue #1116Issue #1290Issue #1433Pull request #1438
    25508a4 Add contributors missed in 8e187cdb.
  • Fix detection of circular symlinks. (Reviewed by Stefan Fercot. Reported by Rohit Raveendran.)
    Links were followed before they were checked, and a circular link sent the manifest build into endless recursion and a crash.
    Issue #1447Pull request #1452Commit 43b8746
  • Only pass selected repo options to the remote. (Reviewed by David Christensen, Cynthia Shang. Reported by Greg Sabino Mullane, David Christensen.)
    Options for other repositories could cause conflicts on the remote, which uses exactly one repository.
    Pull request #1480Commit 935de1e
Improvements:
  • Binary protocol. (Reviewed by Cynthia Shang.)
    Local and remote processes now communicate with the binary pack format rather than JSON. Packs are more compact, faster to render and parse, and strictly typed. Each message is written in one place, which makes an error less likely to leave the protocol in a bad state, and messages have no size limit.
    6a1c033 Binary protocol.Pull request #1424
    2d5cbfc Fix memory leaks introduced by the binary protocol in 6a1c0337.
    f716cb6 Fix use after free introduced by the binary protocol in 6a1c0337.
    4fb6384 Fix more memory leaks introduced by the binary protocol in 6a1c0337.
  • Automatically create data directory on restore. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Chris Bandy.)
    A check migrated from Perl prevented restore from creating the data directory.
    Issue #1445Pull request #1454Commit 056a307
  • Allow restore --type=lsn. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang. Suggested by James Coleman.)
    Set type=lsn and an LSN in target to recover to an LSN on PostgreSQL >= 10.
    Issue #1339Pull request #1462Commit 563d117
  • Change level of backup/restore copied file logging to detail. (Reviewed by Stefan Fercot. Suggested by Jens Wilke.)
    This makes the info log level less noisy. Set the log level to detail to see each copied file.
    Issue #1277Pull request #1455Commit 849ab34
  • Loop while waiting for checkpoint LSN to reach replay LSN. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Fatih Mencutekin.)
    On a standby the checkpoint LSN can lag slightly behind the replay LSN until pg_control is updated. The backup now keeps checking rather than failing.
    Issue #1450Pull request #1460Commit 8fac1ac
  • Log backup file total and restore size/file total. (Reviewed by Cynthia Shang.)
    The backup size is now logged at the end of the backup and includes files added to the manifest after the main copy, e.g. backup_label and WAL. Restore now logs its size and file total.
    Pull request #1489Commit a0bdfa4
Documentation Bug Fixes:
  • Fix incorrect host names in user guide. (Reviewed by Stefan Fercot. Reported by Greg Sabino Mullane.)
    Some examples showed pg1 rather than the host name, and the install block installed packages on the build host no matter which host was specified.
    Pull request #1479Commit 155ef85
Documentation Improvements:
  • Update contributing documentation and add pull request template. (Contributed by Cynthia Shang. Reviewed by David Steele.)
    The contributing guide now has a table of contents and sections on understanding test output and submitting a pull request. GitHub now shows a template for new pull requests.
    e3d05e2 Update contributing documentation and add pull request template.Pull request #1469
    71b654f Fix links and update child process example.Pull request #1487
  • Rearrange backup documentation in user guide. (Reviewed by Cynthia Shang.)
    The Fast Start Option and Archive Timeout sections moved to Quick Start - Perform Backup. The Automatic Stop Option section, which applies only to PostgreSQL <= 9.6, was removed.
    Pull request #1488Commit 6ab18dc
  • Clarify restore --type behavior in command reference. (Contributed by Cynthia Shang. Reviewed by David Steele.)
    The reference now states that set must be specified when the target is before the latest backup and type is not time.
    Commit a6691c6
  • Fix documentation and comment typos. (Contributed by Eric Radman. Reviewed by David Steele.)
    Pull request #1446Commit 23bdc3d
Test Suite Improvements:
  • Add check for test path inside repo path. (Reviewed by Greg Sabino Mullane. Suggested by Greg Sabino Mullane.)
    A test path inside the repository path caused strange failures because the repository is copied into the test path. test.pl now errors in this case.
    Issue #1472Pull request #1474Commit e32f9e1
  • Add CodeQL static code analysis. (Reviewed by Cynthia Shang.)
    CodeQL now runs in CI. Issues it found were fixed, e.g. gmtime()/localtime() were replaced with gmtime_r()/localtime_r().
    Pull request #1457Commit 3c8819e
  • Update tests to use standard patterns. (Contributed by Cynthia Shang. Reviewed by David Steele.)
    3f9fbc3 Update command/expire test to use standard patterns.Pull request #1420
    03021c6 Update command/stanza test to use standard patterns.Pull request #1426
    8dd882b Update info/infoArchive test to use standard patterns.Pull request #1436
    ff95edd Update command/verify test to use standard patterns.Pull request #1435
    17c9ed0 Update command/info test to use standard patterns.Pull request #1432
    bffb43e Update command/check test to use standard patterns.Pull request #1431
    fb2d111 Update info/infoBackup test to use standard patterns.Pull request #1437
    065a8c8 Replace defines with string constants in unit tests.Pull request #1440
    c620811 Add timeModified parameter to HRN_STORAGE_PUT() macro.
    6a90d46 Update info/manifest test to use standard patterns.Pull request #1444
    62e8d97 Update command/archive tests to use standard patterns.Pull request #1449
    8bb0b28 Update command/control and command/command tests to use standard patterns.Pull request #1456
    f2ec4e0 Add cipher parameters to TEST_STORAGE_GET() macro.
    4ad0bbd Update command/backup tests to use standard patterns.Pull request #1463
    117ffe8 Update command/help tests to use standard patterns.Pull request #1464
    ebe5848 Update command/local and command/remote tests to use standard patterns.Pull request #1466
    a6cdf5b Update command/repo test to use standard patterns.Pull request #1467
    4992508 Update command/restore test to use standard patterns.Pull request #1468
    4c9ddf5 Update storage tests to use standard patterns.Pull request #1470
    eeaab6a Update config tests (except parse) to use standard patterns.Pull request #1473
    f653b59 Update db/db test to use standard patterns.Pull request #1486
    e17865a Update protocol/protocol test to use standard patterns.Pull request #1491
    eca2fc6 Update config/parse test to use standard patterns.Pull request #1492
v2.34 Release Notes
PostgreSQL 14 Support
Released June 7, 2021
Bug Fixes:
  • Fix issues with leftover spool files from a prior restore. (Reviewed by Cynthia Shang, Stefan Fercot. Tested by Floris van Nee. Reported by Floris van Nee.)
    A notification left in the spool path by a prior restore could make archive-get report a WAL segment as missing. When this notification is found on the first check, it is now removed and the async process is run again. An error is reported when the async process does not respond, rather than reporting the WAL segment as missing.
    Issue #1357Pull request #1371Commit f45e76f
  • Fix issue when checking links for large numbers of tablespaces. (Reviewed by Cynthia Shang. Tested by Avinash Vallarapu. Reported by Avinash Vallarapu.)
    The link check used a lot of memory with many tablespaces and could crash. The new check is much faster, uses far less memory, and also detects duplicate file links.
    Issue #1354Pull request #1356Commit eba013b
  • Free no longer needed remotes so they do not timeout during restore. (Reviewed by Cynthia Shang. Reported by Francisco Miguel Biete Banon.)
    The connections to remotes were held for the whole restore and could time out, which logged a harmless but distracting warning at the end of a long restore. They are now closed after the manifest is loaded.
    Issue #1368Pull request #1374Commit 06fa185
  • Fix help when a valid option is invalid for the specified command. (Reviewed by Stefan Fercot. Reported by Cynthia Shang.)
    help segfaulted when asked about an option that is not valid for the command.
    Pull request #1375Commit 45f8355
Features:
  • Add PostgreSQL 14 support. (Reviewed by Cynthia Shang.)
    PostgreSQL 14 needs no code changes from PostgreSQL 13. The catalog version may now change during the PostgreSQL beta/rc period without requiring a new pgBackRest release.
    Pull request #1413Commit 2452c4d
  • Add automatic GCS authentication for GCE instances. (Reviewed by Jan Wieck. Tested by Daniel Farina.)
    Set repo-gcs-key-type=auto to get the authentication token from the instance metadata on a GCE instance.
    7b71604 Use existing variable for GCS test server port.
    26fefa6 Refactor GCS token response parsing into a separate function.
    9af0331 Add automatic GCS authentication for GCE instances.Pull request #1395
  • Add repo-retention-history option to expire backup history. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.)
    Set the number of days of backup history to keep. By default all history is kept, and history for backups still in the repository is never expired. When a full backup manifest expires, the manifests of its differential and incremental backups expire as well.
    Pull request #1389Commit 5b332b2
  • Add db-exclude option. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)
    Restore excludes the specified databases by writing them as sparse, zeroed files. PostgreSQL can still perform recovery, and the databases can be dropped after recovery. With db-include, db-exclude applies only to the system databases template0, template1, and postgres.
    Issue #986Pull request #1337Commit 292f836
Improvements:
  • Change archive expiration logging from detail to info level. (Contributed by Cynthia Shang. Reviewed by David Steele.)
    Pull request #1416Commit aceb956
  • Remove stanza archive spool path on restore. (Reviewed by Cynthia Shang, Stefan Fercot.)
    Files in the spool path could interfere with the new cluster, e.g. old archive-push acknowledgements could make it skip archiving WAL.
    Issue #1357Pull request #1386Commit 320c6e1
  • Do not write files atomically or sync paths during backup copy. (Reviewed by Stephen Frost, Stefan Fercot, Cynthia Shang.)
    Atomic writes are not needed since checksums are validated when a failed backup is resumed, and all paths are synced at the end of the backup.
    Pull request #1373Commit aa72c19
Documentation Improvements:
  • Update contributing documentation. (Contributed by Cynthia Shang. Reviewed by David Steele, Stefan Fercot.)
    The contributing guide has more examples of how to write code, add configuration options, and test.
    Pull request #1400Commit 68faf14
  • Consolidate RHEL/CentOS user guide into a single document. (Reviewed by Cynthia Shang.)
    The second guide existed mainly to document stop-auto for PostgreSQL <= 9.5, which is now EOL. The option is still documented in the reference.
    Pull request #1405Commit ccac75e
  • Clarify that repo-s3-role is not an ARN. (Contributed by Isaac Yuen. Reviewed by David Steele.)
    Pull request #1365Commit 5bf1606
v2.33 Release Notes
Multi-Repository and GCS Support
Released April 5, 2021
Bug Fixes:
  • Fix option warnings breaking async archive-get/archive-push. (Reviewed by Cynthia Shang. Reported by Lev Kokotov.)
    An option warning in the async process was logged to a closed stdout, which made the process abort. Since the async role can now have different valid options than the default role, this happened more often. Option warnings are now logged only for the default role.
    Issue #1299Pull request #1301Commit 3837e61
  • Fix memory leak in backup during archive copy. (Reviewed by Cynthia Shang. Reported by Christian ROUX, Efremov Egor.)
    Copying WAL into the backup with archive-copy leaked memory, especially when the WAL needed to be recompressed.
    Issue #1319Commit 778adbf
  • Fix stack overflow in cipher passphrase generation. (Reviewed by Cynthia Shang. Reported by bsiara.)
    The buffer for the generated passphrase had no room for the terminating zero. This was found on arm64 with musl libc.
    Issue #1294Pull request #1295Commit d29855b
  • Fix repo-ls / on S3 repositories. (Reviewed by Cynthia Shang. Reported by Lesovsky Alexey.)
    S3 reports / as a file without the expected headers, which caused an error. / is now reported as not existing on storage that does not support paths.
    Issue #1317Pull request #1336Commit b1d945e
Features:
  • Multiple repository support. (Contributed by Cynthia Shang, David Steele. Reviewed by Stefan Fercot, Stephen Frost.)
    Up to four repositories may be configured, e.g. a local repository for fast restores and a remote repository for redundancy. archive-push pushes WAL to all repositories, and backups are scheduled for each repository. Some commands work with all repositories, while others require repo. repo is not required when only repo1 is configured.
    3408f1e Enhance expire command multi-repo support.Pull request #1289
    e28f6f1 Expire continues if an error occurs processing a repository.Pull request #1308
    bec3e20 Add archive-get command multi-repo support.Pull request #1292
    118d9e6 Enhance restore command multi-repo support.Pull request #1300
    065b2ff Refactor info command repoMin/Max.
    0ddc038 Remove restore default repo from integration tests.
    13dc8e6 Make --repo optional for backup command.Pull request #1316
    a1280c4 Refactor archive-push command warnings to work like archive-get.
    3b8f0ef Add write fault-tolerance to archive-push command.Pull request #1316
    95063f6 Make --repo optional for remaining commands except stanza-delete.
    1dbb3bf Multiple repository support.Pull request #1227
    31c7824 Allow stanza-* commands to be run remotely.Pull request #1326
    2789d3b Improve info command fault tolerance.Pull request #1342
    01b8e22 Improve archive-push command fault tolerance.Pull request #1343
  • GCS support for repository storage. (Reviewed by Cynthia Shang. Tested by Daniel Farina.)
    Set repo-type=gcs to store the repository on GCS or a GCS-compatible object store.
    dcb79ab Decode JSON \u escaped characters.
    1b4b353 Rename uri to path where appropriate in HTTP and storage modules.
    66a4ff4 Encode path before passing to HttpRequest.
    d485609 Add strNewEncode(), strCatEncode(), and bufNewDecode().
    abcbe0f Combine encode module files into a single file.
    edab2a0 Use switch rather than if-else for encoding types.
    f6c3262 Do not expose valid/validate functions from encode module.
    a1f4fd3 Add ASSERT_MSG().
    00b60e5 Add base64url encoding.
    54c4eb0 Make remote storage objects writeable.
    088662d GCS support for repository storage.Issue #631Issue #743Pull request #1320
  • Add archive-header-check option. (Reviewed by Stephen Frost, Cynthia Shang. Suggested by Hans-Jürgen Schönig.)
    The WAL header is checked against the PostgreSQL version and system identifier to make sure WAL is pushed to the correct stanza. The check is enabled by default and can be disabled when needed, e.g. for encrypted WAL.
    Pull request #1298Commit b6106f3
Improvements:
  • Include recreated system databases during selective restore. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)
    A system database (template0, template1, or postgres) that was recreated has an OID that makes it look like a user database. These databases are now always restored, whatever their OID.
    Issue #986Pull request #1307Commit 6942ff5
  • Exclude content-length from S3 signed headers. (Reviewed by Cynthia Shang. Suggested by Brian P Bockelman.)
    Some proxies modify the content-length header, which made authentication fail when it was signed.
    Issue #1340Pull request #1341Commit 7d7ac0e
  • Consolidate less commonly used repository storage options. (Reviewed by Cynthia Shang.)
    The S3 and Azure CA, host, port, and TLS verification options are renamed to repo-storage-*, e.g. repo1-s3-host is now repo1-storage-host. The old names are still accepted.
    Pull request #1321Commit d1aa765
  • Allow custom config-path default with ./configure --with-configdir. (Contributed by Michael Schout. Reviewed by David Steele.)
    Packagers can set a different default for config-path when building. Without the option the default is still /etc/pgbackrest.
    6a717e0 Set config path in configure script.
    9243962 Allow custom config-path default with ./configure --with-configdir.Issue #1251Issue #1293Pull request #1255
  • Log archive copy during backup. (Reviewed by Cynthia Shang, Stefan Fercot.)
    Issue #1319Pull request #1328Commit c862e96
Documentation Improvements:
  • Update reference to include links to user guide examples. (Contributed by Cynthia Shang. Reviewed by David Steele.)
    The command reference now links to the user guide sections with examples, and some descriptions were clarified, e.g. set for expire.
    Pull request #1353Commit d372dd6
  • Update selective restore documentation with caveats. (Reviewed by Cynthia Shang, Stefan Fercot.)
    Recovery may fail unless type=immediate is specified, because PostgreSQL reports zeroed pages as errors after consistency is reached. On PostgreSQL >= 13 ignore_invalid_pages may be used, but check the logs for invalid pages in the selected databases.
    Issue #796Pull request #1329Commit 92d12cc
  • Add compress-type clarification to archive-copy documentation. (Reviewed by Cynthia Shang, Stefan Fercot.)
    WAL must be recompressed when archive-push and backup use a different compress-type, which can be expensive.
    Issue #1319Pull request #1327Commit 9506ffa
  • Add compress-level defaults per compress-type value. (Contributed by Cynthia Shang. Reviewed by David Steele.)
    The defaults are 9 for bz2, 6 for gz, 1 for lz4, and 3 for zst.
    Issue #1242Pull request #1346Commit 3e20608
  • Add note about required NFS settings being the same as PostgreSQL. (Contributed by Cynthia Shang. Reviewed by David Steele.)
    Issue #871Pull request #1347Commit 7598762
v2.32 Release Notes
Repository Commands
Released February 8, 2021
Bug Fixes:
  • Fix resume after partial delete of backup by prior resume. (Reviewed by Cynthia Shang. Reported by Tom Swartz.)
Features:
  • Add repo-ls command. (Reviewed by Cynthia Shang. Tested by Stefan Fercot.)
  • Add repo-get command. (Contributed by Stefan Fercot, David Steele. Reviewed by Cynthia Shang.)
  • Add archive-mode-check option. (Contributed by Stefan Fercot. Reviewed by David Steele. Tested by Michael Banck.)
Improvements:
  • Improve archive-get performance. (Reviewed by Cynthia Shang.)
Documentation Improvements:
  • Improve expire command documentation. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v2.31 Release Notes
Minor Bug Fixes and Improvements
Released December 7, 2020
Bug Fixes:
  • Allow [, #, and space as the first character in database names. (Reviewed by Stefan Fercot, Cynthia Shang. Reported by Jefferson Alexandre.)
  • Create standby.signal only on PostgreSQL 12 when restore type is standby. (Fixed by Stefan Fercot. Reviewed by David Steele. Reported by Keith Fiske.)
Features:
  • Expire history files. (Contributed by Stefan Fercot. Reviewed by David Steele.)
  • Report page checksum errors in info command text output. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)
  • Add repo-azure-endpoint option. (Reviewed by Cynthia Shang. Tested by Brian Peterson. Suggested by Brian Peterson.)
  • Add pg-database option. (Reviewed by Cynthia Shang.)
Improvements:
  • Improve info command output when a stanza is specified but missing. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele. Suggested by uspen.)
  • Improve performance of large file lists in backup/restore commands. (Reviewed by Cynthia Shang. Tested by Oscar.)
  • Add retries to PostgreSQL sleep when starting a backup. (Reviewed by Cynthia Shang. Suggested by Vitaliy Kukharik.)
Documentation Improvements:
  • Replace RHEL/CentOS 6 documentation with RHEL/CentOS 8.
v2.30 Release Notes
PostgreSQL 13 Support
Released October 5, 2020
Bug Fixes:
  • Error with hints when backup user cannot read pg_settings. (Reviewed by Stefan Fercot, Cynthia Shang. Reported by Mohamed Insaf K.)
Features:
  • PostgreSQL 13 support. (Reviewed by Cynthia Shang.)
Improvements:
  • Improve PostgreSQL version identification. (Reviewed by Cynthia Shang, Stephen Frost.)
  • Improve working directory error message. (Reviewed by Stefan Fercot.)
  • Add hint about starting the stanza when WAL segment not found. (Contributed by David Christensen. Reviewed by David Steele.)
  • Add hint for protocol version mismatch. (Reviewed by Cynthia Shang. Suggested by loop-evgeny.)
Documentation Improvements:
  • Add note that pgBackRest versions must match when running remotely. (Reviewed by Cynthia Shang. Suggested by loop-evgeny.)
  • Move info command text to the reference and link to user guide. (Reviewed by Cynthia Shang. Suggested by Christophe Courtois.)
  • Update yum repository path for CentOS/RHEL user guide. (Contributed by Heath Lord. Reviewed by David Steele.)
v2.29 Release Notes
Auto S3 Credentials on AWS
Released August 31, 2020
Bug Fixes:
  • Suppress errors when closing local/remote processes. Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened. (Reviewed by Cynthia Shang. Reported by argdenis.)
  • Fix issue with = character in file or database names. (Reviewed by Bastian Wegge, Cynthia Shang. Reported by Brad Nicholson, Bastian Wegge.)
Features:
  • Automatically retrieve temporary S3 credentials on AWS instances. (Contributed by David Steele, Stephen Frost. Reviewed by Cynthia Shang, Aleš Zelený, Jeanette Bromage. Tested by David Youatt.)
  • Add archive-mode option to disable archiving on restore. (Reviewed by Stephen Frost. Suggested by Stephen Frost.)
Improvements:
  • PostgreSQL 13 beta3 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
  • Asynchronous list/remove for S3/Azure storage. (Reviewed by Cynthia Shang, Stephen Frost.)
  • Improve memory usage of unlogged relation detection in manifest build. (Reviewed by Cynthia Shang, Stephen Frost, Oscar. Tested by Brad Nicholson. Suggested by Oscar, Brad Nicholson.)
  • Proactively close file descriptors after forking async process. (Reviewed by Stephen Frost, Cynthia Shang.)
  • Delay backup remote connection close until after archive check. (Contributed by Floris van Nee. Reviewed by David Steele.)
  • Improve detailed error output. (Reviewed by Cynthia Shang.)
  • Improve TLS error reporting. (Reviewed by Cynthia Shang, Stephen Frost.)
Documentation Bug Fixes:
  • Add none to compress-type option reference and fix example. (Reported by Ugo Bellavance, Don Seiler.)
  • Add missing azure type in repo-type option reference. (Fixed by Don Seiler. Reviewed by David Steele.)
  • Fix typo in repo-cipher-type option reference. (Fixed by Don Seiler. Reviewed by David Steele.)
Documentation Improvements:
  • Clarify that expire must be run regularly when expire-auto is disabled. (Reviewed by Douglas J Hunley. Suggested by Douglas J Hunley.)
v2.28 Release Notes
Azure Repository Storage
Released July 20, 2020
Bug Fixes:
  • Fix restore --force acting like --force --delta. This caused restore to replace files based on timestamp and size rather than overwriting, which meant some files that should have been updated were left unchanged. Normal restore and restore --delta were not affected by this issue. (Reviewed by Cynthia Shang.)
Features:
  • Azure support for repository storage. (Reviewed by Cynthia Shang. Tested by Don Seiler.)
  • Add expire-auto option. This allows automatic expiration after a successful backup to be disabled. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.)
Improvements:
  • Asynchronous S3 multipart upload. (Reviewed by Stephen Frost.)
  • Automatic retry for backup, restore, archive-get, and archive-push. (Reviewed by Cynthia Shang.)
  • Disable query parallelism in PostgreSQL sessions used for backup control. (Reviewed by Stefan Fercot.)
  • PostgreSQL 13 beta2 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
  • Improve handling of invalid HTTP response status. (Reviewed by Cynthia Shang.)
  • Improve error when pg1-path option missing for archive-get command. (Reviewed by Cynthia Shang.)
  • Add hint when checksum delta is enabled after a timeline switch. (Reviewed by Matt Bunter, Cynthia Shang.)
  • Use PostgreSQL instead of postmaster where appropriate. (Reviewed by Cynthia Shang.)
Documentation Bug Fixes:
  • Fix incorrect example for repo-retention-full-type option. (Reported by Höseyin Sönmez.)
  • Remove internal commands from HTML and man command references. (Reported by Cynthia Shang.)
Documentation Improvements:
  • Update PostgreSQL versions used to build user guides. Also add version ranges to indicate that a user guide is accurate for a range of PostgreSQL versions even if it was built for a specific version. (Reviewed by Stephen Frost.)
  • Update FAQ for expiring a specific backup set. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Update FAQ to clarify default PITR behavior. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v2.27 Release Notes
Expiration Improvements and Compression Drivers
Released May 26, 2020
Bug Fixes:
  • Fix issue checking if file links are contained in path links. (Reviewed by Cynthia Shang. Reported by Christophe Cavallié.)
  • Allow pg-path1 to be optional for synchronous archive-push. (Reviewed by Cynthia Shang. Reported by Jerome Peng.)
  • The expire command now checks if a stop file is present. (Fixed by Cynthia Shang. Reviewed by David Steele.)
  • Handle missing reason phrase in HTTP response. (Reviewed by Cynthia Shang. Reported by Tenuun.)
  • Increase buffer size for lz4 compression flush. (Reviewed by Cynthia Shang. Reported by Eric Radman.)
  • Ignore pg-host* and repo-host* options for the remote command. (Reviewed by Cynthia Shang. Reported by Pavel Suderevsky.)
  • Fix possibly missing pg1-* options for the remote command. (Reviewed by Cynthia Shang. Reported by Andrew L'Ecuyer.)
Features:
  • Time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. (Contributed by Cynthia Shang, Pierre Ducroquet. Reviewed by David Steele.)
  • Ad hoc backup expiration. Allow the user to remove a specified backup regardless of retention settings. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Zstandard compression support. Note that setting compress-type=zst will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Reviewed by Cynthia Shang.)
  • bzip2 compression support. Note that setting compress-type=bz2 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Contributed by Stephen Frost. Reviewed by David Steele, Cynthia Shang.)
  • Add backup/expire running status to the info command. (Contributed by Stefan Fercot. Reviewed by David Steele.)
Improvements:
  • Expire WAL archive only when repo-retention-archive threshold is met. WAL prior to the first full backup was previously expired after the first full backup. Now it is preserved according to retention settings. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Add local MD5 implementation so S3 works when FIPS is enabled. (Reviewed by Cynthia Shang, Stephen Frost. Suggested by Brian Almeida, John Kelly.)
  • PostgreSQL 13 beta1 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace. (Reviewed by Cynthia Shang.)
  • Reduce buffer-size default to 1MiB. (Reviewed by Stephen Frost.)
  • Throw user-friendly error if expire is not run on repository host. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v2.26 Release Notes
Non-blocking TLS
Released April 20, 2020
Bug Fixes:
  • Remove empty subexpression from manifest regular expression. MacOS was not happy about this though other platforms seemed to work fine. (Fixed by David Raftis. Reviewed by David Steele.)
Improvements:
  • Non-blocking TLS implementation. (Reviewed by Cynthia Shang, Stephen Frost. Tested by Slava Moudry.)
  • Only limit backup copy size for WAL-logged files. The prior behavior could possibly lead to postgresql.conf or postgresql.auto.conf being truncated in the backup. (Reviewed by Cynthia Shang.)
  • TCP keep-alive options are configurable. (Suggested by Marc Cousin.)
  • Add io-timeout option. (Reviewed by Cynthia Shang.)
v2.25 Release Notes
LZ4 Compression Support
Released March 26, 2020
Features:
  • Add lz4 compression support. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Reviewed by Cynthia Shang.)
  • Add --dry-run option to the expire command. Use dry-run to see which backups/archive would be removed by the expire command without actually removing anything. (Contributed by Cynthia Shang, Luca Ferrari. Reviewed by David Steele. Suggested by Marc Cousin.)
Improvements:
  • Improve performance of remote manifest build. (Suggested by Jens Wilke.)
  • Fix detection of keepalive options on Linux. (Contributed by Marc Cousin. Reviewed by David Steele.)
  • Add configure host detection to set standards flags correctly. (Contributed by Marc Cousin. Reviewed by David Steele.)
  • Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options. (Reviewed by Cynthia Shang.)
  • Limit backup file copy size to size reported at backup start. If a file grows during the backup it will be reconstructed by WAL replay during recovery so there is no need to copy the additional data. (Reviewed by Cynthia Shang.)
v2.24 Release Notes
Auto-Select Backup Set for Time Target
Released February 25, 2020
Bug Fixes:
  • Prevent defunct processes in asynchronous archive commands. (Reviewed by Stephen Frost. Reported by Adam Brusselback, ejberdecia.)
  • Error when archive-get/archive-push/restore are not run on a PostgreSQL host. (Reviewed by Stephen Frost. Reported by Jesper St John.)
  • Read HTTP content to eof when size/encoding not specified. (Reviewed by Cynthia Shang. Reported by Christian ROUX.)
  • Fix resume when the resumable backup was created by Perl. In this case the resumable backup should be ignored, but the C code was not able to load the partial manifest written by Perl since the format differs slightly. Add validations to catch this case and continue gracefully. (Reported by Kacey Holston.)
Features:
  • Auto-select backup set on restore when time target is specified. Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Improvements:
  • Skip pg_internal.init temp file during backup. (Reviewed by Cynthia Shang. Suggested by Michael Paquier.)
  • Add more validations to the manifest on backup. (Reviewed by Cynthia Shang.)
Documentation Improvements:
  • Prevent lock-bot from adding comments to locked issues. (Suggested by Christoph Berg.)
v2.23 Release Notes
Bug Fix
Released January 27, 2020
Bug Fixes:
  • Fix missing files corrupting the manifest. If a file was removed by PostgreSQL during the backup (or was missing from the standby) then the next file might not be copied and updated in the manifest. If this happened then the backup would error when restored. (Reviewed by Cynthia Shang. Reported by Vitaliy Kukharik.)
Improvements:
  • Use pkg-config instead of xml2-config for libxml2 build options. (Contributed by David Steele, Adrian Vondendriesch.)
  • Validate checksums are set in the manifest on backup/restore. (Reviewed by Cynthia Shang.)
v2.22 Release Notes
Bug Fix
Released January 21, 2020
Bug Fixes:
  • Fix error in timeline conversion. The timeline is required to verify WAL segments in the archive after a backup. The conversion was performed base 10 instead of 16, which led to errors when the timeline was ≥ 0xA. (Reported by Lukas Ertl, Eric Veldhuyzen.)
v2.21 Release Notes
C Migration Complete
Released January 15, 2020
Bug Fixes:
  • Fix options being ignored by asynchronous commands. The asynchronous archive-get/archive-push processes were not loading options configured in command configuration sections, e.g. [global:archive-get]. (Reviewed by Cynthia Shang. Reported by Urs Kramer.)
  • Fix handling of \ in filenames. \ was not being properly escaped when calculating the manifest checksum which prevented the manifest from loading. Since instances of \ in cluster filenames should be rare to nonexistent this does not seem likely to be a serious problem in the field.
Features:
  • pgBackRest is now pure C.
  • Add pg-user option. Specifies the database user name when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER, which was the previous behavior. (Contributed by Mike Palmiotto. Reviewed by David Steele.)
  • Allow path-style URIs in S3 driver.
Improvements:
  • The backup command is implemented entirely in C. (Reviewed by Cynthia Shang.)
v2.20 Release Notes
Bug Fixes
Released December 12, 2019
Bug Fixes:
  • Fix archive-push/archive-get when PGDATA is symlinked. These commands tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked. If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call. (Reported by Stephen Frost, Milosz Suchy.)
  • Fix reference list when backup.info is reconstructed in expire command. Since the backup command is still using the Perl version of reconstruct this issue will not express unless 1) there is a backup missing from backup.info and 2) the expire command is run directly instead of running after backup as usual. This unlikely combination of events means this is probably not a problem in the field.
  • Fix segfault on unexpected EOF in gzip decompression. (Reported by Stephen Frost.)
v2.19 Release Notes
C Migrations and Bug Fixes
Released November 12, 2019
Bug Fixes:
  • Fix remote timeout in delta restore. When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout. Add keep-alives to prevent remote timeout. (Reported by James Sewell, Jens Wilke.)
  • Fix handling of repeated HTTP headers. When HTTP headers are repeated they should be considered equivalent to a single comma-separated header rather than generating an error, which was the prior behavior. (Reported by donicrosby.)
Improvements:
  • JSON output from the info command is no longer pretty-printed. Monitoring systems can more easily ingest the JSON without linefeeds. External tools such as jq can be used to pretty-print if desired. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • The check command is implemented entirely in C. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Documentation Improvements:
  • Document how to contribute to pgBackRest. (Contributed by Cynthia Shang, David Steele.)
  • Document maximum version for auto-stop option. (Contributed by Brad Nicholson. Reviewed by David Steele.)
Test Suite Improvements:
  • Fix container test path being used when --vm=none. (Suggested by Stephen Frost.)
  • Fix mismatched timezone in expect test. (Suggested by Stephen Frost.)
  • Don't autogenerate embedded libc code by default. (Suggested by Stephen Frost.)
v2.18 Release Notes
PostgreSQL 12 Support
Released October 1, 2019
Features:
  • PostgreSQL 12 support.
  • Add info command set option for detailed text output. The additional details include databases that can be used for selective restore and a list of tablespaces and symlinks with their default destinations. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by Stephen Frost, ejberdecia.)
  • Add standby restore type. This restore type automatically adds standby_mode=on to recovery.conf for PostgreSQL < 12 and creates standby.signal for PostgreSQL ≥ 12, creating a common interface between PostgreSQL versions. (Reviewed by Cynthia Shang.)
Improvements:
  • The restore command is implemented entirely in C. (Reviewed by Cynthia Shang.)
Documentation Improvements:
  • Document the relationship between db-timeout and protocol-timeout. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by James Chanco Jr.)
  • Add documentation clarifications regarding standby repositories. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Add FAQ for time-based Point-in-Time Recovery. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v2.17 Release Notes
C Migrations and Bug Fixes
Released September 3, 2019
Bug Fixes:
  • Improve slow manifest build for very large quantities of tables/segments. (Reported by Jens Wilke.)
  • Fix exclusions for special files. (Reported by CluelessTechnologist, Janis Puris, Rachid Broum.)
Improvements:
  • The stanza-create/update/delete commands are implemented entirely in C. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • The start/stop commands are implemented entirely in C. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Create log directories/files with 0750/0640 mode. (Suggested by Damiano Albani.)
Documentation Bug Fixes:
  • Fix yum.p.o package being installed when custom package specified. (Reported by Joe Ayers, John Harvey.)
Documentation Improvements:
  • Build pgBackRest as an unprivileged user. (Suggested by Laurenz Albe.)
v2.16 Release Notes
C Migrations and Bug Fixes
Released August 5, 2019
Bug Fixes:
  • Retry S3 RequestTimeTooSkewed errors instead of immediately terminating. (Reported by sean0101n, Tim Garton, Jesper St John, Aleš Zelený.)
  • Fix incorrect handling of transfer-encoding response to HEAD request. (Reported by Pavel Suderevsky.)
  • Fix scoping violations exposed by optimizations in gcc 9. (Reported by Christian Lange, Ned T. Crigler.)
Features:
  • Add repo-s3-port option for setting a non-standard S3 service port.
Improvements:
  • The local command for backup is implemented entirely in C. (Contributed by David Steele, Cynthia Shang.)
  • The check command is implemented partly in C. (Reviewed by Cynthia Shang.)
v2.15 Release Notes
C Implementation of Expire
Released June 25, 2019
Bug Fixes:
  • Fix archive retention expiring too aggressively. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Mohamad El-Rifai.)
Improvements:
  • The expire command is implemented entirely in C. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • The local command for restore is implemented entirely in C.
  • Remove hard-coded PostgreSQL user so $PGUSER works. (Suggested by Julian Zhang, Janis Puris.)
  • Honor configure --prefix option. (Suggested by Daniel Westermann.)
  • Rename repo-s3-verify-ssl option to repo-s3-verify-tls. The new name is preferred because pgBackRest does not support any SSL protocol versions (they are all considered to be insecure). The old name will continue to be accepted.
Documentation Improvements:
  • Add FAQ to the documentation. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Use wal_level=replica in the documentation for PostgreSQL ≥ 9.6. (Suggested by Patrick McLaughlin.)
v2.14 Release Notes
Bug Fix and Improvements
Released May 20, 2019
Bug Fixes:
  • Fix segfault when process-max > 8 for archive-push/archive-get. (Reported by Jens Wilke.)
Improvements:
  • Bypass database checks when stanza-delete issued with force. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by hatifnatt.)
  • Add configure script for improved multi-platform support.
Documentation Features:
  • Add user guides for CentOS/RHEL 6/7.
v2.13 Release Notes
Bug Fixes
Released April 18, 2019
Bug Fixes:
  • Fix zero-length reads causing problems for IO filters that did not expect them. (Reported by brunre01, Jens Wilke, Tomasz Kontusz, guruguruguru.)
  • Fix reliability of error reporting from local/remote processes.
  • Fix Posix/CIFS error messages reporting the wrong filename on write/sync/close.
v2.12 Release Notes
C Implementation of Archive Push
Released April 11, 2019
IMPORTANT NOTE: The new TLS/SSL implementation forbids dots in S3 bucket names per RFC-2818. This security fix is required for compliant hostname verification.
Bug Fixes:
  • Fix issues when a path option is / terminated. (Reported by Marc Cousin.)
  • Fix issues when log-level-file=off is set for the archive-get command. (Reported by Brad Nicholson.)
  • Fix C code to recognize host:port option format like Perl does. (Reported by Kyle Nevins.)
  • Fix issues with remote/local command logging options.
Improvements:
  • The archive-push command is implemented entirely in C.
  • Increase process-max limit to 999. (Suggested by Rakshitha-BR.)
  • Improve error message when an S3 bucket name contains dots.
Documentation Improvements:
  • Clarify that S3-compatible object stores are supported. (Suggested by Magnus Hagander.)
v2.11 Release Notes
C Implementation of Archive Get
Released March 11, 2019
Bug Fixes:
  • Fix possible truncated WAL segments when an error occurs mid-write. (Reported by blogh.)
  • Fix info command missing WAL min/max when stanza specified. (Fixed by Stefan Fercot. Reviewed by David Steele.)
  • Fix non-compliant JSON for options passed from C to Perl. (Reported by Leo Khomenko.)
Improvements:
  • The archive-get command is implemented entirely in C.
  • Enable socket keep-alive on older Perl versions. (Contributed by Marc Cousin. Reviewed by David Steele.)
  • Error when parameters are passed to a command that does not accept parameters. (Suggested by Jason O'Donnell.)
  • Add hints when unable to find a WAL segment in the archive. (Suggested by Hans-Jürgen Schönig.)
  • Improve error when hostname cannot be found in a certificate. (Suggested by James Badger.)
  • Add additional options to backup.manifest for debugging purposes. (Contributed by blogh. Reviewed by David Steele.)
Documentation Improvements:
  • Update default documentation version to PostgreSQL 10.
v2.10 Release Notes
Bug Fixes
Released February 9, 2019
Bug Fixes:
  • Add unimplemented S3 driver method required for archive-get. (Reported by mibiio.)
  • Fix check for improperly configured pg-path. (Reported by James Chanco Jr.)
v2.09 Release Notes
Minor Improvements and Bug Fixes
Released January 30, 2019
Bug Fixes:
  • Fix issue with multiple async status files causing a hard error. (Reported by Vidhya Gurumoorthi, Joe Ayers, Douglas J Hunley.)
Improvements:
  • The info command is implemented entirely in C. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Simplify info command text message when no stanzas are present. Replace the repository path with the repository.
  • Add _DARWIN_C_SOURCE flag to Makefile for MacOS builds. (Contributed by Douglas J Hunley. Reviewed by David Steele.)
  • Update address lookup in C TLS client to use modern methods. (Suggested by Bruno Friedmann.)
  • Include Posix-compliant header for strcasecmp() and fd_set. (Suggested by ucando.)
Documentation Bug Fixes:
  • Fix hard-coded repository path. (Reported by Heath Lord.)
Documentation Improvements:
  • Clarify that encryption is always performed client-side. (Suggested by Bruce Burdick.)
  • Add examples for building a documentation host.
  • Allow if in manifest variables, lists, and list items.
v2.08 Release Notes
Minor Improvements and Bug Fixes
Released January 2, 2019
Bug Fixes:
  • Remove request for S3 object info directly after putting it. (Reported by Matt Kunkel.)
  • Correct archive-get-queue-max to be size type. (Reported by Ronan Dunklau.)
  • Add error message when current user uid/gid does not map to a name. (Reported by Camilo Aguilar.)
  • Error when --target-action=shutdown specified for PostgreSQL < 9.5.
Improvements:
  • Set TCP keepalives on S3 connections. (Suggested by Ronan Dunklau.)
  • Reorder info command text output so most recent backup is output last. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by Ryan Lambert.)
  • Change file ownership only when required.
  • Redact authentication header when throwing S3 errors. (Suggested by Brad Nicholson.)
Documentation Improvements:
  • Clarify when target-action is effective and PostgreSQL version support. (Suggested by Keith Fiske.)
  • Clarify that region/endpoint must be configured correctly for the bucket. (Suggested by Pritam Barhate.)
  • Add documentation for building the documentation.
v2.07 Release Notes
Automatic Backup Checksum Delta
Released November 16, 2018
Bug Fixes:
  • Fix issue with archive-push-queue-max not being honored on connection error. (Reported by Lardière Sébastien.)
  • Fix static WAL segment size used to determine if archive-push-queue-max has been exceeded.
  • Fix error after log file open failure when processing should continue. (Reported by vthriller.)
Features:
  • Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Improvements:
  • Retry all S3 5xx errors rather than just 500 internal errors. (Suggested by Craig A. James.)
v2.06 Release Notes
Checksum Delta Backup and PostgreSQL 11 Support
Released October 15, 2018
Bug Fixes:
  • Fix missing URI encoding in S3 driver. (Reported by Dan Farrell.)
  • Fix incorrect error message for duplicate options in configuration files. (Reported by Jesper St John.)
  • Fix incorrectly reported error return in info logging. A return code of 1 from the archive-get was being logged as an error message at info level but otherwise worked correctly.
Features:
  • Add checksum delta for incremental backups. Checksum delta backups uses checksums rather than timestamps to determine if files have changed. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • PostgreSQL 11 support, including configurable WAL segment size.
Improvements:
  • Ignore all files in a linked tablespace directory except the subdirectory for the current version of PostgreSQL. Previously an error would be generated if other files were present and not owned by the PostgreSQL user.
  • Improve info command to display the stanza cipher type. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by Douglas J Hunley.)
  • Improve support for special characters in filenames.
  • Allow delta option to be specified in the pgBackRest configuration file. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Documentation Improvements:
  • Use command in authorized_hosts to improve SSH security. (Suggested by Stephen Frost, Magnus Hagander.)
  • List allowable values for the buffer-size option in the configuration reference. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by Stéphane Schildknecht.)
v2.05 Release Notes
Environment Variable Options and Exclude Temporary/Unlogged Relations
Released August 31, 2018
Bug Fixes:
  • Fix issue where relative links in $PGDATA could be stored in the backup with the wrong path. This issue did not affect absolute links and relative tablespace links were caught by other checks. (Reported by Cynthia Shang.)
  • Remove incompletely implemented online option from the check command. Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly. (Reported by Jason O'Donnell.)
  • Fix issue where errors raised in C were not logged when called from Perl. pgBackRest properly terminated with the correct error code but lacked an error message to aid in debugging. (Reported by Douglas J Hunley.)
  • Fix issue when a boolean option (e.g. delta) was specified more than once. (Reported by Yogesh Sharma.)
Features:
  • Allow any option to be set in an environment variable. This includes options that previously could only be specified on the command line, e.g. stanza, and secret options that could not be specified on the command-line, e.g. repo1-s3-key-secret.
  • Exclude temporary and unlogged relation (table/index) files from backup. Implemented using the same logic as the patches adding this feature to PostgreSQL, 8694cc96 and 920a5e50. Temporary relation exclusion is enabled in PostgreSQL ≥ 9.0. Unlogged relation exclusion is enabled in PostgreSQL ≥ 9.1, where the feature was introduced. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Allow arbitrary directories and/or files to be excluded from a backup. Misuse of this feature can lead to inconsistent backups so read the --exclude documentation carefully before using. (Reviewed by Cynthia Shang.)
  • Add log-subprocess option to allow file logging for local and remote subprocesses.
  • PostgreSQL 11 Beta 3 support.
Improvements:
  • Allow zero-size files in backup manifest to reference a prior manifest regardless of timestamp delta. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Improve asynchronous archive-get/archive-push performance by directly checking status files. (Contributed by Stephen Frost. Reviewed by David Steele.)
  • Improve error message when a command is missing the stanza option. (Suggested by Sarah Conway.)
Documentation Bug Fixes:
  • Fix invalid log level in log-path option reference. (Reported by Camilo Aguilar.)
Documentation Improvements:
  • Stop trying to arrange contributors in release.xml by last/first name. Contributor names have always been presented in the release notes exactly as given, but we tried to assign internal IDs based on last/first name which can be hard to determine and ultimately doesn't make sense. Inspired by Christophe's PostgresOpen 2017 talk, Human Beings Do Not Have a Primary Key. (Suggested by Christophe Pettus.)
Test Suite Improvements:
  • Error if LibC build is performed outside the test environment. LibC is no longer required for production builds.
v2.04 Release Notes
Critical Bug Fix for Backup Resume
Released July 5, 2018
IMPORTANT NOTE: This release fixes a critical bug in the backup resume feature. All resumed backups prior to this release should be considered inconsistent. A backup will be resumed after a prior backup fails, unless resume=n has been specified. A resumed backup can be identified by checking the backup log for the message aborted backup of same type exists, will be cleaned to remove invalid files and resumed. If the message exists, do not use this backup or any backup in the same set for a restore and check the restore logs to see if a resumed backup was restored. If so, there may be inconsistent data in the cluster.
Bug Fixes:
  • Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. See note above for more details. (Reported by David Youatt, Yogesh Sharma, Stephen Frost.)
  • Fix error in selective restore when only one user database exists in the cluster. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Nj Baliyan.)
  • Fix non-compliant ISO-8601 timestamp format in S3 authorization headers. AWS and some gateways were tolerant of space rather than zero-padded hours while others were not. (Fixed by Andrew Schwartz. Reviewed by David Steele.)
Features:
  • PostgreSQL 11 Beta 2 support.
Improvements:
  • Improve the HTTP client to set content-length to 0 when not specified by the server. S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either. (Suggested by Adam K. Sumner.)
  • Set search_path = 'pg_catalog' on PostgreSQL connections. (Suggested by Stephen Frost.)
Documentation Improvements:
  • Create a new section to describe building pgBackRest and build on a separate host.
  • Add sample S3 policy to restrict bucket privileges. (Suggested by Douglas J Hunley, Jason O'Donnell.)
v2.03 Release Notes
Single Executable to Deploy
Released May 22, 2018
Bug Fixes:
  • Fix potential buffer overrun in error message handling. (Reported by Lætitia.)
  • Fix archive write lock being taken for the synchronous archive-get command. (Reported by uspen.)
Improvements:
  • Embed exported C functions and Perl modules directly into the pgBackRest executable.
  • Use time_t instead of __time_t for better portability. (Suggested by Nick Floersch.)
  • Print total runtime in milliseconds at command end.
v2.02 Release Notes
Parallel Asynchronous Archive Get and Configuration Includes
Released May 6, 2018
Bug Fixes:
  • Fix directory syncs running recursively when only the specified directory should be synced. (Reported by Craig A. James.)
  • Fix archive-copy throwing path not found error for incr/diff backups. (Reported by yummyliu, Vitaliy Kukharik.)
  • Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. (Reported by Vitaliy Kukharik.)
  • Fix delta restore failing when a linked file is missing.
  • Fix rendering of key/value and list options in help. (Reported by Clinton Adams.)
Features:
  • Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
  • Add support for additional pgBackRest configuration files. The directory is specified by the --config-include-path option. Add --config-path option for overriding the default base path of the --config and --config-include-path option. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Add repo-s3-token option to allow temporary credentials tokens to be configured. pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire. (Contributed by Yogesh Sharma. Reviewed by David Steele.)
Improvements:
  • Update the archive-push-queue-max, manifest-save-threshold, and buffer-size options to accept values in KB, MB, GB, TB, or PB where the multiplier is a power of 1024. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Make backup/restore path sync more efficient. Scanning the entire directory can be very expensive if there are a lot of small tables. The backup manifest contains the path list so use it to perform syncs instead of scanning the backup/restore path.
  • Show command parameters as well as command options in initial info log message.
  • Rename archive-queue-max option to archive-push-queue-max. This is consistent with the new archive-get-queue-max option. The old option name will continue to be accepted.
Documentation Bug Fixes:
  • Update docs with 32-bit support and caveats. 32-bit support was added in v1.26. (Reported by Viorel Tabara.)
Documentation Improvements:
  • Add monitoring examples using PostgreSQL and jq. (Suggested by Stephen Frost, Brian Faherty.)
  • Add example of command section usage to archiving configuration. (Suggested by Christophe Courtois.)
  • Remove documentation describing info --output=json as experimental.
  • Update out-of-date description for the spool-path option.
Test Suite Features:
  • Use lcov for C unit test coverage reporting. Switch from Devel::Cover because it would not report on branch coverage for reports converted from gcov. Incomplete branch coverage for a module now generates an error. Coverage of unit tests is not displayed in the report unless they are incomplete for either statement or branch coverage.
v2.01 Release Notes
Minor Bug Fixes and Improvements
Released March 19, 2018
Bug Fixes:
  • Fix --target-action and --recovery-option options being reported as invalid when restoring with --type=immediate. (Reported by Brad Nicholson.)
  • Immediately error when a secure option (e.g. repo1-s3-key) is passed on the command line. Since pgBackRest would not pass secure options on to sub-processes an obscure error was thrown. The new error is much clearer and provides hints about how to fix the problem. Update command documentation to omit secure options that cannot be specified on the command-line. (Reported by Brad Nicholson.)
  • Fix issue passing --no-config to embedded Perl. (Reported by Ibrahim Edib Kokdemir.)
  • Fix issue where specifying log-level-stderr > warn would cause a local/remote process to error on exit due to output found on stderr when none was expected. The max value for a local/remote process is now error since there is no reason for these processes to emit warnings. (Reported by Clinton Adams.)
  • Fix manifest test in the check command when tablespaces are present. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Thomas Flatley.)
Improvements:
  • Error when multiple arguments are set in the config file for an option that does not accept multiple arguments. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Remove extraneous sudo commands from src/Makefile. (Contributed by Adrian Vondendriesch. Reviewed by David Steele.)
Documentation Improvements:
  • Show index in examples for indexed options, i.e. repo-*, pg-*. (Suggested by Stephen Frost.)
  • Simplify table of contents on command page by only listing commands. (Suggested by Stephen Frost.)
  • Remove references to the C library being optional.
Test Suite Features:
  • Add CentOS/RHEL package builds.
  • Use clang for static code analysis. Nothing found initially except for some functions that should have been marked __noreturn__.
v2.00 Release Notes
Performance Improvements for Archive Push
Released February 23, 2018
Features:
  • The archive-push command is now partially coded in C which allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process. (Reviewed by Cynthia Shang.)
Improvements:
  • Improve check command to verify that the backup manifest can be built. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Improve performance of HTTPS client. Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
  • Improve S3 delete performance. The constant S3_BATCH_MAX had been replaced with a hard-coded value of 2, probably during testing.
  • Allow any non-command-line option to be reset to default on the command-line. This allows options in pgbackrest.conf to be reset to default which reduces the need to write new configuration files for specific needs.
  • The C library is now required. This eliminates conditional loading and eases development of new library features.
  • The pgbackrest executable is now a C binary instead of Perl. This allows certain time-critical commands (like async archive-push) to run more quickly.
  • Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. repo-* options are now indexed although currently only one is allowed.
Documentation Features:
  • All clusters in the documentation are initialized with checksums.
Documentation Improvements:
  • List deprecated option names in documentation and command-line help.
  • Clarify that S3 buckets must be created by the user. (Suggested by David Youatt.)
v1.29 Release Notes
Critical Bug Fix for Backup Resume
Released July 5, 2018
IMPORTANT NOTE: This release fixes a critical bug in the backup resume feature. All resumed backups prior to this release should be considered inconsistent. A backup will be resumed after a prior backup fails, unless resume=n has been specified. A resumed backup can be identified by checking the backup log for the message aborted backup of same type exists, will be cleaned to remove invalid files and resumed. If the message exists, do not use this backup or any backup in the same set for a restore and check the restore logs to see if a resumed backup was restored. If so, there may be inconsistent data in the cluster.
Bug Fixes:
  • Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. See note above for more details. (Reported by David Youatt, Yogesh Sharma, Stephen Frost.)
  • Fix non-compliant ISO-8601 timestamp format in S3 authorization headers. AWS and some gateways were tolerant of space rather than zero-padded hours while others were not. (Fixed by Andrew Schwartz. Reviewed by David Steele.)
  • Fix directory syncs running recursively when only the specified directory should be synced. (Reported by Craig A. James.)
  • Fix --target-action and --recovery-option options being reported as invalid when restoring with --type=immediate. (Reported by Brad Nicholson.)
  • Fix archive-copy throwing path not found error for incr/diff backups. (Reported by yummyliu, Vitaliy Kukharik.)
  • Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. (Reported by Vitaliy Kukharik.)
  • Fix delta restore failing when a linked file was missing.
  • Fix error in selective restore when only one user database exists in the cluster. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Nj Baliyan.)
Improvements:
  • Improve the HTTP client to set content-length to 0 when not specified by the server. S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either. (Suggested by Adam K. Sumner.)
  • Improve performance of HTTPS client. Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
  • Improve S3 delete performance. The constant S3_BATCH_MAX had been replaced with a hard-coded value of 2, probably during testing.
  • Make backup/restore path sync more efficient. Scanning the entire directory can be very expensive if there are a lot of small tables. The backup manifest contains the path list so use it to perform syncs instead of scanning the backup/restore path. Remove recursive path sync functionality since it is no longer used.
Documentation Bug Fixes:
  • Update docs with 32-bit support and caveats. 32-bit support was added in v1.26. (Reported by Viorel Tabara.)
Documentation Improvements:
  • Clarify that S3 buckets must be created by the user. (Suggested by David Youatt.)
  • Update out-of-date description for the spool-path option.
v1.28 Release Notes
Stanza Delete
Released February 1, 2018
Bug Fixes:
  • Fixed inability to restore a single database contained in a tablespace using --db-include. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Chiranjeevi Ravilla.)
  • Ensure latest db-id is selected on when matching archive.info to backup.info. This provides correct matching in the event there are system-id and db-version duplicates (e.g. after reverting a pg_upgrade). (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Adam K. Sumner.)
  • Fixed overly chatty error message when reporting an invalid command. (Reported by Jason O'Donnell.)
Features:
  • Add stanza-delete command to cleanup unused stanzas. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by Magnus Hagander.)
Improvements:
  • Improve stanza-create command so that it does not error when the stanza already exists. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Documentation Improvements:
  • Update stanza-create --force documentation to urge caution when using. (Suggested by Jason O'Donnell.)
v1.27 Release Notes
Bug Fixes and Documentation
Released December 19, 2017
Bug Fixes:
  • Fixed an issue that suppressed locality errors for backup and restore. When a backup host is present, backups should only be allowed on the backup host and restores should only be allowed on the database host unless an alternate configuration is created that ignores the remote host. (Reported by Lardière Sébastien.)
  • Fixed an issue where WAL was not expired on PostgreSQL 10. This was caused by a faulty regex that expected all PostgreSQL major versions to be X.X. (Reported by Adam Brusselback.)
  • Fixed an issue where the --no-config option was not passed to child processes. This meant the child processes would still read the local config file and possibly cause unexpected behaviors.
  • Fixed info command to eliminate "db (prior)" output if no backups or archives exist for a prior version of the cluster. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Stephen Frost.)
Documentation Features:
  • Document the relationship between the archive-copy and archive-check options. (Suggested by Markus Nullmeier.)
  • Improve archive-copy reference documentation.
v1.26 Release Notes
Repository Encryption
Released November 21, 2017
Bug Fixes:
  • Fixed an issue that could cause copying large manifests to fail during restore. (Reported by Craig A. James.)
  • Fixed incorrect WAL offset for 32-bit architectures. (Fixed by Javier Wilson. Reviewed by David Steele.)
  • Fixed an issue retrieving WAL for old database versions. After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed. Also clean up some issues when the same db version/id appears multiple times in the history. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Clinton Adams.)
  • Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
  • Fixed an issue passing parameters to remotes. When more than one db was specified the path, port, and socket path would for db1 were passed no matter which db was actually being addressed. (Reported by uspen.)
Features:
  • Repository encryption support. (Contributed by Cynthia Shang, David Steele.)
Improvements:
  • Disable gzip filter when --compress-level-network=0. The filter was used with compress level set to 0 which added overhead without any benefit.
  • Inflate performance improvement for gzip filter.
Documentation Features:
  • Add template to improve initial information gathered for issue submissions. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Documentation Improvements:
  • Clarify usage of the archive-timeout option and describe how it is distinct from the PostgreSQL archive_timeout setting. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by Keith Fiske.)
Test Suite Features:
  • Automated tests for 32-bit i386/i686 architecture.
v1.25 Release Notes
S3 Performance Improvements
Released October 24, 2017
Bug Fixes:
  • Fix custom settings for compress-level option being ignored. (Reported by Jens Wilke.)
  • Remove error when overlapping timelines are detected. Overlapping timelines are valid in many Point-in-Time-Recovery (PITR) scenarios. (Reported by blogh.)
  • Fix instances where database-id was not rendered as an integer in JSON info output. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Jason O'Donnell.)
Features:
  • Improve performance of list requests on S3. Any beginning literal portion of a filter expression is used to generate a search prefix which often helps keep the request small enough to avoid rate limiting. (Suggested by Mihail Shvein.)
Test Suite Features:
  • Add I/O performance tests.
v1.24 Release Notes
New Backup Exclusions
Released September 28, 2017
Bug Fixes:
  • Fixed an issue where warnings were being emitted in place of lower priority log messages during backup from standby initialization. (Reported by uspen.)
  • Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes. (Reported by uspen.)
Features:
  • Exclude contents of pg_snapshots, pg_serial, pg_notify, and pg_dynshmem from backup since they are rebuilt on startup.
  • Exclude pg_internal.init files from backup since they are rebuilt on startup.
Improvements:
  • Open log file after async process is completely separated from the main process to prevent the main process from also logging to the file. (Suggested by Jens Wilke.)
Documentation Features:
  • Add passwordless SSH configuration.
Documentation Improvements:
  • Rename master to primary in documentation to align with PostgreSQL convention.
v1.23 Release Notes
Multiple Standbys and PostgreSQL 10 Support
Released September 3, 2017
Bug Fixes:
  • Fixed an issue that could cause compression to abort on growing files. (Reported by Jesper St John, Aleksandr Rogozin.)
  • Fixed an issue with keep-alives not being sent to the remote from the local process. (Reported by William Cox.)
Features:
  • Up to seven standbys can be configured for backup from standby. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • PostgreSQL 10 support.
  • Allow content-length (in addition to chunked encoding) when reading XML data to improve compatibility with third-party S3 gateways. (Suggested by Victor Gdalevich.)
Improvements:
  • Increase HTTP timeout for S3.
  • Add HTTP retries to harden against transient S3 network errors.
Documentation Bug Fixes:
  • Fixed document generation to include section summaries on the Configuration page. (Fixed by Cynthia Shang. Reviewed by David Steele.)
v1.22 Release Notes
Fixed S3 Retry
Released August 9, 2017
Bug Fixes:
  • Fixed authentication issue in S3 retry.
v1.21 Release Notes
Improved Info Output and SSH Port Option
Released August 8, 2017
Bug Fixes:
  • The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do. (Reported by Stephen Frost.)
  • Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade. (Fixed by Cynthia Shang. Reviewed by David Steele.)
Features:
  • Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Improvements:
  • Retry when S3 returns an internal error (500).
Documentation Bug Fixes:
  • Fix description of --online based on the command context.
Documentation Features:
  • Add creation of /etc/pgbackrest.conf to manual installation instructions.
Documentation Improvements:
  • Move repository options into a separate section in command/command-line help. (Suggested by Stephen Frost.)
v1.20 Release Notes
Critical 8.3/8.4 Bug Fix
Released June 27, 2017
IMPORTANT NOTE: PostgreSQL 8.3 and 8.4 installations utilizing tablespaces should upgrade immediately from any v1 release and run a full backup. A bug prevented tablespaces from being backed up on these versions only. PostgreSQL ≥ 9.0
Bug Fixes:
  • Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4.
  • Fixed missing flag in C library build that resulted in a mismatched binary on 32-bit systems. (Reported by Adrian Vondendriesch.)
Features:
  • Add s3-repo-ca-path and s3-repo-ca-file options to accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs. (Suggested by Scott Frazer.)
Test Suite Features:
  • Add documentation builds to CI.
v1.19 Release Notes
S3 Support
Released June 12, 2017
Bug Fixes:
  • Fixed the info command so the WAL archive min/max displayed is for the current database version. (Fixed by Cynthia Shang. Reviewed by David Steele.)
  • Fixed the backup command so the backup-standby option is reset (and the backup proceeds on the primary) if the standby is not configured and/or reachable. (Fixed by Cynthia Shang. Reviewed by David Steele.)
  • Fixed config warnings raised from a remote process causing errors in the master process. (Fixed by Cynthia Shang. Reviewed by David Steele.)
Features:
  • Amazon S3 repository support. (Reviewed by Cynthia Shang.)
Documentation Bug Fixes:
  • Changed invalid max-archive-mb option in configuration reference to archive-queue-max.
  • Fixed missing sudo in installation section. (Fixed by Lætitia. Reviewed by David Steele.)
v1.18 Release Notes
Stanza Upgrade, Refactoring, and Locking Improvements
Released April 12, 2017
Bug Fixes:
  • Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)
Features:
  • Added the stanza-upgrade command to provide a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Improvements:
  • Simplify locking scheme. Now, only the master process will hold write locks (for archive-push and backup commands) and not all local and remote worker processes as before.
  • Do not set timestamps of files in the backup directories to match timestamps in the cluster directory. This was originally done to enable backup resume, but that process is now implemented with checksums.
  • Improved error message when the restore command detects the presence of postmaster.pid. (Suggested by Yogesh Sharma.)
  • Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions. (Suggested by Yogesh Sharma.)
v1.17 Release Notes
Page Checksum Bug Fix
Released March 13, 2017
Bug Fixes:
  • Fixed an issue where newly initialized (but unused) pages would cause page checksum warnings. (Reported by Stephen Frost.)
v1.16 Release Notes
Page Checksum Improvements, CI, and Package Testing
Released March 2, 2017
Bug Fixes:
  • Fixed an issue where tables over 1GB would report page checksum warnings after the first segment. (Reported by Stephen Frost.)
  • Fixed an issue where databases created with a non-default tablespace would raise bogus warnings about pg_filenode.map and pg_internal.init not being page aligned. (Reported by blogh.)
Test Suite Features:
  • Continuous integration using travis-ci.
  • Automated builds of Debian packages for all supported distributions.
v1.15 Release Notes
Refactoring and Bug Fixes
Released February 13, 2017
Bug Fixes:
  • Fixed a regression introduced in v1.13 that could cause backups to fail if files were removed (e.g. tables dropped) while the manifest was being built. (Reported by Navid Golpayegani.)
v1.14 Release Notes
Refactoring and Bug Fixes
Released February 13, 2017
Bug Fixes:
  • Fixed an issue where an archive-push error would not be retried and would instead return errors to PostgreSQL indefinitely (unless the .error file was manually deleted). (Reported by Jens Wilke.)
  • Fixed a race condition in parallel archiving where creation of new paths generated an error when multiple processes attempted to do so at the same time. (Reported by Jens Wilke.)
Improvements:
  • Improved performance of wal archive min/max provided by the info command. (Suggested by Jens Wilke.)
Documentation Features:
  • Updated async archiving documentation to more accurately describe how the new method works and how it differs from the old method. (Suggested by Jens Wilke.)
v1.13 Release Notes
Parallel Archiving, Stanza Create, Improved Info and Check
Released February 5, 2017
IMPORTANT NOTE: The new implementation of asynchronous archiving no longer copies WAL to a separate queue. If there is any WAL left over in the old queue after upgrading to 1.13, it will be abandoned and not pushed to the repository.

To prevent this outcome, stop archiving by setting archive_command = false. Next, drain the async queue by running pgbackrest --stanza=[stanza-name] archive-push and wait for the process to complete. Check that the queue in [spool-path]/archive/[stanza-name]/out is empty. Finally, install 1.13 and restore the original archive_command.

IMPORTANT NOTE: The stanza-create command is not longer optional and must be executed before backup or archiving can be performed on a new stanza. Pre-existing stanzas do not require stanza-create to be executed.
Bug Fixes:
  • Fixed const assignment giving compiler warning in C library. (Fixed by Adrian Vondendriesch. Reviewed by David Steele.)
  • Fixed a few directory syncs that were missed for the --repo-sync option.
  • Fixed an issue where a missing user/group on restore could cause an uninitialized value error in File->owner(). (Reported by Leonardo GG Avellar.)
  • Fixed an issue where protocol mismatch errors did not output the expected value.
  • Fixed a spurious archive-get log message that indicated an exit code of 1 was an abnormal termination.
Features:
  • Improved, multi-process implementation of asynchronous archiving.
  • Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Improved check command to run on a standby, though only basic checks are done because pg_switch_xlog() cannot be executed on a replica. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Added archive and backup WAL ranges to the info command.
  • Added warning to update pg_tablespace.spclocation when remapping tablespaces in PostgreSQL < 9.2. (Contributed by blogh. Reviewed by David Steele.)
  • Remove remote lock requirements for the archive-get, restore, info, and check commands since they are read-only operations. (Suggested by Michael Vitale.)
Improvements:
  • Log file banner is not output until the first log entry is written. (Suggested by Jens Wilke.)
  • Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN.
  • Remove Intel-specific optimization from C library build flags. (Contributed by Adrian Vondendriesch. Reviewed by David Steele.)
  • Remove --lock option. This option was introduced before the lock directory could be located outside the repository and is now obsolete.
  • Added --log-timestamp option to allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.
  • Return proper error code when unable to convert a relative path to an absolute path. (Suggested by Yogesh Sharma.)
Documentation Features:
  • Added documentation to the User Guide for the process-max option. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v1.12 Release Notes
Page Checksums, Configuration, and Bug Fixes
Released December 12, 2016
IMPORTANT NOTE: In prior releases it was possible to specify options on the command-line that were invalid for the current command without getting an error. An error will now be generated for invalid options so it is important to carefully check command-line options in your environment to prevent disruption.
Bug Fixes:
  • Fixed an issue where options that were invalid for the specified command could be provided on the command-line without generating an error. The options were ignored and did not cause any change in behavior, but it did lead to some confusion. Invalid options will now generate an error. (Reported by Nikhilchandra Kulkarni.)
  • Fixed an issue where internal symlinks were not being created for tablespaces in the repository. This issue was only apparent when trying to bring up clusters in-place manually using filesystem snapshots and did not affect normal backup and restore.
  • Fixed an issue that prevented errors from being output to the console before the logging system was initialized, i.e. while parsing options. Error codes were still being returned accurately so this would not have made a process look like it succeeded when it did not. (Reported by Adrian Vondendriesch.)
  • Fixed an issue where the db-port option specified on the backup server would not be properly passed to the remote unless it was from the first configured database. (Reported by Michael Vitale.)
Features:
  • Added the --checksum-page option to allow pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster. (Suggested by Stephen Frost.)
  • Added the --repo-link option to allow internal symlinks to be suppressed when the repository is located on a filesystem that does not support symlinks. This does not affect any pgBackRest functionality, but the convenience link latest will not be created and neither will internal tablespace symlinks, which will affect the ability to bring up clusters in-place manually using filesystem snapshots.
  • Added the --repo-sync option to allow directory syncs in the repository to be disabled for file systems that do not support them, e.g. NTFS.
  • Added a predictable log entry to signal that a command has completed successfully. For example a backup ends successfully with: INFO: backup command end: completed successfully. (Suggested by Jens Wilke.)
Improvements:
  • For simplicity, the pg_control file is now copied with the rest of the files instead of by itself of at the end of the process. The backup command does not require this behavior and the restore copies to a temporary file which is renamed at the end of the restore.
Documentation Bug Fixes:
  • Fixed an issue that suppressed exceptions in PDF builds.
  • Fixed regression in section links introduced in v1.10.
Documentation Features:
  • Added Retention to QuickStart section.
v1.11 Release Notes
Bug Fix for Asynchronous Archiving Efficiency
Released November 17, 2016
Bug Fixes:
  • Fixed an issue where asynchronous archiving was transferring one file per execution instead of transferring files in batches. This regression was introduced in v1.09 and affected efficiency only, all WAL segments were correctly archived in asynchronous mode. (Reported by Stephen Frost.)
v1.10 Release Notes
Stanza Creation and Minor Bug Fixes
Released November 8, 2016
Bug Fixes:
  • Fixed an issue where a backup could error if no changes were made to a database between backups and only pg_control changed.
  • Fixed an issue where tablespace paths with the same prefix would cause an invalid link error. (Reported by Nikhilchandra Kulkarni.)
Features:
  • Added the stanza-create command to formalize creation of stanzas in the repository. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Improvements:
  • Removed extraneous use lib directives from Perl modules. (Suggested by Devrim Gündüz.)
v1.09 Release Notes
9.6 Support, Configurability, and Bug Fixes
Released October 10, 2016
Bug Fixes:
  • Fixed the check command to prevent an error message from being logged if the backup directory does not exist. (Fixed by Cynthia Shang. Reviewed by David Steele.)
  • Fixed error message to properly display the archive command when an invalid archive command is detected. (Reported by Jason O'Donnell.)
  • Fixed an issue where the async archiver would not be started if archive-push did not have enough space to queue a new WAL segment. This meant that the queue would never be cleared without manual intervention (such as calling archive-push directly). PostgreSQL now receives errors when there is not enough space to store new WAL segments but the async process will still be started so that space is eventually freed. (Reported by Jens Wilke.)
  • Fixed a remote timeout that occurred when a local process generated checksums (during resume or restore) but did not copy files, allowing the remote to go idle. (Reported by Jens Wilke.)
Features:
  • Non-exclusive backups will automatically be used on PostgreSQL 9.6.
  • Added the cmd-ssh option to allow the ssh client to be specified. (Suggested by Jens Wilke.)
  • Added the log-level-stderr option to control whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. (Suggested by Sascha Biberhofer.)
  • Set application_name to "pgBackRest [command]" for database connections. (Suggested by Jens Wilke.)
  • Check that archive_mode is enabled when archive-check option enabled.
Improvements:
  • Clarified error message when unable to acquire pgBackRest advisory lock to make it clear that it is not a PostgreSQL backup lock. (Suggested by Jens Wilke.)
  • pgBackRest version number included in command start INFO log output.
  • Process ID logged for local process start/stop INFO log output.
Documentation Features:
  • Added archive-timeout option documentation to the user guide. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v1.08 Release Notes
Bug Fixes and Log Improvements
Released September 14, 2016
Bug Fixes:
  • Fixed an issue where local processes were not disconnecting when complete and could later timeout. (Reported by Todd Vernick.)
  • Fixed an issue where the protocol layer could timeout while waiting for WAL segments to arrive in the archive. (Reported by Todd Vernick.)
Improvements:
  • Cache file log output until the file is created to create a more complete log.
v1.07 Release Notes
Thread to Process Conversion and Bug Fixes
Released September 7, 2016
Bug Fixes:
  • Fixed an issue where tablespaces were copied from the primary during standby backup.
  • Fixed the check command so backup info is checked remotely and not just locally. (Fixed by Cynthia Shang. Reviewed by David Steele.)
  • Fixed an issue where retention-archive was not automatically being set when retention-archive-type=diff, resulting in a less aggressive than intended expiration of archive. (Fixed by Cynthia Shang. Reviewed by David Steele.)
Features:
  • Converted Perl threads to processes to improve compatibility and performance.
  • Exclude contents of $PGDATA/pg_replslot directory so that replication slots on the primary do not become part of the backup.
  • The archive-start and archive-stop settings are now filled in backup.manifest even when archive-check=n. (Suggested by Jens Wilke.)
  • Additional warnings when archive retention settings may not have the intended effect or would allow indefinite retention. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Experimental support for non-exclusive backups in PostgreSQL 9.6 rc1. Changes to the control/catalog/WAL versions in subsequent release candidates may break compatibility but pgBackRest will be updated with each release to keep pace.
Documentation Bug Fixes:
  • Fixed minor documentation reproducibility issues related to binary paths.
Documentation Features:
  • Documentation for archive retention. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v1.06 Release Notes
Backup from Standby and Bug Fixes
Released August 25, 2016
Bug Fixes:
  • Fixed an issue where a tablespace link that referenced another link would not produce an error, but instead skip the tablespace entirely. (Reported by Michael Vitale.)
  • Fixed an issue where options that should not allow multiple values could be specified multiple times in pgbackrest.conf without an error being raised. (Reported by Michael Vitale.)
  • Fixed an issue where the protocol-timeout option was not automatically increased when the db-timeout option was increased. (Reported by Todd Vernick.)
Features:
  • Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the primary.
  • More flexible configuration for databases. Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the primary. This means no configuration changes for backup are required after failing over from a primary to standby when a separate backup server is used.
  • Exclude directories during backup that are cleaned, recreated, or zeroed by PostgreSQL at startup. These include pgsql_tmp and pg_stat_tmp. The postgresql.auto.conf.tmp file is now excluded in addition to files that were already excluded: backup_label.old, postmaster.opts, postmaster.pid, recovery.conf, recovery.done.
  • Experimental support for non-exclusive backups in PostgreSQL 9.6 beta4. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
Improvements:
  • Improve error message for links that reference links in manifest build.
  • Added hints to error message when relative paths are detected in archive-push or archive-get.
  • Improve backup log messages to indicate which host the files are being copied from.
v1.05 Release Notes
Bug Fix for Tablespace Link Checking
Released August 9, 2016
Bug Fixes:
  • Fixed an issue where tablespace paths that had $PGDATA as a substring would be identified as a subdirectories of $PGDATA even when they were not. Also hardened relative path checking a bit. (Reported by Chris Fort.)
Documentation Features:
  • Added documentation for scheduling backups with cron. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Documentation Improvements:
  • Moved the backlog from the pgBackRest website to the GitHub repository wiki. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v1.04 Release Notes
Various Bug Fixes
Released July 30, 2016
Bug Fixes:
  • Fixed an issue an where an extraneous remote was created causing threaded backup/restore to possibly timeout and/or throw a lock conflict. (Reported by Michael Vitale.)
  • Fixed an issue where db-path was not required for the check command so an assert was raised when it was missing rather than a polite error message. (Reported by Michael Vitale.)
  • Fixed check command to throw an error when database version/id does not match that of the archive. (Fixed by Cynthia Shang. Reviewed by David Steele.)
  • Fixed an issue where a remote could try to start its own remote when the backup-host option was not present in pgbackrest.conf on the database server. (Reported by Lardière Sébastien.)
  • Fixed an issue where the contents of pg_xlog were being backed up if the directory was symlinked. This didn't cause any issues during restore but was a waste of space.
  • Fixed an invalid log() call in lock routines.
Features:
  • Experimental support for non-exclusive backups in PostgreSQL 9.6 beta3. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
Improvements:
  • Suppress banners on SSH protocol connections.
  • Improved remote error messages to identify the host where the error was raised.
  • All remote types now take locks. The exceptions date to when the test harness and pgBackRest were running in the same VM and no longer apply.
Documentation Features:
  • Added clarification on why the default for the backrest-user option is backrest. (Suggested by Michael Vitale.)
  • Updated information about package availability on supported platforms. (Suggested by Michael Vitale.)
v1.03 Release Notes
Check Command and Bug Fixes
Released July 2, 2016
Bug Fixes:
  • Fixed an issue where keep-alives could be starved out by lots of small files during multi-threaded backup. They were also completely absent from single/multi-threaded backup resume and restore checksumming. (Reported by Janice Parkinson, Chris Barber.)
  • Fixed an issue where the expire command would refuse to run when explicitly called from the command line if the db-host option was set. This was not an issue when expire was run automatically after a backup (Reported by Chris Barber.)
  • Fixed an issue where validation was being running on archive_command even when the archive-check option was disabled.
Features:
  • Added check command to validate that pgBackRest is configured correctly for archiving and backups. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Added the protocol-timeout option. Previously protocol-timeout was set as db-timeout + 30 seconds.
  • Failure to shutdown remotes at the end of the backup no longer throws an exception. Instead a warning is generated that recommends a higher protocol-timeout.
  • Experimental support for non-exclusive backups in PostgreSQL 9.6 beta2. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
Improvements:
  • Improved handling of users/groups captured during backup that do not exist on the restore host. Also explicitly handle the case where user/group is not mapped to a name.
  • Option handling is now far more strict. Previously it was possible for a command to use an option that was not explicitly assigned to it. This was especially true for the backup-host and db-host options which are used to determine locality.
Documentation Improvements:
  • Allow a static date to be used for documentation to generate reproducible builds. (Suggested by Adrian Vondendriesch.)
  • Added documentation for asynchronous archiving to the user guide. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Recommended install location for pgBackRest modules is now /usr/share/perl5 since /usr/lib/perl5 has been removed from the search path in newer versions of Perl.
  • Added instructions for removing prior versions of pgBackRest.
v1.02 Release Notes
Bug Fix for Perl 5.22
Released June 2, 2016
Bug Fixes:
  • Fix usage of sprintf() due to new constraints in Perl 5.22. Parameters not referenced in the format string are no longer allowed. (Fixed by Adrian Vondendriesch. Reviewed by David Steele.)
Documentation Bug Fixes:
  • Fixed syntax that was not compatible with Perl 5.2X. (Fixed by Christoph Berg, Adrian Vondendriesch. Reviewed by David Steele.)
  • Fixed absolute paths that were used for the PDF logo. (Reported by Adrian Vondendriesch.)
Documentation Features:
  • Release notes are now broken into sections so that bugs, features, and refactors are clearly delineated. An Additional Notes section has been added for changes to documentation and the test suite that do not affect the core code.
  • Added man page generation. (Contributed by Adrian Vondendriesch, David Steele.)
  • The change log was the last piece of documentation to be rendered in Markdown only. Wrote a converter so the document can be output by the standard renderers. The change log will now be located on the website and has been renamed to Releases. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v1.01 Release Notes
Enhanced Info, Selective Restore, and 9.6 Support
Released May 17, 2016
Features:
  • Enhanced text output of info command to include timestamps, sizes, and the reference list for all backups. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue. (Reviewed by Cynthia Shang, Greg Smith, Stephen Frost. Suggested by Stephen Frost.)
  • Experimental support for non-exclusive backups in PostgreSQL 9.6 beta1. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace. (Reviewed by Cynthia Shang.)
v1.00 Release Notes
New Repository Format and Configuration Scheme, Link Support
Released April 14, 2016
IMPORTANT NOTE: This flag day release breaks compatibility with older versions of pgBackRest. The manifest format, on-disk structure, configuration scheme, and the exe/path names have all changed. You must create a new repository to hold backups for this version of pgBackRest and keep your older repository for a time in case you need to do a restore. Restores from the prior repository will require the prior version of pgBackRest but because of name changes it is possible to have 1.00 and a prior version of pgBackRest installed at the same time. See the notes below for more detailed information on what has changed.
Features:
  • Implemented a new configuration scheme which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. (Suggested by Michael Renner.)
  • The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. A local repository is no longer required.
  • The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally when migrating to 1.00. (Suggested by Michael Renner, Stephen Frost.)
  • The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. (Suggested by Michael Renner, Stephen Frost.)
  • Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest is the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory.
  • Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the log directory. (Suggested by Stephen Frost.)
  • Executable filename changed from pg_backrest to pgbackrest. (Suggested by Michael Renner, Stephen Frost.)
  • All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location.
  • Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function.
  • Added detail log level which will output more information than info without being as verbose as debug.
Pre-Stable Releases
v0.92 Release Notes
Command-line Repository Path Fix
Released April 6, 2016
Bug Fixes:
  • Fixed an issue where the master process was passing --repo-remote-path instead of --repo-path to the remote and causing the lock files to be created in the default repository directory (/var/lib/backup), generally ending in failure. This was only an issue when --repo-remote-path was defined on the command line rather than in pg_backrest.conf. (Reported by Jan Wieck.)
v0.91 Release Notes
Tablespace Bug Fix and Minor Enhancements
Released March 22, 2016
IMPORTANT BUG FIX FOR TABLESPACES: A change to the repository format was accidentally introduced in 0.90 which means the on-disk backup was no longer a valid PostgreSQL cluster when the backup contained tablespaces. This only affected users who directly copied the backups to restore PostgreSQL clusters rather than using the restore command. However, the fix breaks compatibility with older backups that contain tablespaces no matter how they are being restored (pgBackRest will throw errors and refuse to restore). New full backups should be taken immediately after installing version 0.91 for any clusters that contain tablespaces. If older backups need to be restored then use a version of pgBackRest that matches the backup version.
Bug Fixes:
  • Fixed repository incompatibility introduced in pgBackRest 0.90. (Reported by Evan Benoit.)
Features:
  • Copy global/pg_control last during backups.
  • Write .info and .manifest files to temp before moving them to their final locations and fsync'ing.
  • Rename --no-start-stop option to --no-online.
Test Suite Features:
  • Static source analysis using Perl-Critic, currently passes on gentle.
v0.90 Release Notes
9.5 Support, Various Enhancements, and Minor Bug Fixes
Released February 7, 2016
Bug Fixes:
  • Fixed an issue where specifying --no-archive-check would throw a configuration error. (Reported by Jason O'Donnell.)
  • Fixed an issue where a temp WAL file left over after a well-timed system crash could cause the next archive-push to fail.
  • The retention-archive option can now be be safely set to less than backup retention (retention-full or retention-diff) without also specifying archive-copy=n. The WAL required to make the backups that fall outside of archive retention consistent will be preserved in the archive. However, in this case PITR will not be possible for the backups that fall outside of archive retention.
Features:
  • When backing up and restoring tablespaces pgBackRest only operates on the subdirectory created for the version of PostgreSQL being run against. Since multiple versions can live in a tablespace (especially during a binary upgrade) this prevents too many files from being copied during a backup and other versions possibly being wiped out during a restore. This only applies to PostgreSQL >= 9.0 — prior versions of PostgreSQL could not share a tablespace directory.
  • Generate an error when archive-check=y but archive_command does not execute pg_backrest. (Contributed by Jason O'Donnell. Reviewed by David Steele.)
  • Improved error message when repo-path or repo-remote-path does not exist.
  • Added checks for --delta and --force restore options to ensure that the destination is a valid $PGDATA directory. pgBackRest will check for the presence of PG_VERSION or backup.manifest (left over from an aborted restore). If neither file is found then --delta and --force will be disabled but the restore will proceed unless there are files in the $PGDATA directory (or any tablespace directories) in which case the operation will be aborted.
  • When restore --set=latest (the default) the actual backup restored will be output to the log.
  • Support for PostgreSQL 9.5 partial WAL segments and recovery_target_action setting. The archive_mode = 'always' setting is not yet supported.
  • Support for recovery_target = 'immediate' recovery setting introduced in PostgreSQL 9.4.
  • The following tablespace checks have been added: paths or files in pg_tblspc, relative links in pg_tblspc, tablespaces in $PGDATA. All three will generate errors.
v0.89 Release Notes
Timeout Bug Fix and Restore Read-Only Repositories
Released December 24, 2015
Bug Fixes:
  • Fixed an issue where longer-running backups/restores would timeout when remote and threaded. Keepalives are now used to make sure the remote for the main process does not timeout while the thread remotes do all the work. The error message for timeouts was also improved to make debugging easier. (Reported by Stephen Frost.)
Features:
  • Allow restores to be performed on a read-only repository by using --no-lock and --log-level-file=off. The --no-lock option can only be used with restores.
v0.88 Release Notes
Documentation and Minor Bug Fixes
Released November 22, 2015
Bug Fixes:
  • Fixed an issue where the start/stop commands required the --config option. (Reported by Dmitry Didovicher.)
  • Fixed an issue where log files were being overwritten instead of appended. (Reported by Stephen Frost, Dmitry Didovicher.)
  • Fixed an issue where backup-user was not optional.
Features:
  • Symlinks are no longer created in backup directories in the repository. These symlinks could point virtually anywhere and potentially be dangerous. Symlinks are still recreated during a restore. (Suggested by Stephen Frost.)
  • Added better messaging for backup expiration. Full and differential backup expirations are logged on a single line along with a list of all dependent backups expired.
  • Archive retention is automatically set to full backup retention if not explicitly configured.
Documentation Features:
  • Added documentation in the user guide for delta restores, expiration, dedicated backup hosts, starting and stopping pgBackRest, and replication.
v0.87 Release Notes
Website and User Guide
Released October 28, 2015
Features:
  • The backup_label.old and recovery.done files are now excluded from backups.
Documentation Features:
  • Added a new user guide that covers pgBackRest basics and some advanced topics including PITR. Much more to come, but it's a start. (Contributed by David Steele, Stephen Frost. Reviewed by Michael Renner, Cynthia Shang, Eric Radman, Dmitry Didovicher.)
v0.85 Release Notes
Start/Stop Commands and Minor Bug Fixes
Released October 8, 2015
Bug Fixes:
  • Fixed an issue where an error could be returned after a backup or restore completely successfully.
  • Fixed an issue where a resume would fail if temp files were left in the root backup directory when the backup failed. This scenario was likely if the backup process got terminated during the copy phase.
Features:
  • Added stop and start commands to prevent pgBackRest processes from running on a system where PostgreSQL is shutdown or the system needs to be quiesced for some other reason.
  • Experimental support for PostgreSQL 9.5 beta1. This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace. All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.
v0.82 Release Notes
Refactoring, Command-line Help, and Minor Bug Fixes
Released September 14, 2015
Bug Fixes:
  • Fixed an issue where resumed compressed backups were not preserving existing files.
  • Fixed an issue where resume and incr/diff would not ensure that the prior backup had the same compression and hardlink settings.
  • Fixed an issue where a cold backup using --no-start-stop could be started on a running PostgreSQL cluster without --force specified.
  • Fixed an issue where a thread could be started even when none were requested.
  • Fixed an issue where the pgBackRest version number was not being updated in backup.info and archive.info after an upgrade/downgrade.
  • Fixed an issue where the info command was throwing an exception when the repository contained no stanzas. (Reported by Stephen Frost.)
  • Fixed an issue where the PostgreSQL pg_stop_backup() NOTICEs were being output to stderr. (Reported by Stephen Frost.)
Features:
  • Experimental support for PostgreSQL 9.5 alpha2. This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace. All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.
Improvements:
  • Renamed recovery-setting option and section to recovery-option to be more consistent with pgBackRest naming conventions.
  • Added dynamic module loading to speed up commands, especially asynchronous archiving.
Documentation Features:
  • Command-line help is now extracted from the same XML source that is used for the other documentation and includes much more detail.
v0.80 Release Notes
DBI Support, Stability, and Convenience Features
Released August 9, 2015
Bug Fixes:
  • Fixed an issue that caused the formatted timestamp for both the oldest and newest backups to be reported as the current time by the info command. Only text output was affected -- json output reported the correct epoch values. (Reported by Michael Renner.)
  • Fixed protocol issue that was preventing ssh errors (especially on connection) from being logged.
Features:
  • The repository is now created and updated with consistent directory and file modes. By default umask is set to 0000 but this can be disabled with the neutral-umask setting. (Suggested by Cynthia Shang.)
  • Added the stop-auto option to allow failed backups to automatically be stopped when a new backup starts.
  • Added the db-timeout option to limit the amount of time pgBackRest will wait for pg_start_backup() and pg_stop_backup() to return.
  • Remove pg_control file at the beginning of the restore and copy it back at the very end. This prevents the possibility that a partial restore can be started by PostgreSQL.
  • Added checks to be sure the db-path setting is consistent with db-port by comparing the data_directory as reported by the cluster against the db-path setting and the version as reported by the cluster against the value read from pg_control. The db-socket-path setting is checked to be sure it is an absolute path.
  • Experimental support for PostgreSQL 9.5 alpha1. This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace. All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.
Improvements:
  • Now using Perl DBI and DBD::Pg for connections to PostgreSQL rather than psql. The cmd-psql and cmd-psql-option settings have been removed and replaced with db-port and db-socket-path. Follow the instructions in the Installation Guide to install DBD::Pg on your operating system.
Test Suite Features:
  • Added vagrant test configurations for Ubuntu 14.04 and CentOS 7.
v0.78 Release Notes
Remove CPAN Dependencies, Stability Improvements
Released July 13, 2015
Improvements:
  • Removed dependency on CPAN packages for multi-threaded operation. While it might not be a bad idea to update the threads and Thread::Queue packages, it is no longer necessary.
  • Modified wait backoff to use a Fibonacci rather than geometric sequence. This will make wait time grow less aggressively while still giving reasonable values.
Test Suite Features:
  • Added vagrant test configurations for Ubuntu 12.04 and CentOS 6.
v0.77 Release Notes
CentOS/RHEL 6 Support and Protocol Improvements
Released June 30, 2015
Features:
  • Added file and directory syncs to the File object for additional safety during backup/restore and archiving. (Suggested by Andres Freund.)
  • Added support for Perl 5.10.1 and OpenSSH 5.3 which are default for CentOS/RHEL 6. (Suggested by Eric Radman.)
  • Improved error message when backup is run without archive_command set and without --no-archive-check specified. (Suggested by Eric Radman.)
v0.75 Release Notes
New Repository Format, Info Command and Experimental 9.5 Support
Released June 14, 2015
IMPORTANT NOTE: This flag day release breaks compatibility with older versions of pgBackRest. The manifest format, on-disk structure, and the binary names have all changed. You must create a new repository to hold backups for this version of pgBackRest and keep your older repository for a time in case you need to do a restore. The pg_backrest.conf file has not changed but you'll need to change any references to pg_backrest.pl in cron (or elsewhere) to pg_backrest (without the .pl extension).
Features:
  • Added the info command.
  • Logging now uses unbuffered output. This should make log files that are being written by multiple threads less chaotic. (Suggested by Michael Renner.)
  • Experimental support for PostgreSQL 9.5. This may break when the control version or WAL magic changes but will be updated in each release.
Improvements:
  • More efficient file ordering for backup. Files are copied in descending size order so a single thread does not end up copying a large file at the end. This had already been implemented for restore.
v0.70 Release Notes
Stability Improvements for Archiving, Improved Logging and Help
Released June 1, 2015
Bug Fixes:
  • Fixed an issue where archive-copy would fail on an incr/diff backup when hardlink=n. In this case the pg_xlog path does not already exist and must be created. (Reported by Michael Renner.)
  • Fixed an issue in async archiving where archive-push was not properly returning 0 when archive-max-mb was reached and moved the async check after transfer to avoid having to remove the stop file twice. Also added unit tests for this case and improved error messages to make it clearer to the user what went wrong. (Reported by Michael Renner.)
  • Fixed a locking issue that could allow multiple operations of the same type against a single stanza. This appeared to be benign in terms of data integrity but caused spurious errors while archiving and could lead to errors in backup/restore. (Reported by Michael Renner.)
Features:
  • Allow duplicate WAL segments to be archived when the checksum matches. This is necessary for some recovery scenarios.
  • Allow comments/disabling in pg_backrest.conf using the # character. Only # characters in the first character of the line are honored. (Suggested by Michael Renner.)
  • Better logging before pg_start_backup() to make it clear when the backup is waiting on a checkpoint. (Suggested by Michael Renner.)
  • Various command behavior and logging fixes. (Reviewed by Michael Renner. Suggested by Michael Renner.)
Improvements:
  • Replaced JSON module with JSON::PP which ships with core Perl.
Documentation Bug Fixes:
  • Various help fixes. (Reviewed by Michael Renner. Reported by Michael Renner.)
v0.65 Release Notes
Improved Resume and Restore Logging, Compact Restores
Released May 11, 2015
Bug Fixes:
  • Fixed an issue where an absolute path was not written into recovery.conf when the restore was run with a relative path.
Features:
  • Better resume support. Resumed files are checked to be sure they have not been modified and the manifest is saved more often to preserve checksums as the backup progresses. More unit tests to verify each resume case.
  • Resume is now optional. Use the resume setting or --no-resume from the command line to disable.
  • More info messages during restore. Previously, most of the restore messages were debug level so not a lot was output in the log.
  • Added tablespace setting to allow tablespaces to be restored into the pg_tblspc path. This produces compact restores that are convenient for development, staging, etc. Currently these restores cannot be backed up as pgBackRest expects only links in the pg_tblspc path.
v0.61 Release Notes
Bug Fix for Uncompressed Remote Destination
Released April 21, 2015
Bug Fixes:
  • Fixed a buffering error that could occur on large, highly-compressible files when copying to an uncompressed remote destination. The error was detected in the decompression code and resulted in a failed backup rather than corruption so it should not affect successful backups made with previous versions.
v0.60 Release Notes
Better Version Support and WAL Improvements
Released April 19, 2015
Bug Fixes:
  • Pushing duplicate WAL now generates an error. This worked before only if checksums were disabled.
Features:
  • Database System IDs are used to make sure that all WAL in an archive matches up. This should help prevent misconfigurations that send WAL from multiple clusters to the same archive.
Test Suite Features:
  • Regression tests working back to PostgreSQL 8.3.
v0.50 Release Notes
Restore and Much More
Released March 25, 2015
Bug Fixes:
  • Fixed broken checksums and now they work with normal and resumed backups. Finally realized that checksums and checksum deltas should be functionally separated and this simplified a number of things. Issue #28 has been created for checksum deltas.
  • Fixed an issue where a backup could be resumed from an aborted backup that didn't have the same type and prior backup.
Features:
  • Added restore functionality.
  • All options can now be set on the command-line making pg_backrest.conf optional.
  • De/compression is now performed without threads and checksum/size is calculated in stream. That means file checksums are no longer optional.
  • Added option --no-start-stop to allow backups when Postgres is shut down. If postmaster.pid is present then --force is required to make the backup run (though if Postgres is running an inconsistent backup will likely be created). This option was added primarily for the purpose of unit testing, but there may be applications in the real world as well.
  • Checksum for backup.manifest to detect a corrupted/modified manifest.
  • Link latest always points to the last backup. This has been added for convenience and to make restores simpler.
Test Suite Features:
  • More comprehensive unit tests in all areas.
v0.30 Release Notes
Core Restructuring and Unit Tests
Released October 5, 2014
Documentation Features:
  • Added much needed documentation
Test Suite Features:
  • Fairly comprehensive unit tests for all the basic operations. More work to be done here for sure, but then there is always more work to be done on unit tests.
v0.19 Release Notes
Improved Error Reporting/Handling
Released May 13, 2014
Bug Fixes:
  • Found and squashed a nasty bug where file_copy() was defaulted to ignore errors. There was also an issue in file_exists() that was causing the test to fail when the file actually did exist. Together they could have resulted in a corrupt backup with no errors, though it is very unlikely.
v0.18 Release Notes
Return Soft Error When Archive Missing
Released April 13, 2014
Bug Fixes:
  • The archive-get command now returns a 1 when the archive file is missing to differentiate from hard errors (ssh connection failure, file copy error, etc.) This lets PostgreSQL know that the archive stream has terminated normally. However, this does not take into account possible holes in the archive stream. (Reported by Stephen Frost.)
v0.17 Release Notes
Warn When Archive Directories Cannot Be Deleted
Released April 3, 2014
Bug Fixes:
  • If an archive directory which should be empty could not be deleted backrest was throwing an error. There's a good fix for that coming, but for the time being it has been changed to a warning so processing can continue. This was impacting backups as sometimes the final archive file would not get pushed if the first archive file had been in a different directory (plus some bad luck).
v0.16 Release Notes
RequestTTY=yes for SSH Sessions
Released April 1, 2014
Bug Fixes:
  • Added RequestTTY=yes to ssh sessions. Hoping this will prevent random lockups.
v0.15 Release Notes
Added archive-get
Released March 29, 2014
Features:
  • Added archive-get functionality to aid in restores.
  • Added option to force a checkpoint when starting the backup, start-fast=y.
v0.11 Release Notes
Minor Fixes
Released March 26, 2014
Bug Fixes:
  • Removed master_stderr_discard option on database SSH connections. There have been occasional lockups and they could be related to issues originally seen in the file code. (Reported by Stephen Frost.)
  • Changed lock file conflicts on backup and expire commands to ERROR. They were set to DEBUG due to a copy-and-paste from the archive locks.
v0.10 Release Notes
Backup and Archiving are Functional
Released March 5, 2014
Features:
  • No restore functionality, but the backup directories are consistent PostgreSQL data directories. You'll need to either uncompress the files or turn off compression in the backup. Uncompressed backups on a ZFS (or similar) filesystem are a good option because backups can be restored locally via a snapshot to create logical backups or do spot data recovery.
  • Archiving is single-threaded. This has not posed an issue on our multi-terabyte databases with heavy write volume. Recommend a large WAL volume or to use the async option with a large volume nearby.
  • Backups are multi-threaded, but the Net::OpenSSH library does not appear to be 100% thread-safe so it will very occasionally lock up on a thread. There is an overall process timeout that resolves this issue by killing the process. Yes, very ugly.
  • Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes. Checksums from previous backups are correctly recorded and a full backup will reset everything.
  • The backup.manifest is being written as Storable because Config::IniFile does not seem to handle large files well. Would definitely like to save these as human-readable text.
Documentation Features:
  • Absolutely no documentation (outside the code). Well, excepting these release notes.
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated October 18, 2026
././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7770348 pgbackrest-2.60.0/doc/html/slogo.svg0000644000175100017510000005601615265102133016755 0ustar00runnerrunner ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7774975 pgbackrest-2.60.0/doc/html/user-guide-index.html0000644000175100017510000000441515265102133021151 0ustar00runnerrunner pgBackRest - User Guides
pgBackRest
User Guides
Home
User Guides
Releases
Configuration
Commands
FAQ
User Guides
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated October 18, 2026
././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792312410.777414 pgbackrest-2.60.0/doc/html/user-guide-rhel.html0000644000175100017510000071614615265102133021007 0ustar00runnerrunner pgBackRest - RHEL User Guide
pgBackRest
RHEL User Guide
Home
User Guides
Releases
Configuration
Commands
FAQ
Table of Contents
Introduction
Concepts
Backup
Restore
Write Ahead Log (WAL)
Encryption
Upgrading pgBackRest
From v2.x to v2.y
Build
Installation
Quick Start
Setup Demo Cluster
Configure Cluster Stanza
Create the Repository
Configure Archiving
Configure Retention
Configure Repository Encryption
Create the Stanza
Check the Configuration
Performance Tuning
Perform a Backup
Schedule a Backup
Backup Information
Restore a Backup
Monitoring
In PostgreSQL
Backup
File Bundling
Block Incremental
Backup Annotations
Retention
Full Backup Retention
Differential Backup Retention
Archive Retention
Restore
File Ownership
Delta Option
Restore Selected Databases
Point-in-Time Recovery
Delete a Stanza
Multiple Repositories
Azure Support
S3 Support
SFTP Support
GCS Support
Target Time for Repository
Dedicated Repository Host
Installation
Configuration
Setup TLS Server
Create and Check Stanza
Perform a Backup
Restore a Backup
Parallel Backup / Restore
Starting and Stopping
Replication
Installation
Hot Standby
Streaming Replication
Multiple Stanzas
Installation
Configuration
Setup Demo Cluster
Create the Stanza and Check Configuration
Asynchronous Archiving
Archive Push
Archive Get
Backup from a Standby
Upgrading PostgreSQL
This user guide is intended to be followed sequentially from beginning to end — each section depends on the last. For example, the Restore section relies on setup that is performed in the Quick Start section. Once pgBackRest is up and running then skipping around is possible but following the user guide in order is recommended the first time through.
Although the examples in this guide are targeted at RHEL and PostgreSQL 14, it should be fairly easy to apply the examples to any Unix distribution and PostgreSQL version. The only OS-specific commands are those to create, start, stop, and drop PostgreSQL clusters. The pgBackRest commands will be the same on any Unix system though the location of the executable may vary. While pgBackRest strives to operate consistently across versions of PostgreSQL, there are subtle differences between versions of PostgreSQL that may show up in this guide when illustrating certain examples, e.g. PostgreSQL path/file names and settings.
Configuration information and documentation for PostgreSQL can be found in the PostgreSQL Manual.
A somewhat novel approach is taken to documentation in this user guide. Each command is run on a virtual machine when the documentation is built from the XML source. This means you can have a high confidence that the commands work correctly in the order presented. Output is captured and displayed below the command when appropriate. If the output is not included it is because it was deemed not relevant or was considered a distraction from the narrative.
All commands are intended to be run as an unprivileged user that has sudo privileges for both the root and postgres users. It's also possible to run the commands directly as their respective users without modification and in that case the sudo commands can be stripped off.
Concepts
The following concepts are defined as they are relevant to pgBackRest, PostgreSQL, and this user guide.
Backup
A backup is a consistent copy of a database cluster that can be restored to recover from a hardware failure, to perform Point-In-Time Recovery, or to bring up a new standby.
Full Backup: pgBackRest copies the entire contents of the database cluster to the backup. The first backup of the database cluster is always a Full Backup. pgBackRest is always able to restore a full backup directly. The full backup does not depend on any files outside of the full backup for consistency.
Differential Backup: pgBackRest copies only those database cluster files that have changed since the last full backup. pgBackRest restores a differential backup by copying all of the files in the chosen differential backup and the appropriate unchanged files from the previous full backup. The advantage of a differential backup is that it requires less disk space than a full backup, however, the differential backup and the full backup must both be valid to restore the differential backup.
Incremental Backup: pgBackRest copies only those database cluster files that have changed since the last backup (which can be another incremental backup, a differential backup, or a full backup). As an incremental backup only includes those files changed since the prior backup, they are generally much smaller than full or differential backups. As with the differential backup, the incremental backup depends on other backups to be valid to restore the incremental backup. Since the incremental backup includes only those files since the last backup, all prior incremental backups back to the prior differential, the prior differential backup, and the prior full backup must all be valid to perform a restore of the incremental backup. If no differential backup exists then all prior incremental backups back to the prior full backup, which must exist, and the full backup itself must be valid to restore the incremental backup.
Restore
A restore is the act of copying a backup to a system where it will be started as a live database cluster. A restore requires the backup files and one or more WAL segments in order to work correctly.
Write Ahead Log (WAL)
WAL is the mechanism that PostgreSQL uses to ensure that no committed changes are lost. Transactions are written sequentially to the WAL and a transaction is considered to be committed when those writes are flushed to disk. Afterwards, a background process writes the changes into the main database cluster files (also known as the heap). In the event of a crash, the WAL is replayed to make the database consistent.
WAL is conceptually infinite but in practice is broken up into individual 16MB files called segments. WAL segments follow the naming convention 0000000100000A1E000000FE where the first 8 hexadecimal digits represent the timeline and the next 16 digits are the logical sequence number (LSN).
Encryption
Encryption is the process of converting data into a format that is unrecognizable unless the appropriate password (also referred to as passphrase) is provided.
pgBackRest will encrypt the repository based on a user-provided password, thereby preventing unauthorized access to data stored within the repository.
Upgrading pgBackRest
From v2.x to v2.y
Upgrading from v2.x to v2.y is straight-forward. The repository format has not changed, so for most installations it is simply a matter of installing binaries for the new version. It is also possible to downgrade if you have not used new features that are unsupported by the older version.
IMPORTANT:
The local and remote pgBackRest versions must match exactly so they should be upgraded together. If there is a mismatch, WAL archiving and backups will not function until the versions match. In such a case, the following error will be reported: [ProtocolError] expected value '2.x' for greeting key 'version' but got '2.y'.
Build
Installing pgBackRest from a package is preferable to building from source. See Installation for more information about packages.
When building from source it is best to use a build host rather than building on production. Many of the tools required for the build should generally not be installed in production. pgBackRest consists of a single executable so it is easy to copy to a new host once it is built.
build ⇒ Download version 2.60.0 of pgBackRest to /build path 
mkdir -p /build
curl -fsSL \
       https://github.com/pgbackrest/pgbackrest/releases/download/release%2F2.60.0/pgbackrest-2.60.0.tar.gz | \
       tar zx -C /build
build ⇒ Install build dependencies 
sudo yum install meson gcc postgresql14-devel openssl-devel libxml2-devel \
       lz4-devel libzstd-devel bzip2-devel libssh2-devel systemd-devel
build ⇒ Configure and compile pgBackRest 
meson setup /build/pgbackrest /build/pgbackrest-2.60.0
ninja -C /build/pgbackrest
build ⇒ Optionally run smoke tests to verify pgBackRest was built correctly 
meson test -C /build/pgbackrest --suite smoke
Output suppressed for testing
Installation
A new host named pg-primary is created to contain the demo cluster and run pgBackRest examples.
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
RHEL packages for pgBackRest are available at yum.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
pg-primary ⇒ Install dependencies 
sudo yum install postgresql-libs libssh2
pg-primary ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
pg-primary ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown postgres:postgres /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
pgBackRest should now be properly installed but it is best to check. If any dependencies were missed then you will get an error when running pgBackRest from the command line.
pg-primary ⇒ Make sure the installation worked 
sudo -u postgres pgbackrest
Output suppressed for testing
Quick Start
The Quick Start section will cover basic configuration of pgBackRest and PostgreSQL and introduce the backup, restore, and info commands.
Setup Demo Cluster
Creating the demo cluster is optional but is strongly recommended, especially for new users, since the example commands in the user guide reference the demo cluster; the examples assume the demo cluster is running on the default port (i.e. 5432). The cluster will not be started until a later section because there is still some configuration to do.
pg-primary ⇒ Create the demo cluster 
sudo -u postgres /usr/pgsql-14/bin/initdb \
       -D /var/lib/pgsql/14/data -k -A peer
By default RHEL includes the day of the week in the log filename. This makes the user guide a bit more complicated so the log_filename is set to a constant.
pg-primary:/var/lib/pgsql/14/data/postgresql.conf ⇒ Set log_filename 
Config suppressed for testing
Configure Cluster Stanza
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod.
The name 'demo' describes the purpose of this cluster accurately so that will also make a good stanza name.
pgBackRest needs to know where the base data directory for the PostgreSQL cluster is located. The path can be requested from PostgreSQL directly but in a recovery scenario the PostgreSQL process will not be available. During backups the value supplied to pgBackRest will be compared against the path that PostgreSQL is running on and they must be equal or the backup will return an error. Make sure that pg-path is exactly equal to data_directory as reported by PostgreSQL.
By default RHEL stores clusters in /var/lib/pgsql/[version]/data so it is easy to determine the correct path for the data directory.
When creating the /etc/pgbackrest/pgbackrest.conf file, the database owner (usually postgres) must be granted read privileges.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the PostgreSQL cluster data directory 
Config suppressed for testing
pgBackRest configuration files follow a Windows INI-like convention. Sections are denoted by text in brackets and key/value pairs are contained in each section. Lines beginning with # are ignored and can be used as comments, but end-of-line comments following a value on the same line are not supported. Quoting is not supported and whitespace is trimmed from keys and values. Sections will be merged if they appear more than once.
There are multiple ways the pgBackRest configuration files can be loaded:
  • config and config-include-path are default: the default config file will be loaded, if it exists, and *.conf files in the default config include path will be appended, if they exist.
  • config option is specified: only the specified config file will be loaded and is expected to exist.
  • config-include-path is specified: *.conf files in the config include path will be loaded and the path is required to exist. The default config file will be be loaded if it exists. If it is desirable to load only the files in the specified config include path, then the --no-config option can also be passed.
  • config and config-include-path are specified: using the user-specified values, the config file will be loaded and *.conf files in the config include path will be appended. The files are expected to exist.
  • config-path is specified: this setting will override the base path for the default location of the config file and/or the base path of the default config-include-path setting unless the config and/or config-include-path option is explicitly set.
Files are concatenated as if they were one big file and each file must be valid individually. This means sections must be specified in each file where they are needed to store a key/value. Order doesn't matter but there is precedence based on sections. The precedence (highest to lowest) is:
  • [stanza:command]
  • [stanza]
  • [global:command]
  • [global]
NOTE:
--config, --config-include-path and --config-path are command-line only options.
pgBackRest can also be configured using environment variables as described in the command reference.
pg-primary ⇒ Configure log-path using the environment 
sudo -u postgres bash -c ' \
       export PGBACKREST_LOG_PATH=/path/set/by/env && \
       pgbackrest --log-level-console=error help backup log-path'
Output suppressed for testing
Create the Repository
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves.
For this demonstration the repository will be stored on the same host as the PostgreSQL server. This is the simplest configuration and is useful in cases where traditional backup software is employed to backup the database host.
pg-primary ⇒ Create the pgBackRest repository 
sudo mkdir -p /var/lib/pgbackrest
sudo chmod 750 /var/lib/pgbackrest
sudo chown postgres:postgres /var/lib/pgbackrest
The repository path must be configured so pgBackRest knows where to find it.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the pgBackRest repository path 
Config suppressed for testing
Multiple repositories may also be configured. See Multiple Repositories for details.
Configure Archiving
Backing up a running PostgreSQL cluster requires WAL archiving to be enabled. %p is how PostgreSQL specifies the location of the WAL segment to be archived. Note that at least one WAL segment will be created during the backup process even if no explicit writes are made to the cluster.
pg-primary:/var/lib/pgsql/14/data/postgresql.conf ⇒ Configure archive settings 
Config suppressed for testing
The PostgreSQL cluster must be restarted after making these changes and before performing a backup.
pg-primary ⇒ Restart the demo cluster 
sudo systemctl restart postgresql-14.service
The hot_standby setting is enabled by default and should be left that way on every cluster. Any cluster may be restored as a standby later, e.g. a primary that is rebuilt as a standby after failover, and a standby will not accept read-only connections without it.
When archiving a WAL segment is expected to take more than 60 seconds (the default) to reach the pgBackRest repository, then the pgBackRest archive-timeout option should be increased. Note that this option is not the same as the PostgreSQL archive_timeout option which is used to force a WAL segment switch; useful for databases where there are long periods of inactivity. For more information on the PostgreSQL archive_timeout option, see PostgreSQL Write Ahead Log.
The archive-push command can be configured with its own options. For example, a lower compression level may be set to speed archiving without affecting the compression used for backups.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Config archive-push to use a lower compression level 
Config suppressed for testing
This configuration technique can be used for any command and can even target a specific stanza, e.g. demo:archive-push.
Configure Retention
pgBackRest expires backups based on retention options.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure retention to 2 full backups 
Config suppressed for testing
More information about retention can be found in the Retention section.
Configure Repository Encryption
The repository will be configured with a cipher type and key to demonstrate encryption. Encryption is always performed client-side even if the repository type (e.g. S3 or other object store) supports encryption.
It is important to use a long, random passphrase for the cipher key. A good way to generate one is to run: openssl rand -base64 48.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest repository encryption 
Config suppressed for testing
NOTE:
Encryption settings are placed in the [global] section, above, so the info command can read all stanzas. Without the stanza option the info command reads encryption settings only from the [global] section, so encryption settings configured per stanza require the stanza option to read an encrypted stanza.
Once the repository has been configured and the stanza created and checked, the repository encryption settings cannot be changed.
Create the Stanza
The stanza-create command must be run to initialize the stanza. It is recommended that the check command be run after stanza-create to ensure archiving and backups are properly configured.
pg-primary ⇒ Create the stanza and check the configuration 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
Check the Configuration
The check command validates that pgBackRest and the archive_command setting are configured correctly for archiving and backups for the specified stanza. It will attempt to check all repositories and databases that are configured for the host on which the command is run. It detects misconfigurations, particularly in archiving, that result in incomplete backups because required WAL segments did not reach the archive. The command can be run on the PostgreSQL or repository host. The command may also be run on the standby host, however, since pg_switch_xlog()/pg_switch_wal() cannot be performed on the standby, the command will only test the repository configuration.
Note that pg_create_restore_point('pgBackRest Archive Check') and pg_switch_xlog()/pg_switch_wal() are called to force PostgreSQL to archive a WAL segment.
pg-primary ⇒ Check the configuration 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check
Output suppressed for testing
Performance Tuning
pgBackRest has a number of performance options that are not enabled by default to maintain backward compatibility in the repository. However, when creating a new repository the following options are recommended. They can also be used on an existing repository with the caveat that older versions of pgBackRest will not be able to read the repository. This incompatibility depends on when the feature was introduced, as noted in the list below.
  • compress-type - determines the compression algorithm used by the backup and archive-push commands. The default is gz (Gzip) but zst (Zstandard) is recommended because it is much faster and provides compression similar to gz. zst has been supported by the compress-type option since v2.27. See Compress Type for more details.
  • repo-bundle - combines small files during backup to save space and improve the speed of both the backup and restore commands, especially on object stores such as S3. The repo-bundle option was introduced in v2.39. See File Bundling for more details.
  • repo-block - stores only the portions of files that have changed rather than the entire file during diff/incr backup. This saves space and increases the speed of the backup. The repo-block option was introduced in v2.46 but at least v2.52.1 is recommended. See Block Incremental for more details.
There are other performance options that are not enabled by default because they require additional configuration or because the default is safe (but not optimal). These options are available in all v2 versions of pgBackRest.
  • process-max - determines how many processes will be used for commands. The default is 1, which is almost never the appropriate value. Each command uses process-max differently so refer to each command's documentation for details on usage.
  • archive-async - archives WAL files to the repository in batch which greatly increases archiving speed. It is not enabled by default because it requires a spool path to be created. See Asynchronous Archiving for more details.
  • backup-standby - performs the backup on a standby rather than the primary to reduce load on the primary. It is not enabled by default because it requires additional configuration and the presence of one or more standby hosts. See Backup from a Standby for more details.
Perform a Backup
By default pgBackRest will wait for the next regularly scheduled checkpoint before starting a backup. Depending on the checkpoint_timeout and checkpoint_segments settings in PostgreSQL it may be quite some time before a checkpoint completes and the backup can begin. Generally, it is best to set start-fast=y so that the backup starts immediately. This forces a checkpoint, but since backups are usually run once a day an additional checkpoint should not have a noticeable impact on performance. However, on very busy clusters it may be best to pass --start-fast on the command-line as needed.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure backup fast start 
Config suppressed for testing
To perform a backup of the PostgreSQL cluster run pgBackRest with the backup command.
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo \
       --log-level-console=info backup
Output suppressed for testing
By default pgBackRest will attempt to perform an incremental backup. However, an incremental backup must be based on a full backup and since no full backup existed pgBackRest ran a full backup instead.
The type option can be used to specify a full or differential backup.
pg-primary ⇒ Differential backup of the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup
Output suppressed for testing
This time there was no warning because a full backup already existed. While incremental backups can be based on a full or differential backup, differential backups must be based on a full backup. A full backup can be performed by running the backup command with --type=full.
During an online backup pgBackRest waits for WAL segments that are required for backup consistency to be archived. This wait time is governed by the pgBackRest archive-timeout option which defaults to 60 seconds. If archiving an individual segment is known to take longer then this option should be increased.
Schedule a Backup
Backups can be scheduled with utilities such as cron.
In the following example, two cron jobs are configured to run; full backups are scheduled for 6:30 AM every Sunday with differential backups scheduled for 6:30 AM Monday through Saturday. If this crontab is installed for the first time mid-week, then pgBackRest will run a full backup the first time the differential job is executed, followed the next day by a differential backup. 
#m h   dom mon dow   command
30 06  *   *   0     pgbackrest --type=full --stanza=demo backup
30 06  *   *   1-6   pgbackrest --type=diff --stanza=demo backup
Once backups are scheduled it's important to configure retention so backups are expired on a regular schedule, see Retention.
Backup Information
Use the info command to get information about backups.
pg-primary ⇒ Get info for the demo cluster 
sudo -u postgres pgbackrest info
Output suppressed for testing
The info command operates on a single stanza or all stanzas. Text output is the default and gives a human-readable summary of backups for the stanza(s) requested. This format is subject to change with any release.
For machine-readable output use --output=json. The JSON output contains far more information than the text output and is kept stable unless a bug is found.
To speed up execution, limit the output to only progress information by specifying --detail-level=progress. Note that this skips all checks except for availability of the stanza.
Each stanza has a separate section and it is possible to limit output to a single stanza with the --stanza option. The stanza 'status' gives a brief indication of the stanza's health. If this is 'ok' then pgBackRest is functioning normally. If there are multiple repositories, then a status of 'mixed' indicates that the stanza is not in a healthy state on one or more of the repositories; in this case the state of the stanza will be detailed per repository. For cases in which an error on a repository occurred that is not one of the known error codes, then an error code of 'other' will be used and the full error details will be provided. The 'wal archive min/max' shows the minimum and maximum WAL currently stored in the archive and, in the case of multiple repositories, will be reported across all repositories unless the --repo option is set. Note that there may be gaps due to archive retention policies or other reasons.
The 'backup/expire running' and/or 'restore running' messages will appear beside the 'status' information if any of those commands are currently running on the host. Per-repo progress will be also reported in text output and a 'repo' array will be included in JSON output.
The backups are displayed oldest to newest. The oldest backup will always be a full backup (indicated by an F at the end of the label) but the newest backup can be full, differential (ends with D), or incremental (ends with I).
The 'timestamp start/stop' defines the time period when the backup ran. The 'timestamp stop' can be used to determine the backup to use when performing Point-In-Time Recovery. More information about Point-In-Time Recovery can be found in the Point-In-Time Recovery section.
The 'wal start/stop' defines the WAL range that is required to make the database consistent when restoring. The backup command will ensure that this WAL range is in the archive before completing.
The 'database size' is the full uncompressed size of the database while 'database backup size' is the amount of data in the database to actually back up (these will be the same for full backups).
The 'repo' indicates in which repository this backup resides. The 'backup set size' includes all the files from this backup and any referenced backups in the repository that are required to restore the database from this backup while 'backup size' includes only the files in this backup (these will also be the same for full backups). Repository sizes reflect compressed file sizes if compression is enabled in pgBackRest.
The 'backup reference total' summarizes the list of additional backups that are required to restore this backup. Use the --set option to display the complete reference list.
Restore a Backup
Backups can protect you from a number of disaster scenarios, the most common of which are hardware failure and data corruption. The easiest way to simulate data corruption is to remove an important PostgreSQL cluster file.
pg-primary ⇒ Stop the demo cluster and delete the pg_control file 
sudo systemctl stop postgresql-14.service
sudo -u postgres rm /var/lib/pgsql/14/data/global/pg_control
Starting the cluster without this important file will result in an error.
pg-primary ⇒ Attempt to start the corrupted demo cluster 
sudo systemctl start postgresql-14.service
sudo systemctl status postgresql-14.service
Output suppressed for testing
To restore a backup of the PostgreSQL cluster run pgBackRest with the restore command. The cluster needs to be stopped (in this case it is already stopped) and all files must be removed from the PostgreSQL data directory.
pg-primary ⇒ Remove old files from demo cluster 
sudo -u postgres find /var/lib/pgsql/14/data -mindepth 1 -delete
pg-primary ⇒ Restore the demo cluster and start PostgreSQL 
sudo -u postgres pgbackrest --stanza=demo restore
sudo systemctl start postgresql-14.service
This time the cluster started successfully since the restore replaced the missing pg_control file.
More information about the restore command can be found in the Restore section.
Monitoring
Monitoring is an important part of any production system. There are many tools available and pgBackRest can be monitored on any of them with a little work.
pgBackRest can output information about the repository in JSON format which includes a list of all backups for each stanza and WAL archive info.
In PostgreSQL
The PostgreSQL COPY command allows pgBackRest info to be loaded into a table. The following example wraps that logic in a function that can be used to perform real-time queries.
pg-primary ⇒ Load pgBackRest info function for PostgreSQL 
sudo -u postgres cat \
       /var/lib/pgsql/pgbackrest/doc/example/pgsql-pgbackrest-info.sql
Output suppressed for testing
sudo -u postgres psql -f \
       /var/lib/pgsql/pgbackrest/doc/example/pgsql-pgbackrest-info.sql
Now the monitor.pgbackrest_info() function can be used to determine the last successful backup time and archived WAL for a stanza.
pg-primary ⇒ Query last successful backup time and archived WAL 
sudo -u postgres cat \
       /var/lib/pgsql/pgbackrest/doc/example/pgsql-pgbackrest-query.sql
Output suppressed for testing
sudo -u postgres psql -f \
       /var/lib/pgsql/pgbackrest/doc/example/pgsql-pgbackrest-query.sql
Output suppressed for testing
Backup
When multiple repositories are configured, pgBackRest will backup to the highest priority repository (e.g. repo1) unless the --repo option is specified.
pgBackRest does not have a built-in scheduler so it's best to run it from cron or some other scheduling mechanism.
See Perform a Backup for more details and examples.
File Bundling
Bundling files together in the repository saves time during the backup and some space in the repository. This is especially pronounced when the repository is stored on an object store such as S3 or file systems with large block sizes. Per-file creation time on object stores is higher and very small files might cost as much to store as larger files.
The file bundling feature is enabled with the repo-bundle option.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-bundle 
Config suppressed for testing
A full backup without file bundling will have 1000+ files in the backup path, but with bundling the total number of files is greatly reduced. An additional benefit is that zero-length files are not stored (except in the manifest), whereas in a normal backup each zero-length file is stored individually.
pg-primary ⇒ Perform a full backup 
sudo -u postgres pgbackrest --stanza=demo --type=full backup
pg-primary ⇒ Check file total 
sudo -u postgres find /var/lib/pgbackrest/backup/demo/latest/ -type f | wc -l
Output suppressed for testing
The repo-bundle-size and repo-bundle-limit options can be used for tuning, though the defaults should be optimal in most cases.
While file bundling is generally more efficient, the downside is that it is more difficult to manually retrieve files from the repository. It may not be ideal for deduplicated storage since each full backup will arrange files in the bundles differently. Lastly, file bundles cannot be resumed, so be careful not to set repo-bundle-limit too high.
Block Incremental
Block incremental backups save space by only storing the parts of a file that have changed since the prior backup rather than storing the entire file.
The block incremental feature is enabled with the repo-block option and it works best when enabled for all backup types. File bundling must also be enabled.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-block 
Config suppressed for testing
Backup Annotations
Users can attach informative key/value pairs to the backup. This option may be used multiple times to attach multiple annotations.
pg-primary ⇒ Perform a full backup with annotations 
sudo -u postgres pgbackrest --stanza=demo --annotation=source="demo backup" \
       --annotation=key=value --type=full backup
Annotations are output by the info command text output when a backup is specified with --set and always appear in the JSON output.
pg-primary ⇒ Get info for the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --set=[Test Variable] info
Output suppressed for testing
Annotations included with the backup command can be added, modified, or removed afterwards using the annotate command.
pg-primary ⇒ Change backup annotations 
sudo -u postgres pgbackrest --stanza=demo --set=[Test Variable] \
       --annotation=key= --annotation=new_key=new_value annotate
sudo -u postgres pgbackrest --stanza=demo --set=[Test Variable] info
Output suppressed for testing
Retention
Generally it is best to retain as many backups as possible to provide a greater window for Point-in-Time Recovery, but practical concerns such as disk space must also be considered. Retention options remove older backups once they are no longer needed.
pgBackRest does full backup rotation based on the retention type which can be a count or a time period. When a count is specified, then expiration is not concerned with when the backups were created but with how many must be retained. Differential backups are count-based but will always be expired when the full backup they depend on is expired. Incremental backups are not expired by retention independently — they are always expired with their related full or differential backup. See sections Full Backup Retention and Differential Backup Retention for details and examples.
Archived WAL is retained by default for backups that have not expired, however, although not recommended, this schedule can be modified per repository with the retention-archive options. See section Archive Retention for details and examples.
The expire command is run automatically after each successful backup and can also be run by the user. When run by the user, expiration will occur as defined by the retention settings for each configured repository. If the --repo option is provided, expiration will occur only on the specified repository. Expiration can also be limited by the user to a specific backup set with the --set option and, unless the --repo option is specified, all repositories will be searched and any matching the set criteria will be expired. It should be noted that the archive retention schedule will be checked and performed any time the expire command is run.
Full Backup Retention
The repo1-retention-full-type determines how the option repo1-retention-full is interpreted; either as the count of full backups to be retained or how many days to retain full backups. New backups must be completed before expiration will occur — that means if repo1-retention-full-type=count and repo1-retention-full=2 then there will be three full backups stored before the oldest one is expired, or if repo1-retention-full-type=time and repo1-retention-full=20 then there must be one full backup that is at least 20 days old before expiration can occur.
repo1-retention-full=2 was set in Configure Retention but currently there is only one full backup so the next full backup to run will not expire any full backups.
pg-primary ⇒ Perform a full backup 
sudo -u postgres pgbackrest --stanza=demo --type=full \
       --log-level-console=detail backup
Output suppressed for testing
Archive is expired because WAL segments were generated before the oldest backup. These are not useful for recovery — only WAL segments generated after a backup can be used to recover that backup.
pg-primary ⇒ Perform a full backup 
sudo -u postgres pgbackrest --stanza=demo --type=full \
       --log-level-console=info backup
Output suppressed for testing
The [Test Variable] full backup is expired and archive retention is based on the [Test Variable] which is now the oldest full backup.
Differential Backup Retention
Set repo1-retention-diff to the number of differential backups required. Differentials only rely on the prior full backup so it is possible to create a rolling set of differentials for the last day or more. This allows quick restores to recent points-in-time but reduces overall space consumption.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-retention-diff 
Config suppressed for testing
Backup repo1-retention-diff=1 so two differentials will need to be performed before one is expired. An incremental backup is added to demonstrate incremental expiration, which in this case depends on the differential expiration.
pg-primary ⇒ Perform differential and incremental backups 
sudo -u postgres pgbackrest --stanza=demo --type=diff backup
sudo -u postgres pgbackrest --stanza=demo --type=incr backup
Now performing a differential backup will expire the previous differential and incremental backups leaving only one differential backup.
pg-primary ⇒ Perform a differential backup 
sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup
Output suppressed for testing
Archive Retention
Although pgBackRest automatically removes archived WAL segments when expiring backups (the default expires WAL for full backups based on the repo1-retention-full option), it may be useful to expire archive more aggressively to save disk space. Note that full backups are treated as differential backups for the purpose of differential archive retention.
Expiring archive will never remove WAL segments that are required to make a backup consistent. However, since Point-in-Time-Recovery (PITR) only works on a continuous WAL stream, care should be taken when aggressively expiring archive outside of the normal backup expiration process. To determine what will be expired without actually expiring anything, the dry-run option can be provided on the command line with the expire command.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-retention-diff 
Config suppressed for testing
pg-primary ⇒ Perform differential backup 
sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup
Output suppressed for testing
pg-primary ⇒ Expire archive 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=detail \
       --repo1-retention-archive-type=diff --repo1-retention-archive=1 expire
Output suppressed for testing
The [Test Variable] differential backup has archived WAL segments that must be retained to make the older backups consistent even though they cannot be played any further forward with PITR. WAL segments generated after [Test Variable] but before [Test Variable] are removed. WAL segments generated after the new backup [Test Variable] remain and can be used for PITR.
Since full backups are considered differential backups for the purpose of differential archive retention, if a full backup is now performed with the same settings, only the archive for that full backup is retained for PITR.
Restore
The restore command automatically defaults to selecting the latest backup from the first repository where backups exist (see Quick Start - Restore a Backup). The order in which the repositories are checked is dictated by the pgbackrest.conf (e.g. repo1 will be checked before repo2). To select from a specific repository, the --repo option can be passed (e.g. --repo=1). The --set option can be passed if a backup other than the latest is desired.
When PITR of --type=time or --type=lsn is specified, then the target time or target lsn must be specified with the --target option. If a backup is not specified via the --set option, then the configured repositories will be checked, in order, for a backup that contains the requested time or lsn. If no matching backup is found, the latest backup from the first repository containing backups will be used for --type=time while no backup will be selected for --type=lsn. For other types of PITR, e.g. xid, the --set option must be provided if the target is prior to the latest backup. See Point-in-Time Recovery for more details and examples.
Replication slots are not included per recommendation of PostgreSQL. See Backing Up The Data Directory in the PostgreSQL documentation for more information.
The following sections introduce additional restore command features.
File Ownership
If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried.
If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name.
Delta Option
Restore a Backup in Quick Start required the database cluster directory to be cleaned before the restore could be performed. The delta option allows pgBackRest to automatically determine which files in the database cluster directory can be preserved and which ones need to be restored from the backup — it also removes files not present in the backup manifest so it will dispose of divergent changes. This is accomplished by calculating a SHA-1 cryptographic hash for each file in the database cluster directory. If the SHA-1 hash does not match the hash stored in the backup then that file will be restored. This operation is very efficient when combined with the process-max option. Since the PostgreSQL server is shut down during the restore, a larger number of processes can be used than might be desirable during a backup when the PostgreSQL server is running.
pg-primary ⇒ Stop the demo cluster, perform delta restore 
sudo systemctl stop postgresql-14.service
sudo -u postgres pgbackrest --stanza=demo --delta \
       --log-level-console=detail restore
Output suppressed for testing
pg-primary ⇒ Restart PostgreSQL 
sudo systemctl start postgresql-14.service
Restore Selected Databases
There may be cases where it is desirable to selectively restore specific databases from a cluster backup. This could be done for performance reasons or to move selected databases to a machine that does not have enough space to restore the entire cluster backup.
To demonstrate this feature two databases are created: test1 and test2.
pg-primary ⇒ Create two test databases 
sudo -u postgres psql -c "create database test1;"
Output suppressed for testing
sudo -u postgres psql -c "create database test2;"
Output suppressed for testing
Each test database will be seeded with tables and data to demonstrate that recovery works with selective restore.
pg-primary ⇒ Create a test table in each database 
sudo -u postgres psql -c "create table test1_table (id int); \
       insert into test1_table (id) values (1);" test1
Output suppressed for testing
sudo -u postgres psql -c "create table test2_table (id int); \
       insert into test2_table (id) values (2);" test2
Output suppressed for testing
A fresh backup is run so pgBackRest is aware of the new databases.
pg-primary ⇒ Perform a backup 
sudo -u postgres pgbackrest --stanza=demo --type=incr backup
One of the main reasons to use selective restore is to save space. The size of the test1 database is shown here so it can be compared with the disk utilization after a selective restore.
pg-primary ⇒ Show space used by test1 database 
sudo -u postgres du -sh /var/lib/pgsql/14/data/base/[Test Variable]
Output suppressed for testing
If the database to restore is not known, use the info command set option to discover databases that are part of the backup set.
pg-primary ⇒ Show database list for backup 
sudo -u postgres pgbackrest --stanza=demo \
       --set=[Test Variable] info
Output suppressed for testing
Stop the cluster and restore only the test2 database. Built-in databases (template0, template1, and postgres) are always restored.
WARNING:
Recovery may error unless --type=immediate is specified. This is because after consistency is reached PostgreSQL will flag zeroed pages as errors even for a full-page write. For PostgreSQL ≥ 13 the ignore_invalid_pages setting may be used to ignore invalid pages. In this case it is important to check the logs after recovery to ensure that no invalid pages were reported in the selected databases.
pg-primary ⇒ Restore from last backup including only the test2 database 
sudo systemctl stop postgresql-14.service
sudo -u postgres pgbackrest --stanza=demo --delta \
       --db-include=test2 --type=immediate --target-action=promote restore
sudo systemctl start postgresql-14.service
Once recovery is complete the test2 database will contain all previously created tables and data.
pg-primary ⇒ Demonstrate that the test2 database was recovered 
sudo -u postgres psql -c "select * from test2_table;" test2
Output suppressed for testing
The test1 database, despite successful recovery, is not accessible. This is because the entire database was restored as sparse, zeroed files. PostgreSQL can successfully apply WAL on the zeroed files but the database as a whole will not be valid because key files contain no data. This is purposeful to prevent the database from being accidentally used when it might contain partial data that was applied during WAL replay.
pg-primary ⇒ Attempting to connect to the test1 database will produce an error 
sudo -u postgres psql -c "select * from test1_table;" test1
Output suppressed for testing
Since the test1 database is restored with sparse, zeroed files it will only require as much space as the amount of WAL that is written during recovery. While the amount of WAL generated during a backup and applied during recovery can be significant it will generally be a small fraction of the total database size, especially for large databases where this feature is most likely to be useful.
It is clear that the test1 database uses far less disk space during the selective restore than it would have if the entire database had been restored.
pg-primary ⇒ Show space used by test1 database after recovery 
sudo -u postgres du -sh /var/lib/pgsql/14/data/base/[Test Variable]
Output suppressed for testing
At this point the only action that can be taken on the invalid test1 database is drop database. pgBackRest does not automatically drop the database since this cannot be done until recovery is complete and the cluster is accessible.
pg-primary ⇒ Drop the test1 database 
sudo -u postgres psql -c "drop database test1;"
Output suppressed for testing
Now that the invalid test1 database has been dropped only the test2 and built-in databases remain.
pg-primary ⇒ List remaining databases 
sudo -u postgres psql -c "select oid, datname from pg_database order by oid;"
Output suppressed for testing
Point-in-Time Recovery
Restore a Backup in Quick Start performed default recovery, which is to play all the way to the end of the WAL stream. In the case of a hardware failure this is usually the best choice but for data corruption scenarios (whether machine or human in origin) Point-in-Time Recovery (PITR) is often more appropriate.
Point-in-Time Recovery (PITR) allows the WAL to be played from a backup to a specified lsn, time, transaction id, or recovery point. For common recovery scenarios time-based recovery is arguably the most useful. A typical recovery scenario is to restore a table that was accidentally dropped or data that was accidentally deleted. Recovering a dropped table is more dramatic so that's the example given here but deleted data would be recovered in exactly the same way.
pg-primary ⇒ Create a table with very important data 
sudo -u postgres psql -c "begin; \
       create table important_table (message text); \
       insert into important_table values ('Important Data'); \
       commit; \
       select * from important_table;"
Output suppressed for testing
It is important to represent the time as reckoned by PostgreSQL and to include timezone offsets. This reduces the possibility of unintended timezone conversions and an unexpected recovery result.
pg-primary ⇒ Get the time from PostgreSQL 
sudo -u postgres psql -Atc "select current_timestamp"
Output suppressed for testing
Now that the time has been recorded the table is dropped. In practice finding the exact time that the table was dropped is a lot harder than in this example. It may not be possible to find the exact time, but some forensic work should be able to get you close.
pg-primary ⇒ Drop the important table 
sudo -u postgres psql -c "begin; \
       drop table important_table; \
       commit; \
       select * from important_table;"
Output suppressed for testing
If the wrong backup is selected for restore then recovery to the required time target will fail. To demonstrate this a new incremental backup is performed where important_table does not exist.
pg-primary ⇒ Perform an incremental backup 
sudo -u postgres pgbackrest --stanza=demo --type=incr backup
sudo -u postgres pgbackrest info
Output suppressed for testing
It will not be possible to recover the lost table from this backup since PostgreSQL can only play forward, not backward.
pg-primary ⇒ Attempt recovery from an incorrect backup 
sudo systemctl stop postgresql-14.service
sudo -u postgres pgbackrest --stanza=demo --delta \
       --set=[Test Variable] --target-timeline=current \
       --type=time "--target=[Test Variable]" --target-action=promote restore
sudo systemctl start postgresql-14.service
sudo -u postgres cat /var/lib/pgsql/14/data/log/postgresql.log
Output suppressed for testing
A reliable method is to allow pgBackRest to automatically select a backup capable of recovery to the time target, i.e. a backup that ended before the specified time.
NOTE:
pgBackRest cannot automatically select a backup when the restore type is xid or name.
pg-primary ⇒ Restore the demo cluster to [Test Variable] 
sudo -u postgres pgbackrest --stanza=demo --delta \
       --type=time "--target=[Test Variable]" \
       --target-action=promote restore
sudo -u postgres cat /var/lib/pgsql/14/data/postgresql.auto.conf
Output suppressed for testing
pgBackRest has generated the recovery settings in postgresql.auto.conf so PostgreSQL can be started immediately. %f is how PostgreSQL specifies the WAL segment it needs and %p is the location where it should be copied. Once PostgreSQL has finished recovery the table will exist again and can be queried.
pg-primary ⇒ Start PostgreSQL and check that the important table exists 
sudo systemctl start postgresql-14.service
sudo -u postgres psql -c "select * from important_table"
Output suppressed for testing
The PostgreSQL log also contains valuable information. It will indicate the time and transaction where the recovery stopped and also give the time of the last transaction to be applied.
pg-primary ⇒ Examine the PostgreSQL log output 
sudo -u postgres cat /var/lib/pgsql/14/data/log/postgresql.log
Output suppressed for testing
Delete a Stanza
The stanza-delete command removes data in the repository associated with a stanza.
WARNING:
Use this command with caution — it will permanently remove all backups and archives from the pgBackRest repository for the specified stanza.
To delete a stanza:
  • Shut down the PostgreSQL cluster associated with the stanza (or use --force to override).
  • Run the stop command on the host where the stanza-delete command will be run.
  • Run the stanza-delete command.
Once the command successfully completes, it is the responsibility of the user to remove the stanza from all pgBackRest configuration files and/or environment variables.
A stanza may only be deleted from one repository at a time. To delete the stanza from multiple repositories, repeat the stanza-delete command for each repository while specifying the --repo option.
pg-primary ⇒ Stop PostgreSQL cluster to be removed 
sudo systemctl stop postgresql-14.service
pg-primary ⇒ Stop pgBackRest for the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stop
Output suppressed for testing
pg-primary ⇒ Delete the stanza from one repository 
sudo -u postgres pgbackrest --stanza=demo --repo=1 \
       --log-level-console=info stanza-delete
Output suppressed for testing
Multiple Repositories
Multiple repositories may be configured as demonstrated in S3 Support. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy.
Some commands, e.g. stanza-create/stanza-upgrade, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified.
Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later.
The archive-push command will always push WAL to the archive in all configured repositories. When a repository cannot be reached, WAL will still be pushed to other repositories. However, for this to work effectively, archive-async=y must be enabled; otherwise, the other repositories can only get one WAL segment ahead of the unreachable repository. Also, note that if WAL cannot be pushed to any repository, then PostgreSQL will not remove it from the pg_wal directory, which may cause the volume to run out of space.
Backups need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient.
Azure Support
pgBackRest supports locating repositories in Azure-compatible object stores. The container used to store the repository must be created in advance — pgBackRest will not do it automatically. The repository can be located in the container root (/) but it's usually best to place it in a subpath so object store logs or other data can also be stored in the container without conflicts.
WARNING:
Do not enable hierarchical namespace as this will cause errors during expire.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure Azure 
Config suppressed for testing
Shared access signatures may be used by setting the repo2-azure-key-type option to sas and the repo2-azure-key option to the shared access signature token.
Commands are run exactly as if the repository were stored on a local disk.
pg-primary ⇒ Create the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
File creation time in Azure is relatively slow so backup/restore performance is improved by enabling file bundling.
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --repo=2 \
       --log-level-console=info backup
Output suppressed for testing
S3 Support
pgBackRest supports locating repositories in S3-compatible object stores. The bucket used to store the repository must be created in advance — pgBackRest will not do it automatically. The repository can be located in the bucket root (/) but it's usually best to place it in a subpath so object store logs or other data can also be stored in the bucket without conflicts.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure S3 
Config suppressed for testing
NOTE:
The region and endpoint will need to be configured to where the bucket is located. The values given here are for the us-east-1 region.
A role should be created to run pgBackRest and the bucket permissions should be set as restrictively as possible. If the role is associated with an instance in AWS then pgBackRest will automatically retrieve temporary credentials when repo3-s3-key-type=auto, which means that keys do not need to be explicitly set in /etc/pgbackrest/pgbackrest.conf.
This sample Amazon S3 policy will restrict all reads and writes to the bucket and repository path. 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket"
            ],
            "Condition": {
                "StringEquals": {
                    "s3:prefix": [
                        "",
                        "demo-repo"
                    ],
                    "s3:delimiter": [
                        "/"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "demo-repo/*"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket/demo-repo/*"
            ]
        }
    ]
}
Commands are run exactly as if the repository were stored on a local disk.
pg-primary ⇒ Create the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
File creation time in S3 is relatively slow so backup/restore performance is improved by enabling file bundling.
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --repo=3 \
       --log-level-console=info backup
Output suppressed for testing
SFTP Support
pgBackRest supports locating repositories on SFTP hosts. SFTP file transfer is relatively slow so commands benefit by increasing process-max to parallelize file transfer.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure SFTP 
Config suppressed for testing
When utilizing SFTP, if libssh2 is compiled against OpenSSH then repo4-sftp-public-key-file is optional.
pg-primary ⇒ Generate SSH keypair for SFTP backup 
sudo -u postgres mkdir -m 750 -p /var/lib/pgsql/.ssh
sudo -u postgres ssh-keygen -f /var/lib/pgsql/.ssh/id_rsa_sftp \
       -t rsa -b 4096 -N "" -m PEM
sftp-server ⇒ Copy pg-primary SFTP backup public key to sftp-server 
sudo -u pgbackrest mkdir -m 750 -p /home/pgbackrest/.ssh
(sudo ssh root@pg-primary cat /var/lib/pgsql/.ssh/id_rsa_sftp.pub) | \
       sudo -u pgbackrest tee -a /home/pgbackrest/.ssh/authorized_keys
Commands are run exactly as if the repository were stored on a local disk.
pg-primary ⇒ Add sftp-server fingerprint to known_hosts file since repo4-sftp-host-key-check-type defaults to strict 
ssh-keyscan -H sftp-server >> /var/lib/pgsql/.ssh/known_hosts 2>/dev/null
pg-primary ⇒ Create the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --repo=4 \
       --log-level-console=info backup
Output suppressed for testing
GCS Support
pgBackRest supports locating repositories in GCS-compatible object stores. The bucket used to store the repository must be created in advance — pgBackRest will not do it automatically. The repository can be located in the bucket root (/) but it's usually best to place it in a subpath so object store logs or other data can also be stored in the bucket without conflicts.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure GCS 
Config suppressed for testing
When running in GCE set repo5-gcs-key-type=auto to automatically authenticate using the instance service account.
Commands are run exactly as if the repository were stored on a local disk.
File creation time in GCS is relatively slow so backup/restore performance is improved by enabling file bundling.
Target Time for Repository
The target time defines the time that commands use to read a repository on versioned storage. This allows the command to read the repository as it was at a point-in-time in order to recover data that has been deleted or corrupted by user accident or malware.
Versioned storage is supported by S3, GCS, and Azure but is generally not enabled by default. In addition to enabling versioning, it may be useful to enable object locking for S3 and soft delete for GCS or Azure.
When the repo-target-time option is specified then the repo option must also be provided. It is likely that not all repository types will support versioning and in general it makes sense to target a single repository for recovery.
Note that comparisons to the storage timestamp are <= the timestamp provided and milliseconds are truncated from the timestamp when provided.
To demonstrate this feature the demo stanza in the S3 repo is deleted.
pg-primary ⇒ Delete stanza in S3 repository 
sudo systemctl stop postgresql-14.service
sudo -u postgres pgbackrest --stanza=demo stop
Output suppressed for testing
sudo -u postgres pgbackrest --stanza=demo --repo=3 stanza-delete
Output suppressed for testing
Once the stanza is deleted the info command will show the repository in an error state.
pg-primary ⇒ Error on info 
sudo -u postgres pgbackrest --stanza=demo --repo=3 info
Output suppressed for testing
However, since the storage is versioned, it is possible to look at the repository at a time before the stanza was deleted. Finding the target time can be tricky depending on the situation, but in this case the time when the stanza was deleted can be determined by checking when backup.info was deleted.
pg-primary ⇒ List versions of backup.info in the bucket 
key=demo-repo/backup/demo/backup.info; \
       aws s3api list-object-versions --bucket demo-bucket \
       --prefix $key --output table \
       --query "sort_by([Versions[?Key=='$key'].{Action:'PUT', \
       Modified:LastModified,Object:Key}, \
       DeleteMarkers[?Key=='$key'].{Action:'DELETE', \
       Modified:LastModified,Object:Key}][],&Modified)"
Output suppressed for testing
Now the info command can be run with a target time that will show the repository before it was deleted.
pg-primary ⇒ Info with target time 
sudo -u postgres pgbackrest --stanza=demo --repo=3 \
       --repo-target-time="[Test Variable]" info
Output suppressed for testing
If the required backup is shown by the info command then it can be restored using the same target time.
pg-primary ⇒ Restore with target time 
sudo -u postgres pgbackrest --stanza=demo --repo=3 --delta \
       --repo-target-time="[Test Variable]" --log-level-console=info restore
Output suppressed for testing
sudo systemctl start postgresql-14.service
Dedicated Repository Host
The configuration described in Quickstart is suitable for simple installations but for enterprise configurations it is more typical to have a dedicated repository host where the backups and WAL archive files are stored. This separates the backups and WAL archive from the database server so database host failures have less impact. It is still a good idea to employ traditional backup software to backup the repository host.
On PostgreSQL hosts, pg1-path is required to be the path of the local PostgreSQL cluster and no pg1-host should be configured. When configuring a repository host, the pgbackrest configuration file must have the pg-host option configured to connect to the primary and standby (if any) hosts. The repository host has the only pgbackrest configuration that should be aware of more than one PostgreSQL host. Order does not matter, e.g. pg1-path/pg1-host, pg2-path/pg2-host can be primary or standby.
Installation
A new host named repository is created to store the cluster backups.
NOTE:
The pgBackRest version installed on the repository host must exactly match the version installed on the PostgreSQL host.
The pgbackrest user is created to own the pgBackRest repository. Any user can own the repository but it is best not to use postgres (if it exists) to avoid confusion.
NOTE:
When pgBackRest is installed from a package, a logrotate configuration such as /etc/logrotate.d/pgbackrest may be provided that rotates the logs as a specific user via the su directive (e.g. su postgres postgres). Since the files in /var/log/pgbackrest are owned by the user that runs pgBackRest (here pgbackrest), the su directive must be updated to match this user or logrotate will fail with a permission error.
repository ⇒ Create pgbackrest user 
sudo groupadd pgbackrest
sudo adduser -gpgbackrest -n pgbackrest
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
RHEL packages for pgBackRest are available at yum.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
repository ⇒ Install dependencies 
sudo yum install postgresql-libs libssh2
repository ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
repository ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown pgbackrest:pgbackrest /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown pgbackrest:pgbackrest /etc/pgbackrest/pgbackrest.conf
repository ⇒ Create the pgBackRest repository 
sudo mkdir -p /var/lib/pgbackrest
sudo chmod 750 /var/lib/pgbackrest
sudo chown pgbackrest:pgbackrest /var/lib/pgbackrest
Configuration
pgBackRest can use TLS with client certificates to enable communication between the hosts. It is also possible to use SSH, see Setup SSH.
pgBackRest expects client/server certificates to be generated in the same way as PostgreSQL. See Secure TCP/IP Connections with TLS for detailed instructions on generating certificates.
The repository host must be configured with the pg-primary host/user and database path. The primary will be configured as pg1 to allow a standby to be added later.
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pg1-host/pg1-host-user and pg1-path 
Config suppressed for testing
The database host must be configured with the repository host/user. The default for the repo1-host-user option is pgbackrest. If the postgres user does restores on the repository host it is best not to also allow the postgres user to perform backups. However, the postgres user can read the repository directly if it is in the same group as the pgbackrest user.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-host/repo1-host-user 
Config suppressed for testing
PostgreSQL configuration may be found in the Configure Archiving section.
Commands are run the same as on a single host configuration except that some commands such as backup and expire are run from the repository host instead of the database host.
Setup TLS Server
The pgBackRest TLS server must be configured and started on each host.
repository ⇒ Setup pgBackRest Server 
sudo cat /etc/systemd/system/pgbackrest.service
Output suppressed for testing
sudo systemctl enable pgbackrest
sudo systemctl start pgbackrest
pg-primary ⇒ Setup pgBackRest Server 
sudo cat /etc/systemd/system/pgbackrest.service
Output suppressed for testing
sudo systemctl enable pgbackrest
sudo systemctl start pgbackrest
Create and Check Stanza
Create the stanza in the new repository.
repository ⇒ Create the stanza 
sudo -u pgbackrest pgbackrest --stanza=demo stanza-create
Output suppressed for testing
Check that the configuration is correct on both the database and repository hosts. More information about the check command can be found in Check the Configuration.
pg-primary ⇒ Check the configuration 
sudo -u postgres pgbackrest --stanza=demo check
Output suppressed for testing
repository ⇒ Check the configuration 
sudo -u pgbackrest pgbackrest --stanza=demo check
Output suppressed for testing
Perform a Backup
To perform a backup of the PostgreSQL cluster run pgBackRest with the backup command on the repository host.
repository ⇒ Backup the demo cluster 
sudo -u pgbackrest pgbackrest --stanza=demo backup
Output suppressed for testing
Since a new repository was created on the repository host the warning about the incremental backup changing to a full backup was emitted.
Restore a Backup
To perform a restore of the PostgreSQL cluster run pgBackRest with the restore command on the database host.
pg-primary ⇒ Stop the demo cluster, restore, and restart PostgreSQL 
sudo systemctl stop postgresql-14.service
sudo -u postgres pgbackrest --stanza=demo --delta restore
sudo systemctl start postgresql-14.service
Parallel Backup / Restore
pgBackRest offers parallel processing to improve performance of compression and transfer. The number of processes to be used for this feature is set using the --process-max option.
It is usually best not to use more than 25% of available CPUs for the backup command. Backups don't have to run that fast as long as they are performed regularly and the backup process should not impact database performance, if at all possible.
The restore command can and should use all available CPUs because during a restore the PostgreSQL cluster is shut down and there is generally no other important work being done on the host. If the host contains multiple clusters then that should be considered when setting restore parallelism.
repository ⇒ Perform a backup with single process 
sudo -u pgbackrest pgbackrest --stanza=demo --type=full backup
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest to use multiple backup processes 
Config suppressed for testing
repository ⇒ Perform a backup with multiple processes 
sudo -u pgbackrest pgbackrest --stanza=demo --type=full backup
repository ⇒ Get backup info for the demo cluster 
sudo -u pgbackrest pgbackrest info
Output suppressed for testing
The performance of the last backup should be improved by using multiple processes. For very small backups the difference may not be very apparent, but as the size of the database increases so will time savings.
Starting and Stopping
If a standby is promoted for testing, or a test cluster is restored from a production backup, then it is a good idea to prevent those clusters from writing to pgBackRest repositories. This can be accomplished with the stop command.
The commands that write and are blocked by stop are: archive-push, backup, expire, stanza-create, and stanza-upgrade. Note that stanza-delete is an exception to this rule (see Delete a Stanza for more details).
pg-primary ⇒ Stop pgBackRest write commands 
sudo -u postgres pgbackrest stop
New pgBackRest write commands will no longer run.
repository ⇒ Attempt a backup 
sudo -u pgbackrest pgbackrest --stanza=demo backup
Output suppressed for testing
Specify the --force option to terminate any pgBackRest write commands that are currently running. This includes asynchronous archive-get (though it will run again if PostgreSQL requires it). If pgBackRest is already stopped then stopping again will generate a warning.
pg-primary ⇒ Stop the pgBackRest services again 
sudo -u postgres pgbackrest stop
Output suppressed for testing
Start pgBackRest write commands again with the start command. Write commands that were in progress before the stop will not automatically start again, but they are now allowed to start.
pg-primary ⇒ Start pgBackRest write commands 
sudo -u postgres pgbackrest start
It is also possible to stop pgBackRest for a single stanza.
pg-primary ⇒ Stop pgBackRest write commands for the demo stanza 
sudo -u postgres pgbackrest --stanza=demo stop
New pgBackRest write commands for the specified stanza will no longer run.
repository ⇒ Attempt a backup 
sudo -u pgbackrest pgbackrest --stanza=demo backup
Output suppressed for testing
The stanza must also be specified when starting pgBackRest write commands for a single stanza.
pg-primary ⇒ Start pgBackRest write commands for the demo stanza 
sudo -u postgres pgbackrest --stanza=demo start
Replication
Replication allows multiple copies of a PostgreSQL cluster (called standbys) to be created from a single primary. The standbys are useful for balancing reads and to provide redundancy in case the primary host fails.
Installation
A new host named pg-standby is created to run the standby.
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
RHEL packages for pgBackRest are available at yum.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
pg-standby ⇒ Install dependencies 
sudo yum install postgresql-libs libssh2
pg-standby ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
pg-standby ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown postgres:postgres /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
Hot Standby
A hot standby performs replication using the WAL archive and allows read-only queries.
pgBackRest configuration is very similar to pg-primary except that the standby recovery type will be used to keep the cluster in recovery mode when the end of the WAL stream has been reached.
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest on the standby 
Config suppressed for testing
pg-standby ⇒ Setup pgBackRest Server 
sudo cat /etc/systemd/system/pgbackrest.service
Output suppressed for testing
sudo systemctl enable pgbackrest
sudo systemctl start pgbackrest
Create the path where PostgreSQL will be restored.
pg-standby ⇒ Create PostgreSQL path 
sudo -u postgres mkdir -p -m 700 /var/lib/pgsql/14/data
Now the standby can be created with the restore command.
IMPORTANT:
If the cluster is intended to be promoted without becoming the new primary (e.g. for reporting or testing), use --archive-mode=off or set archive_mode=off in postgresql.conf to disable archiving. If archiving is not disabled then the repository may be polluted with WAL that can make restores more difficult.
pg-standby ⇒ Restore the demo standby cluster 
sudo -u postgres pgbackrest --stanza=demo --type=standby restore
sudo -u postgres cat /var/lib/pgsql/14/data/postgresql.auto.conf
Output suppressed for testing
The configuration is in case the standby is promoted to a primary.
pg-standby:/var/lib/pgsql/14/data/postgresql.conf ⇒ Configure PostgreSQL 
Config suppressed for testing
pg-standby ⇒ Start PostgreSQL 
sudo systemctl start postgresql-14.service
The PostgreSQL log gives valuable information about the recovery. Note especially that the cluster has entered standby mode and is ready to accept read-only connections.
pg-standby ⇒ Examine the PostgreSQL log output for log messages indicating success 
sudo -u postgres cat /var/lib/pgsql/14/data/log/postgresql.log
Output suppressed for testing
An easy way to test that replication is properly configured is to create a table on pg-primary.
pg-primary ⇒ Create a new table on the primary 
sudo -u postgres psql -c " \
       begin; \
       create table replicated_table (message text); \
       insert into replicated_table values ('Important Data'); \
       commit; \
       select * from replicated_table";
Output suppressed for testing
And then query the same table on pg-standby.
pg-standby ⇒ Query new table on the standby 
sudo -u postgres psql -c "select * from replicated_table;"
Output suppressed for testing
So, what went wrong? Since PostgreSQL is pulling WAL segments from the archive to perform replication, changes won't be seen on the standby until the WAL segment that contains those changes is pushed from pg-primary.
This can be done manually by calling pg_switch_wal() which pushes the current WAL segment to the archive (a new WAL segment is created to contain further changes).
pg-primary ⇒ Call pg_switch_wal() 
sudo -u postgres psql -c "select *, current_timestamp from pg_switch_wal()";
Output suppressed for testing
Now after a short delay the table will appear on pg-standby.
pg-standby ⇒ Now the new table exists on the standby (may require a few retries) 
sudo -u postgres psql -c " \
       select *, current_timestamp from replicated_table"
Output suppressed for testing
Check the standby configuration for access to the repository.
pg-standby ⇒ Check the configuration 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check
Output suppressed for testing
Streaming Replication
Instead of relying solely on the WAL archive, streaming replication makes a direct connection to the primary and applies changes as soon as they are made on the primary. This results in much less lag between the primary and standby.
Streaming replication requires a user with the replication privilege.
pg-primary ⇒ Create replication user 
sudo -u postgres psql -c " \
       create user replicator password 'jw8s0F4' replication";
Output suppressed for testing
The pg_hba.conf file must be updated to allow the standby to connect as the replication user. Be sure to replace the IP address below with the actual IP address of your pg-standby. A reload will be required after modifying the pg_hba.conf file.
pg-primary ⇒ Create pg_hba.conf entry for replication user 
sudo -u postgres sh -c 'echo \
       "host    replication     replicator      {[host-pg2-ip]}/32           md5" \
       >> /var/lib/pgsql/14/data/pg_hba.conf'
sudo systemctl reload postgresql-14.service
The standby needs to know how to contact the primary so the primary_conninfo setting will be configured in pgBackRest.
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Set primary_conninfo 
Config suppressed for testing
It is possible to configure a password in the primary_conninfo setting but using a .pgpass file is more flexible and secure.
pg-standby ⇒ Configure the replication password in the .pgpass file. 
sudo -u postgres sh -c 'echo \
       "{[host-pg1-ip]}:*:replication:replicator:jw8s0F4" \
       >> /var/lib/pgsql/.pgpass'
sudo -u postgres chmod 600 /var/lib/pgsql/.pgpass
Now the standby can be created with the restore command.
pg-standby ⇒ Stop PostgreSQL and restore the demo standby cluster 
sudo systemctl stop postgresql-14.service
sudo -u postgres pgbackrest --stanza=demo --delta --type=standby restore
sudo -u postgres cat /var/lib/pgsql/14/data/postgresql.auto.conf
Output suppressed for testing
NOTE:
The primary_conninfo setting has been written into the postgresql.auto.conf file because it was configured as a recovery-option in pgbackrest.conf. The --type=preserve option can be used with the restore to leave the existing postgresql.auto.conf file in place if that behavior is preferred.
pg-standby ⇒ Start PostgreSQL 
sudo systemctl start postgresql-14.service
The PostgreSQL log will confirm that streaming replication has started.
pg-standby ⇒ Examine the PostgreSQL log output for log messages indicating success 
sudo -u postgres cat /var/lib/pgsql/14/data/log/postgresql.log
Output suppressed for testing
Now when a table is created on pg-primary it will appear on pg-standby quickly and without the need to call pg_switch_wal().
pg-primary ⇒ Create a new table on the primary 
sudo -u postgres psql -c " \
       begin; \
       create table stream_table (message text); \
       insert into stream_table values ('Important Data'); \
       commit; \
       select *, current_timestamp from stream_table";
Output suppressed for testing
pg-standby ⇒ Query table on the standby 
sudo -u postgres psql -c " \
       select *, current_timestamp from stream_table"
Output suppressed for testing
Multiple Stanzas
pgBackRest supports multiple stanzas. The most common usage is sharing a repository host among multiple stanzas.
Installation
A new host named pg-alt is created to run the new primary.
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
RHEL packages for pgBackRest are available at yum.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
pg-alt ⇒ Install dependencies 
sudo yum install postgresql-libs libssh2
pg-alt ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
pg-alt ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown postgres:postgres /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
Configuration
pgBackRest configuration is nearly identical to pg-primary except that the demo-alt stanza will be used so backups and archive will be stored in a separate location.
pg-alt:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest on the new primary 
Config suppressed for testing
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pg1-host/pg1-host-user and pg1-path 
Config suppressed for testing
pg-alt ⇒ Setup pgBackRest Server 
sudo cat /etc/systemd/system/pgbackrest.service
Output suppressed for testing
sudo systemctl enable pgbackrest
sudo systemctl start pgbackrest
Setup Demo Cluster
pg-alt ⇒ Create the demo cluster 
sudo -u postgres /usr/pgsql-14/bin/initdb \
       -D /var/lib/pgsql/14/data -k -A peer
pg-alt:/var/lib/pgsql/14/data/postgresql.conf ⇒ Configure PostgreSQL settings 
Config suppressed for testing
pg-alt ⇒ Start the demo cluster 
sudo systemctl restart postgresql-14.service
Create the Stanza and Check Configuration
The stanza-create command must be run to initialize the stanza. It is recommended that the check command be run after stanza-create to ensure archiving and backups are properly configured.
pg-alt ⇒ Create the stanza and check the configuration 
sudo -u postgres pgbackrest --stanza=demo-alt --log-level-console=info stanza-create
Output suppressed for testing
sudo -u postgres pgbackrest --log-level-console=info check
Output suppressed for testing
If the check command is run from the repository host then all stanzas will be checked.
repository ⇒ Check the configuration for all stanzas 
sudo -u pgbackrest pgbackrest --log-level-console=info check
Output suppressed for testing
Asynchronous Archiving
Asynchronous archiving is enabled with the archive-async option. This option enables asynchronous operation for both the archive-push and archive-get commands.
A spool path is required. The commands will store transient data here but each command works quite a bit differently so spool path usage is described in detail in each section.
pg-primary ⇒ Create the spool directory 
sudo mkdir -p -m 750 /var/spool/pgbackrest
sudo chown postgres:postgres /var/spool/pgbackrest
pg-standby ⇒ Create the spool directory 
sudo mkdir -p -m 750 /var/spool/pgbackrest
sudo chown postgres:postgres /var/spool/pgbackrest
The spool path must be configured and asynchronous archiving enabled. Asynchronous archiving automatically confers some benefit by reducing the number of connections made to remote storage, but setting process-max can drastically improve performance by parallelizing operations. Be sure not to set process-max so high that it affects normal database operations.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the spool path and asynchronous archiving 
Config suppressed for testing
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the spool path and asynchronous archiving 
Config suppressed for testing
NOTE:
process-max is configured using command sections so that the option is not used by backup and restore. This also allows different values for archive-push and archive-get.
For demonstration purposes streaming replication will be broken to force PostgreSQL to get WAL using the restore_command.
pg-primary ⇒ Break streaming replication by changing the replication password 
sudo -u postgres psql -c "alter user replicator password 'bogus'"
Output suppressed for testing
pg-standby ⇒ Restart standby to break connection 
sudo systemctl restart postgresql-14.service
Archive Push
The asynchronous archive-push command offloads WAL archiving to a separate process (or processes) to improve throughput. It works by looking ahead to see which WAL segments are ready to be archived beyond the request that PostgreSQL is currently making via the archive_command. WAL segments are transferred to the archive directly from the pg_xlog/pg_wal directory and success is only returned by the archive_command when the WAL segment has been safely stored in the archive.
The spool path holds the current status of WAL archiving. Status files written into the spool directory are typically zero length and should consume a minimal amount of space (a few MB at most) and very little IO. All the information in this directory can be recreated so it is not necessary to preserve the spool directory if the cluster is moved to new hardware.
IMPORTANT:
In the original implementation of asynchronous archiving, WAL segments were copied to the spool directory before compression and transfer. The new implementation copies WAL directly from the pg_xlog directory. If asynchronous archiving was utilized in v1.12 or prior, read the v1.13 release notes carefully before upgrading.
The [stanza]-archive-push-async.log file can be used to monitor the activity of the asynchronous process. A good way to test this is to quickly push a number of WAL segments.
pg-primary ⇒ Test parallel asynchronous archiving 
sudo -u postgres psql -c " \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal();"
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check
Output suppressed for testing
Now the log file will contain parallel, asynchronous activity.
pg-primary ⇒ Check results in the log 
sudo -u postgres cat /var/log/pgbackrest/demo-archive-push-async.log
Output suppressed for testing
Archive Get
The asynchronous archive-get command maintains a local queue of WAL to improve throughput. If a WAL segment is not found in the queue it is fetched from the repository along with enough consecutive WAL to fill the queue. The maximum size of the queue is defined by archive-get-queue-max. Whenever the queue is less than half full more WAL will be fetched to fill it.
Asynchronous operation is most useful in environments that generate a lot of WAL or have a high latency connection to the repository storage (i.e., S3 or other object stores). In the case of a high latency connection it may be a good idea to increase process-max.
The [stanza]-archive-get-async.log file can be used to monitor the activity of the asynchronous process.
pg-standby ⇒ Check results in the log 
sudo -u postgres cat /var/log/pgbackrest/demo-archive-get-async.log
Output suppressed for testing
pg-primary ⇒ Fix streaming replication by changing the replication password 
sudo -u postgres psql -c "alter user replicator password 'jw8s0F4'"
Output suppressed for testing
Backup from a Standby
pgBackRest can perform backups on a standby instead of the primary. Standby backups require the pg-standby host to be configured and the backup-standby option enabled. If more than one standby is configured then the first running standby found will be used for the backup.
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pg2-host/pg2-host-user and pg2-path 
Config suppressed for testing
Both the primary and standby databases are required to perform the backup, though the vast majority of the files will be copied from the standby to reduce load on the primary. The database hosts can be configured in any order. pgBackRest will automatically determine which is the primary and which is the standby.
repository ⇒ Backup the demo cluster from pg2 
sudo -u pgbackrest pgbackrest --stanza=demo --log-level-console=detail backup
Output suppressed for testing
This incremental backup shows that most of the files are copied from the pg-standby host and only a few are copied from the pg-primary host.
pgBackRest creates a standby backup that is identical to a backup performed on the primary. It does this by starting/stopping the backup on the pg-primary host, copying only files that are replicated from the pg-standby host, then copying the remaining few files from the pg-primary host. This means that logs and statistics from the primary database will be included in the backup.
Upgrading PostgreSQL
Immediately after upgrading PostgreSQL to a newer major version, the pg-path for all pgBackRest configurations must be set to the new database location and the stanza-upgrade command run. If there is more than one repository configured on the host, the stanza will be upgraded on each. If the database is offline use the --no-online option.
The following instructions are not meant to be a comprehensive guide for upgrading PostgreSQL, rather they outline the general process for upgrading a primary and standby with the intent of demonstrating the steps required to reconfigure pgBackRest. It is recommended that a backup be taken prior to upgrading.
pg-primary ⇒ Stop old cluster 
sudo systemctl stop postgresql-14.service
Stop the old cluster on the standby since it will be restored from the newly upgraded cluster.
pg-standby ⇒ Stop old cluster 
sudo systemctl stop postgresql-14.service
Create the new cluster and perform upgrade.
pg-primary ⇒ Create new cluster and perform the upgrade 
sudo -u postgres /usr/pgsql-15/bin/initdb \
       -D /var/lib/pgsql/15/data -k -A peer
sudo -u postgres sh -c 'cd /var/lib/pgsql && \
       /usr/pgsql-15/bin/pg_upgrade \
       --old-bindir=/usr/pgsql-14/bin \
       --new-bindir=/usr/pgsql-15/bin \
       --old-datadir=/var/lib/pgsql/14/data \
       --new-datadir=/var/lib/pgsql/15/data \
       --old-options=" -c config_file=/var/lib/pgsql/14/data/postgresql.conf" \
       --new-options=" -c config_file=/var/lib/pgsql/15/data/postgresql.conf"'
Output suppressed for testing
Configure the new cluster settings and port.
pg-primary:/var/lib/pgsql/15/data/postgresql.conf ⇒ Configure PostgreSQL 
Config suppressed for testing
Update the pgBackRest configuration on all systems to point to the new cluster.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Upgrade the pg1-path 
Config suppressed for testing
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Upgrade the pg-path 
Config suppressed for testing
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Upgrade pg1-path and pg2-path, disable backup from standby 
Config suppressed for testing
pg-primary ⇒ Copy hba configuration 
sudo cp /var/lib/pgsql/14/data/pg_hba.conf \
       /var/lib/pgsql/15/data/pg_hba.conf
Before starting the new cluster, the stanza-upgrade command must be run.
pg-primary ⇒ Upgrade the stanza 
sudo -u postgres pgbackrest --stanza=demo --no-online \
       --log-level-console=info stanza-upgrade
Output suppressed for testing
Start the new cluster and confirm it is successfully installed.
pg-primary ⇒ Start new cluster 
sudo systemctl start postgresql-15.service
Output suppressed for testing
Test configuration using the check command.
pg-primary ⇒ Check configuration 
sudo systemctl status postgresql-15.service
sudo -u postgres pgbackrest --stanza=demo check
Output suppressed for testing
Remove the old cluster.
pg-primary ⇒ Remove old cluster 
sudo rm -rf /var/lib/pgsql/14/data
Install the new PostgreSQL binaries on the standby and create the cluster.
pg-standby ⇒ Remove old cluster and create the new cluster 
sudo rm -rf /var/lib/pgsql/14/data
sudo -u postgres mkdir -p -m 700 /usr/pgsql-15/bin
Run the check on the repository host. The warning regarding the standby being down is expected since the standby cluster is down. Running this command demonstrates that the repository server is aware of the standby and is configured properly for the primary server.
repository ⇒ Check configuration 
sudo -u pgbackrest pgbackrest --stanza=demo check
Output suppressed for testing
Run a full backup on the new cluster and then restore the standby from the backup. The backup type will automatically be changed to full if incr or diff is requested.
repository ⇒ Run a full backup 
sudo -u pgbackrest pgbackrest --stanza=demo --type=full backup
pg-standby ⇒ Restore the demo standby cluster 
sudo -u postgres pgbackrest --stanza=demo --type=standby restore
pg-standby ⇒ Start PostgreSQL and check the pgBackRest configuration 
sudo systemctl start postgresql-15.service
sudo -u postgres pgbackrest --stanza=demo check
Output suppressed for testing
Backup from standby can be enabled now that the standby is restored.
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Reenable backup from standby 
Config suppressed for testing
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated October 18, 2026
././@PaxHeader0000000000000000000000000000003200000000000010210 xustar0026 mtime=1792312410.77751 pgbackrest-2.60.0/doc/html/user-guide.html0000644000175100017510000074230515265102133020053 0ustar00runnerrunner pgBackRest - Debian & Ubuntu User Guide
pgBackRest
Debian & Ubuntu User Guide
Home
User Guides
Releases
Configuration
Commands
FAQ
Table of Contents
Introduction
Concepts
Backup
Restore
Write Ahead Log (WAL)
Encryption
Upgrading pgBackRest
From v2.x to v2.y
Build
Installation
Quick Start
Setup Demo Cluster
Configure Cluster Stanza
Create the Repository
Configure Archiving
Configure Retention
Configure Repository Encryption
Create the Stanza
Check the Configuration
Performance Tuning
Perform a Backup
Schedule a Backup
Backup Information
Restore a Backup
Monitoring
In PostgreSQL
Using jq
Backup
File Bundling
Block Incremental
Backup Annotations
Retention
Full Backup Retention
Differential Backup Retention
Archive Retention
Restore
File Ownership
Delta Option
Restore Selected Databases
Point-in-Time Recovery
Delete a Stanza
Multiple Repositories
Azure Support
S3 Support
SFTP Support
GCS Support
Target Time for Repository
Dedicated Repository Host
Installation
Setup Passwordless SSH
Configuration
Create and Check Stanza
Perform a Backup
Restore a Backup
Parallel Backup / Restore
Starting and Stopping
Replication
Installation
Setup Passwordless SSH
Hot Standby
Streaming Replication
Multiple Stanzas
Installation
Setup Passwordless SSH
Configuration
Setup Demo Cluster
Create the Stanza and Check Configuration
Asynchronous Archiving
Archive Push
Archive Get
Backup from a Standby
Upgrading PostgreSQL
This user guide is intended to be followed sequentially from beginning to end — each section depends on the last. For example, the Restore section relies on setup that is performed in the Quick Start section. Once pgBackRest is up and running then skipping around is possible but following the user guide in order is recommended the first time through.
Although the examples in this guide are targeted at Debian/Ubuntu and PostgreSQL 17, it should be fairly easy to apply the examples to any Unix distribution and PostgreSQL version. The only OS-specific commands are those to create, start, stop, and drop PostgreSQL clusters. The pgBackRest commands will be the same on any Unix system though the location of the executable may vary. While pgBackRest strives to operate consistently across versions of PostgreSQL, there are subtle differences between versions of PostgreSQL that may show up in this guide when illustrating certain examples, e.g. PostgreSQL path/file names and settings.
Configuration information and documentation for PostgreSQL can be found in the PostgreSQL Manual.
A somewhat novel approach is taken to documentation in this user guide. Each command is run on a virtual machine when the documentation is built from the XML source. This means you can have a high confidence that the commands work correctly in the order presented. Output is captured and displayed below the command when appropriate. If the output is not included it is because it was deemed not relevant or was considered a distraction from the narrative.
All commands are intended to be run as an unprivileged user that has sudo privileges for both the root and postgres users. It's also possible to run the commands directly as their respective users without modification and in that case the sudo commands can be stripped off.
Concepts
The following concepts are defined as they are relevant to pgBackRest, PostgreSQL, and this user guide.
Backup
A backup is a consistent copy of a database cluster that can be restored to recover from a hardware failure, to perform Point-In-Time Recovery, or to bring up a new standby.
Full Backup: pgBackRest copies the entire contents of the database cluster to the backup. The first backup of the database cluster is always a Full Backup. pgBackRest is always able to restore a full backup directly. The full backup does not depend on any files outside of the full backup for consistency.
Differential Backup: pgBackRest copies only those database cluster files that have changed since the last full backup. pgBackRest restores a differential backup by copying all of the files in the chosen differential backup and the appropriate unchanged files from the previous full backup. The advantage of a differential backup is that it requires less disk space than a full backup, however, the differential backup and the full backup must both be valid to restore the differential backup.
Incremental Backup: pgBackRest copies only those database cluster files that have changed since the last backup (which can be another incremental backup, a differential backup, or a full backup). As an incremental backup only includes those files changed since the prior backup, they are generally much smaller than full or differential backups. As with the differential backup, the incremental backup depends on other backups to be valid to restore the incremental backup. Since the incremental backup includes only those files since the last backup, all prior incremental backups back to the prior differential, the prior differential backup, and the prior full backup must all be valid to perform a restore of the incremental backup. If no differential backup exists then all prior incremental backups back to the prior full backup, which must exist, and the full backup itself must be valid to restore the incremental backup.
Restore
A restore is the act of copying a backup to a system where it will be started as a live database cluster. A restore requires the backup files and one or more WAL segments in order to work correctly.
Write Ahead Log (WAL)
WAL is the mechanism that PostgreSQL uses to ensure that no committed changes are lost. Transactions are written sequentially to the WAL and a transaction is considered to be committed when those writes are flushed to disk. Afterwards, a background process writes the changes into the main database cluster files (also known as the heap). In the event of a crash, the WAL is replayed to make the database consistent.
WAL is conceptually infinite but in practice is broken up into individual 16MB files called segments. WAL segments follow the naming convention 0000000100000A1E000000FE where the first 8 hexadecimal digits represent the timeline and the next 16 digits are the logical sequence number (LSN).
Encryption
Encryption is the process of converting data into a format that is unrecognizable unless the appropriate password (also referred to as passphrase) is provided.
pgBackRest will encrypt the repository based on a user-provided password, thereby preventing unauthorized access to data stored within the repository.
Upgrading pgBackRest
From v2.x to v2.y
Upgrading from v2.x to v2.y is straight-forward. The repository format has not changed, so for most installations it is simply a matter of installing binaries for the new version. It is also possible to downgrade if you have not used new features that are unsupported by the older version.
IMPORTANT:
The local and remote pgBackRest versions must match exactly so they should be upgraded together. If there is a mismatch, WAL archiving and backups will not function until the versions match. In such a case, the following error will be reported: [ProtocolError] expected value '2.x' for greeting key 'version' but got '2.y'.
Build
Installing pgBackRest from a package is preferable to building from source. See Installation for more information about packages.
When building from source it is best to use a build host rather than building on production. Many of the tools required for the build should generally not be installed in production. pgBackRest consists of a single executable so it is easy to copy to a new host once it is built.
build ⇒ Download version 2.60.0 of pgBackRest to /build path 
mkdir -p /build
curl -fsSL \
       https://github.com/pgbackrest/pgbackrest/releases/download/release%2F2.60.0/pgbackrest-2.60.0.tar.gz | \
       tar zx -C /build
build ⇒ Install build dependencies 
sudo apt-get install meson gcc libpq-dev libssl-dev libxml2-dev pkg-config \
       liblz4-dev libzstd-dev libbz2-dev libz-dev libssh2-1-dev libsystemd-dev
build ⇒ Configure and compile pgBackRest 
meson setup /build/pgbackrest /build/pgbackrest-2.60.0
ninja -C /build/pgbackrest
build ⇒ Optionally run smoke tests to verify pgBackRest was built correctly 
meson test -C /build/pgbackrest --suite smoke
Output suppressed for testing
Installation
A new host named pg-primary is created to contain the demo cluster and run pgBackRest examples.
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
Debian/Ubuntu packages for pgBackRest are available at apt.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
pg-primary ⇒ Install dependencies 
sudo apt-get install postgresql-client libxml2 libssh2-1
pg-primary ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
pg-primary ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown postgres:postgres /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
pgBackRest should now be properly installed but it is best to check. If any dependencies were missed then you will get an error when running pgBackRest from the command line.
pg-primary ⇒ Make sure the installation worked 
sudo -u postgres pgbackrest
Output suppressed for testing
Quick Start
The Quick Start section will cover basic configuration of pgBackRest and PostgreSQL and introduce the backup, restore, and info commands.
Setup Demo Cluster
Creating the demo cluster is optional but is strongly recommended, especially for new users, since the example commands in the user guide reference the demo cluster; the examples assume the demo cluster is running on the default port (i.e. 5432). The cluster will not be started until a later section because there is still some configuration to do.
pg-primary ⇒ Create the demo cluster 
sudo -u postgres /usr/lib/postgresql/17/bin/initdb \
       -D /var/lib/postgresql/17/demo -k -A peer
sudo pg_createcluster 17 demo
Output suppressed for testing
Configure Cluster Stanza
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod.
The name 'demo' describes the purpose of this cluster accurately so that will also make a good stanza name.
pgBackRest needs to know where the base data directory for the PostgreSQL cluster is located. The path can be requested from PostgreSQL directly but in a recovery scenario the PostgreSQL process will not be available. During backups the value supplied to pgBackRest will be compared against the path that PostgreSQL is running on and they must be equal or the backup will return an error. Make sure that pg-path is exactly equal to data_directory as reported by PostgreSQL.
By default Debian/Ubuntu stores clusters in /var/lib/postgresql/[version]/[cluster] so it is easy to determine the correct path for the data directory.
When creating the /etc/pgbackrest/pgbackrest.conf file, the database owner (usually postgres) must be granted read privileges.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the PostgreSQL cluster data directory 
Config suppressed for testing
pgBackRest configuration files follow a Windows INI-like convention. Sections are denoted by text in brackets and key/value pairs are contained in each section. Lines beginning with # are ignored and can be used as comments, but end-of-line comments following a value on the same line are not supported. Quoting is not supported and whitespace is trimmed from keys and values. Sections will be merged if they appear more than once.
There are multiple ways the pgBackRest configuration files can be loaded:
  • config and config-include-path are default: the default config file will be loaded, if it exists, and *.conf files in the default config include path will be appended, if they exist.
  • config option is specified: only the specified config file will be loaded and is expected to exist.
  • config-include-path is specified: *.conf files in the config include path will be loaded and the path is required to exist. The default config file will be be loaded if it exists. If it is desirable to load only the files in the specified config include path, then the --no-config option can also be passed.
  • config and config-include-path are specified: using the user-specified values, the config file will be loaded and *.conf files in the config include path will be appended. The files are expected to exist.
  • config-path is specified: this setting will override the base path for the default location of the config file and/or the base path of the default config-include-path setting unless the config and/or config-include-path option is explicitly set.
Files are concatenated as if they were one big file and each file must be valid individually. This means sections must be specified in each file where they are needed to store a key/value. Order doesn't matter but there is precedence based on sections. The precedence (highest to lowest) is:
  • [stanza:command]
  • [stanza]
  • [global:command]
  • [global]
NOTE:
--config, --config-include-path and --config-path are command-line only options.
pgBackRest can also be configured using environment variables as described in the command reference.
pg-primary ⇒ Configure log-path using the environment 
sudo -u postgres bash -c ' \
       export PGBACKREST_LOG_PATH=/path/set/by/env && \
       pgbackrest --log-level-console=error help backup log-path'
Output suppressed for testing
Create the Repository
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves.
For this demonstration the repository will be stored on the same host as the PostgreSQL server. This is the simplest configuration and is useful in cases where traditional backup software is employed to backup the database host.
pg-primary ⇒ Create the pgBackRest repository 
sudo mkdir -p /var/lib/pgbackrest
sudo chmod 750 /var/lib/pgbackrest
sudo chown postgres:postgres /var/lib/pgbackrest
The repository path must be configured so pgBackRest knows where to find it.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the pgBackRest repository path 
Config suppressed for testing
Multiple repositories may also be configured. See Multiple Repositories for details.
Configure Archiving
Backing up a running PostgreSQL cluster requires WAL archiving to be enabled. %p is how PostgreSQL specifies the location of the WAL segment to be archived. Note that at least one WAL segment will be created during the backup process even if no explicit writes are made to the cluster.
pg-primary:/etc/postgresql/17/demo/postgresql.conf ⇒ Configure archive settings 
Config suppressed for testing
The PostgreSQL cluster must be restarted after making these changes and before performing a backup.
pg-primary ⇒ Restart the demo cluster 
sudo pg_ctlcluster 17 demo restart
The hot_standby setting is enabled by default and should be left that way on every cluster. Any cluster may be restored as a standby later, e.g. a primary that is rebuilt as a standby after failover, and a standby will not accept read-only connections without it.
When archiving a WAL segment is expected to take more than 60 seconds (the default) to reach the pgBackRest repository, then the pgBackRest archive-timeout option should be increased. Note that this option is not the same as the PostgreSQL archive_timeout option which is used to force a WAL segment switch; useful for databases where there are long periods of inactivity. For more information on the PostgreSQL archive_timeout option, see PostgreSQL Write Ahead Log.
The archive-push command can be configured with its own options. For example, a lower compression level may be set to speed archiving without affecting the compression used for backups.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Config archive-push to use a lower compression level 
Config suppressed for testing
This configuration technique can be used for any command and can even target a specific stanza, e.g. demo:archive-push.
Configure Retention
pgBackRest expires backups based on retention options.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure retention to 2 full backups 
Config suppressed for testing
More information about retention can be found in the Retention section.
Configure Repository Encryption
The repository will be configured with a cipher type and key to demonstrate encryption. Encryption is always performed client-side even if the repository type (e.g. S3 or other object store) supports encryption.
It is important to use a long, random passphrase for the cipher key. A good way to generate one is to run: openssl rand -base64 48.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest repository encryption 
Config suppressed for testing
NOTE:
Encryption settings are placed in the [global] section, above, so the info command can read all stanzas. Without the stanza option the info command reads encryption settings only from the [global] section, so encryption settings configured per stanza require the stanza option to read an encrypted stanza.
Once the repository has been configured and the stanza created and checked, the repository encryption settings cannot be changed.
Create the Stanza
The stanza-create command must be run to initialize the stanza. It is recommended that the check command be run after stanza-create to ensure archiving and backups are properly configured.
pg-primary ⇒ Create the stanza and check the configuration 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
Check the Configuration
The check command validates that pgBackRest and the archive_command setting are configured correctly for archiving and backups for the specified stanza. It will attempt to check all repositories and databases that are configured for the host on which the command is run. It detects misconfigurations, particularly in archiving, that result in incomplete backups because required WAL segments did not reach the archive. The command can be run on the PostgreSQL or repository host. The command may also be run on the standby host, however, since pg_switch_xlog()/pg_switch_wal() cannot be performed on the standby, the command will only test the repository configuration.
Note that pg_create_restore_point('pgBackRest Archive Check') and pg_switch_xlog()/pg_switch_wal() are called to force PostgreSQL to archive a WAL segment.
pg-primary ⇒ Check the configuration 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check
Output suppressed for testing
Performance Tuning
pgBackRest has a number of performance options that are not enabled by default to maintain backward compatibility in the repository. However, when creating a new repository the following options are recommended. They can also be used on an existing repository with the caveat that older versions of pgBackRest will not be able to read the repository. This incompatibility depends on when the feature was introduced, as noted in the list below.
  • compress-type - determines the compression algorithm used by the backup and archive-push commands. The default is gz (Gzip) but zst (Zstandard) is recommended because it is much faster and provides compression similar to gz. zst has been supported by the compress-type option since v2.27. See Compress Type for more details.
  • repo-bundle - combines small files during backup to save space and improve the speed of both the backup and restore commands, especially on object stores such as S3. The repo-bundle option was introduced in v2.39. See File Bundling for more details.
  • repo-block - stores only the portions of files that have changed rather than the entire file during diff/incr backup. This saves space and increases the speed of the backup. The repo-block option was introduced in v2.46 but at least v2.52.1 is recommended. See Block Incremental for more details.
There are other performance options that are not enabled by default because they require additional configuration or because the default is safe (but not optimal). These options are available in all v2 versions of pgBackRest.
  • process-max - determines how many processes will be used for commands. The default is 1, which is almost never the appropriate value. Each command uses process-max differently so refer to each command's documentation for details on usage.
  • archive-async - archives WAL files to the repository in batch which greatly increases archiving speed. It is not enabled by default because it requires a spool path to be created. See Asynchronous Archiving for more details.
  • backup-standby - performs the backup on a standby rather than the primary to reduce load on the primary. It is not enabled by default because it requires additional configuration and the presence of one or more standby hosts. See Backup from a Standby for more details.
Perform a Backup
By default pgBackRest will wait for the next regularly scheduled checkpoint before starting a backup. Depending on the checkpoint_timeout and checkpoint_segments settings in PostgreSQL it may be quite some time before a checkpoint completes and the backup can begin. Generally, it is best to set start-fast=y so that the backup starts immediately. This forces a checkpoint, but since backups are usually run once a day an additional checkpoint should not have a noticeable impact on performance. However, on very busy clusters it may be best to pass --start-fast on the command-line as needed.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure backup fast start 
Config suppressed for testing
To perform a backup of the PostgreSQL cluster run pgBackRest with the backup command.
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo \
       --log-level-console=info backup
Output suppressed for testing
By default pgBackRest will attempt to perform an incremental backup. However, an incremental backup must be based on a full backup and since no full backup existed pgBackRest ran a full backup instead.
The type option can be used to specify a full or differential backup.
pg-primary ⇒ Differential backup of the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup
Output suppressed for testing
This time there was no warning because a full backup already existed. While incremental backups can be based on a full or differential backup, differential backups must be based on a full backup. A full backup can be performed by running the backup command with --type=full.
During an online backup pgBackRest waits for WAL segments that are required for backup consistency to be archived. This wait time is governed by the pgBackRest archive-timeout option which defaults to 60 seconds. If archiving an individual segment is known to take longer then this option should be increased.
Schedule a Backup
Backups can be scheduled with utilities such as cron.
In the following example, two cron jobs are configured to run; full backups are scheduled for 6:30 AM every Sunday with differential backups scheduled for 6:30 AM Monday through Saturday. If this crontab is installed for the first time mid-week, then pgBackRest will run a full backup the first time the differential job is executed, followed the next day by a differential backup. 
#m h   dom mon dow   command
30 06  *   *   0     pgbackrest --type=full --stanza=demo backup
30 06  *   *   1-6   pgbackrest --type=diff --stanza=demo backup
Once backups are scheduled it's important to configure retention so backups are expired on a regular schedule, see Retention.
Backup Information
Use the info command to get information about backups.
pg-primary ⇒ Get info for the demo cluster 
sudo -u postgres pgbackrest info
Output suppressed for testing
The info command operates on a single stanza or all stanzas. Text output is the default and gives a human-readable summary of backups for the stanza(s) requested. This format is subject to change with any release.
For machine-readable output use --output=json. The JSON output contains far more information than the text output and is kept stable unless a bug is found.
To speed up execution, limit the output to only progress information by specifying --detail-level=progress. Note that this skips all checks except for availability of the stanza.
Each stanza has a separate section and it is possible to limit output to a single stanza with the --stanza option. The stanza 'status' gives a brief indication of the stanza's health. If this is 'ok' then pgBackRest is functioning normally. If there are multiple repositories, then a status of 'mixed' indicates that the stanza is not in a healthy state on one or more of the repositories; in this case the state of the stanza will be detailed per repository. For cases in which an error on a repository occurred that is not one of the known error codes, then an error code of 'other' will be used and the full error details will be provided. The 'wal archive min/max' shows the minimum and maximum WAL currently stored in the archive and, in the case of multiple repositories, will be reported across all repositories unless the --repo option is set. Note that there may be gaps due to archive retention policies or other reasons.
The 'backup/expire running' and/or 'restore running' messages will appear beside the 'status' information if any of those commands are currently running on the host. Per-repo progress will be also reported in text output and a 'repo' array will be included in JSON output.
The backups are displayed oldest to newest. The oldest backup will always be a full backup (indicated by an F at the end of the label) but the newest backup can be full, differential (ends with D), or incremental (ends with I).
The 'timestamp start/stop' defines the time period when the backup ran. The 'timestamp stop' can be used to determine the backup to use when performing Point-In-Time Recovery. More information about Point-In-Time Recovery can be found in the Point-In-Time Recovery section.
The 'wal start/stop' defines the WAL range that is required to make the database consistent when restoring. The backup command will ensure that this WAL range is in the archive before completing.
The 'database size' is the full uncompressed size of the database while 'database backup size' is the amount of data in the database to actually back up (these will be the same for full backups).
The 'repo' indicates in which repository this backup resides. The 'backup set size' includes all the files from this backup and any referenced backups in the repository that are required to restore the database from this backup while 'backup size' includes only the files in this backup (these will also be the same for full backups). Repository sizes reflect compressed file sizes if compression is enabled in pgBackRest.
The 'backup reference total' summarizes the list of additional backups that are required to restore this backup. Use the --set option to display the complete reference list.
Restore a Backup
Backups can protect you from a number of disaster scenarios, the most common of which are hardware failure and data corruption. The easiest way to simulate data corruption is to remove an important PostgreSQL cluster file.
pg-primary ⇒ Stop the demo cluster and delete the pg_control file 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres rm /var/lib/postgresql/17/demo/global/pg_control
Starting the cluster without this important file will result in an error.
pg-primary ⇒ Attempt to start the corrupted demo cluster 
sudo pg_ctlcluster 17 demo start
Output suppressed for testing
To restore a backup of the PostgreSQL cluster run pgBackRest with the restore command. The cluster needs to be stopped (in this case it is already stopped) and all files must be removed from the PostgreSQL data directory.
pg-primary ⇒ Remove old files from demo cluster 
sudo -u postgres find /var/lib/postgresql/17/demo -mindepth 1 -delete
pg-primary ⇒ Restore the demo cluster and start PostgreSQL 
sudo -u postgres pgbackrest --stanza=demo restore
sudo pg_ctlcluster 17 demo start
This time the cluster started successfully since the restore replaced the missing pg_control file.
More information about the restore command can be found in the Restore section.
Monitoring
Monitoring is an important part of any production system. There are many tools available and pgBackRest can be monitored on any of them with a little work.
pgBackRest can output information about the repository in JSON format which includes a list of all backups for each stanza and WAL archive info.
In PostgreSQL
The PostgreSQL COPY command allows pgBackRest info to be loaded into a table. The following example wraps that logic in a function that can be used to perform real-time queries.
pg-primary ⇒ Load pgBackRest info function for PostgreSQL 
sudo -u postgres cat \
       /var/lib/postgresql/pgbackrest/doc/example/pgsql-pgbackrest-info.sql
Output suppressed for testing
sudo -u postgres psql -f \
       /var/lib/postgresql/pgbackrest/doc/example/pgsql-pgbackrest-info.sql
Now the monitor.pgbackrest_info() function can be used to determine the last successful backup time and archived WAL for a stanza.
pg-primary ⇒ Query last successful backup time and archived WAL 
sudo -u postgres cat \
       /var/lib/postgresql/pgbackrest/doc/example/pgsql-pgbackrest-query.sql
Output suppressed for testing
sudo -u postgres psql -f \
       /var/lib/postgresql/pgbackrest/doc/example/pgsql-pgbackrest-query.sql
Output suppressed for testing
Using jq
jq is a command-line utility that can easily extract data from JSON.
pg-primary ⇒ Install jq utility 
sudo apt-get install jq
Now jq can be used to query the last successful backup time for a stanza.
pg-primary ⇒ Query last successful backup time 
sudo -u postgres pgbackrest --output=json --stanza=demo info | \
       jq '.[0] | .backup[-1] | .timestamp.stop'
Output suppressed for testing
Or the last archived WAL.
pg-primary ⇒ Query last archived WAL 
sudo -u postgres pgbackrest --output=json --stanza=demo info | \
       jq '.[0] | .archive[-1] | .max'
Output suppressed for testing
NOTE:
This syntax requires jq v1.5.
NOTE:
jq may round large numbers such as system identifiers. Test your queries carefully.
Backup
When multiple repositories are configured, pgBackRest will backup to the highest priority repository (e.g. repo1) unless the --repo option is specified.
pgBackRest does not have a built-in scheduler so it's best to run it from cron or some other scheduling mechanism.
See Perform a Backup for more details and examples.
File Bundling
Bundling files together in the repository saves time during the backup and some space in the repository. This is especially pronounced when the repository is stored on an object store such as S3 or file systems with large block sizes. Per-file creation time on object stores is higher and very small files might cost as much to store as larger files.
The file bundling feature is enabled with the repo-bundle option.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-bundle 
Config suppressed for testing
A full backup without file bundling will have 1000+ files in the backup path, but with bundling the total number of files is greatly reduced. An additional benefit is that zero-length files are not stored (except in the manifest), whereas in a normal backup each zero-length file is stored individually.
pg-primary ⇒ Perform a full backup 
sudo -u postgres pgbackrest --stanza=demo --type=full backup
pg-primary ⇒ Check file total 
sudo -u postgres find /var/lib/pgbackrest/backup/demo/latest/ -type f | wc -l
Output suppressed for testing
The repo-bundle-size and repo-bundle-limit options can be used for tuning, though the defaults should be optimal in most cases.
While file bundling is generally more efficient, the downside is that it is more difficult to manually retrieve files from the repository. It may not be ideal for deduplicated storage since each full backup will arrange files in the bundles differently. Lastly, file bundles cannot be resumed, so be careful not to set repo-bundle-limit too high.
Block Incremental
Block incremental backups save space by only storing the parts of a file that have changed since the prior backup rather than storing the entire file.
The block incremental feature is enabled with the repo-block option and it works best when enabled for all backup types. File bundling must also be enabled.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-block 
Config suppressed for testing
Backup Annotations
Users can attach informative key/value pairs to the backup. This option may be used multiple times to attach multiple annotations.
pg-primary ⇒ Perform a full backup with annotations 
sudo -u postgres pgbackrest --stanza=demo --annotation=source="demo backup" \
       --annotation=key=value --type=full backup
Annotations are output by the info command text output when a backup is specified with --set and always appear in the JSON output.
pg-primary ⇒ Get info for the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --set=[Test Variable] info
Output suppressed for testing
Annotations included with the backup command can be added, modified, or removed afterwards using the annotate command.
pg-primary ⇒ Change backup annotations 
sudo -u postgres pgbackrest --stanza=demo --set=[Test Variable] \
       --annotation=key= --annotation=new_key=new_value annotate
sudo -u postgres pgbackrest --stanza=demo --set=[Test Variable] info
Output suppressed for testing
Retention
Generally it is best to retain as many backups as possible to provide a greater window for Point-in-Time Recovery, but practical concerns such as disk space must also be considered. Retention options remove older backups once they are no longer needed.
pgBackRest does full backup rotation based on the retention type which can be a count or a time period. When a count is specified, then expiration is not concerned with when the backups were created but with how many must be retained. Differential backups are count-based but will always be expired when the full backup they depend on is expired. Incremental backups are not expired by retention independently — they are always expired with their related full or differential backup. See sections Full Backup Retention and Differential Backup Retention for details and examples.
Archived WAL is retained by default for backups that have not expired, however, although not recommended, this schedule can be modified per repository with the retention-archive options. See section Archive Retention for details and examples.
The expire command is run automatically after each successful backup and can also be run by the user. When run by the user, expiration will occur as defined by the retention settings for each configured repository. If the --repo option is provided, expiration will occur only on the specified repository. Expiration can also be limited by the user to a specific backup set with the --set option and, unless the --repo option is specified, all repositories will be searched and any matching the set criteria will be expired. It should be noted that the archive retention schedule will be checked and performed any time the expire command is run.
Full Backup Retention
The repo1-retention-full-type determines how the option repo1-retention-full is interpreted; either as the count of full backups to be retained or how many days to retain full backups. New backups must be completed before expiration will occur — that means if repo1-retention-full-type=count and repo1-retention-full=2 then there will be three full backups stored before the oldest one is expired, or if repo1-retention-full-type=time and repo1-retention-full=20 then there must be one full backup that is at least 20 days old before expiration can occur.
repo1-retention-full=2 was set in Configure Retention but currently there is only one full backup so the next full backup to run will not expire any full backups.
pg-primary ⇒ Perform a full backup 
sudo -u postgres pgbackrest --stanza=demo --type=full \
       --log-level-console=detail backup
Output suppressed for testing
Archive is expired because WAL segments were generated before the oldest backup. These are not useful for recovery — only WAL segments generated after a backup can be used to recover that backup.
pg-primary ⇒ Perform a full backup 
sudo -u postgres pgbackrest --stanza=demo --type=full \
       --log-level-console=info backup
Output suppressed for testing
The [Test Variable] full backup is expired and archive retention is based on the [Test Variable] which is now the oldest full backup.
Differential Backup Retention
Set repo1-retention-diff to the number of differential backups required. Differentials only rely on the prior full backup so it is possible to create a rolling set of differentials for the last day or more. This allows quick restores to recent points-in-time but reduces overall space consumption.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-retention-diff 
Config suppressed for testing
Backup repo1-retention-diff=1 so two differentials will need to be performed before one is expired. An incremental backup is added to demonstrate incremental expiration, which in this case depends on the differential expiration.
pg-primary ⇒ Perform differential and incremental backups 
sudo -u postgres pgbackrest --stanza=demo --type=diff backup
sudo -u postgres pgbackrest --stanza=demo --type=incr backup
Now performing a differential backup will expire the previous differential and incremental backups leaving only one differential backup.
pg-primary ⇒ Perform a differential backup 
sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup
Output suppressed for testing
Archive Retention
Although pgBackRest automatically removes archived WAL segments when expiring backups (the default expires WAL for full backups based on the repo1-retention-full option), it may be useful to expire archive more aggressively to save disk space. Note that full backups are treated as differential backups for the purpose of differential archive retention.
Expiring archive will never remove WAL segments that are required to make a backup consistent. However, since Point-in-Time-Recovery (PITR) only works on a continuous WAL stream, care should be taken when aggressively expiring archive outside of the normal backup expiration process. To determine what will be expired without actually expiring anything, the dry-run option can be provided on the command line with the expire command.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-retention-diff 
Config suppressed for testing
pg-primary ⇒ Perform differential backup 
sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup
Output suppressed for testing
pg-primary ⇒ Expire archive 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=detail \
       --repo1-retention-archive-type=diff --repo1-retention-archive=1 expire
Output suppressed for testing
The [Test Variable] differential backup has archived WAL segments that must be retained to make the older backups consistent even though they cannot be played any further forward with PITR. WAL segments generated after [Test Variable] but before [Test Variable] are removed. WAL segments generated after the new backup [Test Variable] remain and can be used for PITR.
Since full backups are considered differential backups for the purpose of differential archive retention, if a full backup is now performed with the same settings, only the archive for that full backup is retained for PITR.
Restore
The restore command automatically defaults to selecting the latest backup from the first repository where backups exist (see Quick Start - Restore a Backup). The order in which the repositories are checked is dictated by the pgbackrest.conf (e.g. repo1 will be checked before repo2). To select from a specific repository, the --repo option can be passed (e.g. --repo=1). The --set option can be passed if a backup other than the latest is desired.
When PITR of --type=time or --type=lsn is specified, then the target time or target lsn must be specified with the --target option. If a backup is not specified via the --set option, then the configured repositories will be checked, in order, for a backup that contains the requested time or lsn. If no matching backup is found, the latest backup from the first repository containing backups will be used for --type=time while no backup will be selected for --type=lsn. For other types of PITR, e.g. xid, the --set option must be provided if the target is prior to the latest backup. See Point-in-Time Recovery for more details and examples.
Replication slots are not included per recommendation of PostgreSQL. See Backing Up The Data Directory in the PostgreSQL documentation for more information.
The following sections introduce additional restore command features.
File Ownership
If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried.
If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name.
Delta Option
Restore a Backup in Quick Start required the database cluster directory to be cleaned before the restore could be performed. The delta option allows pgBackRest to automatically determine which files in the database cluster directory can be preserved and which ones need to be restored from the backup — it also removes files not present in the backup manifest so it will dispose of divergent changes. This is accomplished by calculating a SHA-1 cryptographic hash for each file in the database cluster directory. If the SHA-1 hash does not match the hash stored in the backup then that file will be restored. This operation is very efficient when combined with the process-max option. Since the PostgreSQL server is shut down during the restore, a larger number of processes can be used than might be desirable during a backup when the PostgreSQL server is running.
pg-primary ⇒ Stop the demo cluster, perform delta restore 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres pgbackrest --stanza=demo --delta \
       --log-level-console=detail restore
Output suppressed for testing
pg-primary ⇒ Restart PostgreSQL 
sudo pg_ctlcluster 17 demo start
Restore Selected Databases
There may be cases where it is desirable to selectively restore specific databases from a cluster backup. This could be done for performance reasons or to move selected databases to a machine that does not have enough space to restore the entire cluster backup.
To demonstrate this feature two databases are created: test1 and test2.
pg-primary ⇒ Create two test databases 
sudo -u postgres psql -c "create database test1;"
Output suppressed for testing
sudo -u postgres psql -c "create database test2;"
Output suppressed for testing
Each test database will be seeded with tables and data to demonstrate that recovery works with selective restore.
pg-primary ⇒ Create a test table in each database 
sudo -u postgres psql -c "create table test1_table (id int); \
       insert into test1_table (id) values (1);" test1
Output suppressed for testing
sudo -u postgres psql -c "create table test2_table (id int); \
       insert into test2_table (id) values (2);" test2
Output suppressed for testing
A fresh backup is run so pgBackRest is aware of the new databases.
pg-primary ⇒ Perform a backup 
sudo -u postgres pgbackrest --stanza=demo --type=incr backup
One of the main reasons to use selective restore is to save space. The size of the test1 database is shown here so it can be compared with the disk utilization after a selective restore.
pg-primary ⇒ Show space used by test1 database 
sudo -u postgres du -sh /var/lib/postgresql/17/demo/base/[Test Variable]
Output suppressed for testing
If the database to restore is not known, use the info command set option to discover databases that are part of the backup set.
pg-primary ⇒ Show database list for backup 
sudo -u postgres pgbackrest --stanza=demo \
       --set=[Test Variable] info
Output suppressed for testing
Stop the cluster and restore only the test2 database. Built-in databases (template0, template1, and postgres) are always restored.
WARNING:
Recovery may error unless --type=immediate is specified. This is because after consistency is reached PostgreSQL will flag zeroed pages as errors even for a full-page write. For PostgreSQL ≥ 13 the ignore_invalid_pages setting may be used to ignore invalid pages. In this case it is important to check the logs after recovery to ensure that no invalid pages were reported in the selected databases.
pg-primary ⇒ Restore from last backup including only the test2 database 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres pgbackrest --stanza=demo --delta \
       --db-include=test2 --type=immediate --target-action=promote restore
sudo pg_ctlcluster 17 demo start
Once recovery is complete the test2 database will contain all previously created tables and data.
pg-primary ⇒ Demonstrate that the test2 database was recovered 
sudo -u postgres psql -c "select * from test2_table;" test2
Output suppressed for testing
The test1 database, despite successful recovery, is not accessible. This is because the entire database was restored as sparse, zeroed files. PostgreSQL can successfully apply WAL on the zeroed files but the database as a whole will not be valid because key files contain no data. This is purposeful to prevent the database from being accidentally used when it might contain partial data that was applied during WAL replay.
pg-primary ⇒ Attempting to connect to the test1 database will produce an error 
sudo -u postgres psql -c "select * from test1_table;" test1
Output suppressed for testing
Since the test1 database is restored with sparse, zeroed files it will only require as much space as the amount of WAL that is written during recovery. While the amount of WAL generated during a backup and applied during recovery can be significant it will generally be a small fraction of the total database size, especially for large databases where this feature is most likely to be useful.
It is clear that the test1 database uses far less disk space during the selective restore than it would have if the entire database had been restored.
pg-primary ⇒ Show space used by test1 database after recovery 
sudo -u postgres du -sh /var/lib/postgresql/17/demo/base/[Test Variable]
Output suppressed for testing
At this point the only action that can be taken on the invalid test1 database is drop database. pgBackRest does not automatically drop the database since this cannot be done until recovery is complete and the cluster is accessible.
pg-primary ⇒ Drop the test1 database 
sudo -u postgres psql -c "drop database test1;"
Output suppressed for testing
Now that the invalid test1 database has been dropped only the test2 and built-in databases remain.
pg-primary ⇒ List remaining databases 
sudo -u postgres psql -c "select oid, datname from pg_database order by oid;"
Output suppressed for testing
Point-in-Time Recovery
Restore a Backup in Quick Start performed default recovery, which is to play all the way to the end of the WAL stream. In the case of a hardware failure this is usually the best choice but for data corruption scenarios (whether machine or human in origin) Point-in-Time Recovery (PITR) is often more appropriate.
Point-in-Time Recovery (PITR) allows the WAL to be played from a backup to a specified lsn, time, transaction id, or recovery point. For common recovery scenarios time-based recovery is arguably the most useful. A typical recovery scenario is to restore a table that was accidentally dropped or data that was accidentally deleted. Recovering a dropped table is more dramatic so that's the example given here but deleted data would be recovered in exactly the same way.
pg-primary ⇒ Create a table with very important data 
sudo -u postgres psql -c "begin; \
       create table important_table (message text); \
       insert into important_table values ('Important Data'); \
       commit; \
       select * from important_table;"
Output suppressed for testing
It is important to represent the time as reckoned by PostgreSQL and to include timezone offsets. This reduces the possibility of unintended timezone conversions and an unexpected recovery result.
pg-primary ⇒ Get the time from PostgreSQL 
sudo -u postgres psql -Atc "select current_timestamp"
Output suppressed for testing
Now that the time has been recorded the table is dropped. In practice finding the exact time that the table was dropped is a lot harder than in this example. It may not be possible to find the exact time, but some forensic work should be able to get you close.
pg-primary ⇒ Drop the important table 
sudo -u postgres psql -c "begin; \
       drop table important_table; \
       commit; \
       select * from important_table;"
Output suppressed for testing
If the wrong backup is selected for restore then recovery to the required time target will fail. To demonstrate this a new incremental backup is performed where important_table does not exist.
pg-primary ⇒ Perform an incremental backup 
sudo -u postgres pgbackrest --stanza=demo --type=incr backup
sudo -u postgres pgbackrest info
Output suppressed for testing
It will not be possible to recover the lost table from this backup since PostgreSQL can only play forward, not backward.
pg-primary ⇒ Attempt recovery from an incorrect backup 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres pgbackrest --stanza=demo --delta \
       --set=[Test Variable] --target-timeline=current \
       --type=time "--target=[Test Variable]" --target-action=promote restore
sudo pg_ctlcluster 17 demo start
Output suppressed for testing
A reliable method is to allow pgBackRest to automatically select a backup capable of recovery to the time target, i.e. a backup that ended before the specified time.
NOTE:
pgBackRest cannot automatically select a backup when the restore type is xid or name.
pg-primary ⇒ Restore the demo cluster to [Test Variable] 
sudo -u postgres pgbackrest --stanza=demo --delta \
       --type=time "--target=[Test Variable]" \
       --target-action=promote restore
sudo -u postgres cat /var/lib/postgresql/17/demo/postgresql.auto.conf
Output suppressed for testing
pgBackRest has generated the recovery settings in postgresql.auto.conf so PostgreSQL can be started immediately. %f is how PostgreSQL specifies the WAL segment it needs and %p is the location where it should be copied. Once PostgreSQL has finished recovery the table will exist again and can be queried.
pg-primary ⇒ Start PostgreSQL and check that the important table exists 
sudo pg_ctlcluster 17 demo start
sudo -u postgres psql -c "select * from important_table"
Output suppressed for testing
The PostgreSQL log also contains valuable information. It will indicate the time and transaction where the recovery stopped and also give the time of the last transaction to be applied.
pg-primary ⇒ Examine the PostgreSQL log output 
sudo -u postgres cat /var/log/postgresql/postgresql-17-demo.log
Output suppressed for testing
Delete a Stanza
The stanza-delete command removes data in the repository associated with a stanza.
WARNING:
Use this command with caution — it will permanently remove all backups and archives from the pgBackRest repository for the specified stanza.
To delete a stanza:
  • Shut down the PostgreSQL cluster associated with the stanza (or use --force to override).
  • Run the stop command on the host where the stanza-delete command will be run.
  • Run the stanza-delete command.
Once the command successfully completes, it is the responsibility of the user to remove the stanza from all pgBackRest configuration files and/or environment variables.
A stanza may only be deleted from one repository at a time. To delete the stanza from multiple repositories, repeat the stanza-delete command for each repository while specifying the --repo option.
pg-primary ⇒ Stop PostgreSQL cluster to be removed 
sudo pg_ctlcluster 17 demo stop
pg-primary ⇒ Stop pgBackRest for the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stop
Output suppressed for testing
pg-primary ⇒ Delete the stanza from one repository 
sudo -u postgres pgbackrest --stanza=demo --repo=1 \
       --log-level-console=info stanza-delete
Output suppressed for testing
Multiple Repositories
Multiple repositories may be configured as demonstrated in S3 Support. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy.
Some commands, e.g. stanza-create/stanza-upgrade, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified.
Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later.
The archive-push command will always push WAL to the archive in all configured repositories. When a repository cannot be reached, WAL will still be pushed to other repositories. However, for this to work effectively, archive-async=y must be enabled; otherwise, the other repositories can only get one WAL segment ahead of the unreachable repository. Also, note that if WAL cannot be pushed to any repository, then PostgreSQL will not remove it from the pg_wal directory, which may cause the volume to run out of space.
Backups need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient.
Azure Support
pgBackRest supports locating repositories in Azure-compatible object stores. The container used to store the repository must be created in advance — pgBackRest will not do it automatically. The repository can be located in the container root (/) but it's usually best to place it in a subpath so object store logs or other data can also be stored in the container without conflicts.
WARNING:
Do not enable hierarchical namespace as this will cause errors during expire.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure Azure 
Config suppressed for testing
Shared access signatures may be used by setting the repo2-azure-key-type option to sas and the repo2-azure-key option to the shared access signature token.
Commands are run exactly as if the repository were stored on a local disk.
pg-primary ⇒ Create the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
File creation time in Azure is relatively slow so backup/restore performance is improved by enabling file bundling.
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --repo=2 \
       --log-level-console=info backup
Output suppressed for testing
S3 Support
pgBackRest supports locating repositories in S3-compatible object stores. The bucket used to store the repository must be created in advance — pgBackRest will not do it automatically. The repository can be located in the bucket root (/) but it's usually best to place it in a subpath so object store logs or other data can also be stored in the bucket without conflicts.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure S3 
Config suppressed for testing
NOTE:
The region and endpoint will need to be configured to where the bucket is located. The values given here are for the us-east-1 region.
A role should be created to run pgBackRest and the bucket permissions should be set as restrictively as possible. If the role is associated with an instance in AWS then pgBackRest will automatically retrieve temporary credentials when repo3-s3-key-type=auto, which means that keys do not need to be explicitly set in /etc/pgbackrest/pgbackrest.conf.
This sample Amazon S3 policy will restrict all reads and writes to the bucket and repository path. 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket"
            ],
            "Condition": {
                "StringEquals": {
                    "s3:prefix": [
                        "",
                        "demo-repo"
                    ],
                    "s3:delimiter": [
                        "/"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "demo-repo/*"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket/demo-repo/*"
            ]
        }
    ]
}
Commands are run exactly as if the repository were stored on a local disk.
pg-primary ⇒ Create the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
File creation time in S3 is relatively slow so backup/restore performance is improved by enabling file bundling.
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --repo=3 \
       --log-level-console=info backup
Output suppressed for testing
SFTP Support
pgBackRest supports locating repositories on SFTP hosts. SFTP file transfer is relatively slow so commands benefit by increasing process-max to parallelize file transfer.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure SFTP 
Config suppressed for testing
When utilizing SFTP, if libssh2 is compiled against OpenSSH then repo4-sftp-public-key-file is optional.
pg-primary ⇒ Generate SSH keypair for SFTP backup 
sudo -u postgres mkdir -m 750 -p /var/lib/postgresql/.ssh
sudo -u postgres ssh-keygen -f /var/lib/postgresql/.ssh/id_rsa_sftp \
       -t rsa -b 4096 -N "" -m PEM
sftp-server ⇒ Copy pg-primary SFTP backup public key to sftp-server 
sudo -u pgbackrest mkdir -m 750 -p /home/pgbackrest/.ssh
(sudo ssh root@pg-primary cat /var/lib/postgresql/.ssh/id_rsa_sftp.pub) | \
       sudo -u pgbackrest tee -a /home/pgbackrest/.ssh/authorized_keys
Commands are run exactly as if the repository were stored on a local disk.
pg-primary ⇒ Add sftp-server fingerprint to known_hosts file since repo4-sftp-host-key-check-type defaults to strict 
ssh-keyscan -H sftp-server >> /var/lib/postgresql/.ssh/known_hosts 2>/dev/null
pg-primary ⇒ Create the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --repo=4 \
       --log-level-console=info backup
Output suppressed for testing
GCS Support
pgBackRest supports locating repositories in GCS-compatible object stores. The bucket used to store the repository must be created in advance — pgBackRest will not do it automatically. The repository can be located in the bucket root (/) but it's usually best to place it in a subpath so object store logs or other data can also be stored in the bucket without conflicts.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure GCS 
Config suppressed for testing
When running in GCE set repo5-gcs-key-type=auto to automatically authenticate using the instance service account.
Commands are run exactly as if the repository were stored on a local disk.
File creation time in GCS is relatively slow so backup/restore performance is improved by enabling file bundling.
Target Time for Repository
The target time defines the time that commands use to read a repository on versioned storage. This allows the command to read the repository as it was at a point-in-time in order to recover data that has been deleted or corrupted by user accident or malware.
Versioned storage is supported by S3, GCS, and Azure but is generally not enabled by default. In addition to enabling versioning, it may be useful to enable object locking for S3 and soft delete for GCS or Azure.
When the repo-target-time option is specified then the repo option must also be provided. It is likely that not all repository types will support versioning and in general it makes sense to target a single repository for recovery.
Note that comparisons to the storage timestamp are <= the timestamp provided and milliseconds are truncated from the timestamp when provided.
To demonstrate this feature the demo stanza in the S3 repo is deleted.
pg-primary ⇒ Delete stanza in S3 repository 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres pgbackrest --stanza=demo stop
Output suppressed for testing
sudo -u postgres pgbackrest --stanza=demo --repo=3 stanza-delete
Output suppressed for testing
Once the stanza is deleted the info command will show the repository in an error state.
pg-primary ⇒ Error on info 
sudo -u postgres pgbackrest --stanza=demo --repo=3 info
Output suppressed for testing
However, since the storage is versioned, it is possible to look at the repository at a time before the stanza was deleted. Finding the target time can be tricky depending on the situation, but in this case the time when the stanza was deleted can be determined by checking when backup.info was deleted.
pg-primary ⇒ List versions of backup.info in the bucket 
key=demo-repo/backup/demo/backup.info; \
       aws s3api list-object-versions --bucket demo-bucket \
       --prefix $key --output table \
       --query "sort_by([Versions[?Key=='$key'].{Action:'PUT', \
       Modified:LastModified,Object:Key}, \
       DeleteMarkers[?Key=='$key'].{Action:'DELETE', \
       Modified:LastModified,Object:Key}][],&Modified)"
Output suppressed for testing
Now the info command can be run with a target time that will show the repository before it was deleted.
pg-primary ⇒ Info with target time 
sudo -u postgres pgbackrest --stanza=demo --repo=3 \
       --repo-target-time="[Test Variable]" info
Output suppressed for testing
If the required backup is shown by the info command then it can be restored using the same target time.
pg-primary ⇒ Restore with target time 
sudo -u postgres pgbackrest --stanza=demo --repo=3 --delta \
       --repo-target-time="[Test Variable]" --log-level-console=info restore
Output suppressed for testing
sudo pg_ctlcluster 17 demo start
Dedicated Repository Host
The configuration described in Quickstart is suitable for simple installations but for enterprise configurations it is more typical to have a dedicated repository host where the backups and WAL archive files are stored. This separates the backups and WAL archive from the database server so database host failures have less impact. It is still a good idea to employ traditional backup software to backup the repository host.
On PostgreSQL hosts, pg1-path is required to be the path of the local PostgreSQL cluster and no pg1-host should be configured. When configuring a repository host, the pgbackrest configuration file must have the pg-host option configured to connect to the primary and standby (if any) hosts. The repository host has the only pgbackrest configuration that should be aware of more than one PostgreSQL host. Order does not matter, e.g. pg1-path/pg1-host, pg2-path/pg2-host can be primary or standby.
Installation
A new host named repository is created to store the cluster backups.
NOTE:
The pgBackRest version installed on the repository host must exactly match the version installed on the PostgreSQL host.
The pgbackrest user is created to own the pgBackRest repository. Any user can own the repository but it is best not to use postgres (if it exists) to avoid confusion.
NOTE:
When pgBackRest is installed from a package, a logrotate configuration such as /etc/logrotate.d/pgbackrest may be provided that rotates the logs as a specific user via the su directive (e.g. su postgres postgres). Since the files in /var/log/pgbackrest are owned by the user that runs pgBackRest (here pgbackrest), the su directive must be updated to match this user or logrotate will fail with a permission error.
repository ⇒ Create pgbackrest user 
sudo adduser --disabled-password --gecos "" pgbackrest
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
Debian/Ubuntu packages for pgBackRest are available at apt.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
repository ⇒ Install dependencies 
sudo apt-get install postgresql-client libxml2 libssh2-1
repository ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
repository ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown pgbackrest:pgbackrest /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown pgbackrest:pgbackrest /etc/pgbackrest/pgbackrest.conf
repository ⇒ Create the pgBackRest repository 
sudo mkdir -p /var/lib/pgbackrest
sudo chmod 750 /var/lib/pgbackrest
sudo chown pgbackrest:pgbackrest /var/lib/pgbackrest
Setup Passwordless SSH
pgBackRest can use passwordless SSH to enable communication between the hosts. It is also possible to use TLS, see Setup TLS.
repository ⇒ Create repository host key pair 
sudo -u pgbackrest mkdir -m 750 /home/pgbackrest/.ssh
sudo -u pgbackrest ssh-keygen -f /home/pgbackrest/.ssh/id_rsa \
       -t rsa -b 4096 -N ""
pg-primary ⇒ Create pg-primary host key pair 
sudo -u postgres mkdir -m 750 -p /var/lib/postgresql/.ssh
sudo -u postgres ssh-keygen -f /var/lib/postgresql/.ssh/id_rsa \
       -t rsa -b 4096 -N ""
Exchange keys between repository and pg-primary.
repository ⇒ Copy pg-primary public key to repository 
(echo -n 'no-agent-forwarding,no-X11-forwarding,no-port-forwarding,' && \
       echo -n 'command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ' && \
       sudo ssh root@pg-primary cat /var/lib/postgresql/.ssh/id_rsa.pub) | \
       sudo -u pgbackrest tee -a /home/pgbackrest/.ssh/authorized_keys
pg-primary ⇒ Copy repository public key to pg-primary 
(echo -n 'no-agent-forwarding,no-X11-forwarding,no-port-forwarding,' && \
       echo -n 'command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ' && \
       sudo ssh root@repository cat /home/pgbackrest/.ssh/id_rsa.pub) | \
       sudo -u postgres tee -a /var/lib/postgresql/.ssh/authorized_keys
Test that connections can be made from repository to pg-primary and vice versa.
repository ⇒ Test connection from repository to pg-primary 
sudo -u pgbackrest ssh postgres@pg-primary
pg-primary ⇒ Test connection from pg-primary to repository 
sudo -u postgres ssh pgbackrest@repository
NOTE:
ssh has been configured to only allow pgBackRest to be run via passwordless ssh. This enhances security in the event that one of the service accounts is hijacked.
Configuration
The repository host must be configured with the pg-primary host/user and database path. The primary will be configured as pg1 to allow a standby to be added later.
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pg1-host/pg1-host-user and pg1-path 
Config suppressed for testing
The database host must be configured with the repository host/user. The default for the repo1-host-user option is pgbackrest. If the postgres user does restores on the repository host it is best not to also allow the postgres user to perform backups. However, the postgres user can read the repository directly if it is in the same group as the pgbackrest user.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-host/repo1-host-user 
Config suppressed for testing
PostgreSQL configuration may be found in the Configure Archiving section.
Commands are run the same as on a single host configuration except that some commands such as backup and expire are run from the repository host instead of the database host.
Create and Check Stanza
Create the stanza in the new repository.
repository ⇒ Create the stanza 
sudo -u pgbackrest pgbackrest --stanza=demo stanza-create
Output suppressed for testing
Check that the configuration is correct on both the database and repository hosts. More information about the check command can be found in Check the Configuration.
pg-primary ⇒ Check the configuration 
sudo -u postgres pgbackrest --stanza=demo check
Output suppressed for testing
repository ⇒ Check the configuration 
sudo -u pgbackrest pgbackrest --stanza=demo check
Output suppressed for testing
Perform a Backup
To perform a backup of the PostgreSQL cluster run pgBackRest with the backup command on the repository host.
repository ⇒ Backup the demo cluster 
sudo -u pgbackrest pgbackrest --stanza=demo backup
Output suppressed for testing
Since a new repository was created on the repository host the warning about the incremental backup changing to a full backup was emitted.
Restore a Backup
To perform a restore of the PostgreSQL cluster run pgBackRest with the restore command on the database host.
pg-primary ⇒ Stop the demo cluster, restore, and restart PostgreSQL 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres pgbackrest --stanza=demo --delta restore
sudo pg_ctlcluster 17 demo start
Parallel Backup / Restore
pgBackRest offers parallel processing to improve performance of compression and transfer. The number of processes to be used for this feature is set using the --process-max option.
It is usually best not to use more than 25% of available CPUs for the backup command. Backups don't have to run that fast as long as they are performed regularly and the backup process should not impact database performance, if at all possible.
The restore command can and should use all available CPUs because during a restore the PostgreSQL cluster is shut down and there is generally no other important work being done on the host. If the host contains multiple clusters then that should be considered when setting restore parallelism.
repository ⇒ Perform a backup with single process 
sudo -u pgbackrest pgbackrest --stanza=demo --type=full backup
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest to use multiple backup processes 
Config suppressed for testing
repository ⇒ Perform a backup with multiple processes 
sudo -u pgbackrest pgbackrest --stanza=demo --type=full backup
repository ⇒ Get backup info for the demo cluster 
sudo -u pgbackrest pgbackrest info
Output suppressed for testing
The performance of the last backup should be improved by using multiple processes. For very small backups the difference may not be very apparent, but as the size of the database increases so will time savings.
Starting and Stopping
If a standby is promoted for testing, or a test cluster is restored from a production backup, then it is a good idea to prevent those clusters from writing to pgBackRest repositories. This can be accomplished with the stop command.
The commands that write and are blocked by stop are: archive-push, backup, expire, stanza-create, and stanza-upgrade. Note that stanza-delete is an exception to this rule (see Delete a Stanza for more details).
pg-primary ⇒ Stop pgBackRest write commands 
sudo -u postgres pgbackrest stop
New pgBackRest write commands will no longer run.
repository ⇒ Attempt a backup 
sudo -u pgbackrest pgbackrest --stanza=demo backup
Output suppressed for testing
Specify the --force option to terminate any pgBackRest write commands that are currently running. This includes asynchronous archive-get (though it will run again if PostgreSQL requires it). If pgBackRest is already stopped then stopping again will generate a warning.
pg-primary ⇒ Stop the pgBackRest services again 
sudo -u postgres pgbackrest stop
Output suppressed for testing
Start pgBackRest write commands again with the start command. Write commands that were in progress before the stop will not automatically start again, but they are now allowed to start.
pg-primary ⇒ Start pgBackRest write commands 
sudo -u postgres pgbackrest start
It is also possible to stop pgBackRest for a single stanza.
pg-primary ⇒ Stop pgBackRest write commands for the demo stanza 
sudo -u postgres pgbackrest --stanza=demo stop
New pgBackRest write commands for the specified stanza will no longer run.
repository ⇒ Attempt a backup 
sudo -u pgbackrest pgbackrest --stanza=demo backup
Output suppressed for testing
The stanza must also be specified when starting pgBackRest write commands for a single stanza.
pg-primary ⇒ Start pgBackRest write commands for the demo stanza 
sudo -u postgres pgbackrest --stanza=demo start
Replication
Replication allows multiple copies of a PostgreSQL cluster (called standbys) to be created from a single primary. The standbys are useful for balancing reads and to provide redundancy in case the primary host fails.
Installation
A new host named pg-standby is created to run the standby.
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
Debian/Ubuntu packages for pgBackRest are available at apt.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
pg-standby ⇒ Install dependencies 
sudo apt-get install postgresql-client libxml2 libssh2-1
pg-standby ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
pg-standby ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown postgres:postgres /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
Setup Passwordless SSH
pgBackRest can use passwordless SSH to enable communication between the hosts. It is also possible to use TLS, see Setup TLS.
pg-standby ⇒ Create pg-standby host key pair 
sudo -u postgres mkdir -m 750 -p /var/lib/postgresql/.ssh
sudo -u postgres ssh-keygen -f /var/lib/postgresql/.ssh/id_rsa \
       -t rsa -b 4096 -N ""
Exchange keys between repository and pg-standby.
repository ⇒ Copy pg-standby public key to repository 
(echo -n 'no-agent-forwarding,no-X11-forwarding,no-port-forwarding,' && \
       echo -n 'command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ' && \
       sudo ssh root@pg-standby cat /var/lib/postgresql/.ssh/id_rsa.pub) | \
       sudo -u pgbackrest tee -a /home/pgbackrest/.ssh/authorized_keys
pg-standby ⇒ Copy repository public key to pg-standby 
(echo -n 'no-agent-forwarding,no-X11-forwarding,no-port-forwarding,' && \
       echo -n 'command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ' && \
       sudo ssh root@repository cat /home/pgbackrest/.ssh/id_rsa.pub) | \
       sudo -u postgres tee -a /var/lib/postgresql/.ssh/authorized_keys
Test that connections can be made from repository to pg-standby and vice versa.
repository ⇒ Test connection from repository to pg-standby 
sudo -u pgbackrest ssh postgres@pg-standby
pg-standby ⇒ Test connection from pg-standby to repository 
sudo -u postgres ssh pgbackrest@repository
Hot Standby
A hot standby performs replication using the WAL archive and allows read-only queries.
pgBackRest configuration is very similar to pg-primary except that the standby recovery type will be used to keep the cluster in recovery mode when the end of the WAL stream has been reached.
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest on the standby 
Config suppressed for testing
The demo cluster must be created (even though it will be overwritten on restore) in order to create the PostgreSQL configuration files.
pg-standby ⇒ Create demo cluster 
sudo pg_createcluster 17 demo
Now the standby can be created with the restore command.
IMPORTANT:
If the cluster is intended to be promoted without becoming the new primary (e.g. for reporting or testing), use --archive-mode=off or set archive_mode=off in postgresql.conf to disable archiving. If archiving is not disabled then the repository may be polluted with WAL that can make restores more difficult.
pg-standby ⇒ Restore the demo standby cluster 
sudo -u postgres pgbackrest --stanza=demo --delta --type=standby restore
sudo -u postgres cat /var/lib/postgresql/17/demo/postgresql.auto.conf
Output suppressed for testing
The configuration is in case the standby is promoted to a primary.
pg-standby:/etc/postgresql/17/demo/postgresql.conf ⇒ Configure PostgreSQL 
Config suppressed for testing
pg-standby ⇒ Start PostgreSQL 
sudo pg_ctlcluster 17 demo start
The PostgreSQL log gives valuable information about the recovery. Note especially that the cluster has entered standby mode and is ready to accept read-only connections.
pg-standby ⇒ Examine the PostgreSQL log output for log messages indicating success 
sudo -u postgres cat /var/log/postgresql/postgresql-17-demo.log
Output suppressed for testing
An easy way to test that replication is properly configured is to create a table on pg-primary.
pg-primary ⇒ Create a new table on the primary 
sudo -u postgres psql -c " \
       begin; \
       create table replicated_table (message text); \
       insert into replicated_table values ('Important Data'); \
       commit; \
       select * from replicated_table";
Output suppressed for testing
And then query the same table on pg-standby.
pg-standby ⇒ Query new table on the standby 
sudo -u postgres psql -c "select * from replicated_table;"
Output suppressed for testing
So, what went wrong? Since PostgreSQL is pulling WAL segments from the archive to perform replication, changes won't be seen on the standby until the WAL segment that contains those changes is pushed from pg-primary.
This can be done manually by calling pg_switch_wal() which pushes the current WAL segment to the archive (a new WAL segment is created to contain further changes).
pg-primary ⇒ Call pg_switch_wal() 
sudo -u postgres psql -c "select *, current_timestamp from pg_switch_wal()";
Output suppressed for testing
Now after a short delay the table will appear on pg-standby.
pg-standby ⇒ Now the new table exists on the standby (may require a few retries) 
sudo -u postgres psql -c " \
       select *, current_timestamp from replicated_table"
Output suppressed for testing
Check the standby configuration for access to the repository.
pg-standby ⇒ Check the configuration 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check
Output suppressed for testing
Streaming Replication
Instead of relying solely on the WAL archive, streaming replication makes a direct connection to the primary and applies changes as soon as they are made on the primary. This results in much less lag between the primary and standby.
Streaming replication requires a user with the replication privilege.
pg-primary ⇒ Create replication user 
sudo -u postgres psql -c " \
       create user replicator password 'jw8s0F4' replication";
Output suppressed for testing
The pg_hba.conf file must be updated to allow the standby to connect as the replication user. Be sure to replace the IP address below with the actual IP address of your pg-standby. A reload will be required after modifying the pg_hba.conf file.
pg-primary ⇒ Create pg_hba.conf entry for replication user 
sudo -u postgres sh -c 'echo \
       "host    replication     replicator      {[host-pg2-ip]}/32           md5" \
       >> /etc/postgresql/17/demo/pg_hba.conf'
sudo pg_ctlcluster 17 demo reload
The standby needs to know how to contact the primary so the primary_conninfo setting will be configured in pgBackRest.
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Set primary_conninfo 
Config suppressed for testing
It is possible to configure a password in the primary_conninfo setting but using a .pgpass file is more flexible and secure.
pg-standby ⇒ Configure the replication password in the .pgpass file. 
sudo -u postgres sh -c 'echo \
       "{[host-pg1-ip]}:*:replication:replicator:jw8s0F4" \
       >> /var/lib/postgresql/.pgpass'
sudo -u postgres chmod 600 /var/lib/postgresql/.pgpass
Now the standby can be created with the restore command.
pg-standby ⇒ Stop PostgreSQL and restore the demo standby cluster 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres pgbackrest --stanza=demo --delta --type=standby restore
sudo -u postgres cat /var/lib/postgresql/17/demo/postgresql.auto.conf
Output suppressed for testing
NOTE:
The primary_conninfo setting has been written into the postgresql.auto.conf file because it was configured as a recovery-option in pgbackrest.conf. The --type=preserve option can be used with the restore to leave the existing postgresql.auto.conf file in place if that behavior is preferred.
pg-standby ⇒ Start PostgreSQL 
sudo pg_ctlcluster 17 demo start
The PostgreSQL log will confirm that streaming replication has started.
pg-standby ⇒ Examine the PostgreSQL log output for log messages indicating success 
sudo -u postgres cat /var/log/postgresql/postgresql-17-demo.log
Output suppressed for testing
Now when a table is created on pg-primary it will appear on pg-standby quickly and without the need to call pg_switch_wal().
pg-primary ⇒ Create a new table on the primary 
sudo -u postgres psql -c " \
       begin; \
       create table stream_table (message text); \
       insert into stream_table values ('Important Data'); \
       commit; \
       select *, current_timestamp from stream_table";
Output suppressed for testing
pg-standby ⇒ Query table on the standby 
sudo -u postgres psql -c " \
       select *, current_timestamp from stream_table"
Output suppressed for testing
Multiple Stanzas
pgBackRest supports multiple stanzas. The most common usage is sharing a repository host among multiple stanzas.
Installation
A new host named pg-alt is created to run the new primary.
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
Debian/Ubuntu packages for pgBackRest are available at apt.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
pg-alt ⇒ Install dependencies 
sudo apt-get install postgresql-client libxml2 libssh2-1
pg-alt ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
pg-alt ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown postgres:postgres /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
Setup Passwordless SSH
pgBackRest can use passwordless SSH to enable communication between the hosts. It is also possible to use TLS, see Setup TLS.
pg-alt ⇒ Create pg-alt host key pair 
sudo -u postgres mkdir -m 750 -p /var/lib/postgresql/.ssh
sudo -u postgres ssh-keygen -f /var/lib/postgresql/.ssh/id_rsa \
       -t rsa -b 4096 -N ""
Exchange keys between repository and pg-alt.
repository ⇒ Copy pg-alt public key to repository 
(echo -n 'no-agent-forwarding,no-X11-forwarding,no-port-forwarding,' && \
       echo -n 'command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ' && \
       sudo ssh root@pg-alt cat /var/lib/postgresql/.ssh/id_rsa.pub) | \
       sudo -u pgbackrest tee -a /home/pgbackrest/.ssh/authorized_keys
pg-alt ⇒ Copy repository public key to pg-alt 
(echo -n 'no-agent-forwarding,no-X11-forwarding,no-port-forwarding,' && \
       echo -n 'command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ' && \
       sudo ssh root@repository cat /home/pgbackrest/.ssh/id_rsa.pub) | \
       sudo -u postgres tee -a /var/lib/postgresql/.ssh/authorized_keys
Test that connections can be made from repository to pg-alt and vice versa.
repository ⇒ Test connection from repository to pg-alt 
sudo -u pgbackrest ssh postgres@pg-alt
pg-alt ⇒ Test connection from pg-alt to repository 
sudo -u postgres ssh pgbackrest@repository
Configuration
pgBackRest configuration is nearly identical to pg-primary except that the demo-alt stanza will be used so backups and archive will be stored in a separate location.
pg-alt:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest on the new primary 
Config suppressed for testing
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pg1-host/pg1-host-user and pg1-path 
Config suppressed for testing
Setup Demo Cluster
pg-alt ⇒ Create the demo cluster 
sudo -u postgres /usr/lib/postgresql/17/bin/initdb \
       -D /var/lib/postgresql/17/demo -k -A peer
sudo pg_createcluster 17 demo
Output suppressed for testing
pg-alt:/etc/postgresql/17/demo/postgresql.conf ⇒ Configure PostgreSQL settings 
Config suppressed for testing
pg-alt ⇒ Start the demo cluster 
sudo pg_ctlcluster 17 demo restart
Create the Stanza and Check Configuration
The stanza-create command must be run to initialize the stanza. It is recommended that the check command be run after stanza-create to ensure archiving and backups are properly configured.
pg-alt ⇒ Create the stanza and check the configuration 
sudo -u postgres pgbackrest --stanza=demo-alt --log-level-console=info stanza-create
Output suppressed for testing
sudo -u postgres pgbackrest --log-level-console=info check
Output suppressed for testing
If the check command is run from the repository host then all stanzas will be checked.
repository ⇒ Check the configuration for all stanzas 
sudo -u pgbackrest pgbackrest --log-level-console=info check
Output suppressed for testing
Asynchronous Archiving
Asynchronous archiving is enabled with the archive-async option. This option enables asynchronous operation for both the archive-push and archive-get commands.
A spool path is required. The commands will store transient data here but each command works quite a bit differently so spool path usage is described in detail in each section.
pg-primary ⇒ Create the spool directory 
sudo mkdir -p -m 750 /var/spool/pgbackrest
sudo chown postgres:postgres /var/spool/pgbackrest
pg-standby ⇒ Create the spool directory 
sudo mkdir -p -m 750 /var/spool/pgbackrest
sudo chown postgres:postgres /var/spool/pgbackrest
The spool path must be configured and asynchronous archiving enabled. Asynchronous archiving automatically confers some benefit by reducing the number of connections made to remote storage, but setting process-max can drastically improve performance by parallelizing operations. Be sure not to set process-max so high that it affects normal database operations.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the spool path and asynchronous archiving 
Config suppressed for testing
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the spool path and asynchronous archiving 
Config suppressed for testing
NOTE:
process-max is configured using command sections so that the option is not used by backup and restore. This also allows different values for archive-push and archive-get.
For demonstration purposes streaming replication will be broken to force PostgreSQL to get WAL using the restore_command.
pg-primary ⇒ Break streaming replication by changing the replication password 
sudo -u postgres psql -c "alter user replicator password 'bogus'"
Output suppressed for testing
pg-standby ⇒ Restart standby to break connection 
sudo pg_ctlcluster 17 demo restart
Archive Push
The asynchronous archive-push command offloads WAL archiving to a separate process (or processes) to improve throughput. It works by looking ahead to see which WAL segments are ready to be archived beyond the request that PostgreSQL is currently making via the archive_command. WAL segments are transferred to the archive directly from the pg_xlog/pg_wal directory and success is only returned by the archive_command when the WAL segment has been safely stored in the archive.
The spool path holds the current status of WAL archiving. Status files written into the spool directory are typically zero length and should consume a minimal amount of space (a few MB at most) and very little IO. All the information in this directory can be recreated so it is not necessary to preserve the spool directory if the cluster is moved to new hardware.
IMPORTANT:
In the original implementation of asynchronous archiving, WAL segments were copied to the spool directory before compression and transfer. The new implementation copies WAL directly from the pg_xlog directory. If asynchronous archiving was utilized in v1.12 or prior, read the v1.13 release notes carefully before upgrading.
The [stanza]-archive-push-async.log file can be used to monitor the activity of the asynchronous process. A good way to test this is to quickly push a number of WAL segments.
pg-primary ⇒ Test parallel asynchronous archiving 
sudo -u postgres psql -c " \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal();"
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check
Output suppressed for testing
Now the log file will contain parallel, asynchronous activity.
pg-primary ⇒ Check results in the log 
sudo -u postgres cat /var/log/pgbackrest/demo-archive-push-async.log
Output suppressed for testing
Archive Get
The asynchronous archive-get command maintains a local queue of WAL to improve throughput. If a WAL segment is not found in the queue it is fetched from the repository along with enough consecutive WAL to fill the queue. The maximum size of the queue is defined by archive-get-queue-max. Whenever the queue is less than half full more WAL will be fetched to fill it.
Asynchronous operation is most useful in environments that generate a lot of WAL or have a high latency connection to the repository storage (i.e., S3 or other object stores). In the case of a high latency connection it may be a good idea to increase process-max.
The [stanza]-archive-get-async.log file can be used to monitor the activity of the asynchronous process.
pg-standby ⇒ Check results in the log 
sudo -u postgres cat /var/log/pgbackrest/demo-archive-get-async.log
Output suppressed for testing
pg-primary ⇒ Fix streaming replication by changing the replication password 
sudo -u postgres psql -c "alter user replicator password 'jw8s0F4'"
Output suppressed for testing
Backup from a Standby
pgBackRest can perform backups on a standby instead of the primary. Standby backups require the pg-standby host to be configured and the backup-standby option enabled. If more than one standby is configured then the first running standby found will be used for the backup.
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pg2-host/pg2-host-user and pg2-path 
Config suppressed for testing
Both the primary and standby databases are required to perform the backup, though the vast majority of the files will be copied from the standby to reduce load on the primary. The database hosts can be configured in any order. pgBackRest will automatically determine which is the primary and which is the standby.
repository ⇒ Backup the demo cluster from pg2 
sudo -u pgbackrest pgbackrest --stanza=demo --log-level-console=detail backup
Output suppressed for testing
This incremental backup shows that most of the files are copied from the pg-standby host and only a few are copied from the pg-primary host.
pgBackRest creates a standby backup that is identical to a backup performed on the primary. It does this by starting/stopping the backup on the pg-primary host, copying only files that are replicated from the pg-standby host, then copying the remaining few files from the pg-primary host. This means that logs and statistics from the primary database will be included in the backup.
Upgrading PostgreSQL
Immediately after upgrading PostgreSQL to a newer major version, the pg-path for all pgBackRest configurations must be set to the new database location and the stanza-upgrade command run. If there is more than one repository configured on the host, the stanza will be upgraded on each. If the database is offline use the --no-online option.
The following instructions are not meant to be a comprehensive guide for upgrading PostgreSQL, rather they outline the general process for upgrading a primary and standby with the intent of demonstrating the steps required to reconfigure pgBackRest. It is recommended that a backup be taken prior to upgrading.
pg-primary ⇒ Stop old cluster 
sudo pg_ctlcluster 17 demo stop
Stop the old cluster on the standby since it will be restored from the newly upgraded cluster.
pg-standby ⇒ Stop old cluster 
sudo pg_ctlcluster 17 demo stop
Create the new cluster and perform upgrade.
pg-primary ⇒ Create new cluster and perform the upgrade 
sudo -u postgres /usr/lib/postgresql/18/bin/initdb \
       -D /var/lib/postgresql/18/demo -k -A peer
sudo pg_createcluster 18 demo
sudo -u postgres sh -c 'cd /var/lib/postgresql && \
       /usr/lib/postgresql/18/bin/pg_upgrade \
       --old-bindir=/usr/lib/postgresql/17/bin \
       --new-bindir=/usr/lib/postgresql/18/bin \
       --old-datadir=/var/lib/postgresql/17/demo \
       --new-datadir=/var/lib/postgresql/18/demo \
       --old-options=" -c config_file=/etc/postgresql/17/demo/postgresql.conf" \
       --new-options=" -c config_file=/etc/postgresql/18/demo/postgresql.conf"'
Output suppressed for testing
Configure the new cluster settings and port.
pg-primary:/etc/postgresql/18/demo/postgresql.conf ⇒ Configure PostgreSQL 
Config suppressed for testing
Update the pgBackRest configuration on all systems to point to the new cluster.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Upgrade the pg1-path 
Config suppressed for testing
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Upgrade the pg-path 
Config suppressed for testing
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Upgrade pg1-path and pg2-path, disable backup from standby 
Config suppressed for testing
pg-primary ⇒ Copy hba configuration 
sudo cp /etc/postgresql/17/demo/pg_hba.conf \
       /etc/postgresql/18/demo/pg_hba.conf
Before starting the new cluster, the stanza-upgrade command must be run.
pg-primary ⇒ Upgrade the stanza 
sudo -u postgres pgbackrest --stanza=demo --no-online \
       --log-level-console=info stanza-upgrade
Output suppressed for testing
Start the new cluster and confirm it is successfully installed.
pg-primary ⇒ Start new cluster 
sudo pg_ctlcluster 18 demo start
Output suppressed for testing
Test configuration using the check command.
pg-primary ⇒ Check configuration 
sudo pg_lsclusters
sudo -u postgres pgbackrest --stanza=demo check
Output suppressed for testing
Remove the old cluster.
pg-primary ⇒ Remove old cluster 
sudo pg_dropcluster 17 demo
Install the new PostgreSQL binaries on the standby and create the cluster.
pg-standby ⇒ Remove old cluster and create the new cluster 
sudo pg_dropcluster 17 demo
sudo pg_createcluster 18 demo
Run the check on the repository host. The warning regarding the standby being down is expected since the standby cluster is down. Running this command demonstrates that the repository server is aware of the standby and is configured properly for the primary server.
repository ⇒ Check configuration 
sudo -u pgbackrest pgbackrest --stanza=demo check
Output suppressed for testing
Run a full backup on the new cluster and then restore the standby from the backup. The backup type will automatically be changed to full if incr or diff is requested.
repository ⇒ Run a full backup 
sudo -u pgbackrest pgbackrest --stanza=demo --type=full backup
pg-standby ⇒ Restore the demo standby cluster 
sudo -u postgres pgbackrest --stanza=demo --delta --type=standby restore
pg-standby ⇒ Start PostgreSQL and check the pgBackRest configuration 
sudo pg_ctlcluster 18 demo start
sudo -u postgres pgbackrest --stanza=demo check
Output suppressed for testing
Backup from standby can be enabled now that the standby is restored.
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Reenable backup from standby 
Config suppressed for testing
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated October 18, 2026
././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7761402 pgbackrest-2.60.0/doc/man/0000750000175100017510000000000015265102133014703 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7761402 pgbackrest-2.60.0/doc/man/pgbackrest.1.txt0000644000175100017510000004305515265102133017744 0ustar00runnerrunnerNAME pgBackRest - Reliable PostgreSQL Backup & Restore SYNOPSIS pgbackrest [options] [command] DESCRIPTION pgBackRest is a reliable backup and restore solution for PostgreSQL that seamlessly scales up to the largest databases and workloads. COMMANDS annotate Add or modify backup annotation. archive-get Get a WAL segment from the archive. archive-push Push a WAL segment to the archive. backup Backup a database cluster. check Check the configuration. expire Expire backups that exceed retention. help Get help. info Retrieve information about backups. repo-get Get a file from a repository. repo-ls List files in a repository. restore Restore a database cluster. server pgBackRest server. server-ping Ping pgBackRest server. stanza-create Create the required stanza data. stanza-delete Delete a stanza. stanza-upgrade Upgrade a stanza. start Allow pgBackRest processes to run. stop Stop pgBackRest processes from running. verify Verify contents of a repository. version Get version. OPTIONS Annotate Options: --set Backup set to annotate. Archive Options: --archive-async Push/get WAL segments asynchronously. --archive-get-queue-max Maximum size of the pgBackRest archive-get queue. --archive-missing-retry Retry missing WAL segment --archive-push-batch-size Maximum amount of WAL to push per asynchronous run. --archive-push-queue-max Maximum size of the PostgreSQL archive queue. --archive-timeout Archive timeout. Backup Options: --annotation Annotate backup with user-defined key/value pairs. --archive-check Check that WAL segments are in the archive before backup completes. --archive-copy Copy WAL segments needed for consistency to the backup. --archive-mode-check Check the PostgreSQL archive_mode setting. --backup-standby Backup from the standby cluster. --checksum-page Validate data page checksums. --checksum-page-error Error on invalid page checksums. --exclude Exclude paths/files from the backup. --expire-auto Automatically run the expire command after a successful backup. --force Force an offline backup. --manifest-save-threshold Manifest save threshold during backup. --online Perform an online backup. --resume Allow resume of failed backup. --start-fast Force a checkpoint to start backup quickly. --type Backup type. Expire Options: --archive-expire-before Remove WAL archive earlier than the specified WAL segment. --checksum-page-error Error on invalid page checksums. --oldest Expire the oldest eligible backup set. --set Backup set to expire. General Options: --allow-root Allow the command to run as the root user. --buffer-size Buffer size for I/O operations. --cmd pgBackRest command. --cmd-ssh SSH client command. --compress Use file compression. --compress-level File compression level. --compress-level-network Network compression level. --compress-type File compression type. --config pgBackRest configuration file. --config-include-path Path to additional pgBackRest configuration files. --config-path Base path of pgBackRest configuration files. --db-timeout Database query timeout. --delta Restore or backup using checksums. --dry-run Execute a dry-run for the command. --io-timeout I/O timeout. --lock-path Path where lock files are stored. --neutral-umask Use a neutral umask. --priority Set process priority. --process-max Max processes to use for compress/transfer. --protocol-timeout Protocol timeout. --raw Do not transform data. --sck-keep-alive Keep-alive enable. --spool-path Path where transient data is stored. --stanza Defines the stanza. --tcp-keep-alive-count Keep-alive count. --tcp-keep-alive-idle Keep-alive idle time. --tcp-keep-alive-interval Keep-alive interval time. --tls-cipher-12 Allowed TLSv1.2 cipher suites. --tls-cipher-13 Allowed TLSv1.3 cipher suites. Help Options: --help Display help. --version Display version. Info Options: --detail-level Output detail level. --output Output format. --set Backup set to detail. --type Filter on backup type. Log Options: --log-level-console Level for console logging. --log-level-file Level for file logging. --log-level-stderr Level for stderr logging. --log-path Path where log files are stored. --log-subprocess Enable logging in subprocesses. --log-timestamp Enable timestamp in logging. Maintainer Options: --archive-header-check Check PostgreSQL version/id in WAL headers. --page-header-check Check PostgreSQL page headers. --pg-version-force Force PostgreSQL version. Repo-get Options: --ignore-missing Ignore missing source file. Repo-ls Options: --filter Filter output with a regular expression. --output Output format. --recurse Include all subpaths in output. --sort Sort output ascending, descending, or none. Repository Options: --repo Set repository. --repo-azure-account Azure repository account. --repo-azure-container Azure repository container. --repo-azure-endpoint Azure repository endpoint. --repo-azure-key Azure repository key. --repo-azure-key-type Azure repository key type. --repo-azure-uri-style Azure URI Style. --repo-block Enable block incremental backup. --repo-bundle Bundle files in repository. --repo-bundle-limit Limit for file bundles. --repo-bundle-size Target size for file bundles. --repo-cipher-pass Repository cipher passphrase. --repo-cipher-type Cipher used to encrypt the repository. --repo-gcs-bucket GCS repository bucket. --repo-gcs-endpoint GCS repository endpoint. --repo-gcs-key GCS repository key. --repo-gcs-key-type GCS repository key type. --repo-gcs-user-project GCS project ID. --repo-hardlink Hardlink files between backups in the repository. --repo-host Repository host when operating remotely. --repo-host-ca-file Repository host certificate authority file. --repo-host-ca-path Repository host certificate authority path. --repo-host-cert-file Repository host certificate file. --repo-host-cmd Repository host pgBackRest command. --repo-host-config pgBackRest repository host configuration file. --repo-host-config-include-path pgBackRest repository host configuration include path. --repo-host-config-path pgBackRest repository host configuration path. --repo-host-key-file Repository host key file. --repo-host-port Repository host port when repo-host is set. --repo-host-type Repository host protocol type. --repo-host-user Repository host user when repo-host is set. --repo-path Path where backups and archive are stored. --repo-retention-archive Number of backups worth of continuous WAL to retain. --repo-retention-archive-type Backup type for WAL retention. --repo-retention-diff Number of differential backups to retain. --repo-retention-full Full backup retention count/time. --repo-retention-full-type Retention type for full backups. --repo-retention-history Days of backup history manifests to retain. --repo-s3-bucket S3 repository bucket. --repo-s3-endpoint S3 repository endpoint. --repo-s3-key S3 repository access key. --repo-s3-key-secret S3 repository secret access key. --repo-s3-key-type S3 repository key type. --repo-s3-kms-key-id S3 repository KMS key. --repo-s3-process-cmd S3 authentication process command. --repo-s3-region S3 repository region. --repo-s3-requester-pays S3 repository requester pays. --repo-s3-role S3 repository role. --repo-s3-service S3 signing service. --repo-s3-sse-customer-key S3 repository SSE customer key. --repo-s3-sts-host S3 repository STS endpoint. --repo-s3-token S3 repository security token. --repo-s3-uri-style S3 URI Style. --repo-sftp-host SFTP repository host. --repo-sftp-host-fingerprint SFTP repository host fingerprint. --repo-sftp-host-key-check-type SFTP host key check type. --repo-sftp-host-key-hash-type SFTP repository host key hash type. --repo-sftp-host-port SFTP repository host port. --repo-sftp-host-user SFTP repository host user. --repo-sftp-known-host SFTP known hosts file. --repo-sftp-private-key-file SFTP private key file. --repo-sftp-private-key-passphrase SFTP private key passphrase. --repo-sftp-public-key-file SFTP public key file. --repo-storage-ca-file Repository storage CA file. --repo-storage-ca-path Repository storage CA path. --repo-storage-host Repository storage host. --repo-storage-port Repository storage port. --repo-storage-prefetch Repository storage prefetch. --repo-storage-read-over Repository storage read over. --repo-storage-tag Repository storage tag(s). --repo-storage-upload-chunk-size Repository storage upload chunk size. --repo-storage-verify-tls Repository storage certificate verify. --repo-symlink Create symlinks within the repository. --repo-target-time Target time for repository. --repo-type Type of storage used for the repository. Restore Options: --archive-mode Preserve or disable archiving on restored cluster. --checksum-page-error Error on invalid page checksums. --db-exclude Restore excluding the specified databases. --db-include Restore only specified databases. --force Force a restore. --link-all Restore all symlinks. --link-map Modify the destination of a symlink. --recovery-option Set an option in postgresql.auto.conf or recovery.conf. --set Backup set to restore. --tablespace-map Restore a tablespace into the specified directory. --tablespace-map-all Restore all tablespaces into the specified directory. --target Recovery target. --target-action Action to take when recovery target is reached. --target-exclusive Stop just before the recovery target is reached. --target-timeline Recover along a timeline. --type Recovery type. Server Options: --tls-server-address TLS server address. --tls-server-auth TLS server authorized clients. --tls-server-ca-file TLS server certificate authorities. --tls-server-cert-file TLS server certificate file. --tls-server-key-file TLS server key file. --tls-server-port TLS server port. Stanza Options: --pg-database PostgreSQL database. --pg-host PostgreSQL host for operating remotely. --pg-host-ca-file PostgreSQL host certificate authority file. --pg-host-ca-path PostgreSQL host certificate authority path. --pg-host-cert-file PostgreSQL host certificate file. --pg-host-cmd PostgreSQL host pgBackRest command. --pg-host-config pgBackRest database host configuration file. --pg-host-config-include-path pgBackRest database host configuration include path. --pg-host-config-path pgBackRest database host configuration path. --pg-host-key-file PostgreSQL host key file. --pg-host-port PostgreSQL host port when pg-host is set. --pg-host-type PostgreSQL host protocol type. --pg-host-user PostgreSQL host logon user when pg-host is set. --pg-path PostgreSQL data directory. --pg-port PostgreSQL port. --pg-socket-path PostgreSQL unix socket path. --pg-user PostgreSQL database user. Stanza-create Options: --online Create on an online cluster. Stanza-delete Options: --force Force stanza delete. Stanza-upgrade Options: --online Update an online cluster. Stop Options: --force Force all pgBackRest processes to stop. Verify Options: --output Output type. --set Backup set to verify. --verbose Verbose output. Version Options: --output Output type. FILES /etc/pgbackrest/pgbackrest.conf /var/lib/pgbackrest /var/log/pgbackrest /var/spool/pgbackrest /tmp/pgbackrest EXAMPLES * Create a backup of the PostgreSQL `main` cluster: $ pgbackrest --stanza=main backup The `main` cluster should be configured in `/etc/pgbackrest/pgbackrest.conf` * Show all available backups: $ pgbackrest info * Show all available backups for a specific cluster: $ pgbackrest --stanza=main info * Show backup specific options: $ pgbackrest help backup SEE ALSO /usr/share/doc/pgbackrest-doc/html/index.html https://pgbackrest.org ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/meson.build0000664000175100017510000003141215265101745015545 0ustar00runnerrunner#################################################################################################################################### # pgBackRest Project #################################################################################################################################### project( 'pgbackrest', ['c'], version: '2.60.0', license: 'MIT', meson_version: '>=0.55.3', default_options: [ # Core options 'buildtype=release', 'warning_level=2', # Base options 'b_ndebug=if-release', # Perform unity builds in a single file 'unity_size=100000', # Compiler options 'c_std=c11', ], ) # Selected C compiler cc = meson.get_compiler('c') #################################################################################################################################### # Check for C11 compiler support (setting c_std has no effect -- works in 1.8.1 so likely a bug in our version) #################################################################################################################################### c11_test = ''' #if !defined __STDC_VERSION__ || __STDC_VERSION__ < 201112L # error "C compiler does not advertise C11 conformance" #endif ''' if not cc.compiles(c11_test) if cc.compiles(c11_test, args: '-std=c11') add_project_arguments('-std=c11', language: 'c') else error('C compiler does not support C11') endif endif #################################################################################################################################### # OS-specific settings #################################################################################################################################### if host_machine.system() == 'linux' add_global_arguments('-D_POSIX_C_SOURCE=200809L', language: 'c') elif host_machine.system() == 'darwin' add_global_arguments('-D_DARWIN_C_SOURCE', language: 'c') elif host_machine.system() == 'sunos' add_global_arguments('-D_POSIX_C_SOURCE=200809L', '-D__EXTENSIONS__', language: 'c') endif #################################################################################################################################### # Enable/disable warnings #################################################################################################################################### # Enable as many additional warnings as possible to catch potential errors warning_enable = [ # Warn whenever a pointer is cast so as to remove a type qualifier from the target type '-Wcast-qual', # Warn for implicit conversions that may alter a value '-Wconversion', # Warn about duplicated conditions in an if-else-if chain '-Wduplicated-cond', # Warn when an if-else has identical branches '-Wduplicated-branches', # Warn if floating-point values are used in equality comparisons '-Wfloat-equal', # Enable -Wformat plus -Wnonnull, -Wformat-nonliteral, -Wformat-security, and -Wformat-y2k '-Wformat=2', # Warn about calls to formatted input/output functions such as sprintf and vsprintf that might overflow the destination buffer '-Wformat-overflow=2', # Warn if the format string requires an unsigned argument and the argument is signed and vice versa '-Wformat-signedness', # Warn about uninitialized variables which are initialized with themselves '-Winit-self', # Warn if a global function is defined without a previous prototype declaration '-Wmissing-prototypes', # Warn if a global variable is defined without a previous declaration '-Wmissing-variable-declarations', # Warn about anything that depends on the "size of" a function type or of void '-Wpointer-arith', # Warn if anything is declared more than once in the same scope, even when the extra declaration is valid and changes nothing '-Wredundant-decls', # Warn if a function is declared or defined without specifying the argument types '-Wstrict-prototypes', # Warn if a variable-length array is used '-Wvla', # Give string constants the type const char[length] so that copying the address of one into a non-const char * pointer produces # a warning '-Wwrite-strings', ] # Disable various unhelpful warnings warning_disable = [ # Warn for variables that might be changed by longjmp or vfork. Disable because of constant false positives/negatives. '-Wno-clobbered', # Warn if a structure's initializer has some fields missing. Disable so we can initialize with {0}. '-Wno-missing-field-initializers', ] add_project_arguments(cc.get_supported_arguments(warning_enable, warning_disable), language: 'c') #################################################################################################################################### # Enable additional optimizations for appropriate levels #################################################################################################################################### if get_option('optimization') in ['2', '3'] optimization_enable = [ # Unroll loops whose number of iterations can be determined at compile time or upon entry to the loop '-funroll-loops', # Perform loop vectorization on trees '-ftree-vectorize', ] add_project_arguments(cc.get_supported_arguments(optimization_enable), language: 'c') endif #################################################################################################################################### # Stop after the first error when error on warn enabled. Subsequent errors are often caused by the first error. #################################################################################################################################### if get_option('fatal-errors') add_project_arguments(cc.get_supported_arguments('-Wfatal-errors'), language: 'c') endif #################################################################################################################################### # Generate -fmacro-prefix-map so the relative path to the source directory is not included in the __FILE__ macro. This provides # reproducible builds and minimizes the file path in debug messages, just like an in-tree make build. For test source, prefix with # test/ in case there are any module name collisions. #################################################################################################################################### python = import('python').find_installation() file_prefix = run_command( [ python, '-c', 'import sys, os; print(os.path.relpath(sys.argv[1], sys.argv[2]))', meson.current_source_dir(), meson.current_build_dir(), ], check: true, ).stdout().strip() add_project_arguments(cc.get_supported_arguments('-fmacro-prefix-map=@0@/src/=' . format(file_prefix)), language: 'c') add_project_arguments(cc.get_supported_arguments('-fmacro-prefix-map=@0@/test/src/=test/' . format(file_prefix)), language: 'c') #################################################################################################################################### # Detect a distribution tarball, which is marked by the dist file written into the tarball root by meson dist. A distribution # tarball ships the generated files, so the code generation tooling (build.py, the doc and test tools) is not used. A normal source # tree such as a git checkout has no marker and generates the files. #################################################################################################################################### dist = run_command( [ python, '-c', 'import os, sys; sys.exit(0 if os.path.exists(os.path.join(sys.argv[1], "dist")) else 1)', meson.current_source_dir(), ], check: false, ).returncode() == 0 #################################################################################################################################### # Build configuration #################################################################################################################################### configuration = configuration_data() # Ensure the C compiler supports __builtin_clzl() if not cc.links('''int main(void) {__builtin_clzl(1);}''') error('compiler must support __builtin_clzl()') endif # Ensure the C compiler supports __builtin_bswap64() if not cc.links('''int main(void) {__builtin_bswap64(1);}''') error('compiler must support __builtin_bswap64()') endif # Check C compiler support for __builtin_types_compatible_p if cc.compiles('''int main(void) {__builtin_types_compatible_p(char *, const char *);}''') configuration.set('HAVE_BUILTIN_TYPES_COMPATIBLE_P', true, description: 'Is __builtin_types_compatible_p present?') endif # Check C compiler support for __builtin_expect() if cc.links('''int main(void) {__builtin_expect(0, 0);}''') configuration.set('HAVE_BUILTIN_EXPECT', true, description: 'Is __builtin_expect present?') endif # Find optional backtrace library lib_backtrace = cc.find_library('backtrace', required: false, has_headers: 'backtrace.h') if lib_backtrace.found() configuration.set('HAVE_LIBBACKTRACE', true, description: 'Is libbacktrace present?') endif # Find required bz2 library lib_bz2 = cc.find_library('bz2') # Find required lz4 library lib_lz4 = dependency('liblz4') # Find required openssl library lib_openssl = dependency('openssl', version : '>=1.1.1') # Find required pq library lib_pq = dependency('libpq') # Find required xml library lib_xml = dependency('libxml-2.0') # Find required gz library lib_z = dependency('zlib') configuration.set('ZLIB_CONST', true, description: 'Require zlib const input buffer') # Find socket library on illumos/Solaris, where sockets are not in libc lib_socket = dependency('', required: false) if host_machine.system() == 'sunos' lib_socket = cc.find_library('socket') endif # Find optional systemd library lib_systemd = dependency('libsystemd', required: get_option('libsystemd')) if lib_systemd.found() configuration.set('HAVE_LIBSYSTEMD', true, description: 'Is libsystemd present?') endif # Find optional libssh2 library lib_ssh2 = dependency('libssh2', required: get_option('libssh2')) if lib_ssh2.found() configuration.set('HAVE_LIBSSH2', true, description: 'Is libssh2 present?') endif # Find optional zstd library lib_zstd = dependency('libzstd', version: '>=1.0', required: get_option('libzstd')) if lib_zstd.found() configuration.set('HAVE_LIBZST', true, description: 'Is libzstd present?') endif # Suppress -Wsign-conversion for C libraries (e.g. musl) that do no accept int without warning for FD_*() macros if not cc.compiles( '''#include int main(void) {int fd = 1; fd_set s; FD_ZERO(&s); FD_SET(fd, &s); FD_ISSET(fd, &s);}''', args: ['-Wsign-conversion', '-Werror']) message('Disabling -Wsign-conversion because int is not accepted without warning by FD_*() macros') add_project_arguments(cc.get_supported_arguments('-Wno-sign-conversion'), language: 'c') endif # Enable debug code if get_option('debug') configuration.set('DEBUG', true, description: 'Enable debug code') # Enable Valgrind memcheck client requests when the header is available so allocators that retain freed memory for reuse keep # use-after-free and uninitialized-read detection accurate under Valgrind. The requests are no-ops when not run under Valgrind. if cc.has_header('valgrind/memcheck.h') add_project_arguments('-DWITH_VALGRIND', language: 'c') endif endif # Set configuration path configuration.set_quoted('CFGOPTDEF_CONFIG_PATH', get_option('configdir'), description: 'Configuration path') # Set VR_NON_STRING macro if cc.compiles( '''int main(void) {__attribute__((nonstring)) static char test[3] = "ABC";}''', args: ['-Wattributes', '-Werror']) configuration.set( 'VR_NON_STRING', '__attribute__((nonstring))', description: 'Indicate that the char array is intentionally not NULL terminated') else configuration.set( 'VR_NON_STRING', '', description: 'Indicate that the char array is intentionally not NULL terminated') endif #################################################################################################################################### # Include subdirs #################################################################################################################################### subdir('src') # The smoke test is shipped in the distribution so packagers can verify a build. It is in a subdir because unit test builds comment # out all subdirs, which excludes the smoke test from them. subdir('test') # A distribution tarball ships the generated files, so it neither generates them nor bakes them in if not dist # Bake the generated files and the dist marker into the tarball so it builds without the code generation tooling meson.add_dist_script('build/dist.sh') endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/meson_options.txt0000664000175100017510000000117115265101745017037 0ustar00runnerrunneroption('configdir', type: 'string', value: '/etc/pgbackrest', description: 'Configuration directory') option('fatal-errors', type: 'boolean', value: false, description: 'Stop compilation on first error') option('fuzz-engine', type: 'string', value: '', description: 'Link args of the fuzzing engine for fuzz targets (empty to disable)') option('libssh2', type: 'feature', value: 'auto', description: 'Enable SFTP storage support') option('libzstd', type: 'feature', value: 'auto', description: 'Enable Zstandard compression support') option('libsystemd', type: 'feature', value: 'auto', description: 'Enable systemd notify support') ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/0000775000175100017510000000000015265101745014171 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/.gitignore0000664000175100017510000000023715265101745016163 0ustar00runnerrunner/.build autom4te.cache /command/help/help.auto.c.inc /config.log /config.status /Makefile /build.auto.h /pgbackrest /postgres/interface.auto.c.inc /build-code ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/build.h0000664000175100017510000000474715265101745015455 0ustar00runnerrunner/*********************************************************************************************************************************** Build Header This header file must be included in all source files. It contains defines that are required in all modules. The build.auto.c header generated by meson is also included here. ***********************************************************************************************************************************/ #ifndef BUILD_H #define BUILD_H /*********************************************************************************************************************************** Ban unsafe functions Poison unsafe functions so any use is a compile error. System headers containing poisoned identifiers must be processed first because build.h is the first include in every file and poisoned identifiers are not exempt inside system headers. #undef prevents a "poisoning existing macro" warning where fortify has defined the function as a macro. ***********************************************************************************************************************************/ #include #include #undef sprintf #undef strcat #undef strcpy #undef strncpy #pragma GCC poison sprintf strcat strcpy strncpy /*********************************************************************************************************************************** Include build.auto.h and other (hopefully small) header files that are commonly used. ***********************************************************************************************************************************/ #include #include /*********************************************************************************************************************************** Function attributes ***********************************************************************************************************************************/ // Indicate that a function should always be inlined #define FN_INLINE_ALWAYS __attribute__((__always_inline__)) static inline // Indicate that a function is formatted like printf (and provide format and args position) #define FN_PRINTF(fmt, args) __attribute__((format(printf, fmt, args))) // Indicate that a function is formatted like strftime (and provide format position) #define FN_STRFTIME(fmt) __attribute__((format(strftime, fmt, 0))) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/0000775000175100017510000000000015265101745015607 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/annotate/0000775000175100017510000000000015265101745017420 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/annotate/annotate.c0000664000175100017510000000731015265101745021376 0ustar00runnerrunner/*********************************************************************************************************************************** Annotate Command ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/annotate/annotate.h" #include "command/backup/backup.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "config/config.h" #include "info/infoBackup.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdAnnotate(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { // Initialize the repo index unsigned int repoIdxMin = 0; unsigned int repoIdxMax = cfgOptionGroupIdxTotal(cfgOptGrpRepo) - 1; // If the repo was specified then set index to the array location and max to loop only once if (cfgOptionTest(cfgOptRepo)) { repoIdxMin = cfgOptionGroupIdxDefault(cfgOptGrpRepo); repoIdxMax = repoIdxMin; } // Check the backup label format const String *const backupLabel = cfgOptionStr(cfgOptSet); if (!regExpMatchOne(backupRegExpP(.full = true, .differential = true, .incremental = true), backupLabel)) THROW_FMT(OptionInvalidValueError, "'%s' is not a valid backup label format", strZ(backupLabel)); // Track the number of backup sets to update in the backup info file and any errors that may occur unsigned int backupTotalProcessed = 0; unsigned int errorTotal = 0; for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { TRY_BEGIN() { // Get the repo storage in case it is remote and encryption settings need to be pulled down storageRepoIdx(repoIdx); // Attempt to load the backup info file InfoBackup *infoBackup = infoBackupLoadFileReconstruct( storageRepoIdx(repoIdx), INFO_BACKUP_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); if (infoBackupLabelExists(infoBackup, backupLabel)) { // Backup label found in backup.info backupTotalProcessed++; LOG_INFO_FMT( "backup set '%s' to annotate found in %s", strZ(backupLabel), cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); // Update annotations infoBackupDataAnnotationSet(infoBackup, backupLabel, cfgOptionKv(cfgOptAnnotation)); // Write the updated backup info infoBackupSaveFile(infoBackup, storageRepoWrite(), INFO_BACKUP_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); } } CATCH_ANY() { LOG_ERROR_FMT(errorTypeCode(errorType()), "%s: %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorMessage()); errorTotal++; } TRY_END(); } // Error if any errors encountered on one or more repos if (errorTotal > 0) THROW_FMT(CommandError, CFGCMD_ANNOTATE " command encountered %u error(s), check the log file for details", errorTotal); if (backupTotalProcessed == 0) THROW(BackupSetInvalidError, "no backup set to annotate found"); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/annotate/annotate.h0000664000175100017510000000130615265101745021402 0ustar00runnerrunner/*********************************************************************************************************************************** Annotate Command ***********************************************************************************************************************************/ #ifndef COMMAND_ANNOTATE_ANNOTATE_H #define COMMAND_ANNOTATE_ANNOTATE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add or modify backup annotations FN_EXTERN void cmdAnnotate(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/0000775000175100017510000000000015265101745017230 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/common.c0000664000175100017510000004020415265101745020664 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Common ***********************************************************************************************************************************/ #include #include #include #include "command/archive/common.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/type/convert.h" #include "common/wait.h" #include "config/config.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/helper.h" /*********************************************************************************************************************************** WAL segment constants ***********************************************************************************************************************************/ STRING_EXTERN(WAL_SEGMENT_REGEXP_STR, WAL_SEGMENT_REGEXP); STRING_EXTERN(WAL_SEGMENT_PARTIAL_REGEXP_STR, WAL_SEGMENT_PARTIAL_REGEXP); STRING_EXTERN(WAL_SEGMENT_DIR_REGEXP_STR, WAL_SEGMENT_DIR_REGEXP); STRING_EXTERN(WAL_SEGMENT_FILE_REGEXP_STR, WAL_SEGMENT_FILE_REGEXP); STRING_EXTERN(WAL_TIMELINE_HISTORY_REGEXP_STR, WAL_TIMELINE_HISTORY_REGEXP); /*********************************************************************************************************************************** Global error file constant ***********************************************************************************************************************************/ #define STATUS_FILE_GLOBAL "global" STRING_STATIC(STATUS_FILE_GLOBAL_STR, STATUS_FILE_GLOBAL); #define STATUS_FILE_GLOBAL_ERROR STATUS_FILE_GLOBAL STATUS_EXT_ERROR STRING_STATIC(STATUS_FILE_GLOBAL_ERROR_STR, STATUS_FILE_GLOBAL_ERROR); /*********************************************************************************************************************************** Get the correct spool queue based on the archive mode ***********************************************************************************************************************************/ static const String * archiveAsyncSpoolQueue(const ArchiveMode archiveMode) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, archiveMode); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST( STRING, archiveMode == archiveModeGet ? STORAGE_SPOOL_ARCHIVE_IN_STR : STORAGE_SPOOL_ARCHIVE_OUT_STR); } /**********************************************************************************************************************************/ FN_EXTERN void archiveAsyncErrorClear(const ArchiveMode archiveMode, const String *const archiveFile) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, archiveMode); FUNCTION_LOG_PARAM(STRING, archiveFile); FUNCTION_LOG_END(); ASSERT(archiveFile != NULL); MEM_CONTEXT_TEMP_BEGIN() { const String *const spoolQueue = archiveAsyncSpoolQueue(archiveMode); storageRemoveP(storageSpoolWrite(), strNewFmt("%s/%s" STATUS_EXT_ERROR, strZ(spoolQueue), strZ(archiveFile))); storageRemoveP(storageSpoolWrite(), strNewFmt("%s/" STATUS_FILE_GLOBAL_ERROR, strZ(spoolQueue))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool archiveAsyncStatus(const ArchiveMode archiveMode, const String *const walSegment, const bool throwOnError, const bool warnOnOk) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, archiveMode); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(BOOL, throwOnError); FUNCTION_LOG_PARAM(BOOL, warnOnOk); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { const String *errorFile = NULL; bool errorFileExists = false; const String *const spoolQueue = archiveAsyncSpoolQueue(archiveMode); String *const okFile = strNewFmt("%s" STATUS_EXT_OK, strZ(walSegment)); const bool okFileExists = storageExistsP(storageSpool(), strNewFmt("%s/%s", strZ(spoolQueue), strZ(okFile))); // If the ok file does not exist then check to see if a file-specific or global error exists if (!okFileExists) { // Check for a file-specific error first errorFile = strNewFmt("%s" STATUS_EXT_ERROR, strZ(walSegment)); errorFileExists = storageExistsP(storageSpool(), strNewFmt("%s/%s", strZ(spoolQueue), strZ(errorFile))); // If that doesn't exist then check for a global error if (!errorFileExists) { errorFile = STATUS_FILE_GLOBAL_ERROR_STR; errorFileExists = storageExistsP(storageSpool(), strNewFmt("%s/%s", strZ(spoolQueue), strZ(errorFile))); } } // If either of them exists then check what happened and report back if (okFileExists || errorFileExists) { // Get the status file content const String *const statusFile = okFileExists ? okFile : errorFile; const String *const content = strNewBuf( storageGetP(storageNewReadP(storageSpool(), strNewFmt("%s/%s", strZ(spoolQueue), strZ(statusFile))))); // Get the code and message if the file has content int code = 0; const String *message = NULL; if (strSize(content) != 0) { // Find the line feed after the error code -- should be the first one const char *const linefeedPtr = strchr(strZ(content), '\n'); // Error if linefeed not found if (linefeedPtr == NULL) THROW_FMT(FormatError, "%s content must have at least two lines", strZ(statusFile)); // Error if message is zero-length if (strlen(linefeedPtr + 1) == 0) THROW_FMT(FormatError, "%s message must be > 0", strZ(statusFile)); // Get contents code = varIntForce(VARSTR(strNewZN(strZ(content), (size_t)(linefeedPtr - strZ(content))))); message = strTrim(strNewZ(linefeedPtr + 1)); } // Process OK files if (okFileExists) { // If there is content in the status file it is a warning if (strSize(content) != 0 && warnOnOk) { // If error code is not success, then this was a renamed error file if (code != 0) { message = strNewFmt( "WAL segment '%s' was not pushed due to error [%d] and was manually skipped: %s", strZ(walSegment), code, strZ(message)); } LOG_WARN(strZ(message)); } result = true; } else if (throwOnError) { // Error status files must have content if (strSize(content) == 0) THROW_FMT(AssertError, "status file '%s' has no content", strZ(statusFile)); // Throw error using the code passed in the file THROW_CODE(code, strZ(message)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN void archiveAsyncStatusErrorWrite( const ArchiveMode archiveMode, const String *const walSegment, const int code, const String *const message) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, archiveMode); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(INT, code); FUNCTION_LOG_PARAM(STRING, message); FUNCTION_LOG_END(); ASSERT(code != 0); ASSERT(message != NULL); MEM_CONTEXT_TEMP_BEGIN() { const String *const errorFile = walSegment == NULL ? STATUS_FILE_GLOBAL_STR : walSegment; storagePutP( storageNewWriteP( storageSpoolWrite(), strNewFmt("%s/%s" STATUS_EXT_ERROR, strZ(archiveAsyncSpoolQueue(archiveMode)), strZ(errorFile))), BUFSTR(strNewFmt("%d\n%s", code, strZ(message)))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void archiveAsyncStatusOkWrite(const ArchiveMode archiveMode, const String *const walSegment, const String *const warning) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, archiveMode); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(STRING, warning); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Write file storagePutP( storageNewWriteP( storageSpoolWrite(), strNewFmt("%s/%s" STATUS_EXT_OK, strZ(archiveAsyncSpoolQueue(archiveMode)), strZ(walSegment))), warning == NULL ? NULL : BUFSTR(strNewFmt("0\n%s", strZ(warning)))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN int archiveIdComparator(const void *const archiveId1, const void *const archiveId2) { ASSERT(strstr(strZ(*(const String *const *)archiveId1), "-") != NULL); ASSERT(strstr(strZ(*(const String *const *)archiveId2), "-") != NULL); const int id1 = cvtZToInt(strstr(strZ(*(const String *const *)archiveId1), "-") + 1); const int id2 = cvtZToInt(strstr(strZ(*(const String *const *)archiveId2), "-") + 1); return LST_COMPARATOR_CMP(id1, id2); } /**********************************************************************************************************************************/ FN_EXTERN bool walIsPartial(const String *const walSegment) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); ASSERT(walIsSegment(walSegment)); FUNCTION_LOG_RETURN(BOOL, strEndsWithZ(walSegment, WAL_SEGMENT_PARTIAL_EXT)); } /**********************************************************************************************************************************/ FN_EXTERN String * walPath(const String *walFile, const String *pgPath, const String *command) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walFile); FUNCTION_LOG_PARAM(STRING, pgPath); FUNCTION_LOG_PARAM(STRING, command); FUNCTION_LOG_END(); ASSERT(walFile != NULL); ASSERT(command != NULL); String *result; if (!strBeginsWithZ(walFile, "/")) { // Error if walFile has a relative path and pgPath is not set if (pgPath == NULL) { THROW_FMT( OptionRequiredError, "option '%s' must be specified when relative wal paths are used\n" "HINT: is %%f passed to %s instead of %%p?\n" "HINT: PostgreSQL may pass relative paths even with %%p depending on the environment.", cfgOptionName(cfgOptPgPath), strZ(command)); } // Get the working directory char currentWorkDir[4096]; THROW_ON_SYS_ERROR(getcwd(currentWorkDir, sizeof(currentWorkDir)) == NULL, FormatError, "unable to get cwd"); // Check if the working directory is the same as pgPath if (!strEqZ(pgPath, currentWorkDir)) { // If not we'll change the working directory to pgPath and see if that equals the working directory we got called with THROW_ON_SYS_ERROR_FMT(chdir(strZ(pgPath)) != 0, PathMissingError, "unable to chdir() to '%s'", strZ(pgPath)); // Get the new working directory char newWorkDir[4096]; THROW_ON_SYS_ERROR(getcwd(newWorkDir, sizeof(newWorkDir)) == NULL, FormatError, "unable to get cwd"); // Error if the new working directory is not equal to the original current working directory. This means that PostgreSQL // and pgBackrest have a different idea about where the PostgreSQL data directory is located. if (strcmp(currentWorkDir, newWorkDir) != 0) { THROW_FMT( OptionInvalidValueError, PG_NAME " working directory '%s' is not the same as option %s '%s'\n" "HINT: is the " PG_NAME " data_directory configured the same as the %s option?", currentWorkDir, cfgOptionName(cfgOptPgPath), strZ(pgPath), cfgOptionName(cfgOptPgPath)); } } result = strNewFmt("%s/%s", strZ(pgPath), strZ(walFile)); } else result = strDup(walFile); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN bool walIsSegment(const String *const walSegment) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); // Create the regular expression to identify WAL segments if it does not already exist static RegExp *regExpSegment = NULL; if (regExpSegment == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { regExpSegment = regExpNew(WAL_SEGMENT_PARTIAL_REGEXP_STR); } MEM_CONTEXT_END(); } FUNCTION_LOG_RETURN(BOOL, regExpMatch(regExpSegment, walSegment)); } /**********************************************************************************************************************************/ FN_EXTERN String * walSegmentNext(const String *const walSegment, const size_t walSegmentSize) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(SIZE, walSegmentSize); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); ASSERT(strSize(walSegment) == 24); ASSERT(UINT32_MAX % walSegmentSize == walSegmentSize - 1); // Extract WAL parts uint32_t timeline; uint32_t major; uint32_t minor; MEM_CONTEXT_TEMP_BEGIN() { timeline = pgTimelineFromWalSegment(walSegment); major = (uint32_t)strtol(strZ(strSubN(walSegment, 8, 8)), NULL, 16); minor = (uint32_t)strtol(strZ(strSubN(walSegment, 16, 8)), NULL, 16); // Increment minor and adjust major dir on overflow minor++; if (minor > UINT32_MAX / walSegmentSize) { major++; minor = 0; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, strNewFmt("%08X%08X%08X", timeline, major, minor)); } /**********************************************************************************************************************************/ FN_EXTERN StringList * walSegmentRange(const String *const walSegmentBegin, const size_t walSegmentSize, const unsigned int range) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walSegmentBegin); FUNCTION_LOG_PARAM(SIZE, walSegmentSize); FUNCTION_LOG_END(); ASSERT(range > 0); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { strLstAdd(result, walSegmentBegin); if (range > 1) { String *current = strDup(walSegmentBegin); for (unsigned int rangeIdx = 0; rangeIdx < range - 1; rangeIdx++) { String *const next = walSegmentNext(current, walSegmentSize); strLstAdd(result, next); strFree(current); current = next; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/common.h0000664000175100017510000001261215265101745020673 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Common ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_COMMON_H #define COMMAND_ARCHIVE_COMMON_H #include #include "common/type/stringId.h" /*********************************************************************************************************************************** Archive mode enum Used for functions that are common to both archive-push and archive-get so they can tailor their behavior to the command being run. ***********************************************************************************************************************************/ typedef enum { archiveModeGet = STRID5("get", 0x50a70), archiveModePush = STRID5("push", 0x44eb00), } ArchiveMode; #include "common/compress/helper.h" #include "common/type/stringList.h" #include "storage/storage.h" /*********************************************************************************************************************************** Status file extension constants ***********************************************************************************************************************************/ #define STATUS_EXT_ERROR ".error" #define STATUS_EXT_ERROR_SIZE (sizeof(STATUS_EXT_ERROR) - 1) #define STATUS_EXT_OK ".ok" #define STATUS_EXT_OK_SIZE (sizeof(STATUS_EXT_OK) - 1) /*********************************************************************************************************************************** WAL segment constants ***********************************************************************************************************************************/ // Extension for partial segments #define WAL_SEGMENT_PARTIAL_EXT ".partial" // Match when the first 24 characters match a WAL segment #define WAL_SEGMENT_PREFIX_REGEXP "^[0-F]{24}" // Match on a WAL segment without checksum appended #define WAL_SEGMENT_REGEXP WAL_SEGMENT_PREFIX_REGEXP "$" STRING_DECLARE(WAL_SEGMENT_REGEXP_STR); // Match on a WAL segment with partial allowed #define WAL_SEGMENT_PARTIAL_REGEXP WAL_SEGMENT_PREFIX_REGEXP "(\\.partial){0,1}$" STRING_DECLARE(WAL_SEGMENT_PARTIAL_REGEXP_STR); // Defines the size of standard WAL segment name -- hopefully this won't change #define WAL_SEGMENT_NAME_SIZE ((unsigned int)24) // WAL segment directory/file #define WAL_SEGMENT_DIR_REGEXP "^[0-F]{16}$" STRING_DECLARE(WAL_SEGMENT_DIR_REGEXP_STR); #define WAL_SEGMENT_FILE_REGEXP "^[0-F]{24}-[0-f]{40}" COMPRESS_TYPE_REGEXP "{0,1}$" STRING_DECLARE(WAL_SEGMENT_FILE_REGEXP_STR); // Timeline history file #define WAL_TIMELINE_HISTORY_REGEXP "^[0-F]{8}.history$" STRING_DECLARE(WAL_TIMELINE_HISTORY_REGEXP_STR); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Remove errors for an archive file. This should be done before forking the async process to prevent a race condition where an old // error may be reported rather than waiting for the async process to succeed or fail. FN_EXTERN void archiveAsyncErrorClear(ArchiveMode archiveMode, const String *archiveFile); // Check for ok/error status files in the spool in/out directory. throwOnError determines whether an error will be thrown when an // error file is found. warnOnOk determines whether a warning will be output when found in an ok file. FN_EXTERN bool archiveAsyncStatus(ArchiveMode archiveMode, const String *walSegment, bool throwOnError, bool warnOnOk); // Write an ok status file FN_EXTERN void archiveAsyncStatusOkWrite(ArchiveMode archiveMode, const String *walSegment, const String *warning); // Write an error status file FN_EXTERN void archiveAsyncStatusErrorWrite(ArchiveMode archiveMode, const String *walSegment, int code, const String *message); // Comparator function for sorting archive ids by the database history id (the number after the dash) e.g. 9.6-1, 10-2 FN_EXTERN int archiveIdComparator(const void *item1, const void *item2); // Is the segment partial? FN_EXTERN bool walIsPartial(const String *walSegment); // Is the file a segment or some other file (e.g. .history, .backup, etc) FN_EXTERN bool walIsSegment(const String *walSegment); // Generates the location of the wal directory using a relative wal path and the supplied pg path FN_EXTERN String *walPath(const String *walFile, const String *pgPath, const String *command); // Get the next WAL segment given a WAL segment and WAL segment size FN_EXTERN String *walSegmentNext(const String *walSegment, size_t walSegmentSize); // Build a list of WAL segments based on a beginning WAL and number of WAL in the range (inclusive) FN_EXTERN StringList *walSegmentRange(const String *walSegmentBegin, size_t walSegmentSize, unsigned int range); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/find.c0000664000175100017510000002231415265101745020316 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Segment Find ***********************************************************************************************************************************/ #include #include "command/archive/common.h" #include "command/archive/find.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/wait.h" #include "storage/helper.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct WalSegmentFind { const Storage *storage; // Storage to find WAL in const String *archiveId; // Archive id to find segments in bool single; // Optimize for a single segment? TimeMSec timeout; // Timeout for each segment String *prefix; // Current list prefix StringList *list; // List of found segments }; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_WAL_SEGMENT_FIND_TYPE \ WalSegmentFind * #define FUNCTION_LOG_WAL_SEGMENT_FIND_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "WalSegmentFind", buffer, bufferSize) /**********************************************************************************************************************************/ FN_EXTERN WalSegmentFind * walSegmentFindNew(const Storage *const storage, const String *const archiveId, const bool single, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, archiveId); FUNCTION_LOG_PARAM(BOOL, single); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(archiveId != NULL); OBJ_NEW_BEGIN(WalSegmentFind, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (WalSegmentFind) { .storage = storage, .archiveId = strDup(archiveId), .single = single, .timeout = timeout, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(WAL_SEGMENT_FIND, this); } /**********************************************************************************************************************************/ FN_EXTERN String * walSegmentFind(WalSegmentFind *const this, const String *const walSegment) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(WAL_SEGMENT_FIND, this); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); ASSERT(walIsSegment(walSegment)); String *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { Wait *const wait = waitNew(this->timeout); const String *const prefix = strSubN(walSegment, 0, 16); const String *const path = strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(this->archiveId), strZ(prefix)); const String *const expression = strNewFmt( "^%s%s-[0-f]{40}" COMPRESS_TYPE_REGEXP "{0,1}$", strZ(strSubN(walSegment, 0, 24)), walIsPartial(walSegment) ? WAL_SEGMENT_PARTIAL_EXT : ""); RegExp *regExp = NULL; do { // Get a list of all WAL segments that match the directory (and prefix when finding a single WAL) if (this->list == NULL || !strEq(prefix, this->prefix)) { MEM_CONTEXT_OBJ_BEGIN(this) { // Free and store prefix if (!strEq(prefix, this->prefix)) { strFree(this->prefix); this->prefix = strDup(prefix); } // Free list strLstFree(this->list); // Get list this->list = strLstSort( storageListP(this->storage, path, .expression = this->single ? expression : NULL), sortOrderAsc); } MEM_CONTEXT_OBJ_END(); } // If there are results if (!strLstEmpty(this->list)) { // By default the match size is the list size since filtering happened above. When not finding a single WAL then // non-matching entries before the matching WAL will need to be removed and then the matching WAL counted. unsigned int match = strLstSize(this->list); if (!this->single) { // Build regexp if not yet built if (regExp == NULL) regExp = regExpNew(expression); // Remove list items that do not match. This prevents us from having check them again on the next find. while (!strLstEmpty(this->list) && !regExpMatch(regExp, strLstGet(this->list, 0))) strLstRemoveIdx(this->list, 0); // Find matches at the beginning of the remaining list match = 0; while (match < strLstSize(this->list) && regExpMatch(regExp, strLstGet(this->list, match))) match++; } // Error if there is more than one match if (match > 1) { // Build list of duplicate WAL StringList *const matchList = strLstNew(); for (unsigned int matchIdx = 0; matchIdx < match; matchIdx++) strLstAdd(matchList, strLstGet(this->list, matchIdx)); // Clear list for next find strLstFree(this->list); this->list = NULL; THROW_FMT( ArchiveDuplicateError, "duplicates found in archive for WAL segment %s: %s\n" "HINT: are multiple primaries archiving to this stanza?", strZ(walSegment), strZ(strLstJoin(matchList, ", "))); } // On match copy file name of WAL segment found into the prior context if (match == 1) { MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(strLstGet(this->list, 0)); } MEM_CONTEXT_PRIOR_END(); } // Remove matching entries so list will be reloaded when empty if (!this->single) { ASSERT(regExp != NULL); while (!strLstEmpty(this->list) && regExpMatch(regExp, strLstGet(this->list, 0))) strLstRemoveIdx(this->list, 0); } } // Clear list for next find if (this->single || strLstEmpty(this->list)) { strLstFree(this->list); this->list = NULL; } } while (result == NULL && waitMore(wait)); } MEM_CONTEXT_TEMP_END(); // Error if segment not found before timeout if (result == NULL && this->timeout != 0) { THROW_FMT( ArchiveTimeoutError, "WAL segment %s was not archived before the %" PRIu64 "ms timeout\n" "HINT: check the archive_command to ensure that all options are correct (especially --stanza).\n" "HINT: check the PostgreSQL server log for errors.\n" "HINT: run the 'start' command if the stanza was previously stopped.", strZ(walSegment), this->timeout); } FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * walSegmentFindOne( const Storage *const storage, const String *const archiveId, const String *const walSegment, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, archiveId); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); String *result; MEM_CONTEXT_TEMP_BEGIN() { WalSegmentFind *const find = walSegmentFindNew(storage, archiveId, true, timeout); MEM_CONTEXT_PRIOR_BEGIN() { result = walSegmentFind(find, walSegment); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/find.h0000664000175100017510000000452015265101745020322 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Segment Find Find a WAL segment (or segments) in a repository. The code paths for finding single or multiple WAL segments are both optimized. ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_FIND_H #define COMMAND_ARCHIVE_FIND_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct WalSegmentFind WalSegmentFind; #include "common/type/string.h" #include "storage/storage.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN WalSegmentFind *walSegmentFindNew(const Storage *storage, const String *archiveId, bool single, TimeMSec timeout); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Find a WAL segment in the repository. The file name can have several things appended such as a hash, compression extension, and // partial extension so it is possible to have multiple files that match the segment, though more than one match is not a good // thing. FN_EXTERN String *walSegmentFind(WalSegmentFind *this, const String *walSegment); /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Find a single WAL segment (see walSegmentFind() for details) FN_EXTERN String *walSegmentFindOne(const Storage *storage, const String *archiveId, const String *walSegment, TimeMSec timeout); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/get/0000775000175100017510000000000015265101745020007 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/get/file.c0000664000175100017510000000772315265101745021103 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Get File ***********************************************************************************************************************************/ #include #include "command/archive/common.h" #include "command/archive/get/file.h" #include "command/control/common.h" #include "common/compress/helper.h" #include "common/debug.h" #include "common/format/cipherBlockFormat.h" #include "common/io/filter/group.h" #include "common/log.h" #include "config/config.h" #include "info/infoArchive.h" #include "postgres/interface.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN ArchiveGetFileResult archiveGetFile( const Storage *const storage, const String *const request, const List *const actualList, const String *const walDestination) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, request); FUNCTION_LOG_PARAM(LIST, actualList); FUNCTION_LOG_PARAM(STRING, walDestination); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(request != NULL); ASSERT(actualList != NULL && !lstEmpty(actualList)); ASSERT(walDestination != NULL); ArchiveGetFileResult result = {.warnList = strLstNew()}; // Check all files in the actual list and return as soon as one is copied bool copied = false; for (unsigned int actualIdx = 0; actualIdx < lstSize(actualList); actualIdx++) { const ArchiveGetFile *const actual = lstGet(actualList, actualIdx); // Is the file compressible during the copy? bool compressible = true; TRY_BEGIN() { MEM_CONTEXT_TEMP_BEGIN() { StorageWrite *const destination = storageNewWriteP( storage, walDestination, .noCreatePath = true, .noSyncFile = true, .noSyncPath = true, .noAtomic = true); // Add decrypt filter to read the format header if (cipherSpecMapSize(actual->cipherSpecArchive) != 0) { cipherBlockFormatFilterGroupReadAddMap( ioWriteFilterGroup(storageWriteIo(destination)), actual->cipherSpecArchive); compressible = false; } // If file is compressed then add the decompression filter CompressType compressType = compressTypeFromName(actual->file); if (compressType != compressTypeNone) { ioFilterGroupAdd(ioWriteFilterGroup(storageWriteIo(destination)), decompressFilterP(compressType)); compressible = false; } // Copy the file storageCopyP( storageNewReadP( storageRepoIdx(actual->repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s", strZ(actual->file)), .compressible = compressible), destination); } MEM_CONTEXT_TEMP_END(); // File was successfully copied result.actualIdx = actualIdx; copied = true; } // Log errors as warnings and continue CATCH_ANY() { strLstAddFmt( result.warnList, "%s: %s [%s] %s", cfgOptionGroupName(cfgOptGrpRepo, actual->repoIdx), strZ(actual->file), errorTypeName(errorType()), errorMessage()); } TRY_END(); // Stop on success if (copied) break; } // If no file was successfully copied then error if (!copied) { ASSERT(!strLstEmpty(result.warnList)); THROW_FMT(FileReadError, "unable to get %s:\n%s", strZ(request), strZ(strLstJoin(result.warnList, "\n"))); } FUNCTION_LOG_RETURN_STRUCT(result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/get/file.h0000664000175100017510000000322615265101745021102 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Get File ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_GET_FILE_H #define COMMAND_ARCHIVE_GET_FILE_H #include "common/format/cipherSpecMap.h" #include "common/type/string.h" #include "storage/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Copy a file from the archive to the specified destination typedef struct ArchiveGetFile { const String *file; // File in the repo (with path, checksum, ext, etc.) unsigned int repoIdx; // Repo idx const String *archiveId; // Repo archive id const CipherSpecMap *cipherSpecArchive; // Repo archive cipher keys } ArchiveGetFile; typedef struct ArchiveGetFileResult { unsigned int actualIdx; // Index of the file from actual list that was retrieved StringList *warnList; // Warnings from a successful operation } ArchiveGetFileResult; FN_EXTERN ArchiveGetFileResult archiveGetFile( const Storage *storage, const String *request, const List *actualList, const String *walDestination); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/get/get.c0000664000175100017510000015027315265101745020742 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Get Command ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "command/archive/common.h" #include "command/archive/get/file.h" #include "command/archive/get/get.h" #include "command/archive/get/protocol.h" #include "command/command.h" #include "command/lock.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/wait.h" #include "config/config.h" #include "config/exec.h" #include "config/load.h" #include "info/infoArchive.h" #include "postgres/interface.h" #include "protocol/helper.h" #include "protocol/parallel.h" #include "storage/helper.h" #include "storage/write.h" /*********************************************************************************************************************************** Constants for log messages that are used multiple times to keep them consistent ***********************************************************************************************************************************/ #define FOUND_IN_ARCHIVE_MSG "found %s in the archive" #define FOUND_IN_REPO_ARCHIVE_MSG "found %s in the %s: %s archive" #define UNABLE_TO_FIND_IN_ARCHIVE_MSG "unable to find %s in the archive" #define UNABLE_TO_FIND_VALID_REPO_MSG "unable to find a valid repository" #define REPO_INVALID_OR_ERR_MSG "some repositories were invalid or encountered errors" /*********************************************************************************************************************************** Check for a list of archive files in the repository ***********************************************************************************************************************************/ typedef struct ArchiveFileMap { const String *request; // Archive file requested by archive_command List *actualList; // Actual files in various repos/archiveIds StringList *warnList; // Warnings that need to be reported by the async process } ArchiveFileMap; typedef struct ArchiveGetCheckResult { List *archiveFileMapList; // List of mapped archive files, i.e. found in the repo // Global error that affects all repos const ErrorType *errorType; // Error type if there was an error const String *errorFile; // Error file if there was an error const String *errorMessage; // Error message if there was an error const StringList *warnList; // Warnings that need to be reported by the async process } ArchiveGetCheckResult; // Helper to add an error to an error list and warn if the error is not already in the list static void archiveGetErrorAdd( StringList *const warnList, const bool log, const unsigned int repoIdx, const ErrorType *const type, const String *const message) { const String *const warn = strNewFmt( "%s: [%s] %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorTypeName(type), strZ(message)); if (!strLstExists(warnList, warn)) { if (log) LOG_WARN(strZ(warn)); strLstAdd(warnList, warn); } } // Helper to find a single archive file in the repository using a cache to speed up the process and minimize storageListP() calls typedef struct ArchiveGetFindCachePath { const String *path; // Cached path in the archiveId const StringList *fileList; // List of files in the cache path } ArchiveGetFindCachePath; typedef struct ArchiveGetFindCacheArchive { const String *archiveId; // ArchiveId in the repo List *pathList; // List of paths cached for archiveId } ArchiveGetFindCacheArchive; typedef struct ArchiveGetFindCacheRepo { unsigned int repoIdx; const CipherSpecMap *cipherSpecArchive; // Repo archive cipher keys List *archiveList; // Cached list of archiveIds and associated paths StringList *warnList; // Track repo warnings so each is only reported once } ArchiveGetFindCacheRepo; static bool archiveGetFind( const String *const archiveFileRequest, ArchiveGetCheckResult *const getCheckResult, List *const cacheRepoList, const StringList *const warnList, const bool single) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, archiveFileRequest); FUNCTION_LOG_PARAM_P(VOID, getCheckResult); FUNCTION_LOG_PARAM(LIST, cacheRepoList); FUNCTION_LOG_PARAM(STRING_LIST, warnList); FUNCTION_LOG_PARAM(BOOL, single); FUNCTION_LOG_END(); ASSERT(archiveFileRequest != NULL); ASSERT(getCheckResult != NULL); ASSERT(cacheRepoList != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { // Is the archive file a WAL segment? const bool isSegment = walIsSegment(archiveFileRequest); // Get the WAL segment path const String *const path = isSegment ? strSubN(archiveFileRequest, 0, 16) : NULL; // List to hold matches for the requested file List *const matchList = lstNewP(sizeof(ArchiveGetFile), .comparator = lstComparatorStr); // List of file level warnings StringList *const fileWarnList = strLstDup(warnList); // Errored repo total to track if all repos errored unsigned int repoErrorTotal = 0; // Check each repo for (unsigned int repoCacheIdx = 0; repoCacheIdx < lstSize(cacheRepoList); repoCacheIdx++) { const ArchiveGetFindCacheRepo *const cacheRepo = lstGet(cacheRepoList, repoCacheIdx); TRY_BEGIN() { // Check each archiveId for (unsigned int archiveCacheIdx = 0; archiveCacheIdx < lstSize(cacheRepo->archiveList); archiveCacheIdx++) { const ArchiveGetFindCacheArchive *const cacheArchive = lstGet(cacheRepo->archiveList, archiveCacheIdx); // If a WAL segment then search among the possible file names if (isSegment) { StringList *segmentList; // If a single file is requested then optimize by adding a restrictive expression to reduce bandwidth if (single) { segmentList = storageListP( storageRepoIdx(cacheRepo->repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(cacheArchive->archiveId), strZ(path)), .expression = strNewFmt( "^%s%s-[0-f]{40}" COMPRESS_TYPE_REGEXP "{0,1}$", strZ(strSubN(archiveFileRequest, 0, 24)), walIsPartial(archiveFileRequest) ? WAL_SEGMENT_PARTIAL_EXT : "")); } // Else multiple files will be requested so cache list results else { // Partial files cannot be in a list with multiple requests ASSERT(!walIsPartial(archiveFileRequest)); // If the path does not exist in the cache then fetch it const ArchiveGetFindCachePath *cachePath = lstFind(cacheArchive->pathList, &path); if (cachePath == NULL) { MEM_CONTEXT_BEGIN(lstMemContext(cacheArchive->pathList)) { const ArchiveGetFindCachePath archiveGetFindCachePath = { .path = strDup(path), .fileList = storageListP( storageRepoIdx(cacheRepo->repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(cacheArchive->archiveId), strZ(path)), .expression = strNewFmt( "^%s[0-F]{8}-[0-f]{40}" COMPRESS_TYPE_REGEXP "{0,1}$", strZ(path))), }; cachePath = lstAdd(cacheArchive->pathList, &archiveGetFindCachePath); } MEM_CONTEXT_END(); } // Get a list of all WAL segments that match segmentList = strLstNew(); for (unsigned int fileIdx = 0; fileIdx < strLstSize(cachePath->fileList); fileIdx++) { if (strBeginsWith(strLstGet(cachePath->fileList, fileIdx), archiveFileRequest)) strLstAdd(segmentList, strLstGet(cachePath->fileList, fileIdx)); } } // Add segments to match list for (unsigned int segmentIdx = 0; segmentIdx < strLstSize(segmentList); segmentIdx++) { MEM_CONTEXT_BEGIN(lstMemContext(getCheckResult->archiveFileMapList)) { const ArchiveGetFile archiveGetFile = { .file = strNewFmt( "%s/%s/%s", strZ(cacheArchive->archiveId), strZ(path), strZ(strLstGet(segmentList, segmentIdx))), .repoIdx = cacheRepo->repoIdx, .archiveId = cacheArchive->archiveId, .cipherSpecArchive = cacheRepo->cipherSpecArchive, }; lstAdd(matchList, &archiveGetFile); } MEM_CONTEXT_END(); } } // Else if not a WAL segment, see if it exists in the archiveId path else if ( storageExistsP( storageRepoIdx(cacheRepo->repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(cacheArchive->archiveId), strZ(archiveFileRequest)))) { MEM_CONTEXT_BEGIN(lstMemContext(getCheckResult->archiveFileMapList)) { const ArchiveGetFile archiveGetFile = { .file = strNewFmt("%s/%s", strZ(cacheArchive->archiveId), strZ(archiveFileRequest)), .repoIdx = cacheRepo->repoIdx, .archiveId = cacheArchive->archiveId, .cipherSpecArchive = cacheRepo->cipherSpecArchive, }; lstAdd(matchList, &archiveGetFile); } MEM_CONTEXT_END(); } } } // Log errors as warnings and continue CATCH_ANY() { repoErrorTotal++; archiveGetErrorAdd(cacheRepo->warnList, true, cacheRepo->repoIdx, errorType(), STR(errorMessage())); archiveGetErrorAdd(fileWarnList, false, cacheRepo->repoIdx, errorType(), STR(errorMessage())); } TRY_END(); } // If all repos errored out then set the global error since processing cannot continue past this segment ASSERT(repoErrorTotal <= lstSize(cacheRepoList)); if (repoErrorTotal == lstSize(cacheRepoList)) { ASSERT(!strLstEmpty(fileWarnList)); MEM_CONTEXT_BEGIN(lstMemContext(getCheckResult->archiveFileMapList)) { getCheckResult->errorType = &RepoInvalidError; getCheckResult->errorFile = strDup(archiveFileRequest); getCheckResult->errorMessage = strNewZ(UNABLE_TO_FIND_VALID_REPO_MSG); getCheckResult->warnList = strLstMove(fileWarnList, memContextCurrent()); } MEM_CONTEXT_END(); } // Else if a file was found else if (!lstEmpty(matchList)) { bool error = false; // If a segment match list is > 1 then check for duplicates if (isSegment && lstSize(matchList) > 1) { // Count the number of unique hashes StringList *const hashList = strLstNew(); for (unsigned int matchIdx = 0; matchIdx < lstSize(matchList); matchIdx++) strLstAddIfMissing(hashList, strSubN(((ArchiveGetFile *)lstGet(matchList, matchIdx))->file, 25, 40)); // If there is more than one unique hash then there are duplicates if (strLstSize(hashList) > 1) { // Build list of duplicates unsigned int repoKeyLast = 0; String *const message = strNew(); bool first = true; // Sort the matches so they are logged in a consistent order lstSort(matchList, sortOrderAsc); for (unsigned int matchIdx = 0; matchIdx < lstSize(matchList); matchIdx++) { const ArchiveGetFile *const file = lstGet(matchList, matchIdx); const unsigned int repoKey = cfgOptionGroupIdxToKey(cfgOptGrpRepo, file->repoIdx); if (repoKey != repoKeyLast) { strCatFmt(message, "\n%s:", cfgOptionGroupName(cfgOptGrpRepo, file->repoIdx)); repoKeyLast = repoKey; first = true; } if (first) first = false; else strCatChr(message, ','); strCatFmt(message, " %s", strZ(file->file)); } // Set as global error since processing cannot continue past this segment MEM_CONTEXT_BEGIN(lstMemContext(getCheckResult->archiveFileMapList)) { getCheckResult->errorType = &ArchiveDuplicateError; getCheckResult->errorFile = strDup(archiveFileRequest); getCheckResult->errorMessage = strNewFmt( "duplicates found for WAL segment %s:%s\n" "HINT: are multiple primaries archiving to this stanza?", strZ(archiveFileRequest), strZ(message)); getCheckResult->warnList = strLstMove(fileWarnList, memContextCurrent()); } MEM_CONTEXT_END(); error = true; } } // Files are valid so add them to the map if (!error) { MEM_CONTEXT_BEGIN(lstMemContext(getCheckResult->archiveFileMapList)) { const ArchiveFileMap map = { .request = strDup(archiveFileRequest), .actualList = lstNewP(sizeof(ArchiveGetFile)), .warnList = strLstMove(fileWarnList, memContextCurrent()), }; for (unsigned int matchIdx = 0; matchIdx < lstSize(matchList); matchIdx++) lstAdd(map.actualList, lstGet(matchList, matchIdx)); lstAdd(getCheckResult->archiveFileMapList, &map); } MEM_CONTEXT_END(); result = true; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } static ArchiveGetCheckResult archiveGetCheck(const StringList *const archiveRequestList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_LIST, archiveRequestList); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(archiveRequestList != NULL); ASSERT(!strLstEmpty(archiveRequestList)); ArchiveGetCheckResult result = {.archiveFileMapList = lstNewP(sizeof(ArchiveFileMap), .comparator = lstComparatorStr)}; MEM_CONTEXT_TEMP_BEGIN() { // List of warnings StringList *const warnList = strLstNew(); // Get pg_control info. Error if the invalid checkpoint written by restore is detected and backup_label is not present. const PgControl controlInfo = pgControlFromFile(storagePg(), cfgOptionStrNull(cfgOptPgVersionForce)); if (controlInfo.checkpoint == PG_CONTROL_CHECKPOINT_INVALID && !storageExistsP(storagePg(), STRDEF(PG_FILE_BACKUPLABEL))) { THROW( FormatError, "pg_control from backup is not valid without backup_label\n" "HINT: was the backup_label file removed?"); } // Build list of repos/archiveIds where WAL may be found List *const cacheRepoList = lstNewP(sizeof(ArchiveGetFindCacheRepo)); for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { // If a repo was specified then skip all other repos if (cfgOptionTest(cfgOptRepo) && cfgOptionUInt(cfgOptRepo) != cfgOptionGroupIdxToKey(cfgOptGrpRepo, repoIdx)) continue; TRY_BEGIN() { // Get the repo storage in case it is remote and encryption settings need to be pulled down storageRepoIdx(repoIdx); ArchiveGetFindCacheRepo cacheRepo = { .repoIdx = repoIdx, .archiveList = lstNewP(sizeof(ArchiveGetFindCacheArchive)), .warnList = strLstNew(), }; // Attempt to load the archive info file const InfoArchive *const info = infoArchiveLoadFile( storageRepoIdx(repoIdx), INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); // Build cipher spec in the result list context once rather than rebuilding it per candidate file later MEM_CONTEXT_BEGIN(lstMemContext(result.archiveFileMapList)) { cacheRepo.cipherSpecArchive = cipherSpecMapDup(infoArchiveCipherSpecMap(info)); } MEM_CONTEXT_END(); // Loop through pg history and determine which archiveIds to use const StringList *archivePathList = NULL; for (unsigned int pgIdx = 0; pgIdx < infoPgDataTotal(infoArchivePg(info)); pgIdx++) { InfoPgData pgData = infoPgData(infoArchivePg(info), pgIdx); // Only use the archive id if it matches the current cluster if (pgData.systemId == controlInfo.systemId && pgData.version == controlInfo.version) { const String *const archiveId = infoPgArchiveId(infoArchivePg(info), pgIdx); bool found = true; // If the archiveId is in the past make sure the path exists if (pgIdx != 0) { // Get list of archiveId paths in the archive path if (archivePathList == NULL) archivePathList = storageListP(storageRepoIdx(repoIdx), STORAGE_REPO_ARCHIVE_STR); if (!strLstExists(archivePathList, archiveId)) found = false; } // If the archiveId is most recent or has files then add it if (found) { ArchiveGetFindCacheArchive cacheArchive = { .pathList = lstNewP(sizeof(ArchiveGetFindCachePath), .comparator = lstComparatorStr), }; // Copy archiveId into the result list context once rather than making a copy per candidate file later MEM_CONTEXT_BEGIN(lstMemContext(result.archiveFileMapList)) { cacheArchive.archiveId = strDup(archiveId); } MEM_CONTEXT_END(); lstAdd(cacheRepo.archiveList, &cacheArchive); } } } // Error if no archive id was found -- this indicates a mismatch with the current cluster if (lstEmpty(cacheRepo.archiveList)) { archiveGetErrorAdd( warnList, true, repoIdx, &ArchiveMismatchError, strNewFmt( "unable to retrieve the archive id for database version '%s' and system-id '%" PRIu64 "'", strZ(pgVersionToStr(controlInfo.version)), controlInfo.systemId)); } // Else add repo to list else lstAdd(cacheRepoList, &cacheRepo); } // Log errors as warnings and continue CATCH_ANY() { archiveGetErrorAdd(warnList, true, repoIdx, errorType(), STR(errorMessage())); } TRY_END(); } // Error if there are no repos to check if (lstEmpty(cacheRepoList)) { ASSERT(!strLstEmpty(warnList)); // Set as global error since processing cannot continue past this segment MEM_CONTEXT_BEGIN(lstMemContext(result.archiveFileMapList)) { result.errorType = &RepoInvalidError; result.errorMessage = strNewZ(UNABLE_TO_FIND_VALID_REPO_MSG); result.warnList = strLstMove(warnList, memContextCurrent()); } MEM_CONTEXT_END(); } else { // Any remaining errors will be reported as warnings since at least one repo is valid MEM_CONTEXT_BEGIN(lstMemContext(result.archiveFileMapList)) { result.warnList = strLstMove(warnList, memContextCurrent()); } MEM_CONTEXT_END(); // Find files in the list for (unsigned int archiveRequestIdx = 0; archiveRequestIdx < strLstSize(archiveRequestList); archiveRequestIdx++) { if (!archiveGetFind( strLstGet(archiveRequestList, archiveRequestIdx), &result, cacheRepoList, warnList, strLstSize(archiveRequestList) == 1)) { break; } } // Sort the list to make searching for files faster lstSort(result.archiveFileMapList, sortOrderAsc); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /*********************************************************************************************************************************** Clean the queue and prepare a list of WAL segments that the async process should get ***********************************************************************************************************************************/ static StringList * queueNeed(const String *const walSegment, const bool found, const uint64_t queueSize, const size_t walSegmentSize) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(BOOL, found); FUNCTION_LOG_PARAM(UINT64, queueSize); FUNCTION_LOG_PARAM(SIZE, walSegmentSize); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Determine the first WAL segment for the async process to get. If the WAL segment requested by PostgreSQL was not found // then use that. If the segment was found but the queue is not full then start with the next segment. const String *const walSegmentFirst = found ? walSegmentNext(walSegment, walSegmentSize) : walSegment; // Determine how many WAL segments should be in the queue. The queue total must be at least 2 or it doesn't make sense to // have async turned on at all. unsigned int walSegmentQueueTotal = (unsigned int)(queueSize / walSegmentSize); if (walSegmentQueueTotal < 2) walSegmentQueueTotal = 2; // Build the ideal queue -- the WAL segments we want in the queue after the async process has run const StringList *const idealQueue = strLstSort( walSegmentRange(walSegmentFirst, walSegmentSize, walSegmentQueueTotal), sortOrderAsc); // Get the list of files actually in the queue const StringList *const actualQueue = strLstSort( storageListP(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN_STR, .errorOnMissing = true), sortOrderAsc); // Build a list of WAL segments that are being kept so we can later make a list of what is needed StringList *const keepQueue = strLstNew(); for (unsigned int actualQueueIdx = 0; actualQueueIdx < strLstSize(actualQueue); actualQueueIdx++) { // Get file from actual queue const String *const file = strLstGet(actualQueue, actualQueueIdx); // Does this match a file we want to preserve? if (strLstExists(idealQueue, file)) { strLstAdd(keepQueue, file); } // Else delete if it does not match an ok file for a WAL segment that has already been preserved. If an ok file exists // in addition to the segment then it contains warnings which need to be preserved. else if ( !strEndsWithZ(file, STATUS_EXT_OK) || !strLstExists(actualQueue, strSubN(file, 0, strSize(file) - STATUS_EXT_OK_SIZE))) { storageRemoveP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s", strZ(file)), .errorOnMissing = true); } } // Generate a list of the WAL that are needed by removing kept WAL from the ideal queue strLstSort(keepQueue, sortOrderAsc); for (unsigned int idealQueueIdx = 0; idealQueueIdx < strLstSize(idealQueue); idealQueueIdx++) { if (!strLstExists(keepQueue, strLstGet(idealQueue, idealQueueIdx))) strLstAdd(result, strLstGet(idealQueue, idealQueueIdx)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /*********************************************************************************************************************************** Get a single WAL segment synchronously, i.e. copy it directly from the repository to the destination rather than using the async spool queue. Returns 0 if the segment was found and copied, else 1. Throws on error. ***********************************************************************************************************************************/ static int archiveGetSingle(const String *const walSegment, const String *const walDestination) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(STRING, walDestination); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); ASSERT(walDestination != NULL); int result = 1; MEM_CONTEXT_TEMP_BEGIN() { // Check for the archive file StringList *const archiveRequestList = strLstNew(); strLstAdd(archiveRequestList, walSegment); const ArchiveGetCheckResult checkResult = archiveGetCheck(archiveRequestList); // If there was an error then throw it if (checkResult.errorType != NULL) THROW_CODE(errorTypeCode(checkResult.errorType), strZ(checkResult.errorMessage)); // Get the archive file if (!lstEmpty(checkResult.archiveFileMapList)) { // There can only be one file mapping since only one file was requested ASSERT(lstSize(checkResult.archiveFileMapList) == 1); const ArchiveFileMap *const fileMap = lstGet(checkResult.archiveFileMapList, 0); // Get the file const ArchiveGetFileResult fileResult = archiveGetFile( storageLocalWrite(), fileMap->request, fileMap->actualList, walDestination); // Output file warnings for (unsigned int warnIdx = 0; warnIdx < strLstSize(fileResult.warnList); warnIdx++) LOG_WARN(strZ(strLstGet(fileResult.warnList, warnIdx))); // If there was no error then the file existed const ArchiveGetFile *const file = lstGet(fileMap->actualList, fileResult.actualIdx); ASSERT(file != NULL); LOG_INFO_FMT( FOUND_IN_REPO_ARCHIVE_MSG, strZ(walSegment), cfgOptionGroupName(cfgOptGrpRepo, file->repoIdx), strZ(file->archiveId)); result = 0; } // Else log that the file was not found else LOG_INFO_FMT(UNABLE_TO_FIND_IN_ARCHIVE_MSG, strZ(walSegment)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INT, result); } /**********************************************************************************************************************************/ FN_EXTERN int cmdArchiveGet(void) { FUNCTION_LOG_VOID(logLevelDebug); // PostgreSQL must be local pgIsLocalVerify(); // Set the result assuming the archive file will not be found int result = 1; MEM_CONTEXT_TEMP_BEGIN() { // Check the parameters const StringList *const commandParam = cfgCommandParam(); if (strLstSize(commandParam) != 2) { if (strLstEmpty(commandParam)) THROW(ParamRequiredError, "WAL segment to get required"); if (strLstSize(commandParam) == 1) THROW(ParamRequiredError, "path to copy WAL segment required"); THROW(ParamInvalidError, "extra parameters found"); } // Get the segment name const String *const walSegment = strBase(strLstGet(commandParam, 0)); // Destination is wherever we were told to move the WAL segment const String *const walDestination = walPath(strLstGet(commandParam, 1), cfgOptionStr(cfgOptPgPath), STR(cfgCommandName())); // Async get can only be performed on WAL segments, history or other files must use synchronous mode if (cfgOptionBool(cfgOptArchiveAsync) && walIsSegment(walSegment)) { bool first = true; // Is this the first time the loop has run? bool found = false; // Has the WAL segment been found yet? bool foundOk = false; // Was an OK file found which confirms the file was missing? bool queueFull = false; // Is the queue half or more full? bool forked = false; // Has the async process been forked yet? bool syncGet = false; // Was the segment resolved by a synchronous get? // Loop and wait for the WAL segment to be pushed Wait *const wait = waitNew(cfgOptionUInt64(cfgOptArchiveTimeout)); do { // Check if the WAL segment is already in the queue found = storageExistsP(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s", strZ(walSegment))); // Determine whether a missing WAL segment will be retried. Retrying is safer, but not retrying lets PostgreSQL // know that there are probably no more WAL segments in the archive which means it can switch to streaming. const bool missingRetry = first && cfgOptionBool(cfgOptArchiveMissingRetry); // Check for errors or missing files. For archive-get ok indicates that the process succeeded but there is no WAL // file to download, or that there was a warning. Do not error on the first run so the async process can be spawned // to correct any errors from a previous run. Do not warn on the first run if the segment was not found so the async // process can be spawned to check for the file again. if (archiveAsyncStatus(archiveModeGet, walSegment, !first, found || !missingRetry)) { storageRemoveP( storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s" STATUS_EXT_OK, strZ(walSegment)), .errorOnMissing = true); // Break if an ok file was found but no segment exists, which means the segment was missing. However, don't // break if this is the first time through the loop since this means the ok file was written by an async process // spawned by a prior archive-get execution, which means we should spawn the async process again to see if the // file exists now. This also prevents spool files from a previous recovery interfering with the current // recovery. if (!found && !missingRetry) { foundOk = true; break; } } // If found then move the WAL segment to the destination directory if (found) { // Source is the WAL segment in the spool queue StorageRead *const source = storageNewReadP( storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s", strZ(walSegment))); // A move will be attempted but if the spool queue and the WAL path are on different file systems then a copy // will be performed instead. // // It looks scary that we are disabling syncs and atomicity (in case we need to copy instead of move) but this // is safe because if the system crashes Postgres will not try to reuse a restored WAL segment but will instead // request it again using the restore_command. In the case of a move this hardly matters since path syncs are // cheap but if a copy is required we could save a lot of writes. StorageWrite *const destination = storageNewWriteP( storageLocalWrite(), walDestination, .noCreatePath = true, .noSyncFile = true, .noSyncPath = true, .noAtomic = true); // Move (or copy if required) the file storageMoveP(storageSpoolWrite(), source, destination); // Return success LOG_INFO_FMT(FOUND_IN_ARCHIVE_MSG " asynchronously", strZ(walSegment)); result = 0; // Get a list of WAL segments left in the queue const StringList *const queue = storageListP( storageSpool(), STORAGE_SPOOL_ARCHIVE_IN_STR, .expression = WAL_SEGMENT_REGEXP_STR, .errorOnMissing = true); if (!strLstEmpty(queue)) { // Get size of the WAL segment const uint64_t walSegmentSize = storageInfoP(storageLocal(), walDestination).size; // Use WAL segment size to estimate queue size and determine if the async process should be launched queueFull = strLstSize(queue) * walSegmentSize > cfgOptionUInt64(cfgOptArchiveGetQueueMax) / 2; } } // If the WAL segment has not already been found then start the async process to get it. There's no point in forking // the async process off more than once so track that as well. Use an archive lock to prevent forking if the async // process was launched by another process. if (!forked && (!found || !queueFull) && cmdLockAcquireP(.returnOnNoLock = true)) { // Get control info const PgControl pgControl = pgControlFromFile(storagePg(), cfgOptionStrNull(cfgOptPgVersionForce)); // Create the queue storagePathCreateP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN_STR); // The async process should not output on the console at all KeyValue *const optionReplace = kvNew(); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_CONSOLE), VARSTRDEF("off")); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_STDERR), VARSTRDEF("off")); // Generate command options StringList *const commandExec = cfgExecParam(cfgCmdArchiveGet, cfgCmdRoleAsync, optionReplace, true, false); strLstInsert(commandExec, 0, cfgBin()); // Clean the current queue using the list of WAL that we ideally want in the queue. queueNeed() will return the // list of WAL needed to fill the queue and this will be passed to the async process. const StringList *const queue = queueNeed( walSegment, found, cfgOptionUInt64(cfgOptArchiveGetQueueMax), pgControl.walSegmentSize); for (unsigned int queueIdx = 0; queueIdx < strLstSize(queue); queueIdx++) strLstAdd(commandExec, strLstGet(queue, queueIdx)); // Clear errors for the current wal segment archiveAsyncErrorClear(archiveModeGet, walSegment); // Release the lock so the child process can acquire it cmdLockReleaseP(); // Execute the async process cmdAsyncExec(CFGCMD_ARCHIVE_GET, commandExec); // Mark the async process as forked so it doesn't get forked again. A single run of the async process should be // enough to do the job, running it again won't help anything. forked = true; } // If the segment is not in the queue and our own async process is not holding the lock then the segment will not be // fetched by us: either a foreign async process holds the lock (spawned by a different main process, e.g. filling // the queue for the old timeline after a timeline switch) and will not fetch our segment, or no async process is // running because ours failed. Rather than wait for it and risk a timeout on a segment that actually exists -- // which can end recovery prematurely -- fetch the segment synchronously. Skip the first run so an async process // spawned above (or by a prior invocation) gets a chance to take the lock first. if (!found && !first && !cmdLockOwn()) { result = archiveGetSingle(walSegment, walDestination); found = result == 0; syncGet = true; break; } // Exit loop if WAL was found if (found) break; // No longer the first run, so errors will be thrown and missing files will be reported first = false; } while (waitMore(wait)); // If the WAL segment was not found and it was not resolved by a synchronous get (which does its own logging) if (!found && !syncGet) { // If no ok file was found then something may be wrong with the async process. It's better to throw an error here // than report not found for debugging purposes. Either way PostgreSQL will halt if it has not reached consistency. if (!foundOk) { THROW_FMT( ArchiveTimeoutError, "unable to get WAL file '%s' from the archive asynchronously after %s second(s)\n" "HINT: check '%s' for errors.", strZ(walSegment), strZ( strNewDivP((uint64_t)cfgOptionInt64(cfgOptArchiveTimeout), MSEC_PER_SEC, .precision = 3, .trim = true)), strZ(cfgLoadLogFileName(cfgCmdRoleAsync))); } // Else report that the WAL segment could not be found else LOG_INFO_FMT(UNABLE_TO_FIND_IN_ARCHIVE_MSG " asynchronously", strZ(walSegment)); } } // Else perform synchronous get else result = archiveGetSingle(walSegment, walDestination); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INT, result); } /**********************************************************************************************************************************/ typedef struct ArchiveGetAsyncData { const List *const archiveFileMapList; // List of wal segments to process unsigned int archiveFileIdx; // Current index in the list to be processed } ArchiveGetAsyncData; static ProtocolParallelJob * archiveGetAsyncCallback(void *const data, const unsigned int clientIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(UINT, clientIdx); FUNCTION_TEST_END(); ProtocolParallelJob *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // No special logic based on the client, we'll just get the next job (void)clientIdx; // Get a new job if there are any left ArchiveGetAsyncData *const jobData = data; if (jobData->archiveFileIdx < lstSize(jobData->archiveFileMapList)) { const ArchiveFileMap *const archiveFileMap = lstGet(jobData->archiveFileMapList, jobData->archiveFileIdx); jobData->archiveFileIdx++; PackWrite *const param = protocolPackNew(); pckWriteStrP(param, archiveFileMap->request); // Add actual files to get for (unsigned int actualIdx = 0; actualIdx < lstSize(archiveFileMap->actualList); actualIdx++) { const ArchiveGetFile *const actual = lstGet(archiveFileMap->actualList, actualIdx); pckWriteStrP(param, actual->file); pckWriteU32P(param, actual->repoIdx); pckWriteStrP(param, actual->archiveId); cipherSpecMapPack(param, actual->cipherSpecArchive); } MEM_CONTEXT_PRIOR_BEGIN() { result = protocolParallelJobNew(VARSTR(archiveFileMap->request), PROTOCOL_COMMAND_ARCHIVE_GET_FILE, param); } MEM_CONTEXT_PRIOR_END(); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } FN_EXTERN void cmdArchiveGetAsync(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { TRY_BEGIN() { // PostgreSQL must be local pgIsLocalVerify(); // Check the parameters if (strLstSize(cfgCommandParam()) < 1) THROW(ParamInvalidError, "at least one wal segment is required"); LOG_INFO_FMT( "get %u WAL file(s) from archive: %s%s", strLstSize(cfgCommandParam()), strZ(strLstGet(cfgCommandParam(), 0)), strLstSize(cfgCommandParam()) == 1 ? "" : zNewFmt("...%s", strZ(strLstGet(cfgCommandParam(), strLstSize(cfgCommandParam()) - 1)))); // Check for archive files const ArchiveGetCheckResult checkResult = archiveGetCheck(cfgCommandParam()); // If any files are missing get the first one (used to construct the "unable to find" warning) const String *archiveFileMissing = NULL; if (lstSize(checkResult.archiveFileMapList) < strLstSize(cfgCommandParam())) archiveFileMissing = strLstGet(cfgCommandParam(), lstSize(checkResult.archiveFileMapList)); // Get archive files that were found if (!lstEmpty(checkResult.archiveFileMapList)) { // Create the parallel executor ArchiveGetAsyncData jobData = {.archiveFileMapList = checkResult.archiveFileMapList}; ProtocolParallel *const parallelExec = protocolParallelNew( cfgOptionUInt64(cfgOptProtocolTimeout) / 2, archiveGetAsyncCallback, &jobData); for (unsigned int processIdx = 1; processIdx <= cfgOptionUInt(cfgOptProcessMax); processIdx++) protocolParallelClientAdd(parallelExec, protocolLocalGet(protocolStorageTypeRepo, 0, processIdx)); // Process jobs MEM_CONTEXT_TEMP_RESET_BEGIN() { do { const unsigned int completed = protocolParallelProcess(parallelExec); for (unsigned int jobIdx = 0; jobIdx < completed; jobIdx++) { // Get the job ProtocolParallelJob *const job = protocolParallelResult(parallelExec); const unsigned int processId = protocolParallelJobProcessId(job); // Get wal segment name and archive file map const String *const walSegment = varStr(protocolParallelJobKey(job)); const ArchiveFileMap *const fileMap = lstFind(checkResult.archiveFileMapList, &walSegment); ASSERT(fileMap != NULL); // Build warnings for status file String *const warning = strNew(); if (!strLstEmpty(fileMap->warnList)) strCatFmt(warning, "%s", strZ(strLstJoin(fileMap->warnList, "\n"))); // The job was successful if (protocolParallelJobErrorCode(job) == 0) { // Get the actual file retrieved PackRead *const fileResult = protocolParallelJobResult(job); const ArchiveGetFile *const file = lstGet(fileMap->actualList, pckReadU32P(fileResult)); ASSERT(file != NULL); // Output file warnings const StringList *const fileWarnList = pckReadStrLstP(fileResult); for (unsigned int warnIdx = 0; warnIdx < strLstSize(fileWarnList); warnIdx++) LOG_WARN_PID(processId, strZ(strLstGet(fileWarnList, warnIdx))); // Build file warnings for status file if (!strLstEmpty(fileWarnList)) { strCatFmt( warning, "%s%s", strSize(warning) == 0 ? "" : "\n", strZ(strLstJoin(fileWarnList, "\n"))); } if (strSize(warning) != 0) archiveAsyncStatusOkWrite(archiveModeGet, walSegment, warning); LOG_DETAIL_PID_FMT( processId, FOUND_IN_REPO_ARCHIVE_MSG, strZ(walSegment), cfgOptionGroupName(cfgOptGrpRepo, file->repoIdx), strZ(file->archiveId)); // Rename temp WAL segment to actual name. This is done after the ok file is written so the ok file // is guaranteed to exist before the foreground process finds the WAL segment. storageMoveP( storageSpoolWrite(), storageNewReadP( storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s." STORAGE_FILE_TEMP_EXT, strZ(walSegment))), storageNewWriteP( storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s", strZ(walSegment)))); } // Else the job errored else { LOG_WARN_PID_FMT( processId, "[%s] %s", errorTypeName(errorTypeFromCode(protocolParallelJobErrorCode(job))), strZ(protocolParallelJobErrorMessage(job))); archiveAsyncStatusErrorWrite( archiveModeGet, walSegment, protocolParallelJobErrorCode(job), strNewFmt( "%s%s", strZ(protocolParallelJobErrorMessage(job)), strSize(warning) == 0 ? "" : zNewFmt("\n%s", strZ(warning)))); } protocolParallelJobFree(job); } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } while (!protocolParallelDone(parallelExec)); } MEM_CONTEXT_TEMP_END(); } // Log an error from archiveGetCheck() after any existing files have been fetched. This ordering is important because we // need to fetch as many valid files as possible before throwing an error. if (checkResult.errorType != NULL) { LOG_WARN_FMT("[%s] %s", errorTypeName(checkResult.errorType), strZ(checkResult.errorMessage)); String *const message = strCat(strNew(), checkResult.errorMessage); if (!strLstEmpty(checkResult.warnList)) strCatFmt(message, "\n%s", strZ(strLstJoin(checkResult.warnList, "\n"))); archiveAsyncStatusErrorWrite( archiveModeGet, checkResult.errorFile, errorTypeCode(checkResult.errorType), message); } // If any files were missing write an ok file for the first missing file and add any warnings. It is important that this // happen right before the async process exits so the main process can immediately respawn the async process to retry // missing files. else if (archiveFileMissing != NULL) { LOG_DETAIL_FMT(UNABLE_TO_FIND_IN_ARCHIVE_MSG, strZ(archiveFileMissing)); const String *message = NULL; if (!strLstEmpty(checkResult.warnList)) message = strLstJoin(checkResult.warnList, "\n"); archiveAsyncStatusOkWrite(archiveModeGet, archiveFileMissing, message); } } // On any global error write a single error file to cover all unprocessed files CATCH_ANY() { archiveAsyncStatusErrorWrite(archiveModeGet, NULL, errorCode(), STR(errorMessage())); RETHROW(); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/get/get.h0000664000175100017510000000153715265101745020745 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Get Command ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_GET_GET_H #define COMMAND_ARCHIVE_GET_GET_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Get an archive file from the repository (WAL segment, history file, etc.) FN_EXTERN int cmdArchiveGet(void); // Async version of archive get that runs in parallel for performance FN_EXTERN void cmdArchiveGetAsync(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/get/protocol.c0000664000175100017510000000403615265101745022017 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Get Protocol Handler ***********************************************************************************************************************************/ #include #include "command/archive/get/file.h" #include "command/archive/get/protocol.h" #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "storage/helper.h" #include "storage/write.h" /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * archiveGetFileProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Get request const String *const request = pckReadStrP(param); // Build the actual list List *const actualList = lstNewP(sizeof(ArchiveGetFile)); while (!pckReadNullP(param)) { ArchiveGetFile actual = {.file = pckReadStrP(param)}; actual.repoIdx = pckReadU32P(param); actual.archiveId = pckReadStrP(param); actual.cipherSpecArchive = cipherSpecMapNewPack(param); lstAdd(actualList, &actual); } // Get file const ArchiveGetFileResult fileResult = archiveGetFile( storageSpoolWrite(), request, actualList, strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s." STORAGE_FILE_TEMP_EXT, strZ(request))); // Return result PackWrite *const data = protocolServerResultData(result); pckWriteU32P(data, fileResult.actualIdx); pckWriteStrLstP(data, fileResult.warnList); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/get/protocol.h0000664000175100017510000000271615265101745022027 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Get Protocol Handler ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_GET_PROTOCOL_H #define COMMAND_ARCHIVE_GET_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process protocol requests FN_EXTERN ProtocolServerResult *archiveGetFileProtocol(PackRead *param); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_ARCHIVE_GET_FILE STRID5("ag-f", 0x36ce10) #define PROTOCOL_SERVER_HANDLER_ARCHIVE_GET_LIST \ {.command = PROTOCOL_COMMAND_ARCHIVE_GET_FILE, .process = archiveGetFileProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/push/0000775000175100017510000000000015265101745020207 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/push/file.c0000664000175100017510000003364515265101745021305 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Push File ***********************************************************************************************************************************/ #include #include "command/archive/common.h" #include "command/archive/find.h" #include "command/archive/push/file.h" #include "command/control/common.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/format/cipherBlockFormat.h" #include "common/io/filter/group.h" #include "common/io/io.h" #include "common/log.h" #include "config/config.h" #include "postgres/interface.h" #include "storage/helper.h" /*********************************************************************************************************************************** Catch write errors during processing We want to continue when there are write errors during processing so add them to a list to be reported later and return false so the caller knows to stop writing on the affected repo. ***********************************************************************************************************************************/ typedef enum { archivePushFileIoTypeOpen = STRID5("open", 0x7160f0), archivePushFileIoTypeWrite = STRID5("write", 0x5a26570), archivePushFileIoTypeClose = STRID5("close", 0x59bd830), } ArchivePushFileIoType; // Helper to add errors to the list static void archivePushErrorAdd(StringList *const errorList, const unsigned int repoIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, errorList); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_END(); strLstAddFmt(errorList, "%s: [%s] %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorTypeName(errorType()), errorMessage()); FUNCTION_TEST_RETURN_VOID(); } static bool archivePushFileIo( const ArchivePushFileIoType type, IoWrite *const write, const Buffer *const buffer, const unsigned int repoIdx, StringList *const errorList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_PARAM(IO_WRITE, write); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_PARAM(STRING_LIST, errorList); FUNCTION_TEST_END(); ASSERT(write != NULL); ASSERT(errorList != NULL); bool result = true; // Process write operation TRY_BEGIN() { switch (type) { case archivePushFileIoTypeOpen: ioWriteOpen(write); break; case archivePushFileIoTypeWrite: ASSERT(buffer != NULL); ioWrite(write, buffer); break; case archivePushFileIoTypeClose: ioWriteClose(write); break; } } // Handle errors CATCH_ANY() { archivePushErrorAdd(errorList, repoIdx); result = false; } TRY_END(); FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN ArchivePushFileResult archivePushFile( const String *const walSource, const bool headerCheck, const bool modeCheck, const unsigned int pgVersion, const uint64_t pgSystemId, const String *const archiveFile, const CompressType compressType, const int compressLevel, const List *const repoList, const StringList *const priorErrorList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walSource); FUNCTION_LOG_PARAM(BOOL, headerCheck); FUNCTION_LOG_PARAM(BOOL, modeCheck); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_PARAM(STRING, archiveFile); FUNCTION_LOG_PARAM(ENUM, compressType); FUNCTION_LOG_PARAM(INT, compressLevel); FUNCTION_LOG_PARAM_P(VOID, repoList); FUNCTION_LOG_PARAM(STRING_LIST, priorErrorList); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(walSource != NULL); ASSERT(archiveFile != NULL); ASSERT(repoList != NULL); ASSERT(priorErrorList != NULL); ASSERT(lstSize(repoList) > 0); ArchivePushFileResult result = {.warnList = strLstNew()}; MEM_CONTEXT_TEMP_BEGIN() { StringList *const errorList = strLstDup(priorErrorList); // Is this a WAL segment? const bool isSegment = walIsSegment(archiveFile); // If this is a segment compare archive version and systemId to the WAL header if (headerCheck && isSegment) { const PgWal walInfo = pgWalFromFile(walSource, storageLocal(), cfgOptionStrNull(cfgOptPgVersionForce)); if (walInfo.version != pgVersion || walInfo.systemId != pgSystemId) { THROW_FMT( ArchiveMismatchError, "WAL file '%s' version %s, system-id %" PRIu64 " do not match stanza version %s, system-id %" PRIu64, strZ(walSource), strZ(pgVersionToStr(walInfo.version)), walInfo.systemId, strZ(pgVersionToStr(pgVersion)), pgSystemId); } } // Set archive destination initially to the archive file, this will be updated later for wal segments String *const archiveDestination = strCat(strNew(), archiveFile); // Assume that all repos need a copy of the archive file bool destinationCopyAny = true; bool *const destinationCopy = memNew(sizeof(bool) * lstSize(repoList)); for (unsigned int repoListIdx = 0; repoListIdx < lstSize(repoList); repoListIdx++) destinationCopy[repoListIdx] = true; // Get wal segment checksum and compare it to what exists in the repo, if any if (isSegment) { // Assume that no repos need a copy of the WAL segment and update when a repo needing a copy is found destinationCopyAny = false; // Generate a sha1 checksum for the wal segment IoRead *const read = storageReadIo(storageNewReadP(storageLocal(), walSource)); ioFilterGroupAdd(ioReadFilterGroup(read), cryptoHashNew(hashTypeSha1)); ioReadDrain(read); const String *const walSegmentChecksum = strNewEncode( encodingHex, pckReadBinP(ioFilterGroupResultP(ioReadFilterGroup(read), CRYPTO_HASH_FILTER_TYPE))); // Check each repo for the WAL segment for (unsigned int repoListIdx = 0; repoListIdx < lstSize(repoList); repoListIdx++) { const ArchivePushFileRepoData *const repoData = lstGet(repoList, repoListIdx); // Check if the WAL segment already exists in the repo const String *walSegmentFile; TRY_BEGIN() { walSegmentFile = walSegmentFindOne(storageRepoIdx(repoData->repoIdx), repoData->archiveId, archiveFile, 0); } CATCH_ANY() { archivePushErrorAdd(errorList, repoData->repoIdx); destinationCopy[repoListIdx] = false; } TRY_END(); // If there was an error try the next repo if (!destinationCopy[repoListIdx]) continue; // If the WAL segment was found validate the checksum if (walSegmentFile != NULL) { const String *const walSegmentRepoChecksum = strSubN( walSegmentFile, strSize(archiveFile) + 1, HASH_TYPE_SHA1_SIZE_HEX); // If the checksums are the same then succeed but warn if archive-mode-check is enabled in case this is a // symptom of some other issue if (strEq(walSegmentChecksum, walSegmentRepoChecksum)) { if (modeCheck) { // Add warning to the result that will be returned to the main process strLstAddFmt( result.warnList, "WAL file '%s' already exists in the %s archive with the same checksum" "\nHINT: this is valid in some recovery scenarios but may also indicate a problem.", strZ(archiveFile), cfgOptionGroupName(cfgOptGrpRepo, repoData->repoIdx)); } // No need to copy to this repo destinationCopy[repoListIdx] = false; } // Else error so we don't overwrite the existing segment. Do not continue processing after this error since it // indicates corruption, split brain, or some other unrecoverable error. else { THROW_FMT( ArchiveDuplicateError, "WAL file '%s' already exists in the %s archive with a different checksum", strZ(archiveFile), cfgOptionGroupName(cfgOptGrpRepo, repoData->repoIdx)); } } // Else the repo needs a copy else destinationCopyAny = true; } // Append the checksum to the archive destination strCatFmt(archiveDestination, "-%s", strZ(walSegmentChecksum)); } // Copy the file if one or more repos require it if (destinationCopyAny) { // Source file is read once and copied to all repos StorageRead *const source = storageNewReadP(storageLocal(), walSource); // Is the file compressible during the copy? bool compressible = true; // If the file will be compressed then add compression filter if (isSegment && compressType != compressTypeNone) { compressExtCat(archiveDestination, compressType); ioFilterGroupAdd(ioReadFilterGroup(storageReadIo(source)), compressFilterP(compressType, compressLevel)); compressible = false; } // Initialize per-repo destination files StorageWrite **const destination = memNew(sizeof(StorageWrite *) * lstSize(repoList)); for (unsigned int repoListIdx = 0; repoListIdx < lstSize(repoList); repoListIdx++) { const ArchivePushFileRepoData *const repoData = lstGet(repoList, repoListIdx); // Does this repo need a copy? if (destinationCopy[repoListIdx]) { // Create destination file destination[repoListIdx] = storageNewWriteP( storageRepoIdxWrite(repoData->repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(repoData->archiveId), strZ(archiveDestination)), .compressible = compressible); // Add the encrypt filter to write the format header when required cipherBlockFormatFilterGroupWriteAddP( ioWriteFilterGroup(storageWriteIo(destination[repoListIdx])), repoData->cipherSpecArchive, repoData->format, .keyId = repoData->cipherIdArchive); } } // Open source file ioReadOpen(storageReadIo(source)); // Open the destination files now that we know the source file exists and is readable for (unsigned int repoListIdx = 0; repoListIdx < lstSize(repoList); repoListIdx++) { const unsigned int repoIdx = ((ArchivePushFileRepoData *)lstGet(repoList, repoListIdx))->repoIdx; if (destinationCopy[repoListIdx]) { destinationCopy[repoListIdx] = archivePushFileIo( archivePushFileIoTypeOpen, storageWriteIo(destination[repoListIdx]), NULL, repoIdx, errorList); } } // Copy data from source to destination Buffer *const read = bufNew(ioBufferSize()); do { // Read from source ioRead(storageReadIo(source), read); // Write to each destination for (unsigned int repoListIdx = 0; repoListIdx < lstSize(repoList); repoListIdx++) { const unsigned int repoIdx = ((ArchivePushFileRepoData *)lstGet(repoList, repoListIdx))->repoIdx; if (destinationCopy[repoListIdx]) { destinationCopy[repoListIdx] = archivePushFileIo( archivePushFileIoTypeWrite, storageWriteIo(destination[repoListIdx]), read, repoIdx, errorList); } } // Clear buffer bufUsedZero(read); } while (!ioReadEof(storageReadIo(source))); // Close the source and destination files ioReadClose(storageReadIo(source)); for (unsigned int repoListIdx = 0; repoListIdx < lstSize(repoList); repoListIdx++) { const unsigned int repoIdx = ((ArchivePushFileRepoData *)lstGet(repoList, repoListIdx))->repoIdx; if (destinationCopy[repoListIdx]) { destinationCopy[repoListIdx] = archivePushFileIo( archivePushFileIoTypeClose, storageWriteIo(destination[repoListIdx]), NULL, repoIdx, errorList); } } } // Throw any errors, even if some pushes were successful. It is important that PostgreSQL receives an error so it does not // remove the file. if (strLstSize(errorList) > 0) THROW_FMT(CommandError, CFGCMD_ARCHIVE_PUSH " command encountered error(s):\n%s", strZ(strLstJoin(errorList, "\n"))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/push/file.h0000664000175100017510000000415315265101745021302 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Push File ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_PUSH_FILE_H #define COMMAND_ARCHIVE_PUSH_FILE_H #include "common/compress/helper.h" #include "common/crypto/spec.h" #include "common/type/string.h" #include "storage/storage.h" /*********************************************************************************************************************************** Structure to hold information for each repository the archive file will be pushed to. An array of these must be passed to archivePushFile() with size equal to cfgOptionGroupIdxTotal(cfgOptGrpRepo). ***********************************************************************************************************************************/ typedef struct ArchivePushFileRepoData { unsigned int repoIdx; const String *archiveId; unsigned int format; // Repo format const CipherSpec *cipherSpecArchive; // Repo archive cipher spec const String *cipherIdArchive; // Archive key id, NULL when no id } ArchivePushFileRepoData; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ typedef struct ArchivePushFileResult { StringList *warnList; // Warnings from a successful operation } ArchivePushFileResult; // Copy a file from the source to the archive FN_EXTERN ArchivePushFileResult archivePushFile( const String *walSource, bool headerCheck, bool modeCheck, unsigned int pgVersion, uint64_t pgSystemId, const String *archiveFile, CompressType compressType, int compressLevel, const List *repoList, const StringList *priorErrorList); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/push/protocol.c0000664000175100017510000000512415265101745022216 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Push Protocol Handler ***********************************************************************************************************************************/ #include #include "command/archive/push/file.h" #include "command/archive/push/protocol.h" #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * archivePushFileProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Read parameters const String *const walSource = pckReadStrP(param); const bool headerCheck = pckReadBoolP(param); const bool modeCheck = pckReadBoolP(param); const unsigned int pgVersion = pckReadU32P(param); const uint64_t pgSystemId = pckReadU64P(param); const String *const archiveFile = pckReadStrP(param); const CompressType compressType = pckReadU32P(param); const int compressLevel = pckReadI32P(param); const StringList *const priorErrorList = pckReadStrLstP(param); // Read repo data List *const repoList = lstNewP(sizeof(ArchivePushFileRepoData)); pckReadArrayBeginP(param); while (!pckReadNullP(param)) { pckReadObjBeginP(param); ArchivePushFileRepoData repo = {.repoIdx = pckReadU32P(param)}; repo.archiveId = pckReadStrP(param); repo.format = pckReadU32P(param); repo.cipherSpecArchive = cipherSpecNewPack(param); repo.cipherIdArchive = pckReadStrP(param); pckReadObjEndP(param); lstAdd(repoList, &repo); } pckReadArrayEndP(param); // Push file const ArchivePushFileResult fileResult = archivePushFile( walSource, headerCheck, modeCheck, pgVersion, pgSystemId, archiveFile, compressType, compressLevel, repoList, priorErrorList); // Return result pckWriteStrLstP(protocolServerResultData(result), fileResult.warnList); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/push/protocol.h0000664000175100017510000000272415265101745022226 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Push Protocol Handler ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_PUSH_PROTOCOL_H #define COMMAND_ARCHIVE_PUSH_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process protocol requests FN_EXTERN ProtocolServerResult *archivePushFileProtocol(PackRead *param); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_ARCHIVE_PUSH_FILE STRID5("ap-f", 0x36e010) #define PROTOCOL_SERVER_HANDLER_ARCHIVE_PUSH_LIST \ {.command = PROTOCOL_COMMAND_ARCHIVE_PUSH_FILE, .process = archivePushFileProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/push/push.c0000664000175100017510000010156715265101745021344 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Push Command ***********************************************************************************************************************************/ #include #include #include #include "command/archive/common.h" #include "command/archive/push/file.h" #include "command/archive/push/protocol.h" #include "command/archive/push/push.h" #include "command/command.h" #include "command/control/common.h" #include "command/lock.h" #include "common/compress/helper.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/wait.h" #include "config/config.h" #include "config/exec.h" #include "config/load.h" #include "info/infoArchive.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "protocol/parallel.h" #include "storage/helper.h" /*********************************************************************************************************************************** Constants for log messages that are used multiple times to keep them consistent ***********************************************************************************************************************************/ #define UNABLE_TO_FIND_VALID_REPO_MSG "unable to find a valid repository" /*********************************************************************************************************************************** Ready file extension constants ***********************************************************************************************************************************/ #define STATUS_EXT_READY ".ready" #define STATUS_EXT_READY_SIZE (sizeof(STATUS_EXT_READY) - 1) /*********************************************************************************************************************************** Format the warning when a file is dropped ***********************************************************************************************************************************/ static String * archivePushDropWarning(const String *const walFile, const uint64_t queueMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, walFile); FUNCTION_TEST_PARAM(UINT64, queueMax); FUNCTION_TEST_END(); String *const size = strSizeFormat(queueMax); String *const result = strNewFmt("dropped WAL file '%s' because archive queue exceeded %s", strZ(walFile), strZ(size)); strFree(size); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Determine if the WAL process list has become large enough to drop ***********************************************************************************************************************************/ static bool archivePushDrop(const String *const walPath, const StringList *const processList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walPath); FUNCTION_LOG_PARAM(STRING_LIST, processList); FUNCTION_LOG_END(); ASSERT(walPath != NULL); ASSERT(processList != NULL); const uint64_t queueMax = cfgOptionUInt64(cfgOptArchivePushQueueMax); uint64_t queueSize = 0; MEM_CONTEXT_TEMP_BEGIN() { if (!strLstEmpty(processList)) { queueSize = storageInfoP( storagePg(), strNewFmt("%s/%s", strZ(walPath), strZ(strLstGet(processList, 0)))).size * strLstSize(processList); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, queueSize > queueMax); } /*********************************************************************************************************************************** Get the list of WAL files ready to be pushed according to PostgreSQL ***********************************************************************************************************************************/ static StringList * archivePushReadyList(const String *const walPath) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, walPath); FUNCTION_LOG_END(); ASSERT(walPath != NULL); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Read the ready files from the archive_status directory const StringList *const readyListRaw = strLstSort( storageListP( storagePg(), strNewFmt("%s/" PG_PATH_ARCHIVE_STATUS, strZ(walPath)), .expression = STRDEF("\\" STATUS_EXT_READY "$"), .errorOnMissing = true), sortOrderAsc); for (unsigned int readyIdx = 0; readyIdx < strLstSize(readyListRaw); readyIdx++) { const String *const ready = strLstGet(readyListRaw, readyIdx); strLstAddSub(result, ready, strSize(ready) - STATUS_EXT_READY_SIZE); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /*********************************************************************************************************************************** Determine which WAL files need to be pushed to the archive when in async mode This is the heart of the "look ahead" functionality in async archiving. Any files in the out directory that do not end in ok are removed and any ok files that do not have a corresponding ready file in archive_status (meaning it has been acknowledged by PostgreSQL) are removed. Then all ready files that do not have a corresponding ok file (meaning it has already been processed) are returned for processing. ***********************************************************************************************************************************/ static StringList * archivePushProcessList(const String *const walPath) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, walPath); FUNCTION_LOG_END(); ASSERT(walPath != NULL); StringList *result; MEM_CONTEXT_TEMP_BEGIN() { // Read the status files from the spool directory, then remove any files that do not end in ok and create a list of the // ok files for further processing. The spool out path is created by the foreground process before the async process is // forked. const StringList *const statusList = strLstSort( storageListP(storageSpool(), STORAGE_SPOOL_ARCHIVE_OUT_STR, .errorOnMissing = true), sortOrderAsc); StringList *const okList = strLstNew(); for (unsigned int statusIdx = 0; statusIdx < strLstSize(statusList); statusIdx++) { const String *const statusFile = strLstGet(statusList, statusIdx); if (strEndsWithZ(statusFile, STATUS_EXT_OK)) strLstAddSub(okList, statusFile, strSize(statusFile) - STATUS_EXT_OK_SIZE); else { storageRemoveP( storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s", strZ(statusFile)), .errorOnMissing = true); } } // Read the ready files from the archive_status directory const StringList *const readyList = archivePushReadyList(walPath); // Remove ok files that are not in the ready list const StringList *const okRemoveList = strLstMergeAnti(okList, readyList); for (unsigned int okRemoveIdx = 0; okRemoveIdx < strLstSize(okRemoveList); okRemoveIdx++) { storageRemoveP( storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s" STATUS_EXT_OK, strZ(strLstGet(okRemoveList, okRemoveIdx))), .errorOnMissing = true); } // Return all ready files that are not in the ok list result = strLstMove(strLstMergeAnti(readyList, okList), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /*********************************************************************************************************************************** Check that pg_control and archive.info match and get the archive id and archive cipher passphrase (if present) As much information as possible is collected here so that async archiving has as little work as possible to do for each file. Sync archiving does not benefit but it makes sense to use the same function. ***********************************************************************************************************************************/ typedef struct ArchivePushCheckResult { unsigned int pgVersion; // PostgreSQL version uint64_t pgSystemId; // PostgreSQL system id List *repoList; // Data for each repo StringList *errorList; // Errors while checking repos } ArchivePushCheckResult; static ArchivePushCheckResult archivePushCheck(const bool pgPathSet) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BOOL, pgPathSet); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ArchivePushCheckResult result = {.repoList = lstNewP(sizeof(ArchivePushFileRepoData)), .errorList = strLstNew()}; MEM_CONTEXT_TEMP_BEGIN() { // If we have access to pg_control then load it to get the pg version and system id. If we can't load pg_control then we'll // still compare the pg info stored in the repo to the WAL segment and also all the repos against each other. if (pgPathSet) { // Get info from pg_control PgControl pgControl = pgControlFromFile(storagePg(), cfgOptionStrNull(cfgOptPgVersionForce)); result.pgVersion = pgControl.version; result.pgSystemId = pgControl.systemId; } for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { TRY_BEGIN() { // Get the repo storage in case it is remote and encryption settings need to be pulled down storageRepoIdx(repoIdx); // Attempt to load the archive info file const InfoArchive *const info = infoArchiveLoadFile( storageRepoIdx(repoIdx), INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); // Get archive id for the most recent version -- archive-push will only operate against the most recent version const String *const archiveId = infoPgArchiveId(infoArchivePg(info), infoPgDataCurrentId(infoArchivePg(info))); const InfoPgData archiveInfo = infoPgData(infoArchivePg(info), infoPgDataCurrentId(infoArchivePg(info))); // Ensure that stanza version and system identifier match pg_control when available or the other repos when // pg_control is not available if (pgPathSet || repoIdx > 0) { if (result.pgVersion != archiveInfo.version || result.pgSystemId != archiveInfo.systemId) { THROW_FMT( ArchiveMismatchError, "%s version %s, system-id %" PRIu64 " do not match %s stanza version %s, system-id %" PRIu64 "\nHINT: are you archiving to the correct stanza?", pgPathSet ? PG_NAME : zNewFmt("%s stanza", cfgOptionGroupName(cfgOptGrpRepo, 0)), strZ(pgVersionToStr(result.pgVersion)), result.pgSystemId, cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(pgVersionToStr(archiveInfo.version)), archiveInfo.systemId); } } // Get the current archive key. Format >= 6 also requires the key id. const CipherSpecMap *const archiveCipherSpecMap = infoArchiveCipherSpecMap(info); const String *const archiveCipherId = cipherSpecMapIdCurrent(archiveCipherSpecMap); const CipherSpec *const archiveCipherSpec = archiveCipherId == NULL ? infoArchiveCipherSpec(info) : cipherSpecMapGet(archiveCipherSpecMap, archiveCipherId); MEM_CONTEXT_PRIOR_BEGIN() { result.pgVersion = archiveInfo.version; result.pgSystemId = archiveInfo.systemId; const ArchivePushFileRepoData archivePushFileRepoData = { .repoIdx = repoIdx, .archiveId = strDup(archiveId), .format = infoArchiveFormat(info), .cipherSpecArchive = cipherSpecDupP(archiveCipherSpec), .cipherIdArchive = strDup(archiveCipherId), }; lstAdd(result.repoList, &archivePushFileRepoData); } MEM_CONTEXT_PRIOR_END(); } CATCH_ANY() { strLstAddFmt( result.errorList, "%s: [%s] %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorTypeName(errorType()), errorMessage()); } TRY_END(); } } MEM_CONTEXT_TEMP_END(); // If no valid repos were found then error if (lstEmpty(result.repoList)) { ASSERT(strLstSize(result.errorList) > 0); THROW_FMT(RepoInvalidError, UNABLE_TO_FIND_VALID_REPO_MSG ":\n%s", strZ(strLstJoin(result.errorList, "\n"))); } FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN void cmdArchivePush(void) { FUNCTION_LOG_VOID(logLevelDebug); ASSERT(cfgCommand() == cfgCmdArchivePush); // PostgreSQL must be local pgIsLocalVerify(); MEM_CONTEXT_TEMP_BEGIN() { // Make sure there is a parameter to retrieve the WAL segment from const StringList *const commandParam = cfgCommandParam(); if (strLstSize(commandParam) != 1) THROW(ParamRequiredError, "WAL segment to push required"); // Test for stop file lockStopTest(); // Get the segment name const String *const walFile = walPath(strLstGet(commandParam, 0), cfgOptionStrNull(cfgOptPgPath), STR(cfgCommandName())); const String *const archiveFile = strBase(walFile); if (cfgOptionBool(cfgOptArchiveAsync)) { bool pushed = false; // Has the WAL segment been pushed yet? bool forked = false; // Has the async process been forked yet? bool throwOnError = false; // Should we throw errors? // pg1-path is not optional for async mode if (!cfgOptionTest(cfgOptPgPath)) { THROW_FMT( OptionRequiredError, "'" CFGCMD_ARCHIVE_PUSH "' command in async mode requires option '%s'", cfgOptionName(cfgOptPgPath)); } // Loop and wait for the WAL segment to be pushed Wait *const wait = waitNew(cfgOptionUInt64(cfgOptArchiveTimeout)); do { // Check if the WAL segment has been pushed. Errors will not be thrown on the first try to allow the async process // a chance to fix them. pushed = archiveAsyncStatus(archiveModePush, archiveFile, throwOnError, true); // If the WAL segment has not already been pushed then start the async process to push it. There's no point in // forking the async process off more than once so track that as well. Use an archive lock to prevent more than one // async process being launched. if (!pushed && !forked && cmdLockAcquireP(.returnOnNoLock = true)) { // The async process should not output on the console at all KeyValue *const optionReplace = kvNew(); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_CONSOLE), VARSTRDEF("off")); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_STDERR), VARSTRDEF("off")); // Generate command options StringList *const commandExec = cfgExecParam(cfgCmdArchivePush, cfgCmdRoleAsync, optionReplace, true, false); strLstInsert(commandExec, 0, cfgBin()); strLstAdd(commandExec, strPath(walFile)); // Create the spool out path in the foreground. If this fails (e.g. permission denied) the error is thrown here // and reaches the PostgreSQL log immediately, rather than only in the async process, where it cannot be written // to the spool as a status file and only appears as an archive timeout. storagePathCreateP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_OUT_STR); // Clear errors for the current archive file archiveAsyncErrorClear(archiveModePush, archiveFile); // Release the lock so the child process can acquire it cmdLockReleaseP(); // Execute the async process cmdAsyncExec(CFGCMD_ARCHIVE_PUSH, commandExec); // Mark the async process as forked so it doesn't get forked again. A single run of the async process should be // enough to do the job, running it again won't help anything. forked = true; } // Now that the async process has been launched, throw any errors that are found throwOnError = true; } while (!pushed && waitMore(wait)); // If the WAL segment was not pushed then error if (!pushed) { THROW_FMT( ArchiveTimeoutError, "unable to push WAL file '%s' to the archive asynchronously after %s second(s)\n" "HINT: check '%s' for errors.", strZ(archiveFile), strZ(cfgOptionDisplay(cfgOptArchiveTimeout)), strZ(cfgLoadLogFileName(cfgCmdRoleAsync))); } // Log success LOG_INFO_FMT("pushed WAL file '%s' to the archive asynchronously", strZ(archiveFile)); } else { // Check if the push queue has been exceeded if (cfgOptionTest(cfgOptArchivePushQueueMax) && archivePushDrop(strPath(walFile), archivePushReadyList(strPath(walFile)))) { LOG_WARN(strZ(archivePushDropWarning(archiveFile, cfgOptionUInt64(cfgOptArchivePushQueueMax)))); } // Else push the file else { // Check archive info for each repo const ArchivePushCheckResult archiveInfo = archivePushCheck(cfgOptionTest(cfgOptPgPath)); // Push the file to the archive const ArchivePushFileResult fileResult = archivePushFile( walFile, cfgOptionBool(cfgOptArchiveHeaderCheck), cfgOptionBool(cfgOptArchiveModeCheck), archiveInfo.pgVersion, archiveInfo.pgSystemId, archiveFile, compressTypeEnum(cfgOptionStrId(cfgOptCompressType)), cfgOptionInt(cfgOptCompressLevel), archiveInfo.repoList, archiveInfo.errorList); // If a warning was returned then log it for (unsigned int warnIdx = 0; warnIdx < strLstSize(fileResult.warnList); warnIdx++) LOG_WARN(strZ(strLstGet(fileResult.warnList, warnIdx))); // Log success LOG_INFO_FMT("pushed WAL file '%s' to the archive", strZ(archiveFile)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ typedef struct ArchivePushAsyncData { const String *walPath; // Path to pg_wal/pg_xlog StringList *walFileList; // List of wal files to process unsigned int walFileIdx; // Current index in the list to be processed CompressType compressType; // Type of compression for WAL segments int compressLevel; // Compression level for wal files ArchivePushCheckResult archiveInfo; // Archive info bool errorFound; // Has a job errored? If so, stop scheduling new jobs } ArchivePushAsyncData; static ProtocolParallelJob * archivePushAsyncCallback(void *const data, const unsigned int clientIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(UINT, clientIdx); FUNCTION_TEST_END(); ProtocolParallelJob *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // No special logic based on the client, we'll just get the next job (void)clientIdx; // Get a new job if there are any left. Stop scheduling new jobs as soon as one errors so the async process exits and the // next run rechecks the queue. Continuing past an error does not relieve disk pressure since PostgreSQL cannot recycle any // WAL past the oldest unarchived segment, so the sooner the queue is rechecked the sooner WAL can be dropped if needed. ArchivePushAsyncData *const jobData = data; if (!jobData->errorFound && jobData->walFileIdx < strLstSize(jobData->walFileList)) { const String *const walFile = strLstGet(jobData->walFileList, jobData->walFileIdx); jobData->walFileIdx++; PackWrite *const param = protocolPackNew(); pckWriteStrP(param, strNewFmt("%s/%s", strZ(jobData->walPath), strZ(walFile))); pckWriteBoolP(param, cfgOptionBool(cfgOptArchiveHeaderCheck)); pckWriteBoolP(param, cfgOptionBool(cfgOptArchiveModeCheck)); pckWriteU32P(param, jobData->archiveInfo.pgVersion); pckWriteU64P(param, jobData->archiveInfo.pgSystemId); pckWriteStrP(param, walFile); pckWriteU32P(param, jobData->compressType); pckWriteI32P(param, jobData->compressLevel); pckWriteStrLstP(param, jobData->archiveInfo.errorList); // Add data for each repo to push to pckWriteArrayBeginP(param); for (unsigned int repoListIdx = 0; repoListIdx < lstSize(jobData->archiveInfo.repoList); repoListIdx++) { ArchivePushFileRepoData *data = lstGet(jobData->archiveInfo.repoList, repoListIdx); pckWriteObjBeginP(param); pckWriteU32P(param, data->repoIdx); pckWriteStrP(param, data->archiveId); pckWriteU32P(param, data->format); cipherSpecPack(param, data->cipherSpecArchive); pckWriteStrP(param, data->cipherIdArchive); pckWriteObjEndP(param); } pckWriteArrayEndP(param); MEM_CONTEXT_PRIOR_BEGIN() { result = protocolParallelJobNew(VARSTR(walFile), PROTOCOL_COMMAND_ARCHIVE_PUSH_FILE, param); } MEM_CONTEXT_PRIOR_END(); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } FN_EXTERN void cmdArchivePushAsync(void) { FUNCTION_LOG_VOID(logLevelDebug); ASSERT(cfgCommand() == cfgCmdArchivePush && cfgCommandRole() == cfgCmdRoleAsync); // PostgreSQL must be local pgIsLocalVerify(); MEM_CONTEXT_TEMP_BEGIN() { // Make sure there is a parameter with the wal path const StringList *const commandParam = cfgCommandParam(); if (strLstSize(commandParam) != 1) THROW(ParamRequiredError, "WAL path to push required"); ArchivePushAsyncData jobData = { .walPath = strLstGet(commandParam, 0), .compressType = compressTypeEnum(cfgOptionStrId(cfgOptCompressType)), .compressLevel = cfgOptionInt(cfgOptCompressLevel), }; TRY_BEGIN() { // Test for stop file lockStopTest(); // Get a list of WAL files that are ready for processing jobData.walFileList = archivePushProcessList(jobData.walPath); // The archive-push:async command should not have been called unless there are WAL files to process if (strLstEmpty(jobData.walFileList)) THROW(AssertError, "no WAL files to process"); // Determine if the queue has grown large enough that all WAL should be dropped. The queue is measured against the ready // list (all WAL that PostgreSQL is waiting to archive) rather than the process list (WAL not yet pushed). If an earlier // WAL file errors repeatedly then PostgreSQL keeps retrying it and never acknowledges the later files that look-ahead // has already pushed, so those files drop out of the process list even though their WAL remains in pg_wal. Using the // process list here would let the queue grow without bound in that case, exactly when dropping is most needed. const bool drop = cfgOptionTest(cfgOptArchivePushQueueMax) && archivePushDrop(jobData.walPath, archivePushReadyList(jobData.walPath)); // If not dropping then limit the number of WAL files pushed in a single run to archive-push-batch-size. The queue is // only rechecked at the start of each run so this bounds how large it can grow before the next run rechecks it. Convert // the size to a segment count using the size of the first ready segment and always push at least one file. Files that // are not segments also count against the batch, so a run may push only those files when the batch size is small, but // they are acknowledged and drop out of the list for the next run. if (!drop) { // Find the first ready WAL segment. The ready list also contains backup and timeline history files, which are // smaller than a segment and may even be zero size when PostgreSQL did not sync them before the cluster was // copied, so they cannot be used to determine the segment size. const String *walSegment = NULL; for (unsigned int walFileIdx = 0; walFileIdx < strLstSize(jobData.walFileList); walFileIdx++) { if (walIsSegment(strLstGet(jobData.walFileList, walFileIdx))) { walSegment = strLstGet(jobData.walFileList, walFileIdx); break; } } // Apply the batch size when a segment is ready. When none is ready every file is small so there is nothing to // limit. A ready WAL segment should never be missing or zero size, so error immediately if it is. if (walSegment != NULL) { const uint64_t segmentSize = storageInfoP( storagePg(), strNewFmt("%s/%s", strZ(jobData.walPath), strZ(walSegment))).size; CHECK_FMT(FormatError, segmentSize != 0, "size of WAL segment '%s' is 0", strZ(walSegment)); const uint64_t batchMax = cfgOptionUInt64(cfgOptArchivePushBatchSize) / segmentSize; while (strLstSize(jobData.walFileList) > 1 && strLstSize(jobData.walFileList) > batchMax) strLstRemoveIdx(jobData.walFileList, strLstSize(jobData.walFileList) - 1); } } LOG_INFO_FMT( "push %u WAL file(s) to archive: %s%s", strLstSize(jobData.walFileList), strZ(strLstGet(jobData.walFileList, 0)), strLstSize(jobData.walFileList) == 1 ? "" : zNewFmt("...%s", strZ(strLstGet(jobData.walFileList, strLstSize(jobData.walFileList) - 1)))); // Drop files if queue max has been exceeded if (drop) { for (unsigned int walFileIdx = 0; walFileIdx < strLstSize(jobData.walFileList); walFileIdx++) { const String *const walFile = strLstGet(jobData.walFileList, walFileIdx); const String *const warning = archivePushDropWarning(walFile, cfgOptionUInt64(cfgOptArchivePushQueueMax)); archiveAsyncStatusOkWrite(archiveModePush, walFile, warning); LOG_WARN(strZ(warning)); } } // Else continue processing else { // Check archive info for each repo jobData.archiveInfo = archivePushCheck(true); // Create the parallel executor ProtocolParallel *const parallelExec = protocolParallelNew( cfgOptionUInt64(cfgOptProtocolTimeout) / 2, archivePushAsyncCallback, &jobData); for (unsigned int processIdx = 1; processIdx <= cfgOptionUInt(cfgOptProcessMax); processIdx++) protocolParallelClientAdd(parallelExec, protocolLocalGet(protocolStorageTypeRepo, 0, processIdx)); // Process jobs MEM_CONTEXT_TEMP_RESET_BEGIN() { do { const unsigned int completed = protocolParallelProcess(parallelExec); for (unsigned int jobIdx = 0; jobIdx < completed; jobIdx++) { protocolKeepAlive(); // Get the job and job key ProtocolParallelJob *const job = protocolParallelResult(parallelExec); const unsigned int processId = protocolParallelJobProcessId(job); const String *const walFile = varStr(protocolParallelJobKey(job)); // The job was successful if (protocolParallelJobErrorCode(job) == 0) { // Output file warnings const StringList *const fileWarnList = pckReadStrLstP(protocolParallelJobResult(job)); for (unsigned int warnIdx = 0; warnIdx < strLstSize(fileWarnList); warnIdx++) LOG_WARN_PID(processId, strZ(strLstGet(fileWarnList, warnIdx))); // Log success LOG_DETAIL_PID_FMT(processId, "pushed WAL file '%s' to the archive", strZ(walFile)); // Write the status file archiveAsyncStatusOkWrite( archiveModePush, walFile, strLstEmpty(fileWarnList) ? NULL : strLstJoin(fileWarnList, "\n")); } // Else the job errored else { jobData.errorFound = true; LOG_WARN_PID_FMT( processId, "could not push WAL file '%s' to the archive (will be retried): [%d] %s", strZ(walFile), protocolParallelJobErrorCode(job), strZ(protocolParallelJobErrorMessage(job))); archiveAsyncStatusErrorWrite( archiveModePush, walFile, protocolParallelJobErrorCode(job), protocolParallelJobErrorMessage(job)); } protocolParallelJobFree(job); } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } while (!protocolParallelDone(parallelExec)); } MEM_CONTEXT_TEMP_END(); // If processing was stopped early because a job errored then log a warning. The remaining WAL is left for the next // run, which will recheck the queue and drop WAL if it now exceeds queue-max. if (jobData.errorFound) LOG_WARN("stopped archive-push after an error, remaining WAL will be processed on the next run"); } } // On any global error write a single error file to cover all unprocessed files CATCH_FATAL() { archiveAsyncStatusErrorWrite(archiveModePush, NULL, errorCode(), STR(errorMessage())); RETHROW(); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/archive/push/push.h0000664000175100017510000000150315265101745021336 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Push Command ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_PUSH_PUSH_H #define COMMAND_ARCHIVE_PUSH_PUSH_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Push a WAL segment to the repository FN_EXTERN void cmdArchivePush(void); // Async version of archive push that runs in parallel for performance FN_EXTERN void cmdArchivePushAsync(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/0000775000175100017510000000000015265101745017054 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/backup.c0000664000175100017510000003455115265101745020475 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Command ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/archive/find.h" #include "command/backup/backup.h" #include "command/backup/common.h" #include "command/backup/protocol.h" #include "command/check/common.h" #include "command/control/common.h" #include "command/lock.h" #include "command/stanza/common.h" #include "common/compress/helper.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/format/cipherBlockFormat.h" #include "common/io/filter/size.h" #include "common/log.h" #include "common/regExp.h" #include "common/time.h" #include "common/type/convert.h" #include "common/type/json.h" #include "config/common.h" #include "config/config.h" #include "config/parse.h" #include "db/helper.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "info/manifest/manifest.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "protocol/parallel.h" #include "storage/helper.h" #include "version.h" /*********************************************************************************************************************************** Get the postgres database and storage objects ***********************************************************************************************************************************/ #define FUNCTION_LOG_BACKUP_DATA_TYPE \ BackupData * #define FUNCTION_LOG_BACKUP_DATA_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "BackupData", buffer, bufferSize) typedef struct BackupData { unsigned int pgIdxPrimary; // cfgOptGrpPg index of the primary Db *dbPrimary; // Database connection to the primary const Storage *storagePrimary; // Storage object for the primary const String *hostPrimary; // Host name of the primary unsigned int pgIdxStandby; // cfgOptGrpPg index of the standby Db *dbStandby; // Database connection to the standby const String *hostStandby; // Host name of the standby const InfoArchive *archiveInfo; // Archive info const String *archiveId; // Archive where backup WAL will be stored unsigned int timeline; // Primary timeline unsigned int version; // PostgreSQL version unsigned int walSegmentSize; // PostgreSQL wal segment size PgPageSize pageSize; // PostgreSQL page size } BackupData; static BackupData * backupInit(const InfoBackup *const infoBackup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(infoBackup != NULL); // Initialize for offline backup BackupData *const result = memNew(sizeof(BackupData)); *result = (BackupData){0}; // Don't allow backup from standby when offline unsigned int backupStandby = cfgOptionSeq(cfgOptBackupStandby); const InfoPgData infoPg = infoPgDataCurrent(infoBackupPg(infoBackup)); if (!cfgOptionBool(cfgOptOnline) && backupStandby != CFGOPTVAL_BACKUP_STANDBY_N) { LOG_WARN( "option " CFGOPT_BACKUP_STANDBY " is enabled but backup is offline - backups will be performed from the primary"); backupStandby = CFGOPTVAL_BACKUP_STANDBY_N; } // Get database info when online PgControl pgControl = {0}; if (cfgOptionBool(cfgOptOnline)) { const DbGetResult dbInfo = dbGet(backupStandby == CFGOPTVAL_BACKUP_STANDBY_N, true, backupStandby); // If no standby was found but using the primary is allowed then warn and proceed if (backupStandby == CFGOPTVAL_BACKUP_STANDBY_PREFER && dbInfo.standby == NULL) LOG_WARN("unable to find a standby to perform the backup, using primary instead"); result->pgIdxPrimary = dbInfo.primaryIdx; result->dbPrimary = dbInfo.primary; if (dbInfo.standby != NULL) { ASSERT(dbInfo.standby != NULL); result->pgIdxStandby = dbInfo.standbyIdx; result->dbStandby = dbInfo.standby; result->hostStandby = cfgOptionIdxStrNull(cfgOptPgHost, result->pgIdxStandby); } // Get pg_control info from the primary pgControl = dbPgControl(result->dbPrimary); } // Else get pg_control info directly from the file else pgControl = pgControlFromFile(storagePgIdx(result->pgIdxPrimary), cfgOptionStrNull(cfgOptPgVersionForce)); // Add primary info result->storagePrimary = storagePgIdx(result->pgIdxPrimary); result->hostPrimary = cfgOptionIdxStrNull(cfgOptPgHost, result->pgIdxPrimary); result->timeline = pgControl.timeline; result->version = pgControl.version; result->walSegmentSize = pgControl.walSegmentSize; result->pageSize = pgControl.pageSize; // Validate pg_control info against the stanza if (result->version != infoPg.version || pgControl.systemId != infoPg.systemId) { THROW_FMT( BackupMismatchError, PG_NAME " version %s, system-id %" PRIu64 " do not match stanza version %s, system-id %" PRIu64 "\n" "HINT: is this the correct stanza?", strZ(pgVersionToStr(pgControl.version)), pgControl.systemId, strZ(pgVersionToStr(infoPg.version)), infoPg.systemId); } // If checksum page is not explicitly set then automatically enable it when checksums are available if (!cfgOptionTest(cfgOptChecksumPage)) { // If online then use the value in pg_control to set checksum-page if (cfgOptionBool(cfgOptOnline)) { cfgOptionSet(cfgOptChecksumPage, cfgSourceParam, VARBOOL(pgControl.pageChecksumVersion != 0)); } // Else set to false. An offline cluster is likely to have false positives so better if the user enables manually. else cfgOptionSet(cfgOptChecksumPage, cfgSourceParam, BOOL_FALSE_VAR); } // Else if checksums have been explicitly enabled but are not available then warn and reset. ??? We should be able to make this // determination when offline as well, but the integration tests don't write pg_control accurately enough to support it. else if (cfgOptionBool(cfgOptOnline) && pgControl.pageChecksumVersion == 0 && cfgOptionBool(cfgOptChecksumPage)) { LOG_WARN(CFGOPT_CHECKSUM_PAGE " option set to true but checksums are not enabled on the cluster, resetting to false"); cfgOptionSet(cfgOptChecksumPage, cfgSourceParam, BOOL_FALSE_VAR); } // Get archive info if (cfgOptionBool(cfgOptArchiveCheck)) { result->archiveInfo = infoArchiveLoadFile(storageRepo(), INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMain()); result->archiveId = infoArchiveId(result->archiveInfo); // Error if the info files do not agree with each other. A stanza left half migrated by an interrupted upgrade should be // repaired before more is written into it, and backup is the command that will run against it next. checkStanzaInfo(cfgOptionGroupIdxDefault(cfgOptGrpRepo), infoArchivePg(result->archiveInfo), infoBackupPg(infoBackup)); } FUNCTION_LOG_RETURN(BACKUP_DATA, result); } // {uncrustify_off - order required for C includes} #include "command/backup/label.c.inc" #include "command/backup/option.c.inc" #include "command/backup/incr.c.inc" #include "command/backup/resume.c.inc" #include "command/backup/db.c.inc" #include "command/backup/complete.c.inc" #include "command/backup/process.c.inc" // {uncrustify_on} /**********************************************************************************************************************************/ FN_EXTERN void cmdBackup(void) { FUNCTION_LOG_VOID(logLevelDebug); // Test for stop file lockStopTest(); MEM_CONTEXT_TEMP_BEGIN() { // If the repo option was not provided and more than one repo is configured, then log the default repo chosen if (!cfgOptionTest(cfgOptRepo) && cfgOptionGroupIdxTotal(cfgOptGrpRepo) > 1) { LOG_INFO_FMT( "repo option not specified, defaulting to %s", cfgOptionGroupName(cfgOptGrpRepo, cfgOptionGroupIdxDefault(cfgOptGrpRepo))); } // Get the repo storage in case it is remote and encryption settings need to be pulled down storageRepo(); // Load backup.info InfoBackup *const infoBackup = infoBackupLoadFileReconstruct(storageRepo(), INFO_BACKUP_PATH_FILE_STR, cfgCipherSpecMain()); const InfoPgData infoPg = infoPgDataCurrent(infoBackupPg(infoBackup)); const CipherSpec *const cipherSpecManifest = infoBackupCipherSpec(infoBackup); // Get pg storage and database objects BackupData *const backupData = backupInit(infoBackup); // Get the start timestamp which will later be written into the manifest to track total backup time const time_t timestampStart = backupTime(backupData, false); // Check if there is a prior manifest when backup type is diff/incr Manifest *const manifestPrior = backupBuildIncrPrior(infoBackup); // Start the backup const BackupStartResult backupStartResult = backupStart(backupData); // Build the manifest const ManifestBlockIncrMap blockIncrMap = backupBlockIncrMap(); Manifest *const manifest = manifestNewBuild( backupData->storagePrimary, infoPg.version, infoPg.catalogVersion, infoBackupFormat(infoBackup), timestampStart, cfgOptionBool(cfgOptOnline), cfgOptionBool(cfgOptChecksumPage), cfgOptionBool(cfgOptRepoBundle), cfgOptionBool(cfgOptRepoBlock), &blockIncrMap, strLstNewVarLst(cfgOptionLst(cfgOptExclude)), backupStartResult.tablespaceList); // Validate the manifest using the copy start time manifestBuildValidate( manifest, cfgOptionBool(cfgOptDelta), backupTime(backupData, true), compressTypeEnum(cfgOptionStrId(cfgOptCompressType))); // Build an incremental backup if type is not full (manifestPrior will be freed in this call) if (!backupBuildIncr(manifest, manifestPrior, backupStartResult.walSegmentName)) manifestCipherSpecSet(manifest, cipherSpecGen(cfgOptionStrId(cfgOptRepoCipherType), manifestFormat(manifest))); // Set delta if it is not already set and the manifest requires it if (!cfgOptionBool(cfgOptDelta) && varBool(manifestData(manifest)->backupOptionDelta)) cfgOptionSet(cfgOptDelta, cfgSourceParam, BOOL_TRUE_VAR); // Resume a backup when possible if (!backupResume(manifest, cipherSpecManifest)) { manifestBackupLabelSet( manifest, backupLabelCreate( (BackupType)cfgOptionStrId(cfgOptType), manifestData(manifest)->backupLabelPrior, timestampStart)); } // Save the manifest before processing starts backupManifestSaveCopy(manifest, cipherSpecManifest, false); // Process the backup manifest const unsigned int warningTotal = backupProcess(backupData, manifest, cipherSpecManifest); // Check that the clusters are alive and correctly configured after the backup backupDbPing(backupData, true); // The standby db object and protocol won't be used anymore so free them if (backupData->dbStandby != NULL) { dbFree(backupData->dbStandby); protocolHelperFree(protocolRemoteGet(protocolStorageTypePg, backupData->pgIdxStandby, false)); } // Stop the backup const BackupStopResult backupStopResult = backupStop(backupData, manifest); // Complete manifest manifestBuildComplete( manifest, backupStartResult.lsn, backupStartResult.walSegmentName, backupStopResult.timestamp, backupStopResult.lsn, backupStopResult.walSegmentName, infoPg.id, infoPg.systemId, backupStartResult.dbList, cfgOptionBool(cfgOptArchiveCheck), cfgOptionBool(cfgOptArchiveCopy), cfgOptionUInt(cfgOptBufferSize), cfgOptionUInt(cfgOptCompressLevel), cfgOptionUInt(cfgOptCompressLevelNetwork), cfgOptionBool(cfgOptRepoHardlink), cfgOptionUInt(cfgOptProcessMax), backupData->dbStandby != NULL, cfgOptionTest(cfgOptAnnotation) ? cfgOptionKv(cfgOptAnnotation) : NULL); // The primary db object won't be used anymore so free it dbFree(backupData->dbPrimary); // Check and copy WAL segments required to make the backup consistent backupArchiveCheckCopy(backupData, manifest, cipherSpecManifest); // The primary protocol connection won't be used anymore so free it. This needs to happen after backupArchiveCheckCopy() so // the backup lock is held on the remote which allows conditional archiving based on the backup lock. Any further access to // the primary storage object may result in an error (likely eof). protocolHelperFree(protocolRemoteGet(protocolStorageTypePg, backupData->pgIdxPrimary, false)); // Complete the backup LOG_INFO_FMT("new backup label = %s", strZ(manifestData(manifest)->backupLabel)); backupComplete(infoBackup, manifest); // Backup info LOG_INFO_FMT( "%s backup size = %s, file total = %u", zNewStrId(manifestData(manifest)->backupType), strZ(strSizeFormat(infoBackupDataByLabel(infoBackup, manifestData(manifest)->backupLabel)->backupInfoSizeDelta)), manifestFileTotal(manifest)); if (warningTotal > 0) LOG_WARN_FMT(CFGCMD_BACKUP " command encountered %u checksum warning(s), check the log file for details", warningTotal); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/backup.h0000664000175100017510000000124715265101745020476 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Command ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_BACKUP_H #define COMMAND_BACKUP_BACKUP_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Make a backup FN_EXTERN void cmdBackup(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/blockIncr.c0000664000175100017510000004771615265101745021145 0ustar00runnerrunner/*********************************************************************************************************************************** Block Incremental Filter ***********************************************************************************************************************************/ #include #include "command/backup/blockIncr.h" #include "command/backup/blockMap.h" #include "common/compress/helper.h" #include "common/crypto/cipherBlock.h" #include "common/crypto/xxhash.h" #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/bufferWrite.h" #include "common/io/filter/buffer.h" #include "common/io/filter/size.h" #include "common/io/io.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct BlockIncr { unsigned int reference; // Current backup reference uint64_t bundleId; // Bundle id StringId compressType; // Compress filter type const Pack *compressParam; // Compress filter parameters const Pack *encryptParam; // Encrypt filter parameters unsigned int blockNo; // Block number uint64_t superBlockNo; // Block no in super block uint64_t blockOffset; // Block offset uint64_t superBlockSize; // Super block size_t blockSize; // Block size size_t checksumSize; // Checksum size Buffer *block; // Block buffer Buffer *blockOut; // Block output buffer IoWrite *blockOutWrite; // Write to the block block buffer List *blockOutList; // List of block map items that need an updated size size_t blockOutSize; // Amount written to block output (excluding block no) size_t blockOutOffset; // Block output offset (already copied to output buffer) const BlockMap *blockMapPrior; // Prior block map BlockMap *blockMapOut; // Output block map uint64_t blockMapOutSize; // Output block map size (if any) bool blockMapWrite; // Write block map (at least one new/changed block) size_t inputOffset; // Input offset bool inputSame; // Input the same data bool done; // Is the filter done? } BlockIncr; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void blockIncrToLog(const BlockIncr *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{superBlockSize, %" PRIu64 ", blockSize: %zu}", this->superBlockSize, this->blockSize); } #define FUNCTION_LOG_BLOCK_INCR_TYPE \ BlockIncr * #define FUNCTION_LOG_BLOCK_INCR_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, blockIncrToLog, buffer, bufferSize) /*********************************************************************************************************************************** Generate block incremental ***********************************************************************************************************************************/ static void blockIncrProcess(THIS_VOID, const Buffer *const input, Buffer *const output) { THIS(BlockIncr); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BLOCK_INCR, this); FUNCTION_LOG_PARAM(BUFFER, input); FUNCTION_LOG_PARAM(BUFFER, output); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(output != NULL); // Is the input done? this->done = input == NULL; // Loop until the input is consumed or there is output do { // If not done and not flushing out then get more block data if (!this->done && this->blockOutOffset == 0) { // If all input can be copied if (bufUsed(input) - this->inputOffset <= bufRemains(this->block)) { bufCatSub(this->block, input, this->inputOffset, bufUsed(input) - this->inputOffset); this->inputOffset = 0; // Same input no longer required this->inputSame = false; } // Else only part of the input can be copied else { const size_t copySize = bufRemains(this->block); bufCatSub(this->block, input, this->inputOffset, copySize); this->inputOffset += copySize; // The same input will be needed again to copy the rest this->inputSame = true; } } // If done with a partial block or block is full if ((this->done && bufUsed(this->block) > 0) || bufUsed(this->block) == this->blockSize) { MEM_CONTEXT_TEMP_BEGIN() { // Get block checksum const Buffer *const checksum = xxHashOne(this->checksumSize, this->block); // Does the block exist in the input map? const BlockMapItem *const blockMapItemIn = this->blockMapPrior != NULL && this->blockNo < blockMapSize(this->blockMapPrior) ? blockMapGet(this->blockMapPrior, this->blockNo) : NULL; // If the block is new or has changed then write it if (blockMapItemIn == NULL || memcmp(blockMapItemIn->checksum, bufPtrConst(checksum), this->checksumSize) != 0) { // Begin the super block if (this->blockOutWrite == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->blockOutWrite = ioBufferWriteNew(this->blockOut); this->blockOutList = lstNewP(sizeof(unsigned int)); } MEM_CONTEXT_OBJ_END(); // Add compress filter if (this->compressParam != NULL) { ioFilterGroupAdd( ioWriteFilterGroup(this->blockOutWrite), compressFilterPack(this->compressType, this->compressParam)); } // Add encrypt filter if (this->encryptParam != NULL) ioFilterGroupAdd(ioWriteFilterGroup(this->blockOutWrite), cipherBlockNewPack(this->encryptParam)); // Add size filter ioFilterGroupAdd(ioWriteFilterGroup(this->blockOutWrite), ioSizeNew()); ioWriteOpen(this->blockOutWrite); } // Copy block data through the filters ioCopyP(ioBufferReadNewOpen(this->block), this->blockOutWrite); this->blockOutSize += bufUsed(this->block); bufUsedZero(this->block); // Write to block map BlockMapItem blockMapItem = { .reference = this->reference, .superBlockSize = this->superBlockSize, .bundleId = this->bundleId, .offset = this->blockOffset, .block = this->superBlockNo, }; memcpy(blockMapItem.checksum, bufPtrConst(checksum), bufUsed(checksum)); const unsigned int blockMapItemIdx = blockMapSize(this->blockMapOut); blockMapAdd(this->blockMapOut, &blockMapItem); lstAdd(this->blockOutList, &blockMapItemIdx); // Increment super block no this->superBlockNo++; } // Else write a reference to the block in the prior backup else { blockMapAdd(this->blockMapOut, blockMapItemIn); bufUsedZero(this->block); } this->blockNo++; } MEM_CONTEXT_TEMP_END(); } // Write the super block if (this->blockOutWrite != NULL && (this->done || this->blockOutSize >= this->superBlockSize)) { // Close write ioWriteClose(this->blockOutWrite); // Update size and super block size for items already added to the block map PackRead *const filter = ioFilterGroupResultP(ioWriteFilterGroup(this->blockOutWrite), SIZE_FILTER_TYPE); const uint64_t blockOutSize = pckReadU64P(filter); for (unsigned int blockMapIdx = 0; blockMapIdx < lstSize(this->blockOutList); blockMapIdx++) { BlockMapItem *const blockMapItem = blockMapGet( this->blockMapOut, *(unsigned int *)lstGet(this->blockOutList, blockMapIdx)); blockMapItem->size = blockOutSize; blockMapItem->superBlockSize = this->blockOutSize; } pckReadFree(filter); lstFree(this->blockOutList); // Set to NULL so the super block can be flushed ioWriteFree(this->blockOutWrite); this->blockOutWrite = NULL; // Increment block offset this->blockOffset += blockOutSize; // Reset block out size and super block no this->blockOutSize = 0; this->superBlockNo = 0; // Block map must be written since there are new/changed blocks this->blockMapWrite = true; } // Write the block map if done processing and there are new/changed blocks or block list has been truncated. A block map // with no blocks is never written because it cannot be read back. That happens when the file is truncated to zero during // the backup, in which case the file is stored as truncated and the map is not needed. if (this->done && this->blockOutOffset == 0 && (this->blockMapWrite || (this->blockMapPrior != NULL && blockMapSize(this->blockMapOut) > 0 && blockMapSize(this->blockMapOut) < blockMapSize(this->blockMapPrior)))) { MEM_CONTEXT_TEMP_BEGIN() { // Size of block output before starting to write the map const size_t blockOutBegin = bufUsed(this->blockOut); // Write the map IoWrite *const write = ioBufferWriteNew(this->blockOut); if (this->encryptParam != NULL) ioFilterGroupAdd(ioWriteFilterGroup(write), cipherBlockNewPack(this->encryptParam)); // Write the map ioWriteOpen(write); blockMapWrite(this->blockMapOut, write, this->blockSize, this->checksumSize); ioWriteClose(write); // Get total bytes written for the map this->blockMapOutSize = bufUsed(this->blockOut) - blockOutBegin; } MEM_CONTEXT_TEMP_END(); } // Copy to output buffer if output has been completely written if (this->blockOutWrite == NULL) { const size_t blockOutSize = bufUsed(this->blockOut) - this->blockOutOffset; if (blockOutSize > 0) { // Output the rest of the block if it will fit if (bufRemains(output) >= blockOutSize) { bufCatSub(output, this->blockOut, this->blockOutOffset, blockOutSize); bufUsedZero(this->blockOut); this->blockOutOffset = 0; this->inputSame = this->inputOffset != 0; } // Else output as much of the block as possible else { const size_t blockOutSize = bufRemains(output); bufCatSub(output, this->blockOut, this->blockOutOffset, blockOutSize); this->blockOutOffset += blockOutSize; this->inputSame = true; } } } } while (this->inputSame && bufEmpty(output)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** The result is the size of the block map ***********************************************************************************************************************************/ static Pack * blockIncrResult(THIS_VOID) { THIS(BlockIncr); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BLOCK_INCR, this); FUNCTION_LOG_END(); ASSERT(this != NULL); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteU64P(packWrite, this->blockMapOutSize); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /*********************************************************************************************************************************** Is filter done? ***********************************************************************************************************************************/ static bool blockIncrDone(const THIS_VOID) { THIS(const BlockIncr); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BLOCK_INCR, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done && !this->inputSame); } /*********************************************************************************************************************************** Should the same input be provided again? ***********************************************************************************************************************************/ static bool blockIncrInputSame(const THIS_VOID) { THIS(const BlockIncr); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BLOCK_INCR, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * blockIncrNew( const uint64_t superBlockSize, const size_t blockSize, const size_t checksumSize, const unsigned int reference, const uint64_t bundleId, const uint64_t bundleOffset, const Buffer *const blockMapPrior, const IoFilter *const compress, const IoFilter *const encrypt) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(UINT64, superBlockSize); FUNCTION_LOG_PARAM(SIZE, blockSize); FUNCTION_LOG_PARAM(SIZE, checksumSize); FUNCTION_LOG_PARAM(UINT, reference); FUNCTION_LOG_PARAM(UINT64, bundleId); FUNCTION_LOG_PARAM(UINT64, bundleOffset); FUNCTION_LOG_PARAM(BUFFER, blockMapPrior); FUNCTION_LOG_PARAM(IO_FILTER, compress); FUNCTION_LOG_PARAM(IO_FILTER, encrypt); FUNCTION_LOG_END(); OBJ_NEW_BEGIN(BlockIncr, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (BlockIncr) { .superBlockSize = (superBlockSize / blockSize + (superBlockSize % blockSize == 0 ? 0 : 1)) * blockSize, .blockSize = blockSize, .checksumSize = checksumSize, .reference = reference, .bundleId = bundleId, .blockOffset = bundleOffset, .block = bufNew(blockSize), .blockOut = bufNew(0), .blockMapOut = blockMapNew(), }; // Duplicate compress filter if (compress != NULL) { this->compressType = ioFilterType(compress); this->compressParam = pckDup(ioFilterParamList(compress)); } // Duplicate encrypt filter if (encrypt != NULL) this->encryptParam = pckDup(ioFilterParamList(encrypt)); // Load prior block map if (blockMapPrior) { MEM_CONTEXT_TEMP_BEGIN() { IoRead *const read = ioBufferReadNewOpen(blockMapPrior); MEM_CONTEXT_PRIOR_BEGIN() { this->blockMapPrior = blockMapNewRead(read, blockSize, checksumSize); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); } } OBJ_NEW_END(); // Create param list Pack *paramList; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteU64P(packWrite, this->superBlockSize); pckWriteU64P(packWrite, blockSize); pckWriteU64P(packWrite, checksumSize); pckWriteU32P(packWrite, reference); pckWriteU64P(packWrite, bundleId); pckWriteU64P(packWrite, bundleOffset); pckWriteBinP(packWrite, blockMapPrior); pckWritePackP(packWrite, this->compressParam); if (this->compressParam != NULL) pckWriteStrIdP(packWrite, this->compressType); pckWritePackP(packWrite, this->encryptParam); pckWriteEndP(packWrite); paramList = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( BLOCK_INCR_FILTER_TYPE, this, paramList, .done = blockIncrDone, .inOut = blockIncrProcess, .inputSame = blockIncrInputSame, .result = blockIncrResult)); } FN_EXTERN IoFilter * blockIncrNewPack(const Pack *const paramList) { IoFilter *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackRead *const paramListPack = pckReadNew(paramList); const uint64_t superBlockSize = pckReadU64P(paramListPack); const size_t blockSize = (size_t)pckReadU64P(paramListPack); const size_t checksumSize = (size_t)pckReadU64P(paramListPack); const unsigned int reference = pckReadU32P(paramListPack); const uint64_t bundleId = pckReadU64P(paramListPack); const uint64_t bundleOffset = pckReadU64P(paramListPack); const Buffer *blockMapPrior = pckReadBinP(paramListPack); // Create compress filter const Pack *const compressParam = pckReadPackP(paramListPack); const IoFilter *compress = NULL; if (compressParam != NULL) compress = compressFilterPack(pckReadStrIdP(paramListPack), compressParam); // Create encrypt filter const Pack *const encryptParam = pckReadPackP(paramListPack); const IoFilter *encrypt = NULL; if (encryptParam != NULL) encrypt = cipherBlockNewPack(encryptParam); result = ioFilterMove( blockIncrNew( superBlockSize, blockSize, checksumSize, reference, bundleId, bundleOffset, blockMapPrior, compress, encrypt), memContextPrior()); } MEM_CONTEXT_TEMP_END(); return result; } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/blockIncr.h0000664000175100017510000000701015265101745021131 0ustar00runnerrunner/*********************************************************************************************************************************** Block Incremental Filter The block incremental filter builds a super block list and map (see BlockMap object) either from a file (the first time) or from a file and a prior map (each subsequent time). The super block list is stored first and the block map second. The filter returns the size of the block map so it can be extracted separately. Super blocks contain a list of blocks that may or may not be contiguous. The advantage is that multiple blocks can be compressed together which leads to better compression. The disadvantage is that the super block must be read sequentially to get at any block in the super block. For large block sizes the block size will usually equal the super block size to negate the read penalty. The super block list is followed by the block map, which is encrypted separately when required but not compressed. The return value of the filter is the stored block map size. Combined with the repo size this allows the block map to be read separately. The map is duplicated in each backup where the file has changed so the block map only needs to be retrieved from the most recent backup. However, the block map may reference super block lists (or parts thereof) in any prior (or the current) backup. The block incremental should be read using BlockDelta since reconstructing the delta is quite involved. The xxHash algorithm is used to determine which blocks have changed. A 128-bit xxHash is generated and then checksumSize bytes are used from the hash depending on the size of the block. xxHash claims to have excellent dispersion characteristics, which has been verified by testing with SMHasher and a custom test suite. xxHash-32 is used for up to 4MiB content blocks in lz4 and the lower 32-bits of xxHash-64 is used for content blocks in zstd. In general 32-bit checksums are pretty common defaults across filesystems such as BtrFS and ZFS. We use at least 5 bytes even for the smallest blocks since we are looking for changes and not just corruption. Ultimately if there is a collision and a block change is not detected it will almost certainly be caught by the overall SHA1 file checksum. This will fail the backup, which is not ideal, but better than restoring corrupted data. ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_BLOCK_INCR_H #define COMMAND_BACKUP_BLOCK_INCR_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define BLOCK_INCR_FILTER_TYPE STRID5("blk-incr", 0x90dc9dad820) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *blockIncrNew( uint64_t superBlockSize, size_t blockSize, size_t checksumSize, unsigned int reference, uint64_t bundleId, uint64_t bundleOffset, const Buffer *blockMapPrior, const IoFilter *compress, const IoFilter *encrypt); FN_EXTERN IoFilter *blockIncrNewPack(const Pack *paramList); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/blockMap.c0000664000175100017510000005241415265101745020756 0ustar00runnerrunner/*********************************************************************************************************************************** Block Incremental Map The block map is stored as a flag and a series of reference, super block, and block info: - Varint-128 flag that contains the version and info about the map (e.g. are the super blocks and blocks equal size). - List of references: - Varint-128 encoded reference (which is an index into the reference list maintained in the manifest). If this is the first time the reference appears it will be followed by a bundle id and an offset if they are not 0. If the reference has appeared before it might update the offset or be a continuation of a prior super block. Continuations happen when a super block is split by a newer super block. The continuation allows the prior super block values for the reference to be used without encoding them again. - List of super blocks: - Varint-128 encoded super block size. The very first size in the map will be encoded directly and subsequent sizes will be encoded as the delta from the last size. - List of blocks: - Varint-128 encoded block number if super block size does not equal block size. If they are equal then there is one block per super block so no reason to encode the block number. - Checksum. References, super blocks, and blocks are encoded with a bit that indicates when the last one has been reached. ***********************************************************************************************************************************/ #include #include "command/backup/blockMap.h" #include "common/debug.h" #include "common/log.h" /**********************************************************************************************************************************/ #define BLOCK_MAP_FLAG_LAST 1 // Last reference, super block, etc. #define BLOCK_MAP_FLAG_OFFSET 4 // Reference has an offset #define BLOCK_MAP_FLAG_BUNDLE_ID 2 // Reference has a bundle id #define BLOCK_MAP_FLAG_CONTINUE 2 // Reference continues a prior super block #define BLOCK_MAP_FLAG_CONTINUE_LAST 4 // Continued super block is last for the reference #define BLOCK_MAP_REFERENCE_SHIFT 3 // Shift bits for reference #define BLOCK_MAP_FLAG_SUPER_BLOCK_SIZE_REMAINDER 1 // Remainder for super block size #define BLOCK_MAP_SUPER_BLOCK_SIZE_SHIFT 1 // Shift bits for super block size #define BLOCK_MAP_FLAG_SUPER_BLOCK_CHANGE 2 // The super block size has changed #define BLOCK_MAP_FLAG_SUPER_BLOCK_TOTAL_OFFSET 4 // Block total/offset when not complete #define BLOCK_MAP_SUPER_BLOCK_SHIFT 3 // Shift bits for super block #define BLOCK_MAP_FLAG_BLOCK_TOTAL_OFFSET 1 // Block total has an offset #define BLOCK_MAP_BLOCK_TOTAL_SHIFT 1 // Shift bits for block total typedef enum { blockMapFlagVersion = 0, // Version (currently always 0) } BlockMapFlag; // Stores current information about a reference to avoid needed to encode it again typedef struct BlockMapReference { unsigned int reference; // Reference uint64_t superBlockSize; // Super block size uint64_t bundleId; // Bundle id uint64_t offset; // Offset uint64_t size; // Stored super block size (with compression, etc.) uint64_t block; // Block no } BlockMapReference; // Reference comparator static int lstComparatorBlockMapReference(const void *const blockMapRef1, const void *const blockMapRef2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, blockMapRef1); FUNCTION_TEST_PARAM_P(VOID, blockMapRef2); FUNCTION_TEST_END(); ASSERT(blockMapRef1 != NULL); ASSERT(blockMapRef2 != NULL); const unsigned int reference1 = ((const BlockMapReference *)blockMapRef1)->reference; const unsigned int reference2 = ((const BlockMapReference *)blockMapRef2)->reference; FUNCTION_TEST_RETURN(INT, LST_COMPARATOR_CMP(reference1, reference2)); } FN_EXTERN BlockMap * blockMapNewRead(IoRead *const map, const size_t blockSize, const size_t checksumSize) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, map); FUNCTION_LOG_END(); // Read flags. Currently the version flag must always be zero. This may be used in the future to indicate if the map version // has changed. CHECK(FormatError, (ioReadVarIntU64(map) & (1 << blockMapFlagVersion)) == 0, "block map version must be zero"); // Read all references in packed format BlockMap *const this = blockMapNew(); List *const refList = lstNewP(sizeof(BlockMapReference), .comparator = lstComparatorBlockMapReference); Buffer *const checksum = bufNew(checksumSize); int64_t sizeLast = 0; bool referenceContinue = false; do { // Read reference const uint64_t referenceEncoded = ioReadVarIntU64(map); BlockMapItem blockMapItem = {.reference = (unsigned int)(referenceEncoded >> BLOCK_MAP_REFERENCE_SHIFT)}; BlockMapReference *referenceData = lstFind(refList, &(BlockMapReference){.reference = blockMapItem.reference}); // If this is the first time this reference has been read if (referenceData == NULL) { // Read bundle id if (referenceEncoded & BLOCK_MAP_FLAG_BUNDLE_ID) blockMapItem.bundleId = ioReadVarIntU64(map); // Read offset if (referenceEncoded & BLOCK_MAP_FLAG_OFFSET) blockMapItem.offset = ioReadVarIntU64(map); // Default super block size blockMapItem.superBlockSize = blockSize; // Add reference to list BlockMapReference referenceDataAdd = { .reference = blockMapItem.reference, .superBlockSize = blockMapItem.superBlockSize, .bundleId = blockMapItem.bundleId, .offset = blockMapItem.offset, }; referenceData = lstAdd(refList, &referenceDataAdd); } // Else this reference has been read before else { blockMapItem.superBlockSize = referenceData->superBlockSize; blockMapItem.bundleId = referenceData->bundleId; // If the reference is continued use the prior offset and size values if (referenceEncoded & BLOCK_MAP_FLAG_CONTINUE) { blockMapItem.offset = referenceData->offset; blockMapItem.size = referenceData->size; referenceContinue = true; } // Else this is a new reference and super block with a possible offset update else { blockMapItem.offset = referenceData->offset + referenceData->size; if (referenceEncoded & BLOCK_MAP_FLAG_OFFSET) blockMapItem.offset += ioReadVarIntU64(map); referenceData->offset = blockMapItem.offset; } } // Read all super blocks in the current reference in packed format bool superBlockFirst = true; do { uint64_t superBlockEncoded = 0; // If the reference was continued check if this is the last super block in the reference if (referenceContinue) { if (referenceEncoded & BLOCK_MAP_FLAG_CONTINUE_LAST) superBlockEncoded = BLOCK_MAP_FLAG_LAST; superBlockEncoded |= BLOCK_MAP_FLAG_SUPER_BLOCK_TOTAL_OFFSET; referenceContinue = false; } // Else read the super block size for the reference else { superBlockEncoded = ioReadVarIntU64(map); // If this is the first size read then just read the size. Otherwise read the difference from the prior size and // add sizeLast. blockMapItem.size = superBlockEncoded >> BLOCK_MAP_SUPER_BLOCK_SHIFT; if (sizeLast != 0) blockMapItem.size = (uint64_t)(cvtInt64FromZigZag(blockMapItem.size) + sizeLast); // If the super block size has changed then read it if (superBlockEncoded & BLOCK_MAP_FLAG_SUPER_BLOCK_CHANGE) { const uint64_t superBlockSizeEncoded = ioReadVarIntU64(map); blockMapItem.superBlockSize = (superBlockSizeEncoded >> BLOCK_MAP_SUPER_BLOCK_SIZE_SHIFT) * blockSize; if (superBlockSizeEncoded & BLOCK_MAP_FLAG_SUPER_BLOCK_SIZE_REMAINDER) blockMapItem.superBlockSize += ioReadVarIntU64(map); referenceData->superBlockSize = blockMapItem.superBlockSize; } // Set offset, size, and block for the super block if (superBlockFirst) referenceData->offset = blockMapItem.offset; else referenceData->offset += (uint64_t)sizeLast; referenceData->size = blockMapItem.size; referenceData->block = 0; } // Update sizeLast with the current size and clear superBlockFirst sizeLast = (int64_t)blockMapItem.size; superBlockFirst = false; // Read or calculate block total uint64_t blockTotal; if (superBlockEncoded & BLOCK_MAP_FLAG_SUPER_BLOCK_TOTAL_OFFSET) { const uint64_t blockTotalEncoded = ioReadVarIntU64(map); blockTotal = (blockTotalEncoded >> BLOCK_MAP_BLOCK_TOTAL_SHIFT) + 1; // Offset block no from expected if (blockTotalEncoded & BLOCK_MAP_FLAG_BLOCK_TOTAL_OFFSET) referenceData->block += ioReadVarIntU64(map); } else blockTotal = blockMapItem.superBlockSize / blockSize + (blockMapItem.superBlockSize % blockSize == 0 ? 0 : 1); // Read checksums for (uint64_t blockIdx = 0; blockIdx < blockTotal; blockIdx++) { // Set block no blockMapItem.block = referenceData->block + blockIdx; // Read checksum bufUsedZero(checksum); ioRead(map, checksum); memcpy(blockMapItem.checksum, bufPtr(checksum), bufUsed(checksum)); // Add to block list lstAdd((List *)this, &blockMapItem); } // Update block in reference with all blocks read referenceData->block += blockTotal; // Update offset with the super block size blockMapItem.offset += blockMapItem.size; // Break when this is the last super block in the reference if (superBlockEncoded & BLOCK_MAP_FLAG_LAST) break; } while (true); // Break when this is the last reference if (referenceEncoded & BLOCK_MAP_FLAG_LAST) break; } while (true); lstFree(refList); bufFree(checksum); FUNCTION_LOG_RETURN(BLOCK_MAP, this); } /**********************************************************************************************************************************/ FN_EXTERN void blockMapWrite(const BlockMap *const this, IoWrite *const output, const size_t blockSize, const size_t checksumSize) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BLOCK_MAP, this); FUNCTION_LOG_PARAM(IO_WRITE, output); FUNCTION_LOG_PARAM(SIZE, blockSize); FUNCTION_LOG_PARAM(SIZE, checksumSize); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(blockMapSize(this) > 0); ASSERT(blockSize > 0); ASSERT(output != NULL); // Write flags ioWriteVarIntU64(output, 0); // Write all references in packed format List *const refList = lstNewP(sizeof(BlockMapReference), .comparator = lstComparatorBlockMapReference); unsigned int referenceIdx = 0; int64_t sizeLast = 0; bool referenceContinue = false; while (referenceIdx < blockMapSize(this)) { const BlockMapItem *const reference = blockMapGet(this, referenceIdx); unsigned int superBlockIdx = referenceIdx; unsigned int blockIdx = referenceIdx; // Determine if this is the last reference uint64_t referenceEncoded = BLOCK_MAP_FLAG_LAST; for (referenceIdx++; referenceIdx < blockMapSize(this); referenceIdx++) { if (reference->reference != blockMapGet(this, referenceIdx)->reference) { referenceEncoded = 0; break; } ASSERT(reference->offset <= blockMapGet(this, referenceIdx)->offset); } // If this is the first time this reference has been written BlockMapReference *referenceData = lstFind(refList, &(BlockMapReference){.reference = reference->reference}); if (referenceData == NULL) { // Add bundle id and offset flags if (reference->bundleId > 0) referenceEncoded |= BLOCK_MAP_FLAG_BUNDLE_ID; if (reference->offset > 0) referenceEncoded |= BLOCK_MAP_FLAG_OFFSET; // Write the references ioWriteVarIntU64(output, referenceEncoded | reference->reference << BLOCK_MAP_REFERENCE_SHIFT); // Write bundle id and offset if (referenceEncoded & BLOCK_MAP_FLAG_BUNDLE_ID) ioWriteVarIntU64(output, reference->bundleId); if (referenceEncoded & BLOCK_MAP_FLAG_OFFSET) ioWriteVarIntU64(output, reference->offset); // Add reference to list const BlockMapReference referenceAdd = { .reference = reference->reference, .superBlockSize = blockSize, .bundleId = reference->bundleId, .offset = reference->offset, }; referenceData = lstAdd(refList, &referenceAdd); } // Else this reference has been written before else { ASSERT(reference->reference == referenceData->reference); ASSERT(reference->bundleId == referenceData->bundleId); ASSERT(reference->offset >= referenceData->offset); // If the offset is identical then reference is continuing an already started super block. The super block size and // block no should be reused. Note that writing the reference is deferred until we know if the continued super block is // the last one for the reference. if (reference->offset == referenceData->offset) { ASSERT(reference->superBlockSize == referenceData->superBlockSize); referenceEncoded |= BLOCK_MAP_FLAG_CONTINUE; referenceContinue = true; } // Else the reference starts a new super block which means the offset may need to be stored if there is a gap from the // prior super block else { if (reference->offset > referenceData->offset + referenceData->size) referenceEncoded |= BLOCK_MAP_FLAG_OFFSET; ioWriteVarIntU64(output, referenceEncoded | reference->reference << BLOCK_MAP_REFERENCE_SHIFT); if (referenceEncoded & BLOCK_MAP_FLAG_OFFSET) ioWriteVarIntU64(output, reference->offset - (referenceData->offset + referenceData->size)); referenceData->offset = reference->offset; referenceData->size = reference->size; } } // Write all super blocks in the current reference in packed format while (superBlockIdx < referenceIdx) { const BlockMapItem *const superBlock = blockMapGet(this, superBlockIdx); // Determine if this is the last super block in the reference uint64_t superBlockEncoded = BLOCK_MAP_FLAG_LAST; for (superBlockIdx++; superBlockIdx < referenceIdx; superBlockIdx++) { if (superBlock->offset != blockMapGet(this, superBlockIdx)->offset) { superBlockEncoded = 0; break; } } // If block offset or total need to be stored then add the flag const unsigned int blockTotal = superBlockIdx - blockIdx; ASSERT(blockTotal > 0); if (referenceContinue || superBlock->block != 0 || blockTotal != superBlock->superBlockSize / blockSize + (superBlock->superBlockSize % blockSize == 0 ? 0 : 1)) { superBlockEncoded |= BLOCK_MAP_FLAG_SUPER_BLOCK_TOTAL_OFFSET; } // Write the continued reference now that we know if this will be the last super block in the reference if (referenceContinue) { ASSERT(superBlock->superBlockSize == referenceData->superBlockSize); if (superBlockEncoded & BLOCK_MAP_FLAG_LAST) referenceEncoded |= BLOCK_MAP_FLAG_CONTINUE_LAST; ioWriteVarIntU64(output, referenceEncoded | reference->reference << BLOCK_MAP_REFERENCE_SHIFT); referenceContinue = false; } // Else write the super block size for the reference else { // Set offset, size, and block for the super block referenceData->offset = superBlock->offset; referenceData->size = superBlock->size; referenceData->block = 0; // If the super block size has changed then add the flag ASSERT(reference->superBlockSize > 0); if (superBlock->superBlockSize != referenceData->superBlockSize) superBlockEncoded |= BLOCK_MAP_FLAG_SUPER_BLOCK_CHANGE; // If this is the first size written then just write the size. Otherwise write the difference from the prior size. // This depends on the expectation that the compressed size of equal-sized blocks will be similar in order to be // most efficient. ioWriteVarIntU64( output, superBlockEncoded | (sizeLast == 0 ? superBlock->size : cvtInt64ToZigZag((int64_t)superBlock->size - sizeLast)) << BLOCK_MAP_SUPER_BLOCK_SHIFT); // If the super block size has changed then write it if (superBlockEncoded & BLOCK_MAP_FLAG_SUPER_BLOCK_CHANGE) { const uint64_t superBlockSizeEncoded = (superBlock->superBlockSize / blockSize) << BLOCK_MAP_SUPER_BLOCK_SIZE_SHIFT | (superBlock->superBlockSize % blockSize == 0 ? 0 : BLOCK_MAP_FLAG_SUPER_BLOCK_SIZE_REMAINDER); ioWriteVarIntU64(output, superBlockSizeEncoded); if (superBlockSizeEncoded & BLOCK_MAP_FLAG_SUPER_BLOCK_SIZE_REMAINDER) ioWriteVarIntU64(output, superBlock->superBlockSize % blockSize); referenceData->superBlockSize = superBlock->superBlockSize; } } sizeLast = (int64_t)superBlock->size; // Write block total if the super block does not include all blocks with no block offset if (superBlockEncoded & BLOCK_MAP_FLAG_SUPER_BLOCK_TOTAL_OFFSET) { // Write total blocks in the super block const uint64_t blockTotalEncoded = (blockTotal - 1) << BLOCK_MAP_BLOCK_TOTAL_SHIFT | (superBlock->block - referenceData->block > 0 ? BLOCK_MAP_FLAG_BLOCK_TOTAL_OFFSET : 0); ioWriteVarIntU64(output, blockTotalEncoded); // If there is a gap in block no from the prior super block. This can happen when a super block is continued or // had blocks at the beginning overridden by a newer super block. if (blockTotalEncoded & BLOCK_MAP_FLAG_BLOCK_TOTAL_OFFSET) { ioWriteVarIntU64(output, superBlock->block - referenceData->block); referenceData->block = superBlock->block; } } ASSERT(superBlock->block >= referenceData->block); // Increment reference block by number of blocks written referenceData->block += blockTotal; // Write checksums for (; blockIdx < superBlockIdx; blockIdx++) { ASSERT( superBlock == blockMapGet(this, blockIdx) || (blockIdx > 0 && (blockMapGet(this, blockIdx)->block == blockMapGet(this, blockIdx - 1)->block + 1))); ioWrite(output, BUF(blockMapGet(this, blockIdx)->checksum, checksumSize)); } } } lstFree(refList); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/blockMap.h0000664000175100017510000001120615265101745020755 0ustar00runnerrunner/*********************************************************************************************************************************** Block Incremental Map The block incremental map stores the location of blocks of data that have been backed up incrementally. When a file changes, instead of copying the entire file, just the blocks that have been changed can be stored. This map does not store the blocks themselves, just the location where they can be found. It must be combined with a super block list to be useful (see BlockIncr filter). ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_BLOCKMAP_H #define COMMAND_BACKUP_BLOCKMAP_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct BlockMap BlockMap; #include "common/crypto/xxhash.h" #include "common/type/list.h" #include "common/type/object.h" typedef struct BlockMapItem { unsigned int reference; // Reference to backup where the block is stored uint64_t superBlockSize; // Super block size uint64_t bundleId; // Bundle where the block is stored (0 if not bundled) uint64_t offset; // Offset of super block into the bundle uint64_t size; // Stored super block size (with compression, etc.) uint64_t block; // Block no inside of super block uint8_t checksum[XX_HASH_SIZE_MAX]; // Checksum of the block } BlockMapItem; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ // Create empty block map FN_INLINE_ALWAYS BlockMap * blockMapNew(void) { return (BlockMap *)OBJ_NAME(lstNewP(sizeof(BlockMapItem)), BlockMap::List); } // New block map from IO FN_EXTERN BlockMap *blockMapNewRead(IoRead *map, size_t blockSize, size_t checksumSize); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add a block map item FN_INLINE_ALWAYS BlockMapItem * blockMapAdd(BlockMap *const this, const BlockMapItem *const item) { ASSERT_INLINE(item != NULL); return (BlockMapItem *)lstAdd((List *const)this, item); } // Write map to IO FN_EXTERN void blockMapWrite(const BlockMap *this, IoWrite *output, size_t blockSize, size_t checksumSize); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Get a block map item FN_INLINE_ALWAYS BlockMapItem * blockMapGet(const BlockMap *const this, const unsigned int mapIdx) { return (BlockMapItem *)lstGet((const List *const)this, mapIdx); } // Block map size FN_INLINE_ALWAYS unsigned int blockMapSize(const BlockMap *const this) { return lstSize((const List *const)this); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void blockMapFree(BlockMap *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_BLOCK_MAP_TYPE \ BlockMap * #define FUNCTION_LOG_BLOCK_MAP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "BlockMap", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/common.c0000664000175100017510000001441115265101745020511 0ustar00runnerrunner/*********************************************************************************************************************************** Common Functions and Definitions for Backup and Expire Commands ***********************************************************************************************************************************/ #include #include #include "command/backup/common.h" #include "common/debug.h" #include "common/log.h" #include "storage/helper.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define BACKUP_LINK_LATEST "latest" /**********************************************************************************************************************************/ FN_EXTERN String * backupFileRepoPath(const String *const backupLabel, const BackupFileRepoPathParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_PARAM(STRING, param.manifestName); FUNCTION_TEST_PARAM(UINT64, param.bundleId); FUNCTION_TEST_PARAM(ENUM, param.compressType); FUNCTION_TEST_PARAM(BOOL, param.blockIncr); FUNCTION_TEST_END(); ASSERT(backupLabel != NULL); ASSERT(param.bundleId != 0 || param.manifestName != NULL); String *const result = strCatFmt(strNew(), STORAGE_REPO_BACKUP "/%s/", strZ(backupLabel)); if (param.bundleId != 0) strCatFmt(result, MANIFEST_PATH_BUNDLE "/%" PRIu64, param.bundleId); else { strCatFmt( result, "%s%s", strZ(param.manifestName), param.blockIncr ? BACKUP_BLOCK_INCR_EXT : strZ(compressExtStr(param.compressType))); } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * backupLabelFormat(const BackupType type, const String *const backupLabelPrior, const time_t timestamp) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(STRING, backupLabelPrior); FUNCTION_LOG_PARAM(TIME, timestamp); FUNCTION_LOG_END(); ASSERT((type == backupTypeFull && backupLabelPrior == NULL) || (type != backupTypeFull && backupLabelPrior != NULL)); ASSERT(timestamp > 0); // Format the timestamp char buffer[DATE_TIME_LEN + 1]; cvtTimeToZP("%Y%m%d-%H%M%S", timestamp, buffer, sizeof(buffer)); // If full label String *result; if (type == backupTypeFull) { result = strNewFmt("%sF", buffer); } // Else diff or incr label else { // Get the full backup portion of the prior backup label and append the diff/incr timestamp result = strNewFmt("%.*s_%s%s", DATE_TIME_LEN + 1, strZ(backupLabelPrior), buffer, type == backupTypeDiff ? "D" : "I"); } FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * backupRegExp(const BackupRegExpParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BOOL, param.full); FUNCTION_LOG_PARAM(BOOL, param.differential); FUNCTION_LOG_PARAM(BOOL, param.incremental); FUNCTION_LOG_PARAM(BOOL, param.noAnchorEnd); FUNCTION_LOG_END(); ASSERT(param.full || param.differential || param.incremental); // Start the expression with the anchor, date/time regexp and full backup indicator String *const result = strCatZ(strNew(), "^" DATE_TIME_REGEX "F"); // Add the diff and/or incr expressions if requested if (param.differential || param.incremental) { // If full requested then diff/incr is optional if (param.full) { strCatZ(result, "(\\_"); } // Else diff/incr is required else { strCatZ(result, "\\_"); } // Append date/time regexp for diff/incr strCatZ(result, DATE_TIME_REGEX); // Filter on both diff/incr if (param.differential && param.incremental) { strCatZ(result, "(D|I)"); } // Else just diff else if (param.differential) { strCatChr(result, 'D'); } // Else just incr else { strCatChr(result, 'I'); } // If full requested then diff/incr is optional if (param.full) { strCatZ(result, "){0,1}"); } } // Append the end anchor if (!param.noAnchorEnd) strCatZ(result, "$"); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void backupLinkLatest(const String *const backupLabel, const unsigned int repoIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { // Create a symlink to the most recent backup if supported. This link is purely informational for the user and is never used // by us since symlinks are not supported on all storage types. // ------------------------------------------------------------------------------------------------------------------------- const String *const latestLink = storagePathP(storageRepoIdx(repoIdx), STRDEF(STORAGE_REPO_BACKUP "/" BACKUP_LINK_LATEST)); // Remove an existing latest link/file in case symlink capabilities have changed storageRemoveP(storageRepoIdxWrite(repoIdx), latestLink); if (storageFeature(storageRepoIdxWrite(repoIdx), storageFeatureSymLink)) storageLinkCreateP(storageRepoIdxWrite(repoIdx), backupLabel, latestLink); // Sync backup path if required if (storageFeature(storageRepoIdxWrite(repoIdx), storageFeaturePathSync)) storagePathSyncP(storageRepoIdxWrite(repoIdx), STORAGE_REPO_BACKUP_STR); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/common.h0000664000175100017510000000615215265101745020521 0ustar00runnerrunner/*********************************************************************************************************************************** Common Functions and Definitions for Backup and Expire Commands ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_COMMON_H #define COMMAND_BACKUP_COMMON_H #include #include #include "common/compress/helper.h" #include "common/type/string.h" #include "info/infoBackup.h" /*********************************************************************************************************************************** Backup constants ***********************************************************************************************************************************/ #define BACKUP_PATH_HISTORY "backup.history" #define BACKUP_BLOCK_INCR_EXT ".pgbi" // Date and time must be in %Y%m%d-%H%M%S format, for example 20220901-193409 #define DATE_TIME_REGEX "[0-9]{8}\\-[0-9]{6}" #define DATE_TIME_LEN (8 + 1 + 6) /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Determine the path/file where the file is backed up in the repo typedef struct BackupFileRepoPathParam { const String *manifestName; // File name in manifest uint64_t bundleId; // Is the file bundled? CompressType compressType; // Is the file compressed? bool blockIncr; // Is the file a block incremental? } BackupFileRepoPathParam; #define backupFileRepoPathP(backupLabel, ...) \ backupFileRepoPath(backupLabel, (BackupFileRepoPathParam){__VA_ARGS__}) FN_EXTERN String *backupFileRepoPath(const String *backupLabel, BackupFileRepoPathParam param); // Format a backup label from a type and timestamp with an optional prior label FN_EXTERN String *backupLabelFormat(BackupType type, const String *backupLabelPrior, time_t timestamp); // Returns an anchored regex string for filtering backups based on the type (at least one type is required to be true) typedef struct BackupRegExpParam { bool full; bool differential; bool incremental; bool noAnchorEnd; } BackupRegExpParam; #define backupRegExpP(...) \ backupRegExp((BackupRegExpParam){__VA_ARGS__}) FN_EXTERN String *backupRegExp(BackupRegExpParam param); // Create a symlink to the specified backup (if symlinks are supported) FN_EXTERN void backupLinkLatest(const String *backupLabel, unsigned int repoIdx); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/complete.c.inc0000664000175100017510000003122115265101745021577 0ustar00runnerrunner/*********************************************************************************************************************************** Save a copy of the backup manifest during processing to preserve checksums for a possible resume. Only save the final copy when resume is disabled since an incremental copy will not be used in a future backup unless resume is enabled beforehand. ***********************************************************************************************************************************/ static void backupManifestSaveCopy(Manifest *const manifest, const CipherSpec *const cipherSpecManifest, const bool final) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpecManifest); FUNCTION_LOG_PARAM(BOOL, final); FUNCTION_LOG_END(); ASSERT(manifest != NULL); if (final || cfgOptionBool(cfgOptResume)) { MEM_CONTEXT_TEMP_BEGIN() { // Open file for write IoWrite *const write = storageWriteIo( storageNewWriteP( storageRepoWrite(), strNewFmt( STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, strZ(manifestData(manifest)->backupLabel)))); // Add encryption filter if required cipherBlockFilterGroupAdd(ioWriteFilterGroup(write), cipherModeEncrypt, cipherSpecManifest); // Save file manifestSave(manifest, write); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Check and copy WAL segments required to make the backup consistent ***********************************************************************************************************************************/ static void backupArchiveCheckCopy(const BackupData *const backupData, Manifest *const manifest, const CipherSpec *const cipherSpecManifest) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpecManifest); FUNCTION_LOG_END(); ASSERT(manifest != NULL); // If archive logs are required to complete the backup, then check them. This is the default, but can be overridden if the // archive logs are going to a different server. Be careful of disabling this option because there is no way to verify that the // backup will be consistent - at least not here. if (cfgOptionBool(cfgOptArchiveCheck)) { MEM_CONTEXT_TEMP_BEGIN() { const uint64_t lsnStart = pgLsnFromStr(manifestData(manifest)->lsnStart); const uint64_t lsnStop = pgLsnFromStr(manifestData(manifest)->lsnStop); LOG_INFO_FMT( "check archive for segment(s) %s:%s", strZ(pgLsnToWalSegment(backupData->timeline, lsnStart, backupData->walSegmentSize)), strZ(pgLsnToWalSegment(backupData->timeline, lsnStop, backupData->walSegmentSize))); // Save the backup manifest before getting archive logs in case of failure backupManifestSaveCopy(manifest, cipherSpecManifest, false); // Use base path to set ownership and mode const ManifestPath *const basePath = manifestPathFind(manifest, MANIFEST_TARGET_PGDATA_STR); // Loop through all the segments in the lsn range const StringList *const walSegmentList = pgLsnRangeToWalSegmentList( backupData->timeline, lsnStart, lsnStop, backupData->walSegmentSize); WalSegmentFind *const find = walSegmentFindNew( storageRepo(), backupData->archiveId, strLstSize(walSegmentList) == 1, cfgOptionUInt64(cfgOptArchiveTimeout)); for (unsigned int walSegmentIdx = 0; walSegmentIdx < strLstSize(walSegmentList); walSegmentIdx++) { MEM_CONTEXT_TEMP_BEGIN() { // Find the actual wal segment file (including checksum compression extension) in the archive const String *const walSegment = strLstGet(walSegmentList, walSegmentIdx); const String *const archiveFile = walSegmentFind(find, walSegment); if (cfgOptionBool(cfgOptArchiveCopy)) { // Copy can be a pretty expensive operation so log it LOG_DETAIL_FMT("copy segment %s to backup", strZ(walSegment)); // Get compression type of the WAL segment and backup const CompressType archiveCompressType = compressTypeFromName(archiveFile); const CompressType backupCompressType = compressTypeEnum(cfgOptionStrId(cfgOptCompressType)); // Open the archive file StorageRead *const read = storageNewReadP( storageRepo(), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(backupData->archiveId), strZ(archiveFile))); IoFilterGroup *const filterGroup = ioReadFilterGroup(storageReadIo(read)); // Decrypt with archive key if encrypted cipherBlockFormatFilterGroupReadAddMap( filterGroup, infoArchiveCipherSpecMap(backupData->archiveInfo)); // Compress/decompress if archive and backup do not have the same compression settings if (archiveCompressType != backupCompressType) { if (archiveCompressType != compressTypeNone) ioFilterGroupAdd(filterGroup, decompressFilterP(archiveCompressType)); if (backupCompressType != compressTypeNone) { ioFilterGroupAdd( filterGroup, compressFilterP(backupCompressType, cfgOptionInt(cfgOptCompressLevel))); } } // Encrypt with backup key if encrypted cipherBlockFilterGroupAdd( filterGroup, cipherModeEncrypt, manifestCipherSpec(manifest)); // Add size filter last to calculate repo size ioFilterGroupAdd(filterGroup, ioSizeNew()); // Copy the file const String *const manifestName = strNewFmt( MANIFEST_TARGET_PGDATA "/%s/%s", strZ(pgWalPath(manifestData(manifest)->pgVersion)), strZ(walSegment)); storageCopyP( read, storageNewWriteP( storageRepoWrite(), backupFileRepoPathP( manifestData(manifest)->backupLabel, manifestName, 0, compressTypeEnum(cfgOptionStrId(cfgOptCompressType)), false))); // Add to manifest ManifestFile file = { .name = manifestName, .mode = basePath->mode & (S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH), .user = basePath->user, .group = basePath->group, .size = backupData->walSegmentSize, .sizeOriginal = backupData->walSegmentSize, .sizeRepo = pckReadU64P(ioFilterGroupResultP(filterGroup, SIZE_FILTER_TYPE)), .timestamp = manifestData(manifest)->backupTimestampStop, .checksumSha1 = bufPtr(bufNewDecode(encodingHex, strSubN(archiveFile, 25, 40))), }; manifestFileAdd(manifest, &file); } } MEM_CONTEXT_TEMP_END(); // A keep-alive is required here for the remote holding the backup lock protocolKeepAlive(); } } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Save and update all files required to complete the backup ***********************************************************************************************************************************/ static void backupComplete(InfoBackup *const infoBackup, Manifest *const manifest) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_END(); ASSERT(manifest != NULL); MEM_CONTEXT_TEMP_BEGIN() { const String *const backupLabel = manifestData(manifest)->backupLabel; // Validation and final save of the backup manifest. Validate in strict mode to catch as many potential issues as possible. // ------------------------------------------------------------------------------------------------------------------------- manifestValidate(manifest, true); backupManifestSaveCopy(manifest, infoBackupCipherSpec(infoBackup), true); storageCopy( storageNewReadP( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, strZ(backupLabel))), storageNewWriteP( storageRepoWrite(), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)))); // Copy a compressed version of the manifest to history. If the repo is encrypted then the passphrase to open the manifest // is required. We can't just do a straight copy since the destination needs to be compressed and that must happen before // encryption in order to be efficient. Compression will always be gz for compatibility and since it is always available. // ------------------------------------------------------------------------------------------------------------------------- StorageRead *const manifestRead = storageNewReadP( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel))); cipherBlockFilterGroupAdd( ioReadFilterGroup(storageReadIo(manifestRead)), cipherModeDecrypt, infoBackupCipherSpec(infoBackup)); StorageWrite *const manifestWrite = storageNewWriteP( storageRepoWrite(), strNewFmt( STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY "/%s/%s.manifest%s", strZ(strSubN(backupLabel, 0, 4)), strZ(backupLabel), strZ(compressExtStr(compressTypeGz)))); ioFilterGroupAdd(ioWriteFilterGroup(storageWriteIo(manifestWrite)), compressFilterP(compressTypeGz, 9)); cipherBlockFilterGroupAdd( ioWriteFilterGroup(storageWriteIo(manifestWrite)), cipherModeEncrypt, infoBackupCipherSpec(infoBackup)); storageCopyP(manifestRead, manifestWrite); // Sync history path if required if (storageFeature(storageRepoWrite(), storageFeaturePathSync)) storagePathSyncP(storageRepoWrite(), STRDEF(STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY)); // Create a symlink to the most recent backup if supported. This link is purely informational for the user and is never used // by us since symlinks are not supported on all storage types. // ------------------------------------------------------------------------------------------------------------------------- backupLinkLatest(backupLabel, cfgOptionGroupIdxDefault(cfgOptGrpRepo)); // Add manifest and save backup.info (infoBackupSaveFile() is responsible for proper syncing) // ------------------------------------------------------------------------------------------------------------------------- infoBackupDataAdd(infoBackup, manifest); infoBackupSaveFile(infoBackup, storageRepoWrite(), INFO_BACKUP_PATH_FILE_STR, cfgCipherSpecMain()); // Save archive.info/copy so the timestamps will be updated to prevent lifecycle settings from removing the files early // ------------------------------------------------------------------------------------------------------------------------- infoArchiveSaveFile( infoArchiveLoadFile(storageRepo(), INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMain()), storageRepoWrite(), INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMain()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/db.c.inc0000664000175100017510000003124315265101745020360 0ustar00runnerrunner/********************************************************************************************************************************** Get time from the database or locally depending on online ***********************************************************************************************************************************/ static time_t backupTime(const BackupData *const backupData, const bool waitRemainder) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_PARAM(BOOL, waitRemainder); FUNCTION_LOG_END(); // Offline backups will just grab the time from the local system since the value of copyStart is not important in this context. // No worries about causing a delta backup since switching online will do that anyway. time_t result = time(NULL); // When online get the time from the database server if (cfgOptionBool(cfgOptOnline)) { // Get time from the database TimeMSec timeMSec = dbTimeMSec(backupData->dbPrimary); result = (time_t)(timeMSec / MSEC_PER_SEC); // Sleep the remainder of the second when requested (this is so copyStart is not subject to one second resolution issues) if (waitRemainder) { unsigned int retry = 0; // Just to be safe we'll loop until PostgreSQL reports that we have slept long enough do { // Error if the clock has not advanced after several attempts if (retry == 3) THROW_FMT(KernelError, PG_NAME " clock has not advanced to the next second after %u tries", retry); // Sleep remainder of current second sleepMSec(((TimeMSec)(result + 1) * MSEC_PER_SEC) - timeMSec); // Check time again to be sure we slept long enough timeMSec = dbTimeMSec(backupData->dbPrimary); // Increment retry to prevent an infinite loop retry++; } while ((time_t)(timeMSec / MSEC_PER_SEC) <= result); } } FUNCTION_LOG_RETURN(TIME, result); } /*********************************************************************************************************************************** Check that the clusters are alive and correctly configured during the backup ***********************************************************************************************************************************/ static void backupDbPing(const BackupData *const backupData, const bool force) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_PARAM(BOOL, force); FUNCTION_LOG_END(); ASSERT(backupData != NULL); if (cfgOptionBool(cfgOptOnline)) { dbPing(backupData->dbPrimary, force); if (backupData->dbStandby != NULL) dbPing(backupData->dbStandby, force); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Start the backup ***********************************************************************************************************************************/ typedef struct BackupStartResult { String *lsn; String *walSegmentName; Pack *dbList; Pack *tablespaceList; } BackupStartResult; static BackupStartResult backupStart(const BackupData *const backupData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); BackupStartResult result = {.lsn = NULL}; MEM_CONTEXT_TEMP_BEGIN() { // If this is an offline backup if (!cfgOptionBool(cfgOptOnline)) { // Check if Postgres is running and if so only continue when forced if (storageExistsP(backupData->storagePrimary, PG_FILE_POSTMTRPID_STR)) { if (cfgOptionBool(cfgOptForce)) { LOG_WARN( "--no-" CFGOPT_ONLINE " passed and " PG_FILE_POSTMTRPID " exists but --" CFGOPT_FORCE " was passed so" " backup will continue though it looks like " PG_NAME " is running and the backup will probably not be" " consistent"); } else { THROW( PgRunningError, "--no-" CFGOPT_ONLINE " passed but " PG_FILE_POSTMTRPID " exists - looks like " PG_NAME " is running. Shut" " down " PG_NAME " and try again, or use --force."); } } } // Else start the backup normally else { // Check database configuration checkDbConfig(backupData->version, backupData->pgIdxPrimary, backupData->dbPrimary, false); // Start backup LOG_INFO_FMT( "execute backup start: backup begins after the %s checkpoint completes", cfgOptionBool(cfgOptStartFast) ? "requested immediate" : "next regular"); const DbBackupStartResult dbBackupStartResult = dbBackupStart( backupData->dbPrimary, cfgOptionBool(cfgOptStartFast), cfgOptionBool(cfgOptArchiveCheck)); MEM_CONTEXT_PRIOR_BEGIN() { result.lsn = strDup(dbBackupStartResult.lsn); result.walSegmentName = strDup(dbBackupStartResult.walSegmentName); result.dbList = dbList(backupData->dbPrimary); result.tablespaceList = dbTablespaceList(backupData->dbPrimary); } MEM_CONTEXT_PRIOR_END(); LOG_INFO_FMT("backup start archive = %s, lsn = %s", strZ(result.walSegmentName), strZ(result.lsn)); // Wait for replay on the standby to catch up if (backupData->dbStandby != NULL) { LOG_INFO_FMT("wait for replay on the standby to reach %s", strZ(result.lsn)); dbReplayWait(backupData->dbStandby, result.lsn, backupData->timeline, cfgOptionUInt64(cfgOptArchiveTimeout)); LOG_INFO_FMT("replay on the standby reached %s", strZ(result.lsn)); } // Check that WAL segments are being archived. If archiving is not working then the backup will eventually fail so // better to catch it as early as possible. A segment to check may not be available on older versions of PostgreSQL or // if archive-check is false. if (dbBackupStartResult.walSegmentCheck != NULL) { LOG_INFO_FMT( "check archive for %ssegment %s", strEq(result.walSegmentName, dbBackupStartResult.walSegmentCheck) ? "" : "prior ", strZ(dbBackupStartResult.walSegmentCheck)); walSegmentFindOne( storageRepo(), backupData->archiveId, dbBackupStartResult.walSegmentCheck, cfgOptionUInt64(cfgOptArchiveTimeout)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /*********************************************************************************************************************************** Stop the backup ***********************************************************************************************************************************/ // Helper to write a file from a string to the repository and update the manifest static void backupFilePut( Manifest *const manifest, const String *const name, const time_t timestamp, const String *const content) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(TIME, timestamp); FUNCTION_LOG_PARAM(STRING, content); FUNCTION_LOG_END(); // Skip files with no content if (content != NULL) { MEM_CONTEXT_TEMP_BEGIN() { // Create file bool repoChecksum = false; const String *const manifestName = strNewFmt(MANIFEST_TARGET_PGDATA "/%s", strZ(name)); const CompressType compressType = compressTypeEnum(cfgOptionStrId(cfgOptCompressType)); StorageWrite *const write = storageNewWriteP( storageRepoWrite(), backupFileRepoPathP( manifestData(manifest)->backupLabel, .manifestName = manifestName, .compressType = compressTypeEnum(cfgOptionStrId(cfgOptCompressType))), .compressible = true); IoFilterGroup *const filterGroup = ioWriteFilterGroup(storageWriteIo(write)); // Add SHA1 filter ioFilterGroupAdd(filterGroup, cryptoHashNew(hashTypeSha1)); // Add compression if (compressType != compressTypeNone) { ioFilterGroupAdd( ioWriteFilterGroup(storageWriteIo(write)), compressFilterP(compressType, cfgOptionInt(cfgOptCompressLevel))); repoChecksum = true; } // Add encryption filter if required if (cipherSpecType(manifestCipherSpec(manifest)) != cipherTypeNone) { ioFilterGroupAdd( ioWriteFilterGroup(storageWriteIo(write)), cipherBlockNewP(cipherModeEncrypt, manifestCipherSpec(manifest))); repoChecksum = true; } // Capture checksum of file stored in the repo if filters that modify the output have been applied if (repoChecksum) ioFilterGroupAdd(filterGroup, cryptoHashNew(hashTypeSha1)); // Add size filter last to calculate repo size ioFilterGroupAdd(filterGroup, ioSizeNew()); // Write file storagePutP(write, BUFSTR(content)); // Use base path to set ownership and mode const ManifestPath *const basePath = manifestPathFind(manifest, MANIFEST_TARGET_PGDATA_STR); // Add to manifest ManifestFile file = { .name = manifestName, .mode = basePath->mode & (S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH), .user = basePath->user, .group = basePath->group, .size = strSize(content), .sizeOriginal = strSize(content), .sizeRepo = pckReadU64P(ioFilterGroupResultP(filterGroup, SIZE_FILTER_TYPE)), .timestamp = timestamp, .checksumSha1 = bufPtr(pckReadBinP(ioFilterGroupResultP(filterGroup, CRYPTO_HASH_FILTER_TYPE, .idx = 0))), }; if (repoChecksum) file.checksumRepoSha1 = bufPtr(pckReadBinP(ioFilterGroupResultP(filterGroup, CRYPTO_HASH_FILTER_TYPE, .idx = 1))); manifestFileAdd(manifest, &file); LOG_DETAIL_FMT("wrote '%s' file returned from backup stop function", strZ(name)); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /*--------------------------------------------------------------------------------------------------------------------------------*/ typedef struct BackupStopResult { String *lsn; String *walSegmentName; time_t timestamp; } BackupStopResult; static BackupStopResult backupStop(BackupData *const backupData, Manifest *const manifest) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); BackupStopResult result = {.lsn = NULL}; if (cfgOptionBool(cfgOptOnline)) { MEM_CONTEXT_TEMP_BEGIN() { // Stop the backup LOG_INFO_FMT( "execute backup stop and wait for all WAL segments to archive"); const DbBackupStopResult dbBackupStopResult = dbBackupStop(backupData->dbPrimary); MEM_CONTEXT_PRIOR_BEGIN() { result.timestamp = backupTime(backupData, false); result.lsn = strDup(dbBackupStopResult.lsn); result.walSegmentName = strDup(dbBackupStopResult.walSegmentName); } MEM_CONTEXT_PRIOR_END(); LOG_INFO_FMT("backup stop archive = %s, lsn = %s", strZ(result.walSegmentName), strZ(result.lsn)); // Save files returned by stop backup backupFilePut(manifest, STRDEF(PG_FILE_BACKUPLABEL), result.timestamp, dbBackupStopResult.backupLabel); backupFilePut(manifest, STRDEF(PG_FILE_TABLESPACEMAP), result.timestamp, dbBackupStopResult.tablespaceMap); } MEM_CONTEXT_TEMP_END(); } else result.timestamp = backupTime(backupData, false); FUNCTION_LOG_RETURN_STRUCT(result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/file.c.inc0000664000175100017510000007400315265101745020713 0ustar00runnerrunner/*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ static unsigned int segmentNumber(const String *const pgFile) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, pgFile); FUNCTION_TEST_END(); // Determine which segment number this is by checking for a numeric extension. No extension means segment 0. FUNCTION_TEST_RETURN(UINT, regExpMatchOne(STRDEF("\\.[0-9]+$"), pgFile) ? cvtZToUInt(strrchr(strZ(pgFile), '.') + 1) : 0); } /*********************************************************************************************************************************** Copy a file from the PostgreSQL data directory to the repository ***********************************************************************************************************************************/ typedef struct BackupFile { const String *pgFile; // Pg file to backup bool pgFileDelta; // Checksum pg file before copying bool pgFileIgnoreMissing; // Ignore missing pg file uint64_t pgFileSize; // Expected pg file size uint64_t pgFileSizeOriginal; // Original size (from manifest build) bool pgFileCopyExactSize; // Copy only pg expected size const Buffer *pgFileChecksum; // Expected pg file checksum bool pgFileChecksumPage; // Validate page checksums? bool pgFilePageHeaderCheck; // Validate page headers? size_t blockIncrSize; // Perform block incremental on this file? size_t blockIncrChecksumSize; // Block checksum size uint64_t blockIncrSuperSize; // Size of the super block const String *blockIncrMapPriorFile; // File containing prior block incremental map (NULL if none) uint64_t blockIncrMapPriorOffset; // Offset of prior block incremental map uint64_t blockIncrMapPriorSize; // Size of prior block incremental map const String *manifestFile; // Repo file const Buffer *repoFileChecksum; // Expected repo file checksum uint64_t repoFileSize; // Expected repo file size bool manifestFileResume; // Checksum repo file before copying bool manifestFileHasReference; // Reference to prior backup, if any } BackupFile; typedef struct BackupFileResult { const String *manifestFile; // Manifest file BackupCopyResult backupCopyResult; bool repoInvalid; // File was recopied because repo file was invalid uint64_t copySize; const Buffer *copyChecksum; // Checksum of pg file const Buffer *repoChecksum; // Checksum of repo file (including compression, etc.) uint64_t bundleOffset; // Offset in bundle if any uint64_t repoSize; uint64_t blockIncrMapSize; // Size of block incremental map (0 if no map) Pack *pageChecksumResult; } BackupFileResult; // Create the block incremental filter, reading the prior block map when there is one. Errors are rethrown with the location of the // map because neither the read nor the map parse know where the map came from, which makes a mismatch between the manifest and the // repository very hard to diagnose. static IoFilter * backupFileBlockIncr( const BackupFile *const file, const unsigned int blockIncrReference, const uint64_t bundleId, const uint64_t bundleOffset, StorageReadMulti *const biMapRead, const CipherSpec *const cipherSpecBackup, const IoFilter *const compress, const IoFilter *const encrypt) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, file); FUNCTION_LOG_PARAM(UINT, blockIncrReference); FUNCTION_LOG_PARAM(UINT64, bundleId); FUNCTION_LOG_PARAM(UINT64, bundleOffset); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI, biMapRead); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpecBackup); FUNCTION_LOG_PARAM(IO_FILTER, compress); FUNCTION_LOG_PARAM(IO_FILTER, encrypt); FUNCTION_LOG_END(); ASSERT(file != NULL); ASSERT(file->blockIncrSize != 0); IoFilter *result; // If there is no prior block map then create the filter without one if (file->blockIncrMapPriorFile == NULL) { result = blockIncrNew( file->blockIncrSuperSize, file->blockIncrSize, file->blockIncrChecksumSize, blockIncrReference, bundleId, bundleOffset, NULL, compress, encrypt); } // Else read the prior block map and create the filter with it else { ASSERT(biMapRead != NULL); MEM_CONTEXT_TEMP_BEGIN() { TRY_BEGIN() { Buffer *const blockMap = bufNew(0); IoWrite *const blockMapWrite = ioBufferWriteNew(blockMap); // Add a size filter before decryption so the size is measured as stored in the repository ioFilterGroupAdd(ioWriteFilterGroup(blockMapWrite), ioSizeNew()); if (cipherSpecType(cipherSpecBackup) != cipherTypeNone) { ioFilterGroupAdd( ioWriteFilterGroup(blockMapWrite), cipherBlockNewP( cipherModeDecrypt, cipherSpecBackup, .header = cipherBlockHeaderNone)); } ioWriteOpen(blockMapWrite); ioCopyP(storageReadMultiIo(biMapRead), blockMapWrite, .limit = VARUINT64(file->blockIncrMapPriorSize)); ioWriteClose(blockMapWrite); // The entire map must have been read. When it was not the map is smaller than the manifest expects and the parse // below would fail without making it clear that data is missing. const uint64_t blockMapSizeRead = pckReadU64P( ioFilterGroupResultP(ioWriteFilterGroup(blockMapWrite), SIZE_FILTER_TYPE)); if (blockMapSizeRead != file->blockIncrMapPriorSize) { THROW_FMT( FileReadError, "expected %" PRIu64 " byte(s) but read %" PRIu64, file->blockIncrMapPriorSize, blockMapSizeRead); } // Create the filter in the prior context so the map read above can be freed MEM_CONTEXT_PRIOR_BEGIN() { result = blockIncrNew( file->blockIncrSuperSize, file->blockIncrSize, file->blockIncrChecksumSize, blockIncrReference, bundleId, bundleOffset, blockMap, compress, encrypt); } MEM_CONTEXT_PRIOR_END(); } CATCH_ANY() { THROWP_FMT( errorType(), "unable to read block map for '%s' from '%s' at offset %" PRIu64 " size %" PRIu64 ": %s", strZ(file->manifestFile), strZ(storagePathP(storageRepo(), file->blockIncrMapPriorFile)), file->blockIncrMapPriorOffset, file->blockIncrMapPriorSize, errorMessage()); } TRY_END(); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(IO_FILTER, result); } static List * backupFile( const String *const repoFile, const uint64_t bundleId, const bool bundleRaw, const unsigned int blockIncrReference, const CompressType repoFileCompressType, const int repoFileCompressLevel, const CipherSpec *const cipherSpecBackup, const String *const pgVersionForce, const PgPageSize pageSize, const List *const fileList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, repoFile); // Repo file FUNCTION_LOG_PARAM(UINT64, bundleId); // Bundle id (0 if none) FUNCTION_LOG_PARAM(BOOL, bundleRaw); // Raw compress/encrypt format in bundles? FUNCTION_LOG_PARAM(UINT, blockIncrReference); // Block incremental reference to use in map FUNCTION_LOG_PARAM(ENUM, repoFileCompressType); // Compress type for repo file FUNCTION_LOG_PARAM(INT, repoFileCompressLevel); // Compression level for repo file FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpecBackup); // Cipher spec to encrypt the backup file FUNCTION_LOG_PARAM(ENUM, pageSize); // Page size FUNCTION_LOG_PARAM(STRING, pgVersionForce); // Force pg version FUNCTION_LOG_PARAM(LIST, fileList); // List of files to backup FUNCTION_LOG_END(); ASSERT(repoFile != NULL); ASSERT(cipherSpecBackup != NULL); ASSERT(fileList != NULL && !lstEmpty(fileList)); ASSERT(pgPageSizeValid(pageSize)); // Backup file results List *const result = lstNewP(sizeof(BackupFileResult)); MEM_CONTEXT_TEMP_BEGIN() { // Multi-read for prior block incremental maps. Created lazily on first use to avoid an open (and a round trip to the repo // host) for jobs that have no maps to read, e.g. full backups or jobs with only whole files. StorageReadMulti *biMapRead = NULL; // Check files to determine which ones need to be copied for (unsigned int fileIdx = 0; fileIdx < lstSize(fileList); fileIdx++) { // Use a per-file mem context to reduce memory usage MEM_CONTEXT_TEMP_BEGIN() { const BackupFile *const file = lstGet(fileList, fileIdx); ASSERT(file->pgFile != NULL); ASSERT(file->manifestFile != NULL); ASSERT((!file->pgFileDelta && !file->manifestFileResume) || file->pgFileChecksum != NULL); BackupFileResult *const fileResult = lstAdd( result, &(BackupFileResult){.manifestFile = file->manifestFile, .backupCopyResult = backupCopyResultCopy}); // Does the file in pg match the checksum and size passed? bool pgFileMatch = false; // If delta then check the pg checksum if (file->pgFileDelta) { // Generate checksum/size for the pg file. Only read as many bytes as passed in pgFileSize. If the file has // grown since the manifest was built we don't need to consider the extra bytes since they will be replayed from // WAL during recovery. IoRead *const read = storageReadIo( storageNewReadP( storagePg(), file->pgFile, .ignoreMissing = file->pgFileIgnoreMissing, .limit = file->pgFileCopyExactSize ? VARUINT64(file->pgFileSizeOriginal) : NULL)); ioFilterGroupAdd(ioReadFilterGroup(read), cryptoHashNew(hashTypeSha1)); ioFilterGroupAdd(ioReadFilterGroup(read), ioSizeNew()); // If the pg file exists check the checksum/size if (ioReadDrain(read)) { const Buffer *const pgTestChecksum = pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(read), CRYPTO_HASH_FILTER_TYPE)); const uint64_t pgTestSize = pckReadU64P(ioFilterGroupResultP(ioReadFilterGroup(read), SIZE_FILTER_TYPE)); // Does the pg file match? if (file->pgFileSize == pgTestSize && bufEq(file->pgFileChecksum, pgTestChecksum)) { pgFileMatch = true; // If it matches and is a reference to a previous backup then no need to copy the file if (file->manifestFileHasReference) { MEM_CONTEXT_BEGIN(lstMemContext(result)) { fileResult->backupCopyResult = backupCopyResultNoOp; fileResult->copySize = file->pgFileSize; fileResult->copyChecksum = file->pgFileChecksum; } MEM_CONTEXT_END(); } } } // Else the source file is missing from the database so skip this file else fileResult->backupCopyResult = backupCopyResultSkip; } // On resume check the manifest file if (file->manifestFileResume) { // Resumed files should never have a reference to a prior backup ASSERT(!file->manifestFileHasReference); // If the file is missing from pg, then remove it from the repo (backupJobResult() will remove it from the // manifest) if (fileResult->backupCopyResult == backupCopyResultSkip) { storageRemoveP(storageRepoWrite(), repoFile); } // Else if the pg file matches or is unknown because delta was not performed then check the repo file else if (!file->pgFileDelta || pgFileMatch) { // Generate checksum/size for the repo file IoRead *const read = storageReadIo(storageNewReadP(storageRepo(), repoFile)); ioFilterGroupAdd(ioReadFilterGroup(read), cryptoHashNew(hashTypeSha1)); ioFilterGroupAdd(ioReadFilterGroup(read), ioSizeNew()); ioReadDrain(read); // Test checksum/size const Buffer *const pgTestChecksum = pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(read), CRYPTO_HASH_FILTER_TYPE)); const uint64_t pgTestSize = pckReadU64P(ioFilterGroupResultP(ioReadFilterGroup(read), SIZE_FILTER_TYPE)); // No need to recopy if checksum/size match. When the repo checksum is missing still compare to repo size // since the repo checksum should only be missing when the repo file was not compressed/encrypted, i.e. the // repo size should match the original size. There is no need to worry about old manifests here since resume // does not work across versions. if (file->repoFileSize == pgTestSize && bufEq(file->repoFileChecksum != NULL ? file->repoFileChecksum : file->pgFileChecksum, pgTestChecksum)) { MEM_CONTEXT_BEGIN(lstMemContext(result)) { fileResult->backupCopyResult = backupCopyResultChecksum; fileResult->copySize = file->pgFileSize; fileResult->copyChecksum = file->pgFileChecksum; } MEM_CONTEXT_END(); } // Else copy when repo file is invalid else fileResult->repoInvalid = true; } } // If block incremental file will be copied add block map to the read if (fileResult->backupCopyResult == backupCopyResultCopy && file->blockIncrSize != 0 && file->blockIncrMapPriorFile != NULL) { // Create the multi-read on first use in the outer context so it survives the per-file context if (biMapRead == NULL) { MEM_CONTEXT_PRIOR_BEGIN() { biMapRead = storageNewReadMultiP(storageRepo()); } MEM_CONTEXT_PRIOR_END(); } storageReadMultiAddP( biMapRead, file->blockIncrMapPriorFile, .offset = file->blockIncrMapPriorOffset, .limit = VARUINT64(file->blockIncrMapPriorSize)); } } MEM_CONTEXT_TEMP_END(); } // Are the files compressible during the copy? const bool compressible = repoFileCompressType == compressTypeNone && cipherSpecType(cipherSpecBackup) == cipherTypeNone; // Copy files that need to be copied StorageWrite *write = NULL; uint64_t bundleOffset = 0; // Open the multi-read if any block incremental maps were added if (biMapRead != NULL) ioReadOpen(storageReadMultiIo(biMapRead)); for (unsigned int fileIdx = 0; fileIdx < lstSize(fileList); fileIdx++) { // Use a per-file mem context to reduce memory usage MEM_CONTEXT_TEMP_BEGIN() { const BackupFile *const file = lstGet(fileList, fileIdx); BackupFileResult *const fileResult = lstGet(result, fileIdx); if (fileResult->backupCopyResult == backupCopyResultCopy) { // Setup pg file for read. Only read as many bytes as passed in pgFileSize. If the file is growing it does no // good to copy data past the end of the size recorded in the manifest since those blocks will need to be // replayed from WAL during recovery. pg_control requires special handling since it needs to be retried on crc // validation failure. bool repoChecksum = false; IoRead *readIo; if (strEqZ(file->pgFile, PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)) readIo = ioBufferReadNew(pgControlBufferFromFile(storagePg(), pgVersionForce)); else { readIo = storageReadIo( storageNewReadP( storagePg(), file->pgFile, .ignoreMissing = file->pgFileIgnoreMissing, .compressible = compressible, .limit = file->pgFileCopyExactSize ? VARUINT64(file->pgFileSizeOriginal) : NULL)); } ioFilterGroupAdd(ioReadFilterGroup(readIo), cryptoHashNew(hashTypeSha1)); ioFilterGroupAdd(ioReadFilterGroup(readIo), ioSizeNew()); // Add page checksum filter if (file->pgFileChecksumPage) { ioFilterGroupAdd( ioReadFilterGroup(readIo), pageChecksumNew( segmentNumber(file->pgFile), PG_SEGMENT_SIZE_DEFAULT / pageSize, pageSize, file->pgFilePageHeaderCheck, storagePathP(storagePg(), file->pgFile))); } // Compress filter IoFilter *const compress = repoFileCompressType != compressTypeNone ? compressFilterP( repoFileCompressType, repoFileCompressLevel, .raw = bundleRaw || file->blockIncrSize != 0) : NULL; // Encrypt filter IoFilter *const encrypt = cipherSpecType(cipherSpecBackup) != cipherTypeNone ? cipherBlockNewP( cipherModeEncrypt, cipherSpecBackup, .header = bundleRaw || file->blockIncrSize != 0 ? cipherBlockHeaderNone : cipherBlockHeaderMagic) : NULL; // If block incremental then add the filter and pass compress/encrypt filters to it since each block is // compressed/encrypted separately if (file->blockIncrSize != 0) { ioFilterGroupAdd( ioReadFilterGroup(readIo), backupFileBlockIncr( file, blockIncrReference, bundleId, bundleOffset, biMapRead, cipherSpecBackup, compress, encrypt)); repoChecksum = true; } // Else apply compress/encrypt filters to the entire file else { // Add compress filter if (compress != NULL) { ioFilterGroupAdd(ioReadFilterGroup(readIo), compress); repoChecksum = true; } // Add encrypt filter if (encrypt != NULL) { ioFilterGroupAdd(ioReadFilterGroup(readIo), encrypt); repoChecksum = true; } } // Capture checksum of file stored in the repo if filters that modify the output have been applied if (repoChecksum) ioFilterGroupAdd(ioReadFilterGroup(readIo), cryptoHashNew(hashTypeSha1)); // Add size filter last to calculate repo size ioFilterGroupAdd(ioReadFilterGroup(readIo), ioSizeNew()); // Open the source if (ioReadOpen(readIo)) { Buffer *const buffer = bufNew(ioBufferSize()); bool readEof = false; // Read the first buffer to determine if the file was truncated or was not changed. Detecting truncation // matters only when bundling is enabled as otherwise the file will be stored anyway. ioRead(readIo, buffer); if (ioReadEof(readIo)) { // Close the source and set eof ioReadClose(readIo); readEof = true; // Get file size fileResult->copySize = pckReadU64P( ioFilterGroupResultP(ioReadFilterGroup(readIo), SIZE_FILTER_TYPE, .idx = 0)); // If file is zero-length then it was truncated during the backup. When bundling we can simply mark // it as truncated since no file needs to be stored. The same is true for block incremental, which // additionally must not be stored because a block map with no blocks cannot be read back. if ((bundleId != 0 || file->blockIncrSize != 0) && fileResult->copySize == 0) { fileResult->backupCopyResult = backupCopyResultTruncate; fileResult->copyChecksum = HASH_TYPE_SHA1_ZERO_BUF; ASSERT( bufEq( fileResult->copyChecksum, pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(readIo), CRYPTO_HASH_FILTER_TYPE, .idx = 0)))); } // Else check if size is equal to prior size else if (file->manifestFileHasReference && fileResult->copySize == file->pgFileSize) { const Buffer *const copyChecksum = pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(readIo), CRYPTO_HASH_FILTER_TYPE)); // If checksum is also equal then no need to copy the file if (bufEq(file->pgFileChecksum, copyChecksum)) { // If block incremental make sure no map was returned but a prior map was provided ASSERT( file->blockIncrSize == 0 || (pckReadU64P( ioFilterGroupResultP(ioReadFilterGroup(readIo), BLOCK_INCR_FILTER_TYPE)) == 0 && file->blockIncrMapPriorFile != NULL)); fileResult->backupCopyResult = backupCopyResultNoOp; fileResult->copyChecksum = file->pgFileChecksum; } } } // Copy the file if (fileResult->backupCopyResult == backupCopyResultCopy) { // Setup the repo file for write. There is no need to write the file atomically (e.g. via a temp file on // Posix) because checksums are tested on resume after a failed backup. The path does not need to be // synced for each file because all paths are synced at the end of the backup. It needs to be created in // the prior context because it will live longer than a single loop when more than one file is being // written. if (write == NULL) { MEM_CONTEXT_PRIOR_BEGIN() { write = storageNewWriteP( storageRepoWrite(), repoFile, .compressible = compressible, .noAtomic = true, .noSyncPath = true); ioWriteOpen(storageWriteIo(write)); } MEM_CONTEXT_PRIOR_END(); } // Write the first buffer ioWrite(storageWriteIo(write), buffer); bufFree(buffer); // Copy remainder of the file if not eof if (!readEof) { ioCopyP(readIo, storageWriteIo(write)); // Close the source ioReadClose(readIo); } // Get copy results MEM_CONTEXT_BEGIN(lstMemContext(result)) { // Get size and checksum fileResult->copySize = pckReadU64P( ioFilterGroupResultP(ioReadFilterGroup(readIo), SIZE_FILTER_TYPE, .idx = 0)); fileResult->copyChecksum = pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(readIo), CRYPTO_HASH_FILTER_TYPE, .idx = 0)); // Get bundle offset fileResult->bundleOffset = bundleOffset; // Get repo size fileResult->repoSize = pckReadU64P( ioFilterGroupResultP(ioReadFilterGroup(readIo), SIZE_FILTER_TYPE, .idx = 1)); // Get results of page checksum validation if (file->pgFileChecksumPage) { fileResult->pageChecksumResult = pckDup( ioFilterGroupResultPackP(ioReadFilterGroup(readIo), PAGE_CHECKSUM_FILTER_TYPE)); } // Get results of block incremental if (file->blockIncrSize != 0) { fileResult->blockIncrMapSize = pckReadU64P( ioFilterGroupResultP(ioReadFilterGroup(readIo), BLOCK_INCR_FILTER_TYPE)); // There must be a map because the file should have changed or shrunk ASSERT(fileResult->blockIncrMapSize > 0); } // Get repo checksum if (repoChecksum) { fileResult->repoChecksum = pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(readIo), CRYPTO_HASH_FILTER_TYPE, .idx = 1)); } } MEM_CONTEXT_END(); bundleOffset += fileResult->repoSize; } } // Else if source file is missing and the read setup indicated ignore a missing file, the database removed it so // skip it else fileResult->backupCopyResult = backupCopyResultSkip; } } MEM_CONTEXT_TEMP_END(); } // Close the repository file if it was opened if (write != NULL) ioWriteClose(storageWriteIo(write)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(LIST, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/incr.c.inc0000664000175100017510000001716515265101745020735 0ustar00runnerrunner/*********************************************************************************************************************************** Create an incremental backup if type is not full and a compatible prior backup exists ***********************************************************************************************************************************/ // Helper to find a compatible prior backup static Manifest * backupBuildIncrPrior(const InfoBackup *const infoBackup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); Manifest *result = NULL; // No incremental if backup type is full const BackupType type = (BackupType)cfgOptionStrId(cfgOptType); if (type != backupTypeFull) { MEM_CONTEXT_TEMP_BEGIN() { const InfoPgData infoPg = infoPgDataCurrent(infoBackupPg(infoBackup)); const String *backupLabelPrior = NULL; const unsigned int backupTotal = infoBackupDataTotal(infoBackup); for (unsigned int backupIdx = backupTotal - 1; backupIdx < backupTotal; backupIdx--) { const InfoBackupData backupPrior = infoBackupData(infoBackup, backupIdx); // The prior backup for a diff must be full if (type == backupTypeDiff && backupPrior.backupType != backupTypeFull) continue; // The backups must come from the same cluster ??? This should enable delta instead if (infoPg.id != backupPrior.backupPgId) continue; // The prior backup must be at the format new backups are written with so that a backup set is never mixed-format. // After the repository format has been upgraded no prior will match and the backup will be changed to full below, // which is the only point where a backup set adopts a new format. if (backupPrior.backrestFormat != infoBackupFormat(infoBackup)) continue; // This backup is a candidate for prior backupLabelPrior = strDup(backupPrior.backupLabel); break; } // If there is a prior backup then check that options for the new backup are compatible if (backupLabelPrior != NULL) { result = manifestLoadFile( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabelPrior)), infoBackupCipherSpec(infoBackup)); const ManifestData *const manifestPriorData = manifestData(result); LOG_INFO_FMT( "last backup label = %s, version = %s", strZ(manifestData(result)->backupLabel), strZ(manifestData(result)->backrestVersion)); // Warn if compress-type option changed if (compressTypeEnum(cfgOptionStrId(cfgOptCompressType)) != manifestPriorData->backupOptionCompressType) { LOG_WARN_FMT( "%s backup cannot alter " CFGOPT_COMPRESS_TYPE " option to '%s', reset to value in %s", strZ(cfgOptionDisplay(cfgOptType)), strZ(cfgOptionDisplay(cfgOptCompressType)), strZ(backupLabelPrior)); // Set the compression type back to whatever was in the prior backup. This is not strictly needed since we could // store compression type on a per file basis, but it seems simplest and safest for now. cfgOptionSet( cfgOptCompressType, cfgSourceParam, VARSTR(compressTypeStr(manifestPriorData->backupOptionCompressType))); // There's a small chance that the prior manifest is old enough that backupOptionCompressLevel was not recorded. // There's an even smaller chance that the user will also alter compression-type in this scenario right after // upgrading to a newer version. Because we judge this combination of events to be nearly impossible just check // here so no test coverage is needed. CHECK( FormatError, manifestPriorData->backupOptionCompressLevel != NULL, "compress-level missing in prior manifest"); // Set the compression level back to whatever was in the prior backup cfgOptionSet( cfgOptCompressLevel, cfgSourceParam, VARINT64(varUInt(manifestPriorData->backupOptionCompressLevel))); } // If not defined this backup was done in a version prior to page checksums being introduced. Just set checksum-page // to false and move on without a warning. Page checksums will start on the next full backup. if (manifestData(result)->backupOptionChecksumPage == NULL) { cfgOptionSet(cfgOptChecksumPage, cfgSourceParam, BOOL_FALSE_VAR); } // Don't allow the checksum-page option to change in a diff or incr backup. This could be confusing as only certain // files would be checksummed and the list could be incomplete during reporting. else { const bool checksumPagePrior = varBool(manifestData(result)->backupOptionChecksumPage); // Warn if an incompatible setting was explicitly requested if (checksumPagePrior != cfgOptionBool(cfgOptChecksumPage)) { LOG_WARN_FMT( "%s backup cannot alter '" CFGOPT_CHECKSUM_PAGE "' option to '%s', reset to '%s' from %s", strZ(cfgOptionDisplay(cfgOptType)), strZ(cfgOptionDisplay(cfgOptChecksumPage)), cvtBoolToConstZ(checksumPagePrior), strZ(manifestData(result)->backupLabel)); } cfgOptionSet(cfgOptChecksumPage, cfgSourceParam, VARBOOL(checksumPagePrior)); } manifestMove(result, memContextPrior()); } else { LOG_WARN_FMT("no prior backup exists, %s backup has been changed to full", strZ(cfgOptionDisplay(cfgOptType))); cfgOptionSet(cfgOptType, cfgSourceParam, VARUINT64(backupTypeFull)); } } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(MANIFEST, result); } static bool backupBuildIncr(Manifest *const manifest, Manifest *const manifestPrior, const String *const archiveStart) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(MANIFEST, manifestPrior); FUNCTION_LOG_PARAM(STRING, archiveStart); FUNCTION_LOG_END(); ASSERT(manifest != NULL); bool result = false; // No incremental if no prior manifest if (manifestPrior != NULL) { MEM_CONTEXT_TEMP_BEGIN() { // Move the manifest to this context so it will be freed when we are done manifestMove(manifestPrior, MEM_CONTEXT_TEMP()); // Build incremental manifest manifestBuildIncr(manifest, manifestPrior, (BackupType)cfgOptionStrId(cfgOptType), archiveStart); // Set the cipher subpass from prior manifest since we want a single subpass for the entire backup set manifestCipherSpecSet(manifest, manifestCipherSpec(manifestPrior)); // Incremental was built result = true; } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(BOOL, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/label.c.inc0000664000175100017510000001174015265101745021052 0ustar00runnerrunner/********************************************************************************************************************************** Generate a unique backup label that does not contain a timestamp from a previous backup ***********************************************************************************************************************************/ static String * backupLabelCreate(const BackupType type, const String *const backupLabelPrior, const time_t timestamp) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(STRING, backupLabelPrior); FUNCTION_LOG_PARAM(TIME, timestamp); FUNCTION_LOG_END(); ASSERT((type == backupTypeFull && backupLabelPrior == NULL) || (type != backupTypeFull && backupLabelPrior != NULL)); ASSERT(timestamp > 0); String *result; MEM_CONTEXT_TEMP_BEGIN() { const String *backupLabelLatest = NULL; // For full backup compare against all backups. For other backup types compare only against backups in the same full backup // set, i.e. those whose name begins with the full backup part of backupLabelPrior. Since the new label is generated from // backupLabelPrior it can never conflict with a label in another full backup set. This prevents label generation from // failing when the last full backup is removed and then a diff/incr is generated from the last remaining full backup, or // when a removed full backup is still returned by the repository listing, which can happen on object stores that retain // empty prefixes. const String *const labelRegExp = type == backupTypeFull ? backupRegExpP(.full = true, .differential = true, .incremental = true, .noAnchorEnd = true) : strNewFmt("^%.*sF\\_" DATE_TIME_REGEX "(D|I)", DATE_TIME_LEN, strZ(backupLabelPrior)); // Get the newest backup const StringList *const backupList = strLstSort( storageListP(storageRepo(), STRDEF(STORAGE_REPO_BACKUP), .expression = strNewFmt("%s$", strZ(labelRegExp))), sortOrderDesc); if (!strLstEmpty(backupList)) backupLabelLatest = strLstGet(backupList, 0); // Get the newest history const StringList *const historyYearList = strLstSort( storageListP(storageRepo(), STRDEF(STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY), .expression = STRDEF("^2[0-9]{3}$")), sortOrderDesc); if (!strLstEmpty(historyYearList)) { const StringList *const historyList = strLstSort( storageListP( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY "/%s", strZ(strLstGet(historyYearList, 0))), .expression = strNewFmt("%s\\.manifest\\.%s$", strZ(labelRegExp), strZ(compressTypeStr(compressTypeGz)))), sortOrderDesc); if (!strLstEmpty(historyList)) { const String *const historyLabelLatest = strLstGet(historyList, 0); if (backupLabelLatest == NULL || strCmp(historyLabelLatest, backupLabelLatest) > 0) { // Strip off the compression and manifest extensions in case this ends up in an error message backupLabelLatest = compressExtStrip(historyLabelLatest, compressTypeFromName(historyLabelLatest)); backupLabelLatest = strSubN(backupLabelLatest, 0, strSize(backupLabelLatest) - sizeof(BACKUP_MANIFEST_EXT) + 1); } } } // Now that we have the latest label check if the provided timestamp will give us an even later label result = backupLabelFormat(type, backupLabelPrior, timestamp); if (backupLabelLatest != NULL && strCmp(result, backupLabelLatest) <= 0) { // If that didn't give us a later label then add one second. It's possible that two backups (they would need to be // offline or halted online) have run very close together. result = backupLabelFormat(type, backupLabelPrior, timestamp + 1); // If the label is still not latest then error. There is probably a timezone change or massive clock skew. if (strCmp(result, backupLabelLatest) <= 0) { THROW_FMT( ClockError, "new backup label '%s' is not later than latest backup label '%s'\n" "HINT: has the timezone changed?\n" "HINT: is there clock skew?", strZ(result), strZ(backupLabelLatest)); } // If adding a second worked then sleep the remainder of the current second so we don't start early sleepMSec(MSEC_PER_SEC - (timeMSec() % MSEC_PER_SEC)); } MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(result); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/option.c.inc0000664000175100017510000001742115265101745021305 0ustar00runnerrunner/********************************************************************************************************************************** Build block incremental maps ***********************************************************************************************************************************/ // Size map. Block size is increased when the block map would be larger than a single block. The break can be calculated with this // formula: [block size in KiB] / (1024 / [block size in KiB] * [checksum size]) * 1073741824. static const ManifestBlockIncrSizeMap manifestBlockIncrSizeMapDefault[] = { {.fileSize = 914 * 1024 * 1024, .blockSize = 88 * 1024}, {.fileSize = 740 * 1024 * 1024, .blockSize = 80 * 1024}, {.fileSize = 585 * 1024 * 1024, .blockSize = 72 * 1024}, {.fileSize = 448 * 1024 * 1024, .blockSize = 64 * 1024}, {.fileSize = 329 * 1024 * 1024, .blockSize = 56 * 1024}, {.fileSize = 228 * 1024 * 1024, .blockSize = 48 * 1024}, {.fileSize = 146 * 1024 * 1024, .blockSize = 40 * 1024}, {.fileSize = 96 * 1024 * 1024, .blockSize = 32 * 1024}, {.fileSize = 43 * 1024 * 1024, .blockSize = 24 * 1024}, {.fileSize = 11 * 1024 * 1024, .blockSize = 16 * 1024}, {.fileSize = 16 * 1024, .blockSize = 8 * 1024}, }; // Age map static const ManifestBlockIncrAgeMap manifestBlockIncrAgeMapDefault[] = { {.fileAge = 4 * 7 * SEC_PER_DAY, .blockMultiplier = 0}, {.fileAge = 2 * 7 * SEC_PER_DAY, .blockMultiplier = 4}, {.fileAge = 7 * SEC_PER_DAY, .blockMultiplier = 2}, }; // Checksum size map static const ManifestBlockIncrChecksumSizeMap manifestBlockIncrChecksumSizeMapDefault[] = { {.blockSize = 4 * 1024 * 1024, .checksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN + 6}, {.blockSize = 2 * 1024 * 1024, .checksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN + 5}, {.blockSize = 1024 * 1024, .checksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN + 4}, {.blockSize = 512 * 1024, .checksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN + 3}, {.blockSize = 128 * 1024, .checksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN + 2}, {.blockSize = 32 * 1024, .checksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN + 1}, }; // All maps static const ManifestBlockIncrMap manifestBlockIncrMap = { .sizeMap = LSTDEF(manifestBlockIncrSizeMapDefault), .ageMap = LSTDEF(manifestBlockIncrAgeMapDefault), .checksumSizeMap = LSTDEF(manifestBlockIncrChecksumSizeMapDefault), }; // Convert map size static unsigned int backupBlockIncrMapSize(const ConfigOption optionId, const unsigned int optionIdx, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); int64_t result; TRY_BEGIN() { result = cfgParseSize(value); // Error if value is out of range (no need for an error message since that will be generated in the catch block) if (result <= 0 || result > UINT_MAX) THROW(OptionInvalidValueError, ""); } CATCH_ANY() { THROW_FMT( OptionInvalidValueError, "'%s' is not valid for '%s' option", strZ(value), cfgOptionIdxName(optionId, optionIdx)); } TRY_END(); ASSERT(result > 0 && result < UINT_MAX); FUNCTION_TEST_RETURN(UINT, (unsigned int)result); } // Convert map checksum size static unsigned int backupBlockIncrMapChecksumSize(const ConfigOption optionId, const unsigned int optionIdx, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); unsigned int result = 0; TRY_BEGIN() { result = varUIntForce(value); } CATCH_ANY() { } TRY_END(); if (result < BLOCK_INCR_CHECKSUM_SIZE_MIN) { THROW_FMT( OptionInvalidValueError, "'%s' is not valid for '%s' option", strZ(varStr(value)), cfgOptionIdxName(optionId, optionIdx)); } FUNCTION_TEST_RETURN(UINT, result); } static ManifestBlockIncrMap backupBlockIncrMap(void) { FUNCTION_TEST_VOID(); FUNCTION_AUDIT_HELPER(); ManifestBlockIncrMap result = manifestBlockIncrMap; if (cfgOptionBool(cfgOptRepoBlock)) { // Build size map const KeyValue *const manifestBlockIncrSizeKv = cfgOptionKvNull(cfgOptRepoBlockSizeMap); if (manifestBlockIncrSizeKv != NULL) { List *const map = lstNewP(sizeof(ManifestBlockIncrSizeMap), .comparator = lstComparatorUInt); const VariantList *const mapKeyList = kvKeyList(manifestBlockIncrSizeKv); for (unsigned int mapKeyIdx = 0; mapKeyIdx < varLstSize(mapKeyList); mapKeyIdx++) { const Variant *mapKey = varLstGet(mapKeyList, mapKeyIdx); ManifestBlockIncrSizeMap manifestBuildBlockIncrSizeMap = { .fileSize = backupBlockIncrMapSize( cfgOptRepoBlockSizeMap, cfgOptionIdxDefault(cfgOptRepoBlockSizeMap), varStr(mapKey)), .blockSize = backupBlockIncrMapSize( cfgOptRepoBlockSizeMap, cfgOptionIdxDefault(cfgOptRepoBlockSizeMap), varStr(kvGet(manifestBlockIncrSizeKv, mapKey))), }; lstAdd(map, &manifestBuildBlockIncrSizeMap); } lstSort(map, sortOrderDesc); result.sizeMap = map; } // Build age map const KeyValue *const manifestBlockIncrAgeKv = cfgOptionKvNull(cfgOptRepoBlockAgeMap); if (manifestBlockIncrAgeKv != NULL) { List *const map = lstNewP(sizeof(ManifestBlockIncrAgeMap), .comparator = lstComparatorUInt); const VariantList *const mapKeyList = kvKeyList(manifestBlockIncrAgeKv); for (unsigned int mapKeyIdx = 0; mapKeyIdx < varLstSize(mapKeyList); mapKeyIdx++) { const Variant *mapKey = varLstGet(mapKeyList, mapKeyIdx); ManifestBlockIncrAgeMap manifestBuildBlockIncrAgeMap = { .fileAge = (unsigned int)(varUIntForce(mapKey) * SEC_PER_DAY), .blockMultiplier = varUIntForce(kvGet(manifestBlockIncrAgeKv, mapKey)), }; lstAdd(map, &manifestBuildBlockIncrAgeMap); } lstSort(map, sortOrderDesc); result.ageMap = map; } // Build checksum size map const KeyValue *const manifestBlockIncrChecksumSizeKv = cfgOptionKvNull(cfgOptRepoBlockChecksumSizeMap); if (manifestBlockIncrChecksumSizeKv != NULL) { List *const map = lstNewP(sizeof(ManifestBlockIncrChecksumSizeMap), .comparator = lstComparatorUInt); const VariantList *const mapKeyList = kvKeyList(manifestBlockIncrChecksumSizeKv); for (unsigned int mapKeyIdx = 0; mapKeyIdx < varLstSize(mapKeyList); mapKeyIdx++) { const Variant *const mapKey = varLstGet(mapKeyList, mapKeyIdx); ManifestBlockIncrChecksumSizeMap manifestBuildBlockIncrChecksumSizeMap = { .blockSize = backupBlockIncrMapSize( cfgOptRepoBlockChecksumSizeMap, cfgOptionIdxDefault(cfgOptRepoBlockChecksumSizeMap), varStr(mapKey)), .checksumSize = backupBlockIncrMapChecksumSize( cfgOptRepoBlockChecksumSizeMap, cfgOptionIdxDefault(cfgOptRepoBlockChecksumSizeMap), kvGet(manifestBlockIncrChecksumSizeKv, mapKey)), }; lstAdd(map, &manifestBuildBlockIncrChecksumSizeMap); } lstSort(map, sortOrderDesc); result.checksumSizeMap = map; } } FUNCTION_TEST_RETURN_TYPE(ManifestBlockIncrMap, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/pageChecksum.c0000664000175100017510000002744515265101745021633 0ustar00runnerrunner/*********************************************************************************************************************************** Page Checksum Filter ***********************************************************************************************************************************/ #include #include "command/backup/pageChecksum.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/macro.h" #include "common/type/json.h" #include "common/type/object.h" #include "postgres/interface/static.vendor.h" #include "storage/posix/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct PageChecksum { unsigned int segmentPageTotal; // Total pages in a segment PgPageSize pageSize; // Page size unsigned int pageNoOffset; // Page number offset for subsequent segments bool headerCheck; // Perform additional header checks? const String *fileName; // Used to load the file to retry pages uint8_t *pageBuffer; // Buffer to hold a page while verifying the checksum bool valid; // Is the relation structure valid? bool align; // Is the relation alignment valid? PackWrite *error; // List of checksum errors } PageChecksum; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void pageChecksumToLog(const PageChecksum *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{valid: %s, align: %s}", cvtBoolToConstZ(this->valid), cvtBoolToConstZ(this->align)); } #define FUNCTION_LOG_PAGE_CHECKSUM_TYPE \ PageChecksum * #define FUNCTION_LOG_PAGE_CHECKSUM_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, pageChecksumToLog, buffer, bufferSize) /*********************************************************************************************************************************** Verify page checksums ***********************************************************************************************************************************/ static void pageChecksumProcess(THIS_VOID, const Buffer *const input) { THIS(PageChecksum); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PAGE_CHECKSUM, this); FUNCTION_LOG_PARAM(BUFFER, input); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(input != NULL); // Calculate total pages in the buffer unsigned int pageTotal = (unsigned int)(bufUsed(input) / this->pageSize); // If there is a partial page make sure there is enough of it to validate the checksum const unsigned int pageRemainder = (unsigned int)(bufUsed(input) % this->pageSize); if (pageRemainder != 0) { // Misaligned blocks, if any, should only be at the end of the file if (!this->align) THROW(AssertError, "should not be possible to see two misaligned pages in a row"); // Mark this buffer as misaligned in case we see another one this->align = false; // If there at least 512 bytes then we'll treat this as a partial write (modern file systems will have at least 4096) if (pageRemainder >= 512) { pageTotal++; } // Else this appears to be a corrupted file and we'll stop doing page checksums else this->valid = false; } // Verify the checksums of complete pages in the buffer if (this->valid) { for (unsigned int pageIdx = 0; pageIdx < pageTotal; pageIdx++) { // Get a pointer to the page header const PageHeaderData *const pageHeader = (const PageHeaderData *)(bufPtrConst(input) + pageIdx * this->pageSize); // Block number relative to all segments in the relation const unsigned int blockNo = this->pageNoOffset + pageIdx; // Only validate page checksum if the page is complete if (this->align || pageIdx < pageTotal - 1) { bool pageValid = true; // Skip new all-zero pages. To speed this up first check pd_upper when header check is enabled or pd_checksum when // header check is disabled. The latter is required when the page is encrypted. if ((this->headerCheck && pageHeader->pd_upper == 0) || (!this->headerCheck && pageHeader->pd_checksum == 0)) { // Check that the entire page is zero for (unsigned int pageIdx = 0; pageIdx < this->pageSize / sizeof(size_t); pageIdx++) { if (((const size_t *)pageHeader)[pageIdx] != 0) { pageValid = false; break; } } // If the entire page is zero it is valid if (pageValid) continue; } // Only validate the checksum if the page is valid if (pageValid) { // Make a copy of the page since it will be modified by the page checksum function memcpy(this->pageBuffer, pageHeader, this->pageSize); // Continue if the checksum matches if (pageHeader->pd_checksum == pgPageChecksum(this->pageBuffer, blockNo, this->pageSize)) continue; } // On error retry the page bool changed = false; MEM_CONTEXT_TEMP_BEGIN() { // Reload the page const Buffer *const pageRetry = storageGetP( storageNewReadP( storagePosixNewP(FSLASH_STR), this->fileName, .offset = (uint64_t)(blockNo % this->segmentPageTotal) * this->pageSize, .limit = VARUINT64(this->pageSize))); // Check if the page has changed since it was last read changed = !bufEq(pageRetry, BUF(pageHeader, this->pageSize)); } MEM_CONTEXT_TEMP_END(); // If the page has changed then PostgreSQL must be updating it so we won't consider it to be invalid. We are not // concerned about whether this new page has a valid checksum or not, since the page will be replayed from WAL on // recovery. This prevents (theoretically) limitless retries for a hot page. if (changed) continue; } // Create the error list if it does not exist yet if (this->error == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->error = pckWriteNewP(); pckWriteArrayBeginP(this->error); } MEM_CONTEXT_OBJ_END(); } // Add page number and lsn to the error list pckWriteObjBeginP(this->error, .id = blockNo + 1); pckWriteObjEndP(this->error); } this->pageNoOffset += pageTotal; } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Return filter result ***********************************************************************************************************************************/ static Pack * pageChecksumResult(THIS_VOID) { THIS(PageChecksum); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PAGE_CHECKSUM, this); FUNCTION_LOG_END(); ASSERT(this != NULL); Pack *result; MEM_CONTEXT_OBJ_BEGIN(this) { // End the error array if (this->error != NULL) { pckWriteArrayEndP(this->error); this->valid = false; } // Else create a pack to hold the flags else { this->error = pckWriteNewP(); pckWriteNullP(this->error); } // Valid and align flags pckWriteBoolP(this->error, this->valid, .defaultWrite = true); pckWriteBoolP(this->error, this->align, .defaultWrite = true); // End pack pckWriteEndP(this->error); result = pckMove(pckWriteResult(this->error), memContextPrior()); } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * pageChecksumNew( const unsigned int segmentNo, const unsigned int segmentPageTotal, const PgPageSize pageSize, const bool headerCheck, const String *const fileName) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(UINT, segmentNo); FUNCTION_LOG_PARAM(UINT, segmentPageTotal); FUNCTION_LOG_PARAM(ENUM, pageSize); FUNCTION_LOG_PARAM(BOOL, headerCheck); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_END(); ASSERT(pgPageSizeValid(pageSize)); ASSERT(segmentPageTotal > 0 && segmentPageTotal % pageSize == 0); OBJ_NEW_BEGIN(PageChecksum, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (PageChecksum) { .segmentPageTotal = segmentPageTotal, .pageSize = pageSize, .pageNoOffset = segmentNo * segmentPageTotal, .headerCheck = headerCheck, .fileName = strDup(fileName), .pageBuffer = bufPtr(bufNew(pageSize)), .valid = true, .align = true, }; } OBJ_NEW_END(); // Create param list Pack *paramList; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteU32P(packWrite, segmentNo); pckWriteU32P(packWrite, segmentPageTotal); pckWriteU32P(packWrite, pageSize); pckWriteBoolP(packWrite, headerCheck); pckWriteStrP(packWrite, fileName); pckWriteEndP(packWrite); paramList = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP(PAGE_CHECKSUM_FILTER_TYPE, this, paramList, .in = pageChecksumProcess, .result = pageChecksumResult)); } FN_EXTERN IoFilter * pageChecksumNewPack(const Pack *const paramList) { IoFilter *result; MEM_CONTEXT_TEMP_BEGIN() { PackRead *const paramListPack = pckReadNew(paramList); const unsigned int segmentNo = pckReadU32P(paramListPack); const unsigned int segmentPageTotal = pckReadU32P(paramListPack); const PgPageSize pageSize = (PgPageSize)pckReadU32P(paramListPack); const bool headerCheck = pckReadBoolP(paramListPack); const String *const fileName = pckReadStrP(paramListPack); result = ioFilterMove(pageChecksumNew(segmentNo, segmentPageTotal, pageSize, headerCheck, fileName), memContextPrior()); } MEM_CONTEXT_TEMP_END(); return result; } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/pageChecksum.h0000664000175100017510000000261015265101745021623 0ustar00runnerrunner/*********************************************************************************************************************************** Page Checksum Filter Check all pages in a PostgreSQL relation to ensure the checksums are valid. ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_PAGE_CHECKSUM_H #define COMMAND_BACKUP_PAGE_CHECKSUM_H #include "common/io/filter/filter.h" #include "postgres/interface.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define PAGE_CHECKSUM_FILTER_TYPE STRID5("pg-chksum", 0xdacd681ecf00) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *pageChecksumNew( unsigned int segmentNo, unsigned int segmentPageTotal, PgPageSize pageSize, bool headerCheck, const String *fileName); FN_EXTERN IoFilter *pageChecksumNewPack(const Pack *paramList); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/process.c.inc0000664000175100017510000013306315265101745021454 0ustar00runnerrunner/*********************************************************************************************************************************** Convert page checksum error pack to a VariantList ***********************************************************************************************************************************/ // Helper to output pages and page ranges static void backupJobResultPageChecksumOut(VariantList *const result, const unsigned int pageBegin, const unsigned int pageEnd) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT_LIST, result); FUNCTION_TEST_PARAM(UINT, pageBegin); FUNCTION_TEST_PARAM(UINT, pageEnd); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); // Output a single page if (pageBegin == pageEnd) { varLstAdd(result, varNewUInt64(pageBegin)); } // Else output a page range else { VariantList *errorListSub = varLstNew(); varLstAdd(errorListSub, varNewUInt64(pageBegin)); varLstAdd(errorListSub, varNewUInt64(pageEnd)); varLstAdd(result, varNewVarLst(errorListSub)); } FUNCTION_TEST_RETURN_VOID(); } static VariantList * backupJobResultPageChecksum(PackRead *const checksumPageResult) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, checksumPageResult); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); VariantList *result = NULL; // If there is an error result array if (!pckReadNullP(checksumPageResult)) { result = varLstNew(); pckReadArrayBeginP(checksumPageResult); bool first = false; unsigned int pageBegin = 0; unsigned int pageEnd = 0; // Combine results into a more compact form while (pckReadNext(checksumPageResult)) { const unsigned int pageId = pckReadId(checksumPageResult) - 1; pckReadObjBeginP(checksumPageResult, .id = pageId + 1); // If first error then just store page if (!first) { pageBegin = pageId; pageEnd = pageId; first = true; } // Expand list when the page is in sequence else if (pageId == pageEnd + 1) { pageEnd = pageId; } // Else output the page or page range else { backupJobResultPageChecksumOut(result, pageBegin, pageEnd); // Start again with a single page range pageBegin = pageId; pageEnd = pageId; } pckReadObjEndP(checksumPageResult); } // Check that the array was not empty CHECK(FormatError, first, "page checksum result array is empty"); // Output last page or page range backupJobResultPageChecksumOut(result, pageBegin, pageEnd); pckReadArrayEndP(checksumPageResult); } FUNCTION_LOG_RETURN(VARIANT_LIST, result); } /*********************************************************************************************************************************** Log the results of a job and throw errors ***********************************************************************************************************************************/ static void backupJobResult( Manifest *const manifest, const String *const host, const Storage *const storagePg, StringList *const fileRemove, ProtocolParallelJob *const job, const PgPageSize pageSize, unsigned int *const warningTotal, const uint64_t sizeTotal, uint64_t *const sizeProgress, unsigned int *const currentPercentComplete) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(STORAGE, storagePg); FUNCTION_LOG_PARAM(STRING_LIST, fileRemove); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, job); FUNCTION_LOG_PARAM(ENUM, pageSize); FUNCTION_LOG_PARAM_P(UINT, warningTotal); FUNCTION_LOG_PARAM(UINT64, sizeTotal); FUNCTION_LOG_PARAM_P(UINT64, sizeProgress); FUNCTION_LOG_PARAM_P(UINT, currentPercentComplete); FUNCTION_LOG_END(); ASSERT(manifest != NULL); ASSERT(storagePg != NULL); ASSERT(fileRemove != NULL); ASSERT(job != NULL); // The job was successful if (protocolParallelJobErrorCode(job) == 0) { MEM_CONTEXT_TEMP_BEGIN() { const unsigned int processId = protocolParallelJobProcessId(job); const uint64_t bundleId = varType(protocolParallelJobKey(job)) == varTypeUInt64 ? varUInt64(protocolParallelJobKey(job)) : 0; PackRead *const jobResult = protocolParallelJobResult(job); unsigned int percentComplete = 0; while (!pckReadNullP(jobResult)) { ManifestFile file = manifestFileFind(manifest, pckReadStrP(jobResult)); const BackupCopyResult copyResult = (BackupCopyResult)pckReadU32P(jobResult); const bool repoInvalid = pckReadBoolP(jobResult); const uint64_t copySize = pckReadU64P(jobResult); const uint64_t bundleOffset = pckReadU64P(jobResult); const uint64_t blockIncrMapSize = pckReadU64P(jobResult); const uint64_t repoSize = pckReadU64P(jobResult); const Buffer *const copyChecksum = pckReadBinP(jobResult); const Buffer *const repoChecksum = pckReadBinP(jobResult); PackRead *const checksumPageResult = pckReadPackReadP(jobResult); // Increment backup copy progress. Use the original size since the size may have changed during the copy but for the // purpose of reporting progress we need to increment by the original size used to generate the total size. *sizeProgress += file.sizeOriginal; // Create log file name const String *const fileName = storagePathP(storagePg, manifestPathPg(file.name)); const String *const fileLog = host == NULL ? fileName : strNewFmt("%s:%s", strZ(host), strZ(fileName)); // Format log progress String *const logProgress = strNew(); if (bundleId != 0 && copyResult != backupCopyResultNoOp && copyResult != backupCopyResultTruncate) strCatFmt(logProgress, "bundle %" PRIu64 "/%" PRIu64 ", ", bundleId, bundleOffset); // Log original manifest size if copy size differs if (copySize != file.sizeOriginal) strCatFmt(logProgress, "%s->", strZ(strSizeFormat(file.sizeOriginal))); // Store percentComplete as an integer (used to update progress in the lock file) percentComplete = cvtPctToUInt(*sizeProgress, sizeTotal); strCatFmt(logProgress, "%s, %s", strZ(strSizeFormat(copySize)), strZ(strNewPct(*sizeProgress, sizeTotal))); // Format log checksum const String *const logChecksum = copySize != 0 ? strNewFmt(" checksum %s", strZ(strNewEncode(encodingHex, copyChecksum))) : EMPTY_STR; // If the file is in a prior backup and nothing changed, just log it if (copyResult == backupCopyResultNoOp) { LOG_DETAIL_PID_FMT( processId, "match file from prior backup %s (%s)%s", strZ(fileLog), strZ(logProgress), strZ(logChecksum)); } // Else if the repo matched the expect checksum, just log it else if (copyResult == backupCopyResultChecksum) { LOG_DETAIL_PID_FMT( processId, "checksum resumed file %s (%s)%s", strZ(fileLog), strZ(logProgress), strZ(logChecksum)); } // Else if the file was removed during backup add it to the list of files to be removed from the manifest when the // backup is complete. It can't be removed right now because that will invalidate the pointers that are being used // for processing. else if (copyResult == backupCopyResultSkip) { LOG_DETAIL_PID_FMT(processId, "skip file removed by database %s", strZ(fileLog)); strLstAdd(fileRemove, file.name); } // Else file was copied so update manifest else { LOG_DETAIL_PID_FMT( processId, "%s file %s (%s)%s", copyResult != backupCopyResultTruncate ? "backup" : "store truncated", strZ(fileLog), strZ(logProgress), strZ(logChecksum)); // If the repo file was invalid warn that there might be corruption in the repository if (repoInvalid) { ASSERT(copyResult == backupCopyResultCopy); LOG_WARN_FMT( "resumed backup file %s did not have expected checksum %s. The file was recopied and backup will" " continue but this may be an issue unless the resumed backup path in the repository is known to be" " corrupted.\n" "NOTE: this does not indicate a problem with the PostgreSQL page checksums.", strZ(file.name), strZ(strNewEncode(encodingHex, BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE)))); } // If the file had page checksums calculated during the copy ASSERT((!file.checksumPage && checksumPageResult == NULL) || (file.checksumPage && checksumPageResult != NULL)); bool checksumPageError = false; const VariantList *checksumPageErrorList = NULL; if (checksumPageResult != NULL) { checksumPageErrorList = backupJobResultPageChecksum(checksumPageResult); // If the checksum was valid if (!pckReadBoolP(checksumPageResult)) { checksumPageError = true; String *const message = strNew(); if (!pckReadBoolP(checksumPageResult)) { checksumPageErrorList = NULL; strCatFmt( message, "page misalignment in file %s: file size %" PRIu64 " is not divisible by page size %u", strZ(fileLog), copySize, pageSize); } else { // Format the page checksum errors CHECK(FormatError, checksumPageErrorList != NULL, "page checksum error list is missing"); CHECK(FormatError, !varLstEmpty(checksumPageErrorList), "page checksum error list is empty"); String *const error = strNew(); unsigned int errorTotalMin = 0; for (unsigned int errorIdx = 0; errorIdx < varLstSize(checksumPageErrorList); errorIdx++) { const Variant *const errorItem = varLstGet(checksumPageErrorList, errorIdx); // Add a comma if this is not the first item if (errorIdx != 0) strCatZ(error, ", "); // If an error range if (varType(errorItem) == varTypeVariantList) { const VariantList *const errorItemList = varVarLst(errorItem); ASSERT(varLstSize(errorItemList) == 2); strCatFmt( error, "%" PRIu64 "-%" PRIu64, varUInt64(varLstGet(errorItemList, 0)), varUInt64(varLstGet(errorItemList, 1))); errorTotalMin += 2; } // Else a single error else { ASSERT(varType(errorItem) == varTypeUInt64); strCatFmt(error, "%" PRIu64, varUInt64(errorItem)); errorTotalMin++; } } // Make message plural when appropriate const char *const plural = errorTotalMin > 1 ? "s" : ""; strCatFmt( message, "invalid page checksum%s found in file %s at page%s %s", plural, strZ(fileLog), plural, strZ(error)); } if (cfgOptionBool(cfgOptChecksumPageError)) THROW(ChecksumError, strZ(message)); LOG_WARN(strZ(message)); (*warningTotal)++; } } // Update file info and remove any reference to the file's existence in a prior backup file.size = copySize; file.sizeRepo = repoSize; file.checksumSha1 = bufPtrConst(copyChecksum); file.checksumRepoSha1 = repoChecksum != NULL ? bufPtrConst(repoChecksum) : NULL; file.reference = NULL; file.checksumPageError = checksumPageError; file.checksumPageErrorList = checksumPageErrorList != NULL ? jsonFromVar(varNewVarLst(checksumPageErrorList)) : NULL; // Truncated file is not put in bundle file.bundleId = copyResult != backupCopyResultTruncate ? bundleId : 0; file.bundleOffset = bundleOffset; file.blockIncrMapSize = blockIncrMapSize; manifestFileUpdate(manifest, &file); } } // Update currentPercentComplete and lock file when the change is significant enough if (percentComplete - *currentPercentComplete > 10) { *currentPercentComplete = percentComplete; cmdLockWriteP( .percentComplete = VARUINT(*currentPercentComplete), .sizeComplete = VARUINT64(*sizeProgress), .size = VARUINT64(sizeTotal)); } } MEM_CONTEXT_TEMP_END(); // Free the job protocolParallelJobFree(job); } // Else the job errored else THROW_CODE(protocolParallelJobErrorCode(job), strZ(protocolParallelJobErrorMessage(job))); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Build the processing queues and generate jobs ***********************************************************************************************************************************/ typedef struct BackupJobData { const Manifest *const manifest; // Backup manifest const String *const backupLabel; // Backup label (defines the backup path) const bool backupStandby; // Backup from standby RegExp *standbyExp; // Identify files that may be copied from the standby const CipherSpec *const cipherSpecBackup; // Cipher spec used to encrypt files in the backup const PgPageSize pageSize; // Page size const CompressType compressType; // Backup compression type const int compressLevel; // Compress level if backup is compressed const bool delta; // Is this a checksum delta backup? const bool bundle; // Bundle files? uint64_t bundleSize; // Target bundle size uint64_t bundleLimit; // Limit on files to bundle uint64_t bundleId; // Bundle id const bool blockIncr; // Block incremental? size_t blockIncrSizeSuper; // Super block size List *queueList; // List of processing queues } BackupJobData; // Identify files that must be copied from the primary static bool backupProcessFilePrimary(RegExp *const standbyExp, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(REGEXP, standbyExp); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(standbyExp != NULL); ASSERT(name != NULL); FUNCTION_TEST_RETURN( BOOL, strEqZ(name, MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL) || !regExpMatch(standbyExp, name)); } // Comparator to order files in a backup processing queue. Unbundled files (too large to bundle) sort first, largest first, so the // longest copies start early and processes are less likely to stall at the end of the backup. Bundled files follow, ordered to // mirror the layout of the prior backups they reference so files keep the same relative order from one backup to the next. That // lets block incremental map reads (and reads of whole files in prior bundles) run in sequence rather than seeking back and forth. // // This is the counterpart to backupFileComparator(), which applies the same prior-backup locality within a bundle once the files // reach the remote process. See that comparator's header for the full rationale. static const Manifest *backupProcessQueueComparatorManifest = NULL; static bool backupProcessQueueComparatorBundle; static uint64_t backupProcessQueueComparatorBundleLimit; static int backupProcessQueueComparator(const void *const item1, const void *const item2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, item1); FUNCTION_TEST_PARAM_P(VOID, item2); FUNCTION_TEST_END(); ASSERT(item1 != NULL); ASSERT(item2 != NULL); // Unpack files const ManifestFile file1 = manifestFileUnpack(backupProcessQueueComparatorManifest, *(const ManifestFilePack *const *)item1); const ManifestFile file2 = manifestFileUnpack(backupProcessQueueComparatorManifest, *(const ManifestFilePack *const *)item2); // A file is bundled when bundling is enabled and it is within the bundle limit; larger files are stored unbundled. Use the // original size since that is how many bytes will be copied -- size may have been set to the prior size during manifest build. const bool file1Bundle = backupProcessQueueComparatorBundle && file1.sizeOriginal <= backupProcessQueueComparatorBundleLimit; const bool file2Bundle = backupProcessQueueComparatorBundle && file2.sizeOriginal <= backupProcessQueueComparatorBundleLimit; // Order unbundled files (each copied as its own job) before bundled files if (file1Bundle != file2Bundle) FUNCTION_TEST_RETURN(INT, file1Bundle ? 1 : -1); // Order unbundled files by size descending so larger files are processed first. This helps prevent stalls at the end of the // backup where one process is still copying a large file after the others are idle. This only matters when process-max > 1. if (!file1Bundle) { if (file1.sizeOriginal < file2.sizeOriginal) FUNCTION_TEST_RETURN(INT, 1); else if (file1.sizeOriginal > file2.sizeOriginal) FUNCTION_TEST_RETURN(INT, -1); // If size is equal then use name to generate a deterministic ordering (names are unique) FUNCTION_TEST_RETURN(INT, strCmp(file1.name, file2.name)); } // The rest are bundled files. Order first by reference (the prior backup the file is based on) so files based on the same prior // backup are grouped and processed in the order those backups were created. New files have no reference and sort last. if (file1.reference == NULL) { if (file2.reference != NULL) FUNCTION_TEST_RETURN(INT, 1); } else if (file2.reference == NULL) FUNCTION_TEST_RETURN(INT, -1); else { const int backupLabelCmp = strCmp(file1.reference, file2.reference); if (backupLabelCmp != 0) FUNCTION_TEST_RETURN(INT, backupLabelCmp); // Then by bundle id so files that shared a bundle in the prior backup stay grouped. Bundle id and offset only apply to // files based on the same prior backup so they are ordered here rather than for new files, which have no prior location. if (file1.bundleId < file2.bundleId) FUNCTION_TEST_RETURN(INT, -1); else if (file1.bundleId > file2.bundleId) FUNCTION_TEST_RETURN(INT, 1); // Then by offset within that bundle so prior maps are read in the order they are stored if (file1.bundleOffset < file2.bundleOffset) FUNCTION_TEST_RETURN(INT, -1); else if (file1.bundleOffset > file2.bundleOffset) FUNCTION_TEST_RETURN(INT, 1); } // Order new files eligible for bundling by time ascending so files of a similar age are bundled together. A full backup has no // references so all its bundled files use this ordering. if (file1.timestamp < file2.timestamp) FUNCTION_TEST_RETURN(INT, -1); else if (file1.timestamp > file2.timestamp) FUNCTION_TEST_RETURN(INT, 1); // If everything above is equal then use name to generate a deterministic ordering (names must be unique) FUNCTION_TEST_RETURN(INT, strCmp(file1.name, file2.name)); } // Helper to generate the backup queues static uint64_t backupProcessQueue(const BackupData *const backupData, Manifest *const manifest, BackupJobData *const jobData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM_P(VOID, jobData); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(manifest != NULL); uint64_t result = 0; MEM_CONTEXT_TEMP_BEGIN() { // Create list of process queues (use void * instead of List * to avoid Coverity false positive) jobData->queueList = lstNewP(sizeof(void *)); // Generate the list of targets StringList *const targetList = strLstNew(); strLstAddZ(targetList, MANIFEST_TARGET_PGDATA "/"); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); if (target->tablespaceId != 0) strLstAddFmt(targetList, "%s/", strZ(target->name)); } // Generate the processing queues (there is always at least one) const unsigned int queueOffset = jobData->backupStandby ? 1 : 0; MEM_CONTEXT_BEGIN(lstMemContext(jobData->queueList)) { for (unsigned int queueIdx = 0; queueIdx < strLstSize(targetList) + queueOffset; queueIdx++) { List *const queue = lstNewP(sizeof(ManifestFile *), .comparator = backupProcessQueueComparator); lstAdd(jobData->queueList, &queue); } } MEM_CONTEXT_END(); // Now put all files into the processing queues uint64_t fileTotal = 0; bool pgControlFound = false; for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFilePack *const filePack = manifestFilePackGet(manifest, fileIdx); const ManifestFile file = manifestFileUnpack(manifest, filePack); // Only process files that need to be copied if (!file.copy) { // If bundling log zero-length files as stored since they will never be copied if (file.size == 0 && jobData->bundle) { LOG_DETAIL_FMT( "store zero-length file %s", strZ(storagePathP(backupData->storagePrimary, manifestPathPg(file.name)))); } continue; } // Is pg_control in the backup? if (strEq(file.name, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL))) pgControlFound = true; // Files that must be copied from the primary are always put in queue 0 when backup from standby if (jobData->backupStandby && backupProcessFilePrimary(jobData->standbyExp, file.name)) { lstAdd(*(List **)lstGet(jobData->queueList, 0), &filePack); } // Else find the correct queue by matching the file to a target else { // Find the target that contains this file unsigned int targetIdx = 0; do { CHECK(AssertError, targetIdx < strLstSize(targetList), "backup target not found"); if (strBeginsWith(file.name, strLstGet(targetList, targetIdx))) break; targetIdx++; } while (1); // Add file to queue lstAdd(*(List **)lstGet(jobData->queueList, targetIdx + queueOffset), &filePack); } // Add size to total result += file.sizeOriginal; // Increment total files fileTotal++; } // pg_control should always be in an online backup if (!pgControlFound && cfgOptionBool(cfgOptOnline)) { THROW( FileMissingError, PG_FILE_PGCONTROL " must be present in all online backups\n" "HINT: is something wrong with the clock or filesystem timestamps?"); } // If there are no files to backup then we'll exit with an error. This could happen if the database is down and backup is // called with --no-online twice in a row. if (fileTotal == 0) THROW(FileMissingError, "no files have changed since the last backup - this seems unlikely"); // Sort the queues backupProcessQueueComparatorManifest = manifest; backupProcessQueueComparatorBundle = jobData->bundle; backupProcessQueueComparatorBundleLimit = jobData->bundleLimit; for (unsigned int queueIdx = 0; queueIdx < lstSize(jobData->queueList); queueIdx++) lstSort(*(List **)lstGet(jobData->queueList, queueIdx), sortOrderAsc); // Move process queues to prior context lstMove(jobData->queueList, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT64, result); } // Helper to calculate the next queue to scan based on the client index static int backupJobQueueNext(const unsigned int clientIdx, int queueIdx, const unsigned int queueTotal) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, clientIdx); FUNCTION_TEST_PARAM(INT, queueIdx); FUNCTION_TEST_PARAM(UINT, queueTotal); FUNCTION_TEST_END(); // Move (forward or back) to the next queue queueIdx += clientIdx % 2 ? -1 : 1; // Deal with wrapping on either end if (queueIdx < 0) FUNCTION_TEST_RETURN(INT, (int)queueTotal - 1); else if (queueIdx == (int)queueTotal) FUNCTION_TEST_RETURN(INT, 0); FUNCTION_TEST_RETURN(INT, queueIdx); } // Callback to fetch backup jobs for the parallel executor static ProtocolParallelJob * backupJobCallback(void *const data, const unsigned int clientIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(UINT, clientIdx); FUNCTION_TEST_END(); ASSERT(data != NULL); ProtocolParallelJob *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get a new job if there are any left BackupJobData *const jobData = data; // Determine where to begin scanning the queue (we'll stop when we get back here). When copying from the primary during // backup from standby only queue 0 will be used. const unsigned int queueOffset = jobData->backupStandby && clientIdx > 0 ? 1 : 0; int queueIdx = jobData->backupStandby && clientIdx == 0 ? 0 : (int)(clientIdx % (lstSize(jobData->queueList) - queueOffset)); const int queueEnd = queueIdx; // Create backup job PackWrite *param = NULL; uint64_t fileTotal = 0; uint64_t fileSize = 0; do { List *const queue = *(List **)lstGet(jobData->queueList, (unsigned int)queueIdx + queueOffset); unsigned int fileIdx = 0; bool bundle = jobData->bundle; const String *fileName = NULL; while (fileIdx < lstSize(queue)) { const ManifestFile file = manifestFileUnpack(jobData->manifest, *(ManifestFilePack **)lstGet(queue, fileIdx)); // Is this file a block incremental? const bool blockIncr = jobData->blockIncr && file.blockIncrSize > 0; // Add common parameters before first file if (param == NULL) { param = protocolPackNew(); if (bundle && file.sizeOriginal <= jobData->bundleLimit) { pckWriteStrP(param, backupFileRepoPathP(jobData->backupLabel, .bundleId = jobData->bundleId)); pckWriteU64P(param, jobData->bundleId); pckWriteBoolP(param, manifestData(jobData->manifest)->bundleRaw); } else { CHECK(AssertError, fileTotal == 0, "cannot bundle file"); pckWriteStrP( param, backupFileRepoPathP( jobData->backupLabel, .manifestName = file.name, .compressType = jobData->compressType, .blockIncr = blockIncr)); pckWriteU64P(param, 0); fileName = file.name; bundle = false; } // Provide the backup reference pckWriteU64P(param, strLstSize(manifestReferenceList(jobData->manifest)) - 1); pckWriteU32P(param, jobData->compressType); pckWriteI32P(param, jobData->compressLevel); cipherSpecPack(param, jobData->cipherSpecBackup); pckWriteU32P(param, jobData->pageSize); pckWriteStrP(param, cfgOptionStrNull(cfgOptPgVersionForce)); } pckWriteStrP(param, manifestPathPg(file.name)); pckWriteBoolP(param, file.delta); pckWriteBoolP(param, !strEq(file.name, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL))); pckWriteU64P(param, file.size); pckWriteU64P(param, file.sizeOriginal); pckWriteBoolP(param, !backupProcessFilePrimary(jobData->standbyExp, file.name)); pckWriteBinP(param, file.checksumSha1 != NULL ? BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE) : NULL); pckWriteBoolP(param, file.checksumPage); pckWriteBoolP(param, cfgOptionBool(cfgOptPageHeaderCheck)); // If block incremental then provide the location of the prior map when available if (blockIncr) { pckWriteU64P(param, file.blockIncrSize); pckWriteU64P(param, file.blockIncrChecksumSize); pckWriteU64P(param, jobData->blockIncrSizeSuper); if (file.blockIncrMapSize != 0 && !file.resume) { pckWriteStrP( param, backupFileRepoPathP( file.reference, .manifestName = file.name, .bundleId = file.bundleId, .blockIncr = true)); pckWriteU64P(param, file.bundleOffset + file.sizeRepo - file.blockIncrMapSize); pckWriteU64P(param, file.blockIncrMapSize); } else pckWriteNullP(param); } else pckWriteU64P(param, 0); pckWriteStrP(param, file.name); pckWriteBinP(param, file.checksumRepoSha1 != NULL ? BUF(file.checksumRepoSha1, HASH_TYPE_SHA1_SIZE) : NULL); pckWriteU64P(param, file.sizeRepo); pckWriteBoolP(param, file.resume); pckWriteBoolP(param, file.reference != NULL); fileTotal++; fileSize += file.sizeOriginal; // Remove job from the queue lstRemoveIdx(queue, fileIdx); // Break if not bundling or bundle size has been reached if (!bundle || fileSize >= jobData->bundleSize) break; } if (fileTotal > 0) { // Assign job to result MEM_CONTEXT_PRIOR_BEGIN() { result = protocolParallelJobNew( bundle ? VARUINT64(jobData->bundleId) : VARSTR(fileName), PROTOCOL_COMMAND_BACKUP_FILE, param); if (bundle) jobData->bundleId++; } MEM_CONTEXT_PRIOR_END(); break; } // Don't get next queue when copying from primary during backup from standby since the primary only has one queue if (!jobData->backupStandby || clientIdx > 0) queueIdx = backupJobQueueNext(clientIdx, queueIdx, lstSize(jobData->queueList) - queueOffset); } while (queueIdx != queueEnd); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } /*********************************************************************************************************************************** Process the backup manifest ***********************************************************************************************************************************/ static unsigned int backupProcess(const BackupData *const backupData, Manifest *const manifest, const CipherSpec *const cipherSpecManifest) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpecManifest); FUNCTION_LOG_END(); ASSERT(manifest != NULL); uint64_t sizeTotal = 0; unsigned int warningTotal = 0; MEM_CONTEXT_TEMP_BEGIN() { // Get backup info const BackupType backupType = manifestData(manifest)->backupType; const String *const backupLabel = manifestData(manifest)->backupLabel; const String *const backupPathExp = strNewFmt(STORAGE_REPO_BACKUP "/%s", strZ(backupLabel)); const bool hardLink = cfgOptionBool(cfgOptRepoHardlink) && storageFeature(storageRepoWrite(), storageFeatureHardLink); const bool backupStandby = backupData->dbStandby != NULL; BackupJobData jobData = { .manifest = manifest, .backupLabel = backupLabel, .backupStandby = backupData->dbStandby != NULL, .compressType = compressTypeEnum(cfgOptionStrId(cfgOptCompressType)), .compressLevel = cfgOptionInt(cfgOptCompressLevel), .cipherSpecBackup = manifestCipherSpec(manifest), .pageSize = backupData->pageSize, .delta = cfgOptionBool(cfgOptDelta), .bundle = cfgOptionBool(cfgOptRepoBundle), .bundleId = 1, .blockIncr = cfgOptionBool(cfgOptRepoBlock), // Build expression to identify files that can be copied from the standby when standby backup is supported .standbyExp = regExpNew( strNewFmt( "^((" MANIFEST_TARGET_PGDATA "/(" PG_PATH_BASE "|" PG_PATH_GLOBAL "|%s|" PG_PATH_PGMULTIXACT "))" "|" MANIFEST_TARGET_PGTBLSPC ")/", strZ(pgXactPath(backupData->version)))), }; if (jobData.bundle) { jobData.bundleSize = cfgOptionUInt64(cfgOptRepoBundleSize); jobData.bundleLimit = cfgOptionUInt64(cfgOptRepoBundleLimit); } if (jobData.blockIncr) { // Set super block size based on the backup type jobData.blockIncrSizeSuper = backupType == backupTypeFull ? (size_t)cfgOptionUInt64(cfgOptRepoBlockSizeSuperFull) : (size_t)cfgOptionUInt64(cfgOptRepoBlockSizeSuper); } // If this is a full backup or hard-linked and paths are supported then create all paths explicitly so that empty paths will // exist in the repo. Also create tablespace symlinks when symlinks are available. This makes it possible for the user to // make a copy of the backup path and get a valid cluster. if ((backupType == backupTypeFull && !jobData.bundle) || hardLink) { // Create paths when available if (storageFeature(storageRepoWrite(), storageFeaturePath)) { for (unsigned int pathIdx = 0; pathIdx < manifestPathTotal(manifest); pathIdx++) { storagePathCreateP( storageRepoWrite(), strNewFmt("%s/%s", strZ(backupPathExp), strZ(manifestPath(manifest, pathIdx)->name))); } } // Create tablespace symlinks when available if (storageFeature(storageRepoWrite(), storageFeatureSymLink)) { for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); if (target->tablespaceId != 0) { const String *const link = storagePathP( storageRepo(), strNewFmt("%s/" MANIFEST_TARGET_PGDATA "/%s", strZ(backupPathExp), strZ(target->name))); const String *const linkDestination = strNewFmt( "../../" MANIFEST_TARGET_PGTBLSPC "/%u", target->tablespaceId); storageLinkCreateP(storageRepoWrite(), linkDestination, link); } } } } // Generate processing queues sizeTotal = backupProcessQueue(backupData, manifest, &jobData); // Create the parallel executor ProtocolParallel *const parallelExec = protocolParallelNew( cfgOptionUInt64(cfgOptProtocolTimeout) / 2, backupJobCallback, &jobData); // First client is always on the primary protocolParallelClientAdd(parallelExec, protocolLocalGet(protocolStorageTypePg, backupData->pgIdxPrimary, 1)); // Create the rest of the clients on the primary or standby depending on the value of backup-standby. Note that standby // backups don't count the primary client in process-max. const unsigned int processMax = cfgOptionUInt(cfgOptProcessMax) + (jobData.backupStandby ? 1 : 0); const unsigned int pgIdx = jobData.backupStandby ? backupData->pgIdxStandby : backupData->pgIdxPrimary; for (unsigned int processIdx = 2; processIdx <= processMax; processIdx++) protocolParallelClientAdd(parallelExec, protocolLocalGet(protocolStorageTypePg, pgIdx, processIdx)); // Maintain a list of files that need to be removed from the manifest when the backup is complete StringList *const fileRemove = strLstNew(); // Determine how often the manifest will be saved (every one percent or threshold size, whichever is greater) uint64_t manifestSaveLast = 0; uint64_t manifestSaveSize = sizeTotal / 100; if (manifestSaveSize < cfgOptionUInt64(cfgOptManifestSaveThreshold)) manifestSaveSize = cfgOptionUInt64(cfgOptManifestSaveThreshold); // Process jobs uint64_t sizeProgress = 0; // Initialize percent complete and bytes completed/total unsigned int currentPercentComplete = 0; cmdLockWriteP( .percentComplete = VARUINT(currentPercentComplete), .sizeComplete = VARUINT64(sizeProgress), .size = VARUINT64(sizeTotal)); MEM_CONTEXT_TEMP_RESET_BEGIN() { do { const unsigned int completed = protocolParallelProcess(parallelExec); for (unsigned int jobIdx = 0; jobIdx < completed; jobIdx++) { ProtocolParallelJob *const job = protocolParallelResult(parallelExec); backupJobResult( manifest, backupStandby && protocolParallelJobProcessId(job) > 1 ? backupData->hostStandby : backupData->hostPrimary, protocolParallelJobProcessId(job) > 1 ? storagePgIdx(pgIdx) : backupData->storagePrimary, fileRemove, job, jobData.pageSize, &warningTotal, sizeTotal, &sizeProgress, ¤tPercentComplete); } // A keep-alive is required here for the remote holding open the backup connection protocolKeepAlive(); // Check that the clusters are alive and correctly configured during the backup backupDbPing(backupData, false); // Save the manifest periodically to preserve checksums for resume if (sizeProgress - manifestSaveLast >= manifestSaveSize) { backupManifestSaveCopy(manifest, cipherSpecManifest, false); manifestSaveLast = sizeProgress; } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } while (!protocolParallelDone(parallelExec)); } MEM_CONTEXT_TEMP_END(); #ifdef DEBUG // Ensure that all processing queues are empty for (unsigned int queueIdx = 0; queueIdx < lstSize(jobData.queueList); queueIdx++) ASSERT(lstEmpty(*(List **)lstGet(jobData.queueList, queueIdx))); #endif // Remove files from the manifest that were removed during the backup. This must happen after processing to avoid // invalidating pointers by deleting items from the list. for (unsigned int fileRemoveIdx = 0; fileRemoveIdx < strLstSize(fileRemove); fileRemoveIdx++) manifestFileRemove(manifest, strLstGet(fileRemove, fileRemoveIdx)); // Log references or create hardlinks for all files const char *const compressExt = strZ(compressExtStr(jobData.compressType)); for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFile file = manifestFile(manifest, fileIdx); // If the file has a reference, then it was not copied since it can be retrieved from the referenced backup. However, // if hardlinking is enabled the link will need to be created. if (file.reference != NULL) { // If hardlinking is enabled then create a hardlink for files that have not changed since the last backup if (hardLink) { LOG_DETAIL_FMT("hardlink %s to %s", strZ(file.name), strZ(file.reference)); const String *const linkName = storagePathP( storageRepo(), strNewFmt("%s/%s%s", strZ(backupPathExp), strZ(file.name), compressExt)); const String *const linkDestination = storagePathP( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/%s/%s%s", strZ(file.reference), strZ(file.name), compressExt)); storageLinkCreateP(storageRepoWrite(), linkDestination, linkName, .linkType = storageLinkHard); } // Else log the reference. With delta, it is possible that references may have been removed if a file needed to be // recopied. else LOG_DETAIL_FMT("reference %s to %s", strZ(file.name), strZ(file.reference)); } } // Sync backup paths if required if (storageFeature(storageRepoWrite(), storageFeaturePathSync)) { for (unsigned int pathIdx = 0; pathIdx < manifestPathTotal(manifest); pathIdx++) { const String *const path = strNewFmt("%s/%s", strZ(backupPathExp), strZ(manifestPath(manifest, pathIdx)->name)); // Always sync the path if it exists or if the backup is full (without bundling) or hardlinked. In the latter cases // the directory should always exist so we want to error if it does not. if ((backupType == backupTypeFull && !jobData.bundle) || hardLink || storagePathExistsP(storageRepo(), path)) storagePathSyncP(storageRepoWrite(), path); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, warningTotal); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/protocol.c0000664000175100017510000002234515265101745021067 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Protocol Handler ***********************************************************************************************************************************/ #include #include "command/backup/blockIncr.h" #include "command/backup/pageChecksum.h" #include "command/backup/protocol.h" #include "common/compress/helper.h" #include "common/crypto/cipherBlock.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/bufferWrite.h" #include "common/io/filter/size.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/type/json.h" #include "config/config.h" #include "storage/helper.h" #include "command/backup/file.c.inc" /*********************************************************************************************************************************** Comparator to order the files processed together in a single backup job for efficient copying. Block incremental files sort before whole files, then by the prior backup map file they read and the offset within it. This lets the prior block maps be read from the repo in one combined, in-order pass (see StorageReadMulti) rather than opening and seeking each one separately, and it keeps files in the same relative order from backup to backup so the same read pattern benefits future backups and restore. Whole files sort by size so the smallest sit nearest the block incremental data and benefit from read over. The rationale for each ordering step is in the inline comments below. This is the counterpart to backupProcessQueueComparator(), which applies the same prior-backup locality across the processing queues before jobs are assembled. The two sort different structs (BackupFile vs ManifestFile) at different stages and cannot share code, so a change to one ordering rule should be checked against the other. ***********************************************************************************************************************************/ static int backupFileComparator(const void *const item1, const void *const item2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, item1); FUNCTION_TEST_PARAM_P(VOID, item2); FUNCTION_TEST_END(); ASSERT(item1 != NULL); ASSERT(item2 != NULL); const BackupFile *const file1 = item1; const BackupFile *const file2 = item2; // Order block incremental files before whole files. This produces slightly smaller maps since the offsets are smaller. Also // whole files can have reads combined and read over more often without block maps/lists in between them. const bool file1BlockIncr = file1->blockIncrSize != 0; const bool file2BlockIncr = file2->blockIncrSize != 0; if (file1BlockIncr != file2BlockIncr) FUNCTION_TEST_RETURN(INT, file1BlockIncr ? -1 : 1); // If both files are block incremental order by the prior map file and then the prior map offset. The map file encodes the // reference and prior bundle, so ordering on it groups reads on the same file together (and is exactly what the multi-read uses // to identify a file), while the offset orders the reads within that file so they can be combined and benefit from read over. // Files with no prior map (NULL) sort first and contribute no reads. if (file1BlockIncr) { // Order by prior map file so reads to the same file are grouped const int compare = strCmp(file1->blockIncrMapPriorFile, file2->blockIncrMapPriorFile); if (compare != 0) FUNCTION_TEST_RETURN(INT, compare); // Files sharing a prior map file are ordered by map offset so reads in the repo are ordered and more likely to be combined // and benefit from read over. Distinct files have non-overlapping map regions so their offsets are never equal here. Files // with no prior map have offset 0 and fall through to the size ordering below. if (file1->blockIncrMapPriorFile != NULL) { ASSERT(file1->blockIncrMapPriorOffset != file2->blockIncrMapPriorOffset); FUNCTION_TEST_RETURN(INT, file1->blockIncrMapPriorOffset < file2->blockIncrMapPriorOffset ? -1 : 1); } } // Order by size ascending so the smaller whole files sort nearest the block incremental maps (which will be stored at the end // of the block incremental data in the future). Both tend to be small so keeping them together benefits read over on restore. // Ascending also makes new block incremental maps slightly smaller since the first offset of each reference is stored in full, // so files placed earlier get smaller offsets. if (file1->pgFileSize < file2->pgFileSize) FUNCTION_TEST_RETURN(INT, -1); else if (file1->pgFileSize > file2->pgFileSize) FUNCTION_TEST_RETURN(INT, 1); // If all the above are the same then use name asc to generate a deterministic ordering (names must be unique) ASSERT(!strEq(file1->pgFile, file2->pgFile)); FUNCTION_TEST_RETURN(INT, strCmp(file1->pgFile, file2->pgFile)); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * backupFileProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Backup options that apply to all files const String *const repoFile = pckReadStrP(param); const uint64_t bundleId = pckReadU64P(param); const bool bundleRaw = bundleId != 0 ? pckReadBoolP(param) : false; const unsigned int blockIncrReference = (unsigned int)pckReadU64P(param); const CompressType repoFileCompressType = (CompressType)pckReadU32P(param); const int repoFileCompressLevel = pckReadI32P(param); const CipherSpec *const cipherSpecBackup = cipherSpecNewPack(param); const PgPageSize pageSize = pckReadU32P(param); const String *const pgVersionForce = pckReadStrP(param); // Build the file list List *const fileList = lstNewP(sizeof(BackupFile), .comparator = backupFileComparator); while (!pckReadNullP(param)) { BackupFile file = {.pgFile = pckReadStrP(param)}; file.pgFileDelta = pckReadBoolP(param); file.pgFileIgnoreMissing = pckReadBoolP(param); file.pgFileSize = pckReadU64P(param); file.pgFileSizeOriginal = pckReadU64P(param); file.pgFileCopyExactSize = pckReadBoolP(param); file.pgFileChecksum = pckReadBinP(param); file.pgFileChecksumPage = pckReadBoolP(param); file.pgFilePageHeaderCheck = pckReadBoolP(param); file.blockIncrSize = (size_t)pckReadU64P(param); if (file.blockIncrSize > 0) { file.blockIncrChecksumSize = (size_t)pckReadU64P(param); file.blockIncrSuperSize = pckReadU64P(param); file.blockIncrMapPriorFile = pckReadStrP(param); if (file.blockIncrMapPriorFile != NULL) { file.blockIncrMapPriorOffset = pckReadU64P(param); file.blockIncrMapPriorSize = pckReadU64P(param); } } file.manifestFile = pckReadStrP(param); file.repoFileChecksum = pckReadBinP(param); file.repoFileSize = pckReadU64P(param); file.manifestFileResume = pckReadBoolP(param); file.manifestFileHasReference = pckReadBoolP(param); lstAdd(fileList, &file); } // Sort files for efficient processing lstSort(fileList, sortOrderAsc); // Backup file const List *const resultList = backupFile( repoFile, bundleId, bundleRaw, blockIncrReference, repoFileCompressType, repoFileCompressLevel, cipherSpecBackup, pgVersionForce, pageSize, fileList); // Return result PackWrite *const data = protocolServerResultData(result); for (unsigned int resultIdx = 0; resultIdx < lstSize(resultList); resultIdx++) { const BackupFileResult *const fileResult = lstGet(resultList, resultIdx); ASSERT( fileResult->backupCopyResult == backupCopyResultSkip || fileResult->copySize != 0 || bufEq(fileResult->copyChecksum, HASH_TYPE_SHA1_ZERO_BUF)); pckWriteStrP(data, fileResult->manifestFile); pckWriteU32P(data, fileResult->backupCopyResult); pckWriteBoolP(data, fileResult->repoInvalid); pckWriteU64P(data, fileResult->copySize); pckWriteU64P(data, fileResult->bundleOffset); pckWriteU64P(data, fileResult->blockIncrMapSize); pckWriteU64P(data, fileResult->repoSize); pckWriteBinP(data, fileResult->copyChecksum); pckWriteBinP(data, fileResult->repoChecksum); pckWritePackP(data, fileResult->pageChecksumResult); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/protocol.h0000664000175100017510000000357415265101745021077 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Protocol Handler ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_PROTOCOL_H #define COMMAND_BACKUP_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Backup file types ***********************************************************************************************************************************/ typedef enum { backupCopyResultChecksum, backupCopyResultCopy, backupCopyResultSkip, backupCopyResultNoOp, backupCopyResultTruncate, } BackupCopyResult; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process protocol requests FN_EXTERN ProtocolServerResult *backupFileProtocol(PackRead *param); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_BACKUP_FILE STRID5("bp-f", 0x36e020) #define PROTOCOL_SERVER_HANDLER_BACKUP_LIST \ {.command = PROTOCOL_COMMAND_BACKUP_FILE, .process = backupFileProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/backup/resume.c.inc0000664000175100017510000004332615265101745021300 0ustar00runnerrunner/*********************************************************************************************************************************** Check for a backup that can be resumed and merge into the manifest if found ***********************************************************************************************************************************/ // Helper to clean invalid paths/files/links out of the resumable backup path static void backupResumeClean( StorageIterator *const storageItr, Manifest *const manifest, const Manifest *const manifestResume, const CompressType compressType, const bool delta, const String *const backupParentPath, const String *const manifestParentName) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_ITERATOR, storageItr); // Storage info FUNCTION_LOG_PARAM(MANIFEST, manifest); // New manifest FUNCTION_LOG_PARAM(MANIFEST, manifestResume); // Resumed manifest FUNCTION_LOG_PARAM(ENUM, compressType); // Backup compression type FUNCTION_LOG_PARAM(BOOL, delta); // Is this a delta backup? FUNCTION_LOG_PARAM(STRING, backupParentPath); // Path to the current level of the backup being cleaned FUNCTION_LOG_PARAM(STRING, manifestParentName); // Parent manifest name used to construct manifest name FUNCTION_LOG_END(); ASSERT(storageItr != NULL); ASSERT(manifest != NULL); ASSERT(manifestResume != NULL); ASSERT(backupParentPath != NULL); MEM_CONTEXT_TEMP_RESET_BEGIN() { while (storageItrMore(storageItr)) { const StorageInfo info = storageItrNext(storageItr); // Skip backup.manifest.copy -- it must be preserved to allow resume again if this process throws an error before // writing the manifest for the first time if (manifestParentName == NULL && strEqZ(info.name, BACKUP_MANIFEST_FILE INFO_COPY_EXT)) continue; // Build the name used to lookup files in the manifest const String *manifestName = manifestParentName != NULL ? strNewFmt("%s/%s", strZ(manifestParentName), strZ(info.name)) : info.name; // Build the backup path used to remove files/links/paths that are invalid const String *const backupPath = strNewFmt("%s/%s", strZ(backupParentPath), strZ(info.name)); // Process file types switch (info.type) { // Check paths // ----------------------------------------------------------------------------------------------------------------- case storageTypePath: { // If the path was not found in the new manifest then remove it if (manifestPathFindDefault(manifest, manifestName, NULL) == NULL) { LOG_DETAIL_FMT("remove path '%s' from resumed backup", strZ(storagePathP(storageRepo(), backupPath))); storagePathRemoveP(storageRepoWrite(), backupPath, .recurse = true); } // Else recurse into the path else { backupResumeClean( storageNewItrP(storageRepo(), backupPath, .sortOrder = sortOrderAsc), manifest, manifestResume, compressType, delta, backupPath, manifestName); } break; } // Check files // ----------------------------------------------------------------------------------------------------------------- case storageTypeFile: { // If the file is compressed then strip off the extension before doing the lookup const CompressType fileCompressType = compressTypeFromName(manifestName); if (fileCompressType != compressTypeNone) manifestName = compressExtStrip(manifestName, fileCompressType); // If the file is block incremental then strip off the extension before doing the lookup const bool blockIncr = strEndsWithZ(manifestName, BACKUP_BLOCK_INCR_EXT); if (blockIncr) manifestName = strSubN(manifestName, 0, strSize(manifestName) - (sizeof(BACKUP_BLOCK_INCR_EXT) - 1)); // Check if the file can be resumed or must be removed const char *removeReason = NULL; if (fileCompressType != compressType && !blockIncr) removeReason = "mismatched compression type"; else if (!manifestFileExists(manifest, manifestName)) removeReason = "missing in manifest"; else { ManifestFile file = manifestFileFind(manifest, manifestName); ASSERT(file.reference == NULL); if (!manifestFileExists(manifestResume, manifestName)) removeReason = "missing in resumed manifest"; else { const ManifestFile fileResume = manifestFileFind(manifestResume, manifestName); ASSERT(fileResume.reference == NULL); if (fileResume.checksumSha1 == NULL) removeReason = "no checksum in resumed manifest"; else if (file.size != fileResume.size) removeReason = "mismatched size"; else if (!delta && file.timestamp != fileResume.timestamp) removeReason = "mismatched timestamp"; else if (file.size == 0) // ??? don't resume zero size files because Perl wouldn't -- can be removed after the migration) removeReason = "zero size"; else { ASSERT(file.copy); ASSERT(file.bundleId == 0); ASSERT(file.blockIncrMapSize == 0); file.sizeRepo = fileResume.sizeRepo; file.checksumSha1 = fileResume.checksumSha1; file.checksumRepoSha1 = fileResume.checksumRepoSha1; file.blockIncrSize = fileResume.blockIncrSize; file.blockIncrChecksumSize = fileResume.blockIncrChecksumSize; file.blockIncrMapSize = fileResume.blockIncrMapSize; file.checksumPage = fileResume.checksumPage; file.checksumPageError = fileResume.checksumPageError; file.checksumPageErrorList = fileResume.checksumPageErrorList; file.resume = true; file.delta = delta; manifestFileUpdate(manifest, &file); } } } // Remove the file if it could not be resumed if (removeReason != NULL) { LOG_DETAIL_FMT( "remove file '%s' from resumed backup (%s)", strZ(storagePathP(storageRepo(), backupPath)), removeReason); storageRemoveP(storageRepoWrite(), backupPath); } break; } // Remove links. We could check that the link has not changed and preserve it but it doesn't seem worth the extra // testing. The link will be recreated during the backup if needed. // ----------------------------------------------------------------------------------------------------------------- case storageTypeLink: storageRemoveP(storageRepoWrite(), backupPath); break; // Remove special files // ----------------------------------------------------------------------------------------------------------------- case storageTypeSpecial: LOG_WARN_FMT("remove special file '%s' from resumed backup", strZ(storagePathP(storageRepo(), backupPath))); storageRemoveP(storageRepoWrite(), backupPath); break; } } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } // Helper to find a resumable backup static const Manifest * backupResumeFind(const Manifest *const manifest, const CipherSpec *const cipherSpecManifest) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpecManifest); FUNCTION_LOG_END(); ASSERT(manifest != NULL); Manifest *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Only the last backup can be resumed const StringList *const backupList = strLstSort( storageListP( storageRepo(), STRDEF(STORAGE_REPO_BACKUP), .expression = backupRegExpP(.full = true, .differential = true, .incremental = true)), sortOrderDesc); if (!strLstEmpty(backupList)) { const String *const backupLabel = strLstGet(backupList, 0); const String *const manifestFile = strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)); // Resumable backups do not have backup.manifest if (!storageExistsP(storageRepo(), manifestFile)) { const bool resume = cfgOptionBool(cfgOptResume) && cfgOptionStrId(cfgOptType) == backupTypeFull; bool usable = false; const char *reason = "resume only valid for full backup"; Manifest *manifestResume = NULL; if (cfgOptionStrId(cfgOptType) == backupTypeFull) { reason = "partially deleted by prior resume or invalid"; // Resumable backups must have backup.manifest.copy if (storageExistsP(storageRepo(), strNewFmt("%s" INFO_COPY_EXT, strZ(manifestFile)))) { reason = "resume is disabled"; // Attempt to read the manifest file in the resumable backup to see if it can be used. If any error at all // occurs then the backup will be considered unusable and a resume will not be attempted. if (resume) { TRY_BEGIN() { manifestResume = manifestLoadFile(storageRepo(), manifestFile, cipherSpecManifest); } CATCH_ANY() { reason = zNewFmt("unable to read %s" INFO_COPY_EXT, strZ(manifestFile)); } TRY_END(); if (manifestResume != NULL) { const ManifestData *manifestResumeData = manifestData(manifestResume); // Check pgBackRest version. This allows the resume implementation to be changed with each version // of pgBackRest at the expense of users losing a resumable back after an upgrade, which seems worth // the cost. if (!strEq(manifestResumeData->backrestVersion, manifestData(manifest)->backrestVersion)) { reason = zNewFmt( "new " PROJECT_NAME " version '%s' does not match resumable " PROJECT_NAME " version '%s'", strZ(manifestData(manifest)->backrestVersion), strZ(manifestResumeData->backrestVersion)); } // Check repository format. The format applies to an entire backup, so files written at another // format cannot be reused no matter how little else has changed. else if (manifestFormat(manifestResume) != manifestFormat(manifest)) { reason = zNewFmt( "new repository format %u does not match resumable repository format %u", manifestFormat(manifest), manifestFormat(manifestResume)); } // Check backup type because new backup label must be the same type as resume backup label else if (manifestResumeData->backupType != cfgOptionStrId(cfgOptType)) { reason = zNewFmt( "new backup type '%s' does not match resumable backup type '%s'", strZ(cfgOptionDisplay(cfgOptType)), zNewStrId(manifestResumeData->backupType)); } // Check prior backup label ??? Do we really care about the prior backup label? else if (!strEq(manifestResumeData->backupLabelPrior, manifestData(manifest)->backupLabelPrior)) { reason = zNewFmt( "new prior backup label '%s' does not match resumable prior backup label '%s'", manifestResumeData->backupLabelPrior ? strZ(manifestResumeData->backupLabelPrior) : "", manifestData(manifest)->backupLabelPrior ? strZ(manifestData(manifest)->backupLabelPrior) : ""); } // Check compression. Compression can't be changed between backups so resume won't work either. else if ( manifestResumeData->backupOptionCompressType != compressTypeEnum(cfgOptionStrId(cfgOptCompressType))) { reason = zNewFmt( "new compression '%s' does not match resumable compression '%s'", strZ(cfgOptionDisplay(cfgOptCompressType)), strZ(compressTypeStr(manifestResumeData->backupOptionCompressType))); } else usable = true; } } } } // If the backup is usable then return the manifest if (usable) { result = manifestMove(manifestResume, memContextPrior()); } // Else warn and remove the unusable backup else { LOG_FMT( resume ? logLevelWarn : logLevelInfo, 0, "backup '%s' cannot be resumed: %s", strZ(backupLabel), reason); storagePathRemoveP( storageRepoWrite(), strNewFmt(STORAGE_REPO_BACKUP "/%s", strZ(backupLabel)), .recurse = true); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(MANIFEST, result); } static bool backupResume(Manifest *const manifest, const CipherSpec *const cipherSpecManifest) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpecManifest); FUNCTION_LOG_END(); ASSERT(manifest != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { const Manifest *const manifestResume = backupResumeFind(manifest, cipherSpecManifest); // If a resumable backup was found set the label and cipher subpass if (manifestResume) { // Resuming result = true; // Set the backup label to the resumed backup manifestBackupLabelSet(manifest, manifestData(manifestResume)->backupLabel); LOG_WARN_FMT( "resumable backup %s of same type exists -- invalid files will be removed then the backup will resume", strZ(manifestData(manifest)->backupLabel)); // Copy cipher subpass since it was used to encrypt the resumable files manifestCipherSpecSet(manifest, manifestCipherSpec(manifestResume)); // Clean resumed backup const String *const backupPath = strNewFmt(STORAGE_REPO_BACKUP "/%s", strZ(manifestData(manifest)->backupLabel)); backupResumeClean( storageNewItrP(storageRepo(), backupPath, .sortOrder = sortOrderAsc), manifest, manifestResume, compressTypeEnum(cfgOptionStrId(cfgOptCompressType)), cfgOptionBool(cfgOptDelta), backupPath, NULL); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/check/0000775000175100017510000000000015265101745016664 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/check/check.c0000664000175100017510000002231315265101745020106 0ustar00runnerrunner/*********************************************************************************************************************************** Check Command ***********************************************************************************************************************************/ #include #include #include "command/archive/find.h" #include "command/check/check.h" #include "command/check/common.h" #include "command/check/report.h" #include "common/debug.h" #include "common/io/fdWrite.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "config/load.h" #include "config/parse.h" #include "db/helper.h" #include "info/infoArchive.h" #include "postgres/interface.h" #include "protocol/helper.h" #include "storage/helper.h" /*********************************************************************************************************************************** Helper functions (to assist with testing) ***********************************************************************************************************************************/ static unsigned int checkManifest(void) { FUNCTION_LOG_VOID(logLevelDebug); // Return the actual number of pg* defined unsigned int result = 0; MEM_CONTEXT_TEMP_BEGIN() { // Loop through all defined databases and attempt to build a manifest for (unsigned int pgIdx = 0; pgIdx < cfgOptionGroupIdxTotal(cfgOptGrpPg); pgIdx++) { result++; // ??? Placeholder for manifest build storageListP(storagePgIdx(pgIdx), cfgOptionIdxStr(cfgOptPgPath, pgIdx)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } static void checkStandby(const DbGetResult dbGroup, const unsigned int pgPathDefinedTotal) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB_GET_RESULT, dbGroup); FUNCTION_LOG_PARAM(UINT, pgPathDefinedTotal); FUNCTION_LOG_END(); // If a standby is defined, check the configuration if (dbGroup.standby != NULL) { // If primary was not found (only have 1 pg configured locally, and we want to still run because this is a standby) if (dbGroup.primary == NULL) { // If any repo is local or more than one pg-path is found then a primary should have been found so error bool error = pgPathDefinedTotal > 1; unsigned int repoIdx = 0; while (!error && repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo)) { if (repoIsLocal(repoIdx)) error = true; repoIdx++; } if (error) THROW(ConfigError, "primary database not found\nHINT: check indexed pg-path/pg-host configurations"); } // Check the user configured path and version against the database checkDbConfig(dbPgControl(dbGroup.standby).version, dbGroup.standbyIdx, dbGroup.standby, true); // Check each repository configured for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { LOG_INFO_FMT(CFGCMD_CHECK " %s (standby)", cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); // Get the repo storage in case it is remote and encryption settings need to be pulled down (performed here for testing) const Storage *const storageRepo = storageRepoIdx(repoIdx); // Check that the backup and archive info files exist and are valid for the current database of the stanza checkStanzaInfoPg( repoIdx, storageRepo, dbPgControl(dbGroup.standby).version, dbPgControl(dbGroup.standby).systemId, cfgCipherSpecMainIdx(repoIdx)); } LOG_INFO("switch wal not performed because this is a standby"); // Free the standby connection dbFree(dbGroup.standby); } // If backup from standby is true then warn when a standby not found else if (cfgOptionSeq(cfgOptBackupStandby) != CFGOPTVAL_BACKUP_STANDBY_N) { LOG_WARN("option '" CFGOPT_BACKUP_STANDBY "' is enabled but standby is not properly configured"); } FUNCTION_LOG_RETURN_VOID(); } static void checkPrimary(const DbGetResult dbGroup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB_GET_RESULT, dbGroup); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); // If a primary is defined, check the configuration and perform a WAL switch and make sure the WAL is archived if (dbGroup.primary != NULL) { // Check the user configured path and version against the database checkDbConfig(dbPgControl(dbGroup.primary).version, dbGroup.primaryIdx, dbGroup.primary, false); // Check configuration of each repo const String **const repoArchiveId = memNew(sizeof(String *) * cfgOptionGroupIdxTotal(cfgOptGrpRepo)); for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { LOG_INFO_FMT(CFGCMD_CHECK " %s configuration (primary)", cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); // Get the repo storage in case it is remote and encryption settings need to be pulled down (performed here for testing) const Storage *const storageRepo = storageRepoIdx(repoIdx); // Check that the backup and archive info files exist and are valid for the current database of the stanza checkStanzaInfoPg( repoIdx, storageRepo, dbPgControl(dbGroup.primary).version, dbPgControl(dbGroup.primary).systemId, cfgCipherSpecMainIdx(repoIdx)); // Attempt to load the archive info file and retrieve the archiveId const InfoArchive *const archiveInfo = infoArchiveLoadFile( storageRepo, INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); repoArchiveId[repoIdx] = infoArchiveId(archiveInfo); } // Perform a WAL switch const String *const walSegment = dbWalSwitch(dbGroup.primary); // Wait for the WAL to appear in each repo for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { LOG_INFO_FMT(CFGCMD_CHECK " %s archive for WAL (primary)", cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); const String *const walSegmentFile = walSegmentFindOne( storageRepoIdx(repoIdx), repoArchiveId[repoIdx], walSegment, cfgOptionUInt64(cfgOptArchiveTimeout)); LOG_INFO_FMT( "WAL segment %s successfully archived to '%s' on %s", strZ(walSegment), strZ( storagePathP( storageRepoIdx(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(repoArchiveId[repoIdx]), strZ(walSegmentFile)))), cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); } dbFree(dbGroup.primary); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cmdCheck(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { if (cfgOptionBool(cfgOptReport)) ioFdWriteOneStr(STDOUT_FILENO, checkReport()); else { // Build stanza list based on whether a stanza was specified or not StringList *stanzaList; bool stanzaSpecified = cfgOptionTest(cfgOptStanza); if (stanzaSpecified) { stanzaList = strLstNew(); strLstAdd(stanzaList, cfgOptionStr(cfgOptStanza)); } else { stanzaList = cfgParseStanzaList(); if (strLstSize(stanzaList) == 0) { LOG_WARN( "no stanzas found to check\n" "HINT: are there non-empty stanza sections in the configuration?"); } } // Iterate stanzas for (unsigned int stanzaIdx = 0; stanzaIdx < strLstSize(stanzaList); stanzaIdx++) { // Switch stanza if required if (!stanzaSpecified) { const String *const stanza = strLstGet(stanzaList, stanzaIdx); LOG_INFO_FMT("check stanza '%s'", strZ(stanza)); // Free storage and protocol cache storageHelperFree(); protocolFree(); // Reload config with new stanza cfgLoadStanza(stanza); } // Get the primary/standby connections (standby is only required if backup from standby is enabled) DbGetResult dbGroup = dbGet(false, false, CFGOPTVAL_BACKUP_STANDBY_N); if (dbGroup.standby == NULL && dbGroup.primary == NULL) THROW(ConfigError, "no database found\nHINT: check indexed pg-path/pg-host configurations"); const unsigned int pgPathDefinedTotal = checkManifest(); checkStandby(dbGroup, pgPathDefinedTotal); checkPrimary(dbGroup); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/check/check.h0000664000175100017510000000125315265101745020113 0ustar00runnerrunner/*********************************************************************************************************************************** Check Command ***********************************************************************************************************************************/ #ifndef COMMAND_CHECK_CHECK_H #define COMMAND_CHECK_CHECK_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Perform standard checks FN_EXTERN void cmdCheck(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/check/common.c0000664000175100017510000001767215265101745020335 0ustar00runnerrunner/*********************************************************************************************************************************** Check Common Handler ***********************************************************************************************************************************/ #include #include #include "command/check/common.h" #include "common/debug.h" #include "config/config.h" #include "db/helper.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "postgres/interface.h" #include "storage/helper.h" #include "version.h" /*********************************************************************************************************************************** Helper function ***********************************************************************************************************************************/ static bool checkArchiveCommand(const String *const archiveCommand) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, archiveCommand); FUNCTION_TEST_END(); bool result = archiveCommand != NULL; if (result && strstr(strZ(archiveCommand), PROJECT_BIN) == NULL) result = false; if (!result) { THROW_FMT( ArchiveCommandInvalidError, "archive_command '%s' must contain %s", archiveCommand != NULL ? strZ(archiveCommand) : "[" NULL_Z "]", PROJECT_BIN); } FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN void checkDbConfig(const unsigned int pgVersion, const unsigned int pgIdx, const Db *const dbObject, const bool isStandby) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_PARAM(UINT, pgIdx); FUNCTION_TEST_PARAM(DB, dbObject); FUNCTION_TEST_PARAM(BOOL, isStandby); FUNCTION_TEST_END(); ASSERT(dbObject != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Error if the version from the control file and the configured pg-path do not match the values obtained from the database const unsigned int dbVersion = dbPgVersion(dbObject); const String *const dbPath = dbPgDataPath(dbObject); if (pgVersion != dbVersion || strCmp(cfgOptionIdxStr(cfgOptPgPath, pgIdx), dbPath) != 0) { THROW_FMT( DbMismatchError, "version '%s' and path '%s' queried from cluster do not match version '%s' and '%s' read from '%s/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "'\nHINT: the %s and %s settings likely reference different clusters.", strZ(pgVersionToStr(dbVersion)), strZ(dbPath), strZ(pgVersionToStr(pgVersion)), strZ(cfgOptionIdxDisplay(cfgOptPgPath, pgIdx)), strZ(cfgOptionIdxDisplay(cfgOptPgPath, pgIdx)), cfgOptionIdxName(cfgOptPgPath, pgIdx), cfgOptionIdxName(cfgOptPgPort, pgIdx)); } // Check archive configuration if option is valid for the command and set if (!isStandby && cfgOptionValid(cfgOptArchiveCheck) && cfgOptionBool(cfgOptArchiveCheck)) { // Error if archive_mode = off since backup start will fail if (strCmpZ(dbArchiveMode(dbObject), "off") == 0) { THROW(ArchiveDisabledError, "archive_mode must be enabled"); } // Error if archive_mode = always unless check is disabled (support has not been added yet) if (cfgOptionBool(cfgOptArchiveModeCheck) && strCmpZ(dbArchiveMode(dbObject), "always") == 0) { THROW(FeatureNotSupportedError, "archive_mode=always not supported"); } // Check if archive_command is set and is valid checkArchiveCommand(dbArchiveCommand(dbObject)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void checkStanzaInfo(const unsigned int repoIdx, const InfoPg *const archiveInfoPg, const InfoPg *const backupInfoPg) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_PARAM(INFO_PG, archiveInfoPg); FUNCTION_TEST_PARAM(INFO_PG, backupInfoPg); FUNCTION_TEST_END(); ASSERT(archiveInfoPg != NULL); ASSERT(backupInfoPg != NULL); const InfoPgData archiveInfo = infoPgData(archiveInfoPg, infoPgDataCurrentId(archiveInfoPg)); const InfoPgData backupInfo = infoPgData(backupInfoPg, infoPgDataCurrentId(backupInfoPg)); // Error if there is a mismatch between the archive and backup info files if (archiveInfo.id != backupInfo.id || archiveInfo.systemId != backupInfo.systemId || archiveInfo.version != backupInfo.version) { THROW_FMT( FileInvalidError, "backup info file and archive info file do not match\n" "archive: id = %u, version = %s, system-id = %" PRIu64 "\n" "backup : id = %u, version = %s, system-id = %" PRIu64 "\n" "HINT: this may be a symptom of repository corruption!", archiveInfo.id, strZ(pgVersionToStr(archiveInfo.version)), archiveInfo.systemId, backupInfo.id, strZ(pgVersionToStr(backupInfo.version)), backupInfo.systemId); } // Error if the info files are at different repository formats. The formats are written together but stored apart, so an // upgrade interrupted between the two saves leaves them mismatched. if (infoPgFormat(archiveInfoPg) != infoPgFormat(backupInfoPg)) { const unsigned int formatArchive = infoPgFormat(archiveInfoPg); const unsigned int formatBackup = infoPgFormat(backupInfoPg); THROW_FMT( FileInvalidError, "backup info file and archive info file are at different repository formats\n" "archive: format = %u\n" "backup : format = %u\n" "HINT: run " CFGCMD_STANZA_UPGRADE " with --%s=%u to complete an interrupted upgrade.", formatArchive, formatBackup, cfgOptionIdxName(cfgOptRepoFormat, repoIdx), formatArchive > formatBackup ? formatArchive : formatBackup); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void checkStanzaInfoPg( const unsigned int repoIdx, const Storage *const storage, const unsigned int pgVersion, const uint64_t pgSystemId, const CipherSpec *const cipherSpecMain) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_PARAM(STORAGE, storage); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_PARAM(UINT64, pgSystemId); FUNCTION_TEST_PARAM(CIPHER_SPEC, cipherSpecMain); FUNCTION_TEST_END(); ASSERT(storage != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Check that the backup and archive info files exist const InfoArchive *const infoArchive = infoArchiveLoadFile(storage, INFO_ARCHIVE_PATH_FILE_STR, cipherSpecMain); const InfoPgData archiveInfoPg = infoPgData(infoArchivePg(infoArchive), infoPgDataCurrentId(infoArchivePg(infoArchive))); const InfoBackup *const infoBackup = infoBackupLoadFile(storage, INFO_BACKUP_PATH_FILE_STR, cipherSpecMain); // Check that the info files pg data and repository format match each other checkStanzaInfo(repoIdx, infoArchivePg(infoArchive), infoBackupPg(infoBackup)); // Check that the version and system id match the current database if (pgVersion != archiveInfoPg.version || pgSystemId != archiveInfoPg.systemId) { THROW( FileInvalidError, "backup and archive info files exist but do not match the database\n" "HINT: is this the correct stanza?\n" "HINT: did an error occur during stanza-upgrade?"); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/check/common.h0000664000175100017510000000253115265101745020326 0ustar00runnerrunner/*********************************************************************************************************************************** Check Command Common ***********************************************************************************************************************************/ #ifndef COMMAND_CHECK_COMMON_H #define COMMAND_CHECK_COMMON_H #include "common/crypto/spec.h" #include "common/type/string.h" #include "db/db.h" #include "info/infoPg.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Check the database path and version are configured correctly FN_EXTERN void checkDbConfig(const unsigned int pgVersion, const unsigned int pgIdx, const Db *dbObject, bool isStandby); // Validate the archive and backup info files FN_EXTERN void checkStanzaInfo(unsigned int repoIdx, const InfoPg *archiveInfoPg, const InfoPg *backupInfoPg); // Load and validate the database data of the info files against each other and the current database FN_EXTERN void checkStanzaInfoPg( unsigned int repoIdx, const Storage *storage, const unsigned int pgVersion, const uint64_t pgSystemId, const CipherSpec *cipherSpecMain); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/check/report.c0000664000175100017510000002022215265101745020341 0ustar00runnerrunner/*********************************************************************************************************************************** Support Command ***********************************************************************************************************************************/ #include #include "command/check/report.h" #include "common/debug.h" #include "common/log.h" #include "common/type/json.h" #include "config/config.h" #include "config/parse.h" /*********************************************************************************************************************************** Render config ***********************************************************************************************************************************/ // Helper to render an individual config value static void checkReportConfigVal(JsonWrite *const json, const String *const optionName, const StringList *valueList, const bool env) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, json); FUNCTION_TEST_PARAM(STRING, optionName); FUNCTION_TEST_PARAM(STRING_LIST, valueList); FUNCTION_TEST_PARAM(BOOL, env); FUNCTION_TEST_END(); ASSERT(json != NULL); ASSERT(optionName != NULL); ASSERT(valueList != NULL); MEM_CONTEXT_TEMP_BEGIN() { jsonWriteObjectBegin(json); const CfgParseOptionResult option = cfgParseOptionP(optionName); // Render value if (option.found) { jsonWriteKeyStrId(json, STRID5("val", 0x30360)); // Multiple values if (option.multi) { ASSERT(strLstSize(valueList) >= 1); ASSERT(!cfgParseOptionSecure(option.id)); // Split environment values if (env) { ASSERT(strLstSize(valueList) == 1); valueList = strLstNewSplitZ(strLstGet(valueList, 0), ":"); } // Render each value jsonWriteArrayBegin(json); for (unsigned int valueIdx = 0; valueIdx < strLstSize(valueList); valueIdx++) jsonWriteStr(json, strLstGet(valueList, valueIdx)); jsonWriteArrayEnd(json); } // Single value else { ASSERT(strLstSize(valueList) == 1); if (cfgParseOptionSecure(option.id)) jsonWriteStr(json, strNewFmt("<%s>", strZ(optionName))); else jsonWriteStr(json, strLstGet(valueList, 0)); } } // Warn on invalid options or option modifiers if (!option.found || option.negate || option.reset) { jsonWriteKeyStrId(json, STRID5("warn", 0x748370)); // Warn if the option not found if (!option.found) { jsonWriteZ(json, "invalid option"); } // Warn if negate option else if (option.negate) { jsonWriteZ(json, "invalid negate option"); } // Warn if reset option else { ASSERT(option.reset); jsonWriteZ(json, "invalid reset option"); } } jsonWriteObjectEnd(json); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } // Helper to render the environment static void checkReportConfigEnv(JsonWrite *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, json); FUNCTION_TEST_END(); ASSERT(json != NULL); MEM_CONTEXT_TEMP_BEGIN() { jsonWriteKeyStrId(json, STRID5("env", 0x59c50)); jsonWriteObjectBegin(json); // Loop through all environment variables and look for our env vars by matching the prefix unsigned int environIdx = 0; KeyValue *const keyValue = kvNew(); while (environ[environIdx] != NULL) { const char *const environKeyValue = environ[environIdx]; environIdx++; if (strstr(environKeyValue, PGBACKREST_ENV) == environKeyValue) { // Find the first = char const char *const equalPtr = strchr(environKeyValue, '='); ASSERT(equalPtr != NULL); // Get key and value const String *const key = strNewZN(environKeyValue, (size_t)(equalPtr - environKeyValue)); const String *const value = STR(equalPtr + 1); kvPut(keyValue, VARSTR(key), VARSTR(value)); } } // Render environment variables const StringList *const keyList = strLstSort(strLstNewVarLst(kvKeyList(keyValue)), sortOrderAsc); for (unsigned int keyIdx = 0; keyIdx < strLstSize(keyList); keyIdx++) { const String *const key = strLstGet(keyList, keyIdx); StringList *const valueList = strLstNew(); strLstAdd(valueList, varStr(kvGet(keyValue, VARSTR(key)))); jsonWriteKey(json, key); checkReportConfigVal( json, strReplaceChr(strLower(strNewZ(strZ(key) + PGBACKREST_ENV_SIZE)), '_', '-'), valueList, true); } jsonWriteObjectEnd(json); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } // Helper to render the config file static void checkReportConfigFile(JsonWrite *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, json); FUNCTION_TEST_END(); ASSERT(json != NULL); MEM_CONTEXT_TEMP_BEGIN() { const Ini *const ini = cfgParseIni(); jsonWriteKeyStrId(json, STRID5("file", 0x2b1260)); // If no config if (ini == NULL) { jsonWriteNull(json); } // Else config exists else { // Iterate config sections const StringList *const sectionList = strLstSort(iniSectionList(ini), sortOrderAsc); jsonWriteObjectBegin(json); for (unsigned int sectionIdx = 0; sectionIdx < strLstSize(sectionList); sectionIdx++) { const String *const section = strLstGet(sectionList, sectionIdx); jsonWriteKey(json, section); jsonWriteObjectBegin(json); // Iterate config keys const StringList *const keyList = strLstSort(iniSectionKeyList(ini, section), sortOrderAsc); for (unsigned int keyIdx = 0; keyIdx < strLstSize(keyList); keyIdx++) { const String *const key = strLstGet(keyList, keyIdx); // Render value StringList *valueList; if (iniSectionKeyIsList(ini, section, key)) valueList = iniGetList(ini, section, key); else { valueList = strLstNew(); strLstAdd(valueList, iniGet(ini, section, key)); } jsonWriteKey(json, key); checkReportConfigVal(json, key, valueList, false); } jsonWriteObjectEnd(json); } jsonWriteObjectEnd(json); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } static void checkReportConfig(JsonWrite *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, json); FUNCTION_TEST_END(); ASSERT(json != NULL); jsonWriteKeyStrId(json, STRID5("cfg", 0x1cc30)); jsonWriteObjectBegin(json); checkReportConfigEnv(json); checkReportConfigFile(json); jsonWriteObjectEnd(json); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * checkReport(void) { FUNCTION_TEST_VOID(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { JsonWrite *const json = jsonWriteNewP(.json = result); // Render report jsonWriteObjectBegin(json); checkReportConfig(json); jsonWriteObjectEnd(json); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/check/report.h0000664000175100017510000000132015265101745020344 0ustar00runnerrunner/*********************************************************************************************************************************** Check Command Report ***********************************************************************************************************************************/ #ifndef COMMAND_CHECK_REPORT_H #define COMMAND_CHECK_REPORT_H #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Check report FN_EXTERN String *checkReport(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/command.c0000664000175100017510000003041115265101745017370 0ustar00runnerrunner/*********************************************************************************************************************************** Common Command Routines ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "command/command.h" #include "common/debug.h" #include "common/fork.h" #include "common/log.h" #include "common/memContext.h" #include "common/stat.h" #include "common/time.h" #include "common/type/json.h" #include "config/config.intern.h" #include "config/parse.h" #include "version.h" /*********************************************************************************************************************************** Track time command started ***********************************************************************************************************************************/ static TimeMSec timeBegin; static String *cmdOptionStr; /**********************************************************************************************************************************/ FN_EXTERN void cmdInit(void) { FUNCTION_LOG_VOID(logLevelTrace); timeBegin = timeMSec(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN const String * cmdOption(void) { FUNCTION_TEST_VOID(); if (cmdOptionStr == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { cmdOptionStr = strNew(); MEM_CONTEXT_TEMP_BEGIN() { // Add command parameters if they exist const StringList *const commandParamList = cfgCommandParam(); if (!strLstEmpty(commandParamList)) { strCatZ(cmdOptionStr, " ["); for (unsigned int commandParamIdx = 0; commandParamIdx < strLstSize(commandParamList); commandParamIdx++) { const String *commandParam = strLstGet(commandParamList, commandParamIdx); if (commandParamIdx != 0) strCatZ(cmdOptionStr, ", "); if (strchr(strZ(commandParam), ' ') != NULL) commandParam = strNewFmt("\"%s\"", strZ(commandParam)); strCat(cmdOptionStr, commandParam); } strCatZ(cmdOptionStr, "]"); } // Loop though options and add the ones that are interesting for (ConfigOption optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++) { // Skip the option if not valid for this command. Generally only one command runs at a time, but sometimes // commands are chained together (e.g. backup and expire) and the second command may not use all the options of // the first command. Displaying them is harmless but might cause confusion. if (!cfgOptionValid(optionId) || !cfgParseOptionValid(cfgCommand(), cfgCommandRole(), optionId)) continue; // Loop through option indexes const unsigned int optionIdxTotal = cfgOptionGroup(optionId) ? cfgOptionGroupIdxTotal(cfgOptionGroupId(optionId)) : 1; for (unsigned int optionIdx = 0; optionIdx < optionIdxTotal; optionIdx++) { // If option was negated if (cfgOptionIdxNegate(optionId, optionIdx)) strCatFmt(cmdOptionStr, " --no-%s", cfgOptionIdxName(optionId, optionIdx)); // If option was reset else if (cfgOptionIdxReset(optionId, optionIdx)) strCatFmt(cmdOptionStr, " --reset-%s", cfgOptionIdxName(optionId, optionIdx)); // Else not default else if (cfgOptionIdxSource(optionId, optionIdx) != cfgSourceDefault) { // Don't show redacted options if (cfgParseOptionSecure(optionId)) strCatFmt(cmdOptionStr, " --%s=", cfgOptionIdxName(optionId, optionIdx)); // Output boolean option else if (cfgParseOptionType(optionId) == cfgOptTypeBoolean) strCatFmt(cmdOptionStr, " --%s", cfgOptionIdxName(optionId, optionIdx)); // Output other options else { StringList *valueList = NULL; // Generate the values of hash options if (cfgParseOptionType(optionId) == cfgOptTypeHash) { valueList = strLstNew(); const KeyValue *const optionKv = cfgOptionIdxKv(optionId, optionIdx); const VariantList *const keyList = kvKeyList(optionKv); for (unsigned int keyIdx = 0; keyIdx < varLstSize(keyList); keyIdx++) { strLstAddFmt( valueList, "%s=%s", strZ(varStr(varLstGet(keyList, keyIdx))), strZ(varStrForce(kvGet(optionKv, varLstGet(keyList, keyIdx))))); } } // Generate values for list options else if (cfgParseOptionType(optionId) == cfgOptTypeList) { valueList = strLstNewVarLst(cfgOptionIdxLst(optionId, optionIdx)); } // Else only one value else { valueList = strLstNew(); strLstAdd(valueList, cfgOptionIdxDisplay(optionId, optionIdx)); } // Output options and values for (unsigned int valueListIdx = 0; valueListIdx < strLstSize(valueList); valueListIdx++) { const String *value = strLstGet(valueList, valueListIdx); strCatFmt(cmdOptionStr, " --%s", cfgOptionIdxName(optionId, optionIdx)); if (strchr(strZ(value), ' ') != NULL) value = strNewFmt("\"%s\"", strZ(value)); strCatFmt(cmdOptionStr, "=%s", strZ(value)); } } } } } } MEM_CONTEXT_TEMP_END(); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN(STRING, cmdOptionStr); } /**********************************************************************************************************************************/ FN_EXTERN void cmdBegin(void) { FUNCTION_LOG_VOID(logLevelTrace); ASSERT(cfgInited()); // This is fairly expensive log message to generate so skip it if it won't be output if (logAny(cfgLogLevelDefault())) { MEM_CONTEXT_TEMP_BEGIN() { // Basic info on command start String *const info = strCatFmt(strNew(), "%s command begin", strZ(cfgCommandRoleName())); // Free the old option string if it exists. This is needed when more than one command is run in a row so an option // string gets created for the new command. strFree(cmdOptionStr); cmdOptionStr = NULL; // Add version and options strCatFmt(info, " %s:%s", PROJECT_VERSION, strZ(cmdOption())); LOG(cfgLogLevelDefault(), 0, strZ(info)); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cmdEnd(const String *const errorMessage) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, errorMessage); FUNCTION_LOG_END(); ASSERT(cfgInited()); // Skip this log message if it won't be output. It's not too expensive but since we skipped cmdBegin(), may as well. if (logAny(cfgLogLevelDefault())) { MEM_CONTEXT_TEMP_BEGIN() { // Output statistics if there are any const String *const statJson = statToJson(); if (statJson != NULL) LOG_DETAIL_FMT("statistics: %s", strZ(statJson)); // Basic info on command end String *const info = strCatFmt(strNew(), "%s command end: ", strZ(cfgCommandRoleName())); if (errorMessage == NULL) { strCatZ(info, "completed successfully"); if (cfgOptionBool(cfgOptLogTimestamp)) strCatFmt(info, " (%" PRIu64 "ms)", timeMSec() - timeBegin); } else strCat(info, errorMessage); LOG(cfgLogLevelDefault(), 0, strZ(info)); } MEM_CONTEXT_TEMP_END(); } // Reset timeBegin in case there is another command following this one timeBegin = timeMSec(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cmdAsyncExec(const char *const command, const StringList *const commandExec) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRINGZ, command); FUNCTION_LOG_PARAM(STRING_LIST, commandExec); FUNCTION_LOG_END(); ASSERT(commandExec != NULL); // Fork off the async process const pid_t pid = forkSafe(); if (pid == 0) { // Disable logging and close log file logClose(); // Detach from parent process forkDetach(); // Close any open file descriptors above the standard three (stdin, stdout, stderr). Don't check the return value since we // don't know which file descriptors are actually open (might be none). It's possible that there are open files >= 1024 but // there is no easy way to detect that and this should give us enough descriptors to do our work. for (int fd = 3; fd < 1024; fd++) close(fd); // Execute the binary. This statement will not return if it is successful. execvp() requires non-const parameters because it // modifies them after the fork. #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wcast-qual" THROW_ON_SYS_ERROR_FMT( execvp(strZ(strLstGet(commandExec, 0)), UNCONSTIFY(char **, strLstPtr(commandExec))) == -1, ExecuteError, "unable to execute asynchronous '%s'", command); #pragma GCC diagnostic pop } #ifdef DEBUG_EXEC_TIME // Get the time to measure how long it takes for the forked process to exit const TimeMSec timeBegin = timeMSec(); #endif // The process that was just forked should return immediately int processStatus; THROW_ON_SYS_ERROR(waitpid(pid, &processStatus, 0) == -1, ExecuteError, "unable to wait for forked process"); // The first fork should exit with success. If not, something went wrong during the second fork. CHECK(ExecuteError, WIFEXITED(processStatus) && WEXITSTATUS(processStatus) == 0, "error on first fork"); #ifdef DEBUG_EXEC_TIME // If the process does not exit immediately then something probably went wrong with the double fork. It's possible that this // test will fail on very slow systems so it may need to be tuned. The idea is to make sure that the waitpid() above is not // waiting on the async process. CHECK(AssertError, timeMSec() - timeBegin < 10, "the process does not exit immediately"); #endif FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/command.h0000664000175100017510000000244015265101745017376 0ustar00runnerrunner/*********************************************************************************************************************************** Common Command Routines ***********************************************************************************************************************************/ #ifndef COMMAND_COMMAND_H #define COMMAND_COMMAND_H #include "common/type/stringList.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Capture time at the very start of main so total time is more accurate FN_EXTERN void cmdInit(void); // Begin the command FN_EXTERN void cmdBegin(void); // Get the command options as a string. This is output in cmdBegin() if the log level is high enough but can also be used for // debugging or error reporting later on. FN_EXTERN const String *cmdOption(void); // End the command FN_EXTERN void cmdEnd(const String *errorMessage); // Execute an async process. This function will only return in the calling process and the implementation is platform dependent. FN_EXTERN void cmdAsyncExec(const char *command, const StringList *commandExec); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/control/0000775000175100017510000000000015265101745017267 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/control/common.c0000664000175100017510000000321615265101745020725 0ustar00runnerrunner/*********************************************************************************************************************************** Common Handler for Control Commands ***********************************************************************************************************************************/ #include #include "command/control/common.h" #include "common/debug.h" #include "config/config.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN String * lockStopFileName(const String *const stanza) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, stanza); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN( STRING, strNewFmt("%s/%s" STOP_FILE_EXT, strZ(cfgOptionStr(cfgOptLockPath)), stanza != NULL ? strZ(stanza) : "all")); } /**********************************************************************************************************************************/ FN_EXTERN void lockStopTest(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { // Check the current stanza (if any) if (cfgOptionTest(cfgOptStanza)) { if (storageExistsP(storageLocal(), lockStopFileName(cfgOptionStr(cfgOptStanza)))) THROW_FMT(StopError, "stop file exists for stanza %s", strZ(cfgOptionDisplay(cfgOptStanza))); } // Check all stanzas if (storageExistsP(storageLocal(), lockStopFileName(NULL))) THROW(StopError, "stop file exists for all stanzas"); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/control/common.h0000664000175100017510000000226215265101745020732 0ustar00runnerrunner/*********************************************************************************************************************************** Common Handler for Control Commands ***********************************************************************************************************************************/ #ifndef COMMAND_CONTROL_COMMON_H #define COMMAND_CONTROL_COMMON_H #include "common/type/string.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define STOP_FILE_EXT ".stop" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Create the stop filename FN_EXTERN String *lockStopFileName(const String *stanza); // Test for the existence of a stop file FN_EXTERN void lockStopTest(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/control/start.c0000664000175100017510000000265215265101745020575 0ustar00runnerrunner/*********************************************************************************************************************************** Start Command ***********************************************************************************************************************************/ #include #include "command/control/common.h" #include "command/control/start.h" #include "common/debug.h" #include "config/config.h" #include "storage/helper.h" #include "storage/storage.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdStart(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { // Remove the stop file so processes can run const String *const stopFile = lockStopFileName(cfgOptionStrNull(cfgOptStanza)); // If the stop file exists, then remove it if (storageExistsP(storageLocal(), stopFile)) { // If the file cannot be removed, storageRemove() will throw an error if the error is not ENOENT storageRemoveP(storageLocalWrite(), stopFile); } else { LOG_WARN_FMT( "stop file does not exist%s", (cfgOptionTest(cfgOptStanza) ? zNewFmt(" for stanza %s", strZ(cfgOptionDisplay(cfgOptStanza))) : "")); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/control/start.h0000664000175100017510000000122415265101745020574 0ustar00runnerrunner/*********************************************************************************************************************************** Start Command ***********************************************************************************************************************************/ #ifndef COMMAND_CONTROL_START_H #define COMMAND_CONTROL_START_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN void cmdStart(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/control/stop.c0000664000175100017510000001062115265101745020420 0ustar00runnerrunner/*********************************************************************************************************************************** Stop Command ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include "command/control/common.h" #include "command/control/stop.h" #include "common/debug.h" #include "common/lock.h" #include "common/type/convert.h" #include "config/config.h" #include "storage/helper.h" #include "storage/storage.h" #include "storage/storage.intern.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdStop(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { const String *const stopFile = lockStopFileName(cfgOptionStrNull(cfgOptStanza)); // If the stop file does not already exist, then create it if (!storageExistsP(storageLocal(), stopFile)) { // Create the lock path (ignore if already created) storagePathCreateP(storageLocalWrite(), strPath(stopFile), .mode = 0770); // Create the stop file with without truncating an existing file IoWrite *const stopWrite = storageWriteIo( storageNewWriteP(storageLocalWrite(), stopFile, .noAtomic = true, .noTruncate = true, .modePath = 0770)); ioWriteOpen(stopWrite); ioWriteClose(stopWrite); // If --force was specified then send term signals to running processes if (cfgOptionBool(cfgOptForce)) { const String *const lockPath = cfgOptionStr(cfgOptLockPath); const String *const stanzaPrefix = cfgOptionTest(cfgOptStanza) ? strNewFmt("%s-", strZ(cfgOptionStr(cfgOptStanza))) : NULL; const StringList *const lockPathFileList = strLstSort( storageListP(storageLocal(), lockPath, .errorOnMissing = true), sortOrderAsc); List *const pidList = lstNewP(sizeof(int), .comparator = lstComparatorInt); // Find each lock file and send term signals to the processes for (unsigned int lockPathFileIdx = 0; lockPathFileIdx < strLstSize(lockPathFileList); lockPathFileIdx++) { const String *const lockFile = strLstGet(lockPathFileList, lockPathFileIdx); // Skip any file that is not a lock file. Skip lock files for other stanzas if a stanza is provided. if (!strEndsWithZ(lockFile, LOCK_FILE_EXT) || (stanzaPrefix != NULL && !strBeginsWith(lockFile, stanzaPrefix))) continue; // Read the lock file const LockReadResult lockResult = lockReadP(lockFile, .remove = true); // If we cannot read the lock file for any reason then warn and continue to next file if (lockResult.status != lockReadStatusValid) { LOG_WARN_FMT("unable to read lock file %s/%s", strZ(lockPath), strZ(lockFile)); continue; } // If the process has already been killed then skip it. This is possible because a single process may be holding // multiple lock files. if (lstFind(pidList, &lockResult.data.processId) != NULL) continue; lstAdd(pidList, &lockResult.data.processId); // The lock file is valid so that means there is a running process -- send a term signal to the process if (kill(lockResult.data.processId, SIGTERM) != 0) LOG_WARN_FMT("unable to send term signal to process %d", lockResult.data.processId); else LOG_INFO_FMT("sent term signal to process %d", lockResult.data.processId); } } } else { LOG_WARN_FMT( "stop file already exists for %s", cfgOptionTest(cfgOptStanza) ? zNewFmt("stanza %s", strZ(cfgOptionDisplay(cfgOptStanza))) : "all stanzas"); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/control/stop.h0000664000175100017510000000122015265101745020420 0ustar00runnerrunner/*********************************************************************************************************************************** Stop Command ***********************************************************************************************************************************/ #ifndef COMMAND_CONTROL_STOP_H #define COMMAND_CONTROL_STOP_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN void cmdStop(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/exit.c0000664000175100017510000001343215265101745016727 0ustar00runnerrunner/*********************************************************************************************************************************** Exit Routines ***********************************************************************************************************************************/ #include #include #include #include #include "command/command.h" #include "command/exit.h" #include "command/lock.h" #include "common/debug.h" #include "common/log.h" #include "config/config.intern.h" #include "protocol/helper.h" #include "version.h" /*********************************************************************************************************************************** Indicates that exit() is in progress so a signal cannot call exit() recursively, which is undefined behavior and crashes on musl libc >= 1.2.6 ***********************************************************************************************************************************/ static volatile sig_atomic_t exitInProgress = false; /*********************************************************************************************************************************** Return signal names ***********************************************************************************************************************************/ static const char * exitSignalName(const SignalType signalType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, signalType); FUNCTION_TEST_END(); const char *name = NULL; switch (signalType) { case signalTypeHup: name = "SIGHUP"; break; case signalTypeInt: name = "SIGINT"; break; case signalTypeTerm: name = "SIGTERM"; break; case signalTypeNone: THROW(AssertError, "no name for signal none"); } FUNCTION_TEST_RETURN_CONST(STRINGZ, name); } /*********************************************************************************************************************************** Catch signals ***********************************************************************************************************************************/ static void exitOnSignal(const int signalType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, signalType); FUNCTION_TEST_END(); // Ignore the signal when exit() is already in progress and allow it to complete if (!exitInProgress) { exitInProgress = true; logSignal( cfgLogLevelDefault(), signalType == signalTypeNone ? "from child process" : exitSignalName((SignalType)signalType)); exit(errorTypeCode(&TermError)); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void exitInit(void) { FUNCTION_LOG_VOID(logLevelTrace); // Reset exit in progress in case this process was forked by a process that was exiting exitInProgress = false; signal(SIGHUP, exitOnSignal); signal(SIGINT, exitOnSignal); signal(SIGTERM, exitOnSignal); // Ignore SIGPIPE and check for EPIPE errors on write() instead signal(SIGPIPE, SIG_IGN); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ // Helper to provide details for error logging static String * exitErrorDetail(void) { FUNCTION_TEST_VOID(); String *const role = cfgCommandRoleName(); String *const result = strNewFmt( "--------------------------------------------------------------------\n" "If SUBMITTING AN ISSUE please provide the following information:\n" "\n" "version: " PROJECT_VERSION "\n" "command: %s\n" "options:%s\n" "\n" "stack trace:\n%s\n" "--------------------------------------------------------------------", strZ(role), strZ(cmdOption()), errorStackTrace()); strFree(role); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN int exitSafe(int result, const bool error) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INT, result); FUNCTION_LOG_PARAM(BOOL, error); FUNCTION_LOG_END(); // Report error if one was thrown if (error) { LogLevel logLevel = errorCode() == errorTypeCode(&AssertError) ? logLevelAssert : logLevelError; // Output stack track on assert or when an output (e.g. console) is at least debug level if (logLevel == logLevelAssert || logAny(logLevelDebug)) { String *const detail = exitErrorDetail(); LOG_INTERNAL_FMT( logLevel, logLevel == logLevelAssert ? LOG_LEVEL_MIN : logLevelDebug, LOG_LEVEL_MAX, 0, errorCode(), "%s\n%s", errorMessage(), strZ(detail)); strFree(detail); } // Log just the error to non-debug levels if (logLevel != logLevelAssert) LOG_INTERNAL(logLevel, LOG_LEVEL_MIN, logLevelDetail, 0, errorCode(), errorMessage()); result = errorCode(); } // On error generate an error message for command end when config is initialized if (cfgInited()) { String *errorMessage = NULL; if (result != 0 && error) errorMessage = strNewFmt("aborted with exception [%03d]", result); cmdEnd(errorMessage); strFree(errorMessage); } // Release any locks but ignore errors TRY_BEGIN() { cmdLockReleaseP(.returnOnNoLock = true); } TRY_END(); // Set exit in progress so a signal arriving while exit() runs will not call exit() recursively exitInProgress = true; // Return result - caller should immediate pass this result to exit() FUNCTION_LOG_RETURN(INT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/exit.h0000664000175100017510000000227515265101745016737 0ustar00runnerrunner/*********************************************************************************************************************************** Exit Routines ***********************************************************************************************************************************/ #ifndef COMMAND_EXIT_H #define COMMAND_EXIT_H #include #include /*********************************************************************************************************************************** Signal type ***********************************************************************************************************************************/ typedef enum { signalTypeNone = 0, signalTypeHup = SIGHUP, signalTypeInt = SIGINT, signalTypeTerm = SIGTERM, } SignalType; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Setup signal handlers FN_EXTERN void exitInit(void); // Do cleanup and return result code FN_EXTERN int exitSafe(int result, bool error); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/expire/0000775000175100017510000000000015265101745017103 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/expire/expire.c0000664000175100017510000020714215265101745020551 0ustar00runnerrunner/*********************************************************************************************************************************** Expire Command ***********************************************************************************************************************************/ #include #include "command/archive/common.h" #include "command/backup/common.h" #include "command/control/common.h" #include "command/expire/expire.h" #include "command/stanza/common.h" #include "common/debug.h" #include "common/format/format.h" #include "common/regExp.h" #include "common/time.h" #include "common/type/list.h" #include "config/config.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "protocol/helper.h" #include "storage/helper.h" #include /*********************************************************************************************************************************** Helper functions and structures ***********************************************************************************************************************************/ typedef struct ArchiveExpired { uint64_t total; String *start; String *stop; } ArchiveExpired; typedef struct ArchiveRange { const String *start; const String *stop; } ArchiveRange; /*********************************************************************************************************************************** Given a backup label, expire a backup and all its dependents (if any). ***********************************************************************************************************************************/ static StringList * expireBackup(InfoBackup *const infoBackup, const String *const backupLabel, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(STRING, backupLabel); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); ASSERT(backupLabel != NULL); // Return a list of all backups being expired StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Get the backup and all its dependents sorted from newest to oldest - that way if the process is aborted before all // dependencies have been dealt with, the backups remaining should still be usable. const StringList *const backupList = strLstSort(infoBackupDataDependentList(infoBackup, backupLabel), sortOrderDesc); // Expire each backup in the list for (unsigned int backupIdx = 0; backupIdx < strLstSize(backupList); backupIdx++) { const String *const removeBackupLabel = strLstGet(backupList, backupIdx); // Execute the real expiration and deletion only if the dry-run option is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { // Remove the manifest files to invalidate the backup storageRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(removeBackupLabel))); storageRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, strZ(removeBackupLabel))); } // Remove the backup from the info object infoBackupDataDelete(infoBackup, removeBackupLabel); strLstAdd(result, removeBackupLabel); } // Sort from oldest to newest and return the list of expired backups strLstSort(result, sortOrderAsc); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /*********************************************************************************************************************************** Checks the given backup for checksum errors and throws an error or logs a warning depending on shouldFail ***********************************************************************************************************************************/ static void expireChecksumErrorCheck( const InfoBackup *const infoBackup, const String *const backupLabel, const unsigned int repoIdx, const bool error) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(STRING, backupLabel); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_PARAM(BOOL, error); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); if (infoBackupDataTotal(infoBackup) > 0) { const InfoBackupData *const backupData = infoBackupDataByLabel(infoBackup, backupLabel); if (backupData->backupError != NULL && varBool(backupData->backupError)) { MEM_CONTEXT_TEMP_BEGIN() { const String *message = strNewFmt( "oldest retained backup %s contains invalid page checksum(s)\n" "HINT: use info --set command to get details about errors in the backup.", strZ(backupLabel)); if (error) THROW(ChecksumError, strZ(message)); LOG_WARN_FMT("%s: %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(message)); } MEM_CONTEXT_TEMP_END(); } } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Function to expire a selected backup (and all its dependents) regardless of retention rules. ***********************************************************************************************************************************/ static unsigned int expireAdhocBackup(InfoBackup *const infoBackup, const String *const backupLabel, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(STRING, backupLabel); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); ASSERT(backupLabel != NULL); unsigned int result = 0; MEM_CONTEXT_TEMP_BEGIN() { // Get a list of all full backups with most recent in position 0 const StringList *const fullList = strLstSort( infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true)), sortOrderDesc); // If the requested backup to expire is the latest full backup if (strCmp(strLstGet(fullList, 0), backupLabel) == 0) { // If the latest full backup requested is the only backup or the prior full backup is not for the same db-id // then the backup requested cannot be expired if (strLstSize(fullList) == 1 || infoBackupDataByLabel(infoBackup, backupLabel)->backupPgId != infoBackupDataByLabel(infoBackup, strLstGet(fullList, 1))->backupPgId) { THROW_FMT( BackupSetInvalidError, "full backup %s cannot be expired until another full backup has been created on this repo", strZ(backupLabel)); } } // Save off what is currently the latest backup (it may be removed if it is the adhoc backup or is a dependent of the // adhoc backup const String *const latestBackup = infoBackupData(infoBackup, infoBackupDataTotal(infoBackup) - 1).backupLabel; // Expire the requested backup and any dependents const StringList *const backupExpired = expireBackup(infoBackup, backupLabel, repoIdx); // If the latest backup was removed, then update the latest link if not a dry-run if (!infoBackupLabelExists(infoBackup, latestBackup)) { // If retention settings have been configured, then there may be holes in the archives. For example, if the archive // for db-id=1 has 01,02,03,04,05 and F1 backup has archive start-stop 02-03 and retention-full=1 // (hence retention-archive=1 and retention-archive-type=full), then when F2 backup is created and assuming its // archive start-stop=05-06 then archives 01 and 04 will be removed resulting in F1 not being able to play through // PITR, which is expected. Now adhoc expire is attempted on F2 - it will be allowed but now there will be no // backups that can be recovered through PITR until the next full backup is created. Same problem for differential // backups with retention-diff. LOG_WARN_FMT( "%s: expiring latest backup %s - the ability to perform point-in-time-recovery (PITR) may be affected\n" "HINT: non-default settings for '%s'/'%s' (even in prior expires) can cause gaps in the WAL.", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(latestBackup), cfgOptionIdxName(cfgOptRepoRetentionArchive, repoIdx), cfgOptionIdxName(cfgOptRepoRetentionArchiveType, repoIdx)); // Adhoc expire is never performed through backup command so only check to determine if dry-run has been set or not if (!cfgOptionBool(cfgOptDryRun)) backupLinkLatest(infoBackupData(infoBackup, infoBackupDataTotal(infoBackup) - 1).backupLabel, repoIdx); } result = strLstSize(backupExpired); // Log the expired backup list (prepend "set:" if there were any dependents that were also expired) LOG_INFO_FMT( "%s: expire adhoc backup %s%s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), (result > 1 ? "set " : ""), strZ(strLstJoin(backupExpired, ", "))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } /*********************************************************************************************************************************** Expire differential backups ***********************************************************************************************************************************/ static unsigned int expireDiffBackup(InfoBackup *const infoBackup, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); // Total of all backups being expired unsigned int result = 0; MEM_CONTEXT_TEMP_BEGIN() { const unsigned int differentialRetention = cfgOptionIdxTest(cfgOptRepoRetentionDiff, repoIdx) ? cfgOptionIdxUInt(cfgOptRepoRetentionDiff, repoIdx) : 0; // Find all the expired differential backups if (differentialRetention > 0) { // Get a list of full and differential backups. Full are considered differential for the purpose of retention. // Example: F1, D1, D2, F2, repo-retention-diff=2, then F1,D2,F2 will be retained, not D2 and D1 as might be expected. const StringList *const currentBackupList = infoBackupDataLabelList( infoBackup, backupRegExpP(.full = true, .differential = true)); // If there are more backups than the number to retain, then expire the oldest ones if (strLstSize(currentBackupList) > differentialRetention) { for (unsigned int diffIdx = 0; diffIdx < strLstSize(currentBackupList) - differentialRetention; diffIdx++) { // Skip if this is a full backup. Full backups only count as differential when deciding which differential // backups to expire. if (regExpMatchOne(backupRegExpP(.full = true), strLstGet(currentBackupList, diffIdx))) continue; // Expire the differential and any dependent backups const StringList *const backupExpired = expireBackup( infoBackup, strLstGet(currentBackupList, diffIdx), repoIdx); result += strLstSize(backupExpired); // Log the expired backups. If there is more than one backup, then prepend "set:" LOG_INFO_FMT( "%s: expire diff backup %s%s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), (strLstSize(backupExpired) > 1 ? "set " : ""), strZ(strLstJoin(backupExpired, ", "))); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } /*********************************************************************************************************************************** Expire full backups ***********************************************************************************************************************************/ static unsigned int expireFullBackup(InfoBackup *const infoBackup, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); // Total of all backups being expired unsigned int result = 0; MEM_CONTEXT_TEMP_BEGIN() { const unsigned int fullRetention = cfgOptionIdxTest(cfgOptRepoRetentionFull, repoIdx) ? cfgOptionIdxUInt(cfgOptRepoRetentionFull, repoIdx) : 0; // Find all the expired full backups if (fullRetention > 0) { // Get list of current full backups (default order is oldest to newest) const StringList *const currentBackupList = infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true)); // If there are more full backups then the number to retain, then expire the oldest ones if (strLstSize(currentBackupList) > fullRetention) { // If checksum-page-error is set, check the would-be-oldest retained backup and terminate with an error if it has // checksum errors. if (cfgOptionBool(cfgOptChecksumPageError)) { const String *const oldestRetainedBackupLabel = strLstGet( currentBackupList, strLstSize(currentBackupList) - fullRetention); expireChecksumErrorCheck(infoBackup, oldestRetainedBackupLabel, repoIdx, true); } // Expire all backups that depend on the full backup for (unsigned int fullIdx = 0; fullIdx < strLstSize(currentBackupList) - fullRetention; fullIdx++) { // Expire the full backup and all its dependents const StringList *const backupExpired = expireBackup( infoBackup, strLstGet(currentBackupList, fullIdx), repoIdx); result += strLstSize(backupExpired); // Log the expired backups. If there is more than one backup, then prepend "set:" LOG_INFO_FMT( "%s: expire full backup %s%s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), (strLstSize(backupExpired) > 1 ? "set " : ""), strZ(strLstJoin(backupExpired, ", "))); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } /*********************************************************************************************************************************** Expire backups based on time ***********************************************************************************************************************************/ static unsigned int expireTimeBasedBackup(InfoBackup *const infoBackup, const time_t minTimestamp, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(TIME, minTimestamp); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); ASSERT(minTimestamp > 0); unsigned int result = 0; MEM_CONTEXT_TEMP_BEGIN() { // Get the list of full backups const StringList *const currentBackupList = strLstSort( infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true)), sortOrderAsc); unsigned int backupIdx = strLstSize(currentBackupList); // Find out the point where we will have to stop purging backups. Starting with the newest backup (the end of the list), // find the first backup that is older than the expire time period by checking the backup stop time. This way, if the // backups are F1 D1a D1b D1c F2 D2a D2b F3 D3a D3b and the expiration time period is at D2b, then purge only F1, D1a, D1b // and D1c, and keep the next full backups (F2 and F3) and all intermediate non-full backups. if (backupIdx > 0) { const String *retentionMetBackupLabel = NULL; do { backupIdx--; InfoBackupData *info = infoBackupDataByLabel(infoBackup, strLstGet(currentBackupList, backupIdx)); // We can start deleting before this backup. This way, we keep one full backup and its dependents. if (info->backupTimestampStop < minTimestamp) { retentionMetBackupLabel = info->backupLabel; break; } } while (backupIdx != 0); // If retention has not been met there is nothing to expire if (retentionMetBackupLabel != NULL) { // If checksum-page-error is set, check the would-be-oldest retained backup and terminate with an error if it has // checksum errors. if (cfgOptionBool(cfgOptChecksumPageError)) expireChecksumErrorCheck(infoBackup, retentionMetBackupLabel, repoIdx, true); // Count number of full backups being expired unsigned int numFullExpired = 0; // Since expireBackup() will remove the requested entry from the backup list, we keep checking the first entry which // is always the oldest so if it is not the backup to retain then we can remove it while (!strEq(infoBackupData(infoBackup, 0).backupLabel, retentionMetBackupLabel)) { const StringList *const backupExpired = expireBackup( infoBackup, infoBackupData(infoBackup, 0).backupLabel, repoIdx); result += strLstSize(backupExpired); numFullExpired++; // Log the expired backups. If there is more than one backup, then prepend "set:" LOG_INFO_FMT( "%s: expire time-based backup %s%s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), (strLstSize(backupExpired) > 1 ? "set " : ""), strZ(strLstJoin(backupExpired, ", "))); } // Set archive retention (if it is not already set) based on the number of full backups retained if (cfgOptionIdxStrId(cfgOptRepoRetentionArchiveType, repoIdx) == backupTypeFull && !cfgOptionIdxTest(cfgOptRepoRetentionArchive, repoIdx)) { cfgOptionIdxSet( cfgOptRepoRetentionArchive, repoIdx, cfgSourceDefault, VARINT64(strLstSize(currentBackupList) - numFullExpired)); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } /*********************************************************************************************************************************** Log detailed information about archive logs removed ***********************************************************************************************************************************/ static void logExpire(ArchiveExpired *const archiveExpire, const String *const archiveId, const unsigned int repoIdx) { if (archiveExpire->start != NULL) { // Force out any remaining message LOG_INFO_FMT( "%s: %s remove archive, start = %s, stop = %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(archiveId), strZ(archiveExpire->start), strZ(archiveExpire->stop)); archiveExpire->start = NULL; } } /*********************************************************************************************************************************** Expire WAL for a single archive id, removing everything not covered by the supplied retention ranges ***********************************************************************************************************************************/ static void expireArchiveId( const String *const archiveId, const List *const archiveRangeList, const String *const archiveExpireMax, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, archiveId); FUNCTION_LOG_PARAM(LIST, archiveRangeList); FUNCTION_LOG_PARAM(STRING, archiveExpireMax); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(archiveId != NULL); ASSERT(archiveRangeList != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Initialize the expired archive information for this archive id ArchiveExpired archiveExpire = {.total = 0, .start = NULL, .stop = NULL}; // Get all major archive paths (timeline and first 32 bits of LSN) const StringList *const walPathList = strLstSort( storageListP( storageRepoIdx(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s", strZ(archiveId)), .expression = STRDEF(WAL_SEGMENT_DIR_REGEXP)), sortOrderAsc); for (unsigned int walIdx = 0; walIdx < strLstSize(walPathList); walIdx++) { const String *const walPath = strLstGet(walPathList, walIdx); bool removeArchive = true; // Keep the path if it falls in the range of any backup in retention for (unsigned int rangeIdx = 0; rangeIdx < lstSize(archiveRangeList); rangeIdx++) { const ArchiveRange *const archiveRange = lstGet(archiveRangeList, rangeIdx); if (strCmp(walPath, strSubN(archiveRange->start, 0, 16)) >= 0 && (archiveRange->stop == NULL || strCmp(walPath, strSubN(archiveRange->stop, 0, 16)) <= 0)) { removeArchive = false; break; } } // Remove the entire directory if all archive is expired if (removeArchive) { // Execute the real expiration and deletion only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { storagePathRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(archiveId), strZ(walPath)), .recurse = true); } LOG_DETAIL_FMT( "%s: %s remove archive path %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(archiveId), strZ(walPath)); archiveExpire.total++; archiveExpire.stop = strDup(walPath); // Update start for the first WAL path removed if (archiveExpire.start == NULL) archiveExpire.start = strDup(walPath); } // Else delete individual files instead if the major path is at or below the scan boundary. This optimization prevents // scanning through major paths that could not possibly have anything to expire. When there is no retention backup // (archiveExpireMax is NULL, e.g. archive-expire-before with no backup) the optimization does not apply, so scan // whenever the path was not removed in bulk. else if (archiveExpireMax == NULL || strCmp(walPath, strSubN(archiveExpireMax, 0, 16)) <= 0) { // Look for files in the archive directory const StringList *const walSubPathList = strLstSort( storageListP( storageRepoIdx(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(archiveId), strZ(walPath)), .expression = STRDEF("^[0-F]{24}.*$")), sortOrderAsc); for (unsigned int subIdx = 0; subIdx < strLstSize(walSubPathList); subIdx++) { removeArchive = true; const String *const walSubPath = strLstGet(walSubPathList, subIdx); // Determine if the individual archive log is used in a backup for (unsigned int rangeIdx = 0; rangeIdx < lstSize(archiveRangeList); rangeIdx++) { const ArchiveRange *const archiveRange = lstGet(archiveRangeList, rangeIdx); if (strCmp(strSubN(walSubPath, 0, 24), archiveRange->start) >= 0 && (archiveRange->stop == NULL || strCmp(strSubN(walSubPath, 0, 24), archiveRange->stop) <= 0)) { removeArchive = false; break; } } // Remove archive log if it is not used in a backup if (removeArchive) { // Execute the real expiration and deletion only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { storageRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s/%s", strZ(archiveId), strZ(walPath), strZ(walSubPath))); } // Track that this archive was removed archiveExpire.total++; archiveExpire.stop = strDup(strSubN(walSubPath, 0, 24)); if (archiveExpire.start == NULL) archiveExpire.start = strDup(strSubN(walSubPath, 0, 24)); } else logExpire(&archiveExpire, archiveId, repoIdx); } } } // Log if no archive was expired if (archiveExpire.total == 0) LOG_INFO_FMT("%s: %s no archive to remove", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(archiveId)); // Log if there is more to log else logExpire(&archiveExpire, archiveId, repoIdx); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Process archive retention ***********************************************************************************************************************************/ static void removeExpiredArchive(const InfoBackup *const infoBackup, const bool timeBasedFullRetention, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(BOOL, timeBasedFullRetention); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Get the retention options. repo-archive-retention-type always has a value as it defaults to "full" const BackupType archiveRetentionType = (BackupType)cfgOptionIdxStrId(cfgOptRepoRetentionArchiveType, repoIdx); const unsigned int archiveRetention = cfgOptionIdxTest( cfgOptRepoRetentionArchive, repoIdx) ? cfgOptionIdxUInt(cfgOptRepoRetentionArchive, repoIdx) : 0; // If archive retention is undefined, then ignore archiving. The user does not have to set this - it will be defaulted in // cfgLoadUpdateOption based on certain rules. if (archiveRetention == 0) { const String *const msg = strNewZ("- archive logs will not be expired"); // Only notify user if not time-based retention if (!timeBasedFullRetention) LOG_INFO_FMT("option '%s' is not set %s", cfgOptionIdxName(cfgOptRepoRetentionArchive, repoIdx), strZ(msg)); else { LOG_INFO_FMT( "%s: time-based archive retention not met %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(msg)); } } else { // Determine which backup type to use for archive retention (full, differential, incremental) and get a list of the // remaining non-expired backups, from newest to oldest, based on the type. const StringList *globalBackupRetentionList = NULL; switch (archiveRetentionType) { case backupTypeFull: globalBackupRetentionList = strLstSort( infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true)), sortOrderDesc); break; case backupTypeDiff: globalBackupRetentionList = strLstSort( infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true, .differential = true)), sortOrderDesc); break; case backupTypeIncr: // Incrementals can depend on Full or Diff so get a list of all incrementals globalBackupRetentionList = strLstSort( infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true, .differential = true, .incremental = true)), sortOrderDesc); break; } // Cap removal at the requested segment when --archive-expire-before is set: only WAL strictly before it may be removed const String *const walExpireBefore = cfgOptionTest(cfgOptArchiveExpireBefore) ? cfgOptionStr(cfgOptArchiveExpireBefore) : NULL; // Expire archives. If no backups were found or the number of backups found is not enough to satisfy archive retention // then preserve current archive logs - too soon to expire them. if (walExpireBefore != NULL || (!strLstEmpty(globalBackupRetentionList) && archiveRetention <= strLstSize(globalBackupRetentionList))) { // Attempt to load the archive info file const InfoArchive *const infoArchive = infoArchiveLoadFile( storageRepoIdx(repoIdx), INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); const InfoPg *const infoArchivePgData = infoArchivePg(infoArchive); // Get a list of archive directories (e.g. 9.6-1, 10-2, etc) sorted by the db-id (number after the dash). const StringList *const listArchiveDisk = strLstSort( strLstComparatorSet( storageListP( storageRepoIdx(repoIdx), STORAGE_REPO_ARCHIVE_STR, .expression = STRDEF(REGEX_ARCHIVE_DIR_DB_VERSION)), archiveIdComparator), sortOrderAsc); StringList *const globalBackupArchiveRetentionList = strLstNew(); // globalBackupRetentionList is ordered newest to oldest backup, so create globalBackupArchiveRetentionList of the // newest backups whose archives will be retained const unsigned int retentionSize = archiveRetention < strLstSize(globalBackupRetentionList) ? archiveRetention : strLstSize(globalBackupRetentionList); for (unsigned int retentionIdx = 0; retentionIdx < retentionSize; retentionIdx++) strLstAdd(globalBackupArchiveRetentionList, strLstGet(globalBackupRetentionList, retentionIdx)); // From newest to oldest, confirm the pgVersion and pgSystemId from the archive.info history id match that of the // same history id of the backup.info and if not, there is a mismatch between the info files so do not continue. // NOTE: If the archive.info file was reconstructed and contains history 4: version 10, sys-id 456 and history 3: // version 9.6, sys-id 123, but backup.info contains history 4 and 3 (identical to archive.info) but also 2 and 1 // then an error will not be thrown since only archive 4 and 3 would have existed and the others expired so // nothing really to do. for (unsigned int infoPgIdx = 0; infoPgIdx < infoPgDataTotal(infoArchivePgData); infoPgIdx++) { const InfoPgData archiveInfoPgHistory = infoPgData(infoArchivePgData, infoPgIdx); const InfoPgData backupInfoPgHistory = infoPgData(infoBackupPg(infoBackup), infoPgIdx); if (archiveInfoPgHistory.id != backupInfoPgHistory.id || archiveInfoPgHistory.systemId != backupInfoPgHistory.systemId || archiveInfoPgHistory.version != backupInfoPgHistory.version) { THROW( FormatError, "archive expiration cannot continue - archive and backup history lists do not match"); } } // Loop through the archive.info history from oldest to newest and if there is a corresponding directory on disk // then remove WAL that are not part of retention as long as the db:history id version/system-id matches backup.info for (unsigned int pgIdx = infoPgDataTotal(infoArchivePgData) - 1; (int)pgIdx >= 0; pgIdx--) { const String *const archiveId = infoPgArchiveId(infoArchivePgData, pgIdx); StringList *const localBackupRetentionList = strLstNew(); for (unsigned int archiveIdx = 0; archiveIdx < strLstSize(listArchiveDisk); archiveIdx++) { // Is there an archive directory for this archiveId? If not, move on to the next. if (strCmp(archiveId, strLstGet(listArchiveDisk, archiveIdx)) != 0) continue; const unsigned int archivePgId = cvtZToUInt(strrchr(strZ(archiveId), '-') + 1); // From the global list of backups to retain, create a list of backups, oldest to newest, associated with // this archiveId (e.g. 9.6-1), e.g. If globalBackupRetention has 4F, 3F, 2F, 1F then // localBackupRetentionList will have 1F, 2F, 3F, 4F (assuming they all have same history id) for (unsigned int retentionIdx = strLstSize(globalBackupRetentionList) - 1; (int)retentionIdx >= 0; retentionIdx--) { for (unsigned int backupIdx = 0; backupIdx < infoBackupDataTotal(infoBackup); backupIdx++) { InfoBackupData backupData = infoBackupData(infoBackup, backupIdx); if (strCmp(backupData.backupLabel, strLstGet(globalBackupRetentionList, retentionIdx)) == 0 && backupData.backupPgId == archivePgId) { strLstAdd(localBackupRetentionList, backupData.backupLabel); } } } // If no backup to retain was found if (strLstEmpty(localBackupRetentionList)) { const InfoPgData currentPg = infoPgDataCurrent(infoArchivePgData); // If archive-expire-before is set, remove only WAL earlier than the requested segment, keeping the // segment and everything after it (still required to restart recovery). Model the boundary as a single // open range [walExpireBefore, NULL] so the shared expiration logic keeps all WAL >= the segment. if (walExpireBefore != NULL && currentPg.id == archivePgId) { List *const archiveRangeList = lstNewP(sizeof(ArchiveRange)); ArchiveRange archiveRange = {.start = strDup(walExpireBefore), .stop = NULL}; lstAdd(archiveRangeList, &archiveRange); expireArchiveId(archiveId, archiveRangeList, NULL, repoIdx); } // Otherwise, if this is not the current database, delete the whole archive directory. The current DB // archive directory is the live destination for archive-push and must not be deleted. else if (currentPg.id != archivePgId) { String *fullPath = storagePathP( storageRepoIdx(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s", strZ(archiveId))); LOG_INFO_FMT( "%s: remove archive path %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(fullPath)); // Execute the real expiration and deletion only if the dry-run option is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) storagePathRemoveP(storageRepoIdxWrite(repoIdx), fullPath, .recurse = true); } // Continue to next directory break; } // If we get here, then a local backup was found for retention StringList *const localBackupArchiveRetentionList = strLstNew(); // From the full list of backups in archive retention, find the intersection of local backups to retain // from oldest to newest for (unsigned int globalIdx = strLstSize(globalBackupArchiveRetentionList) - 1; (int)globalIdx >= 0; globalIdx--) { for (unsigned int localIdx = 0; localIdx < strLstSize(localBackupRetentionList); localIdx++) { if (strCmp( strLstGet(globalBackupArchiveRetentionList, globalIdx), strLstGet(localBackupRetentionList, localIdx)) == 0) { strLstAdd(localBackupArchiveRetentionList, strLstGet(localBackupRetentionList, localIdx)); } } } // If no local backups were found as part of retention then set the backup archive retention to the newest // backup so that the database is fully recoverable (can be recovered from the last backup through pitr) if (strLstEmpty(localBackupArchiveRetentionList)) { strLstAdd( localBackupArchiveRetentionList, strLstGet(localBackupRetentionList, strLstSize(localBackupRetentionList) - 1)); } // Get the data for the backup selected for retention and all backups associated with this archive id List *const archiveIdBackupList = lstNewP(sizeof(InfoBackupData)); InfoBackupData archiveRetentionBackup = {0}; for (unsigned int infoBackupIdx = 0; infoBackupIdx < infoBackupDataTotal(infoBackup); infoBackupIdx++) { const InfoBackupData archiveIdBackup = infoBackupData(infoBackup, infoBackupIdx); // If this is the backup selected for retention, store its data if (strCmp(archiveIdBackup.backupLabel, strLstGet(localBackupArchiveRetentionList, 0)) == 0) archiveRetentionBackup = infoBackupData(infoBackup, infoBackupIdx); // If this is a backup associated with this archive Id, then add it to the list to check if (archiveIdBackup.backupPgId == archivePgId) lstAdd(archiveIdBackupList, &archiveIdBackup); } // Only expire if the selected backup has archive data - backups performed with --no-online will // not have archive data and cannot be used for expiration. if (archiveRetentionBackup.backupArchiveStart != NULL) { // Get archive ranges to preserve. Because archive retention can be less than total retention it is // important to preserve archive that is required to make the older backups consistent even though they // cannot be played any further forward with PITR. const String *archiveExpireMax = NULL; List *const archiveRangeList = lstNewP(sizeof(ArchiveRange)); // From the full list of backups, loop through those associated with this archiveId for (unsigned int backupListIdx = 0; backupListIdx < lstSize(archiveIdBackupList); backupListIdx++) { const InfoBackupData *const backupData = lstGet(archiveIdBackupList, backupListIdx); // If the backup is earlier than or the same as the retention backup and the backup has an // archive start if (strCmp(backupData->backupLabel, archiveRetentionBackup.backupLabel) <= 0 && backupData->backupArchiveStart != NULL) { ArchiveRange archiveRange = { .start = strDup(backupData->backupArchiveStart), .stop = NULL, }; // If this is not the retention backup, then set the stop, otherwise set the expire max to // the archive start of the archive to retain if (strCmp(backupData->backupLabel, archiveRetentionBackup.backupLabel) != 0) archiveRange.stop = strDup(backupData->backupArchiveStop); else archiveExpireMax = strDup(archiveRange.start); LOG_DETAIL_FMT( "%s: %s archive retention on backup %s, start = %s%s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(archiveId), strZ(backupData->backupLabel), strZ(archiveRange.start), archiveRange.stop != NULL ? zNewFmt(", stop = %s", strZ(archiveRange.stop)) : ""); // Add the archive range to the list lstAdd(archiveRangeList, &archiveRange); } } // If --archive-expire-before is set, preserve everything from the requested segment up to the retention // backup start, so that only WAL strictly before the segment is eligible for removal. The condition // walExpireBefore < archiveExpireMax avoids an inverted range and skips the case where the segment is // newer than the retention backup start (nothing extra to preserve there). archiveExpireMax is always // set here: the retention backup is in archiveIdBackupList and has an archive start (checked above). if (walExpireBefore != NULL && strCmp(walExpireBefore, archiveExpireMax) < 0) { ArchiveRange archiveRange = { .start = strDup(walExpireBefore), .stop = strDup(archiveExpireMax), }; lstAdd(archiveRangeList, &archiveRange); } // Expire WAL for this archive id according to the retention ranges computed above expireArchiveId(archiveId, archiveRangeList, archiveExpireMax, repoIdx); // Look for history files to expire based on the timeline of backupArchiveStart const String *const backupArchiveStartTimeline = strSubN( archiveRetentionBackup.backupArchiveStart, 0, 8); const StringList *const historyFilesList = strLstSort( storageListP( storageRepoIdx(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s", strZ(archiveId)), .expression = WAL_TIMELINE_HISTORY_REGEXP_STR), sortOrderAsc); for (unsigned int historyFileIdx = 0; historyFileIdx < strLstSize(historyFilesList); historyFileIdx++) { const String *const historyFile = strLstGet(historyFilesList, historyFileIdx); // Expire history files older than the oldest retained timeline if (strCmp(strSubN(historyFile, 0, 8), backupArchiveStartTimeline) < 0) { // Execute the real expiration and deletion only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { storageRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(archiveId), strZ(historyFile))); } LOG_INFO_FMT( "%s: %s remove history file %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(archiveId), strZ(historyFile)); } } } } } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Remove expired backups from repo ***********************************************************************************************************************************/ static void removeExpiredBackup(const InfoBackup *const infoBackup, const String *const adhocBackupLabel, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(STRING, adhocBackupLabel); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Get all the current backups in backup.info - these will not be expired const StringList *const currentBackupList = strLstSort(infoBackupDataLabelList(infoBackup, NULL), sortOrderDesc); // Get all the backups on disk const StringList *const backupList = strLstSort( storageListP( storageRepoIdx(repoIdx), STORAGE_REPO_BACKUP_STR, .expression = backupRegExpP(.full = true, .differential = true, .incremental = true)), sortOrderDesc); // Initialize the index to the latest backup on disk unsigned int backupIdx = 0; // The latest backup on disk may be resumable or in-progress (has a backup.manifest.copy but no backup.manifest). A backup // in this state must not be removed by default: it may be a crashed backup that the next backup will resume or clean, or it // may be actively running on another host (e.g. backup runs on the primary while expire runs on a standby). Removing it in // the latter case corrupts the in-progress backup. Only the latest backup can be in this state. if (!strLstEmpty(backupList)) { const String *const manifestFileName = strNewFmt( STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(strLstGet(backupList, 0))); const String *const manifestCopyFileName = strNewFmt("%s" INFO_COPY_EXT, strZ(manifestFileName)); // If the latest backup is resumable (has a backup.manifest.copy but no backup.manifest) if (!storageExistsP(storageRepoIdx(repoIdx), manifestFileName) && storageExistsP(storageRepoIdx(repoIdx), manifestCopyFileName)) { // Protect the resumable/in-progress backup by default backupIdx = 1; // During adhoc expiration the resumable backup may be a dependent of the label being expired, in which case it // should be removed along with it -- but only when its ancestor no longer exists in backup.info if (adhocBackupLabel != NULL && strBeginsWith(strLstGet(backupList, 0), strSubN(adhocBackupLabel, 0, 16))) { const Manifest *const manifestResume = manifestLoadFile( storageRepoIdx(repoIdx), manifestFileName, infoBackupCipherSpec(infoBackup)); // If the ancestor of the resumable backup no longer exists in backup.info then it can be removed if (!infoBackupLabelExists(infoBackup, manifestData(manifestResume)->backupLabelPrior)) backupIdx = 0; } } } // Remove non-current backups from disk for (; backupIdx < strLstSize(backupList); backupIdx++) { if (!strLstExists(currentBackupList, strLstGet(backupList, backupIdx))) { LOG_INFO_FMT( "%s: remove expired backup %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(strLstGet(backupList, backupIdx))); // Execute the real expiration and deletion only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { storagePathRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_BACKUP "/%s", strZ(strLstGet(backupList, backupIdx))), .recurse = true); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Remove expired backup history manifests from repo ***********************************************************************************************************************************/ static void removeExpiredHistory(const InfoBackup *const infoBackup, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); MEM_CONTEXT_TEMP_BEGIN() { if (cfgOptionIdxTest(cfgOptRepoRetentionHistory, repoIdx)) { // Get current backups in backup.info - these will not be expired const StringList *const currentBackupList = strLstSort(infoBackupDataLabelList(infoBackup, NULL), sortOrderDesc); // If the full backup history manifests are expired, we should expire diff and incr too. So, format the oldest full // backup label to retain. const time_t minTimestamp = time(NULL) - (time_t)(cfgOptionIdxUInt(cfgOptRepoRetentionHistory, repoIdx) * SEC_PER_DAY); const String *const minBackupLabel = backupLabelFormat(backupTypeFull, NULL, minTimestamp); // Get all history years const StringList *const historyYearList = strLstSort( storageListP( storageRepo(), STRDEF(STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY), .expression = STRDEF("^2[0-9]{3}$")), sortOrderAsc); for (unsigned int historyYearIdx = 0; historyYearIdx < strLstSize(historyYearList); historyYearIdx++) { // Get all the backup history manifests const String *const historyYear = strLstGet(historyYearList, historyYearIdx); // If the entire year is less than the year of the minimum backup to retain, then remove completely the directory if (strCmp(historyYear, strSubN(minBackupLabel, 0, 4)) < 0) { LOG_INFO_FMT( "%s: remove expired backup history path %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(historyYear)); // Execute the real expiration and deletion only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { storagePathRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY "/%s", strZ(historyYear)), .recurse = true); } } // Else find and remove individual files else if (strEq(historyYear, strSubN(minBackupLabel, 0, 4))) { const StringList *const historyList = strLstSort( storageListP( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY "/%s", strZ(historyYear)), .expression = strNewFmt( "%s\\.manifest\\.%s$", strZ(backupRegExpP(.full = true, .differential = true, .incremental = true, .noAnchorEnd = true)), strZ(compressTypeStr(compressTypeGz)))), sortOrderDesc); for (unsigned int historyIdx = 0; historyIdx < strLstSize(historyList); historyIdx++) { const String *const historyBackupFile = strLstGet(historyList, historyIdx); const String *const historyBackupLabel = strLstGet(strLstNewSplitZ(historyBackupFile, "."), 0); // Keep backup history manifests for unexpired backups and backups newer than the oldest backup to retain if (!strLstExists(currentBackupList, historyBackupLabel) && strCmp(historyBackupLabel, minBackupLabel) < 0) { LOG_INFO_FMT( "%s: remove expired backup history manifest %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(historyBackupFile)); // Execute the real expiration and deletion only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { storageRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt( STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY "/%s/%s", strZ(historyYear), strZ(historyBackupFile))); } } } } else break; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Rotate the archive key when it is older than the rotation period ***********************************************************************************************************************************/ static void expireArchiveCipherRotate(const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); if (cfgOptionIdxTest(cfgOptRepoCipherRotate, repoIdx)) { MEM_CONTEXT_TEMP_BEGIN() { InfoArchive *const infoArchive = infoArchiveLoadFile( storageRepoIdx(repoIdx), INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); // Format < 6 has a single archive key that cannot be rotated. Warn only when the option is not the default. if (infoArchiveFormat(infoArchive) < REPOSITORY_FORMAT_6) { if (cfgOptionIdxSource(cfgOptRepoCipherRotate, repoIdx) != cfgSourceDefault) { LOG_WARN_FMT( "option '%s' requires repository format %d or higher", cfgOptionIdxName(cfgOptRepoCipherRotate, repoIdx), REPOSITORY_FORMAT_6); } } else { const time_t timeNow = time(NULL); if (infoArchiveCipherRotateTime(infoArchive) < timeNow - (time_t)(cfgOptionIdxUInt64(cfgOptRepoCipherRotate, repoIdx) / MSEC_PER_SEC)) { infoArchiveCipherRotate( infoArchive, cipherSpecGen(cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), infoArchiveFormat(infoArchive)), timeNow); // Store the new archive info only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { infoArchiveSaveFile( infoArchive, storageRepoIdxWrite(repoIdx), INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); } LOG_INFO_FMT( "%s: rotate archive key to id %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(cipherSpecMapIdCurrent(infoArchiveCipherSpecMap(infoArchive)))); } } } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cmdExpire(void) { FUNCTION_LOG_VOID(logLevelDebug); // Test for stop file lockStopTest(); MEM_CONTEXT_TEMP_BEGIN() { // Initialize the repo index unsigned int repoIdxMin = 0; unsigned int repoIdxMax = cfgOptionGroupIdxTotal(cfgOptGrpRepo) - 1; // If the repo was specified then set index to the array location and max to loop only once if (cfgOptionTest(cfgOptRepo)) { repoIdxMin = cfgOptionGroupIdxDefault(cfgOptGrpRepo); repoIdxMax = repoIdxMin; } // Get the backup label if specified const String *adhocBackupLabel = NULL; bool adhocBackupFound = false; // Check if this run should expire the --oldest backup set const bool expireOldest = cfgOptionValid(cfgOptOldest) ? cfgOptionBool(cfgOptOldest) : false; // If the --set option is valid (i.e. expire is called on its own) then check the label format if (cfgOptionTest(cfgOptSet)) { adhocBackupLabel = cfgOptionStr(cfgOptSet); // Conflict check with the --oldest option if (expireOldest) THROW(OptionInvalidError, "--oldest and --set cannot be used together"); // If the label format is invalid, then error if (!regExpMatchOne(backupRegExpP(.full = true, .differential = true, .incremental = true), adhocBackupLabel)) THROW_FMT(OptionInvalidValueError, "'%s' is not a valid backup label format", strZ(adhocBackupLabel)); } // If --archive-expire-before is set, the value must be an exact WAL segment name. Match the strict segment expression // rather than walIsSegment(), which also accepts a .partial suffix that would shift the removal boundary by a segment. if (cfgOptionTest(cfgOptArchiveExpireBefore) && !regExpMatchOne(WAL_SEGMENT_REGEXP_STR, cfgOptionStr(cfgOptArchiveExpireBefore))) { THROW_FMT( OptionInvalidValueError, "'%s' is not a valid WAL segment name for option '--archive-expire-before'", strZ(cfgOptionStr(cfgOptArchiveExpireBefore))); } // Track any errors that may occur unsigned int errorTotal = 0; for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { // Get the repo storage in case it is remote and encryption settings need to be pulled down const Storage *const storageRepo = storageRepoIdx(repoIdx); InfoBackup *infoBackup = NULL; // Is full retention time-based? bool timeBasedFullRetention = cfgOptionIdxSeq(cfgOptRepoRetentionFullType, repoIdx) == CFGOPTVAL_REPO_RETENTION_FULL_TYPE_TIME; TRY_BEGIN() { // Load backup.info for this repo infoBackup = infoBackupLoadFileReconstruct(storageRepo, INFO_BACKUP_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); // When expire oldest requested, expire the oldest full backup by lowering retention for this execution if (expireOldest) { // Count full backups in this repo (we want to expire the oldest one) const unsigned int fullLstSize = strLstSize(infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true))); // If there is more than one full, temporarily keep one fewer if (fullLstSize > 1) { // If time-based retention is configured, force count-based behaviour for this run if (timeBasedFullRetention) { LOG_INFO_FMT( "repo%u: expire oldest overrides configured time-based retention with count-based retention for" " this execution", repoIdx + 1); timeBasedFullRetention = false; } cfgOptionIdxSet(cfgOptRepoRetentionFull, repoIdx, cfgSourceParam, VARINT64(fullLstSize - 1)); // Also lower archive retention so WAL for the expired backups will be removed cfgOptionIdxSet(cfgOptRepoRetentionArchiveType, repoIdx, cfgSourceParam, VARSTRDEF("full")); cfgOptionIdxSet(cfgOptRepoRetentionArchive, repoIdx, cfgSourceParam, VARINT64(fullLstSize - 1)); LOG_DETAIL_FMT( "repo%u: expire oldest enforcing %s=" CFGOPTVAL_REPO_RETENTION_FULL_TYPE_COUNT_Z ", %s=%u, %s=" CFGOPTVAL_DETAIL_LEVEL_FULL_Z ", and %s=%u for this execution", repoIdx + 1, cfgOptionIdxName(cfgOptRepoRetentionFullType, repoIdx), cfgOptionIdxName(cfgOptRepoRetentionFull, repoIdx), fullLstSize - 1, cfgOptionIdxName(cfgOptRepoRetentionArchiveType, repoIdx), cfgOptionIdxName(cfgOptRepoRetentionArchive, repoIdx), fullLstSize - 1); } else LOG_WARN_FMT("repo%u: expire oldest requested but no eligible full backup to expire", repoIdx + 1); } // If a backupLabel was set, then attempt to expire the requested backup if (adhocBackupLabel != NULL) { if (infoBackupLabelExists(infoBackup, adhocBackupLabel)) { adhocBackupFound = true; expireAdhocBackup(infoBackup, adhocBackupLabel, repoIdx); } // If the adhoc backup was not found and this was the last repo to check, then log a warning but continue to // process the expiration based on retention if (!adhocBackupFound && repoIdx == repoIdxMax) { LOG_WARN_FMT( "backup %s does not exist\nHINT: run the info command and confirm the backup is listed", strZ(adhocBackupLabel)); } } else { // If time-based retention for full backups is set, then expire based on time period if (timeBasedFullRetention) { // If a time period was provided then run time-based expiration otherwise do nothing (the user has already // been warned by the config system that retention-full was not set) if (cfgOptionIdxTest(cfgOptRepoRetentionFull, repoIdx)) { expireTimeBasedBackup( infoBackup, time(NULL) - (time_t)(cfgOptionIdxUInt(cfgOptRepoRetentionFull, repoIdx) * SEC_PER_DAY), repoIdx); } } else expireFullBackup(infoBackup, repoIdx); expireDiffBackup(infoBackup, repoIdx); } // Store the new backup info only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { infoBackupSaveFile( infoBackup, storageRepoIdxWrite(repoIdx), INFO_BACKUP_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); } // Remove all files on disk that are now expired removeExpiredBackup(infoBackup, adhocBackupLabel, repoIdx); removeExpiredArchive(infoBackup, timeBasedFullRetention, repoIdx); removeExpiredHistory(infoBackup, repoIdx); // Rotate the archive key when it is due expireArchiveCipherRotate(repoIdx); // Check the oldest retained backup for page checksum errors and issue a warning if any if (infoBackupDataTotal(infoBackup) > 0) expireChecksumErrorCheck(infoBackup, infoBackupData(infoBackup, 0).backupLabel, repoIdx, false); } CATCH_ANY() { LOG_ERROR_FMT(errorTypeCode(errorType()), "%s: %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorMessage()); errorTotal++; } TRY_END(); } // Error if any errors encountered on one or more repos if (errorTotal > 0) THROW_FMT(CommandError, CFGCMD_EXPIRE " command encountered %u error(s), check the log file for details", errorTotal); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/expire/expire.h0000664000175100017510000000126515265101745020554 0ustar00runnerrunner/*********************************************************************************************************************************** Expire Command ***********************************************************************************************************************************/ #ifndef COMMAND_EXPIRE_EXPIRE_H #define COMMAND_EXPIRE_EXPIRE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Expire backups and archives FN_EXTERN void cmdExpire(void); #endif ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792312410.703075 pgbackrest-2.60.0/src/command/help/0000775000175100017510000000000015265102133016527 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7049444 pgbackrest-2.60.0/src/command/help/help.auto.c.inc0000644000175100017510000017564215265102133021357 0ustar00runnerrunner/*********************************************************************************************************************************** Help Data Automatically generated by 'build.py help' -- do not modify directly. ***********************************************************************************************************************************/ VR_NON_STRING static const char helpData[22445] = "BZh91AY&SYH\222\207\377\000\001\373\177\377\377\377\377\377\377\377\377\377\277\377\377\376\377\377\377\377\?.\177\361\236\273C|\3\ 77\277\307-\333\3769c\340y]\302\037C\207\252}h\364\301\022\b\252P\020\225IA\263E\002\n\220\250\221B\225*\225\000\244\001@\245J\212R\ \020\333J)!%\002R\251\024'Jh\320j\251B\320\030j}\r>\351\200\350\037{u\342\363\"\333\355\327j6\213D\334\344\267U\264\016\355\322\212\ heT\245m\215\023\336\a@hPh\240\304jt`\350\016t\212\266\321\202c8]\357wm\334\350b\001F\251\204\201\"S+\3307f\225\317\016\347\206\264\ \264t\320r\256\354\000qk\262\b\254\240t;L\024\325\032\030@9I\246\222i@\325\000Zi\026\320U\254\006\332cZ\251\224nz\001\350\241\251;\\ 262k\333\000\322\265\255q@\000\000\207Y\330\025g\a\t\334\343R\256\203f\333H\313\204\333\33404\356\203M\004\002hM\002\b&\211\352M\22\ 0M\352\223\324f\210d\000\000\320\000\000\000\000\r\000\000\000\000\000\000z\200z\2004\323 z\203!\265\006\230\202 \204M\020Shi2H\036\ \240\320\365\0324\032\000\000\000\000\000\000\000\032\r\000\000\000\000\000\000\000\r\000\000\00054\205&\365M=OiG\2502h\365\032i\34\ 4F\324~\244\36521=@\310\003M\031\001\246\200\006\206\200\0004\000\0004\3656\240\000\365\000\036\241\344\201\246#\3226MO\325==4PBRJi\ !\223'\251\r\017P4\003@\310h\036\240d\000\032\000\001\240h\323#F@\000\000\000\000\000\000\000\000\000\000\0004\b\222 A4\320LM\003M&\ )\372S\021\264&&Dz\247\246\214\236\250yOS\324\336\251\352z\232h\332\233I\264'\250\0004\000\000\000\000\000\003@\000\000\017$\364\32\ 2z\200B\221\020@A1\0051\246\212g\244\236\232SjmG\244\000\364\201\352\000\000\006\200\031\000\000\000\000\000\000\000\000\000\000\00\ 0\000\031\a\264\314'\232\2375R\327\250\272G\316^\234\354\364\tG\024\364gx\364^\217\350\272F\264\210\266\273=\336\331$\223N\3568\206\ \341\215}i\034<\344\375:>\346\177\355h\377\ \340w\317\241\264\210\274{i\030\227\212\227h\201\320z\364 \253\357\266\237\214\275\223[\245\276\315\355bY\322%\220\263\233Zs\r\215\ \307\341\351\022\242UF\177wn\004),Z;\207xo.\267\356\026\023-\n,:77L\242R\377\235\2573s\026\344\031\322M\243\265b4\346-r\315\347s\36\ 3N\251\363Q\333N\324b\025^D\231\246^\247\377\275k\377g\322\332\r\017\251\3477\b\203\226\213\333\254\362\3501\233e\242\237Q\267\016`\ ;\375|\177o\212\371.\277L@\371\324\373\373\362\344W\323\302iU\364\360\a\325\037Bg^\233\037=\3527\374s\267OX\314\375\033\034y\363\37\ 0\340q\203\227)\202\b\216v\343\271\267\215\272\350\304PQ'M5g2\277\233\002\201\353\363u\264\221\360\203rn\003\342C\250(\321\266\030\\ 212\247E\032\333QM7n\?-\330\336\253\217o\341\242\bif\365\033y@\036\030\347\f\344\203\277.`\374\212,\313\330\241-\344\366\206\212.\3\ 66`1\212\032(>\003b\222\226\"\222\222\205\3525\340\374\257-\340\362\377\363p\350\242\214\023\365\356~\223}\222\252\017\330\375\230\\ 331\264\?e]\033\316\033\a\373\r!$L\252\356\343Z\001\253\327\322$\246\203\355\346\036\240\233\375\306\?gY\v$\312\236.\323*!G\321\250\ \243\244\n.\272\377{\377s\354\t2\304\260=\243\350\a\341\324A\372\003.\017d\3250\223K\245\023TXR\363\247R\273\344Q\265\021\325\325k\\ 376\033\302G\275\207\323\350I\005\307\334\322\305\320IW\f\275\001\200\373_\210\217\?%\367\017t7\351T\\\247\002'\323\313\240\001\332\ \023\200pA\240\241h\370w\273\331:\tinH\344A\201\377/\243C\364I\254V\031\246E9\270g7h\?\207\372`}\220a\216\207c6\240|\fw\357\367\342\ \362\207\306\276v\240u\354\256a%3T\025H\260\365\314y-G\324\220\222!\031\024\241\231\3005#YCT\003\232\362\227=\245\215\241*\212\254C\ \a\\\305\221\205\244\025\351\0178@\212P\213_S\231\201\205ca\261\266\373\217\373l\304\027l\025\276\320\322\325[f\245\v\201}\261\200`\ \217q}\034n,\240{\324\tN\346\236\273\353\331\207\026\235\316\b\r\246.\367E'7#\317\311\016\351@\370 ,Q\036\376\224\vT=LM/\320\326\27\ 2\311\277X\216=!\232\206\312\263z\341\037\023\200\342\341B\215\345U\335\224\245\017B\304\236K\342\033\233\370\b\316\367\323_\373V\0\ 22\324\001\312\t:\277\025N\340K\325\016\vo>\332\242\200\221R\214\373\300dq\265tx@\313\r\321F\311\366\353\366\240\370\016\352\221\32\ 3a\331\033\240;`\a\3060\226\205\252\363\262A)\244N\337M\357>sy\a\003\255e\344WV\225)\341\002S;tf#]\2242\221'y\036@\221@dgph\372L\25\ 5\355h\317%G0(\330~\366\\\001PIAE^G:o4I\347n\257\277S\335i\203\036PT\364~S\3076\261{r=\310\310'Y9\272\343\024\301\302\372\342\204I\\ 222\337\277\201S\255\213\221\200\220\177\3746%)\345\357\315]\335\217KZjk_RJ\363x8\235\301\v\336&dB\b!\006Y\231\003\222Q\346*\310\006\\\ UZ\026U\213!\021sa0\020#t\337\347}\366\353\341Y\230\032\373}K\037\243Juo\277W\335\323\342\a\n\215\032\"\215\034\252\212\363\370\327\ \0245p\216\250\357\016\374I\006\256\026j\220[\017\361\347\242\n\2351Dz%\000UQ\325\3247\202\344o\305\354\3677\342\352(q\334\340\242\\ f#\323\241\325\351q\233\371\325\360\252\006\032\027P\245!\227\264)\300\230\353\2448\370\274\256\377fn2\017\333\006\030\006\240\372\\ 356\240la\221\350\201^\366\340\265@\261\203\320v9\n\021\366}}rL(\371&\f_V\342\315Kh\311\021E\323q\366XPA\211 ]\254\335S\325F\210\24\ 6AS\"\212$\253z\026Sb\376\221\305\332\0236j1\330\263\367v\032\202\016\006)]\245\322\376\375\265\345#\362\224v\"\330RrF\354\024\021\\ 3142\243\024$\020C\t\002\bC\362\372\231\217\222 \220rP\371\367\362vLRV\222\317(u\027\3400\250\006B\214|\201n\200\354n\362\323\232\2\ 62\241z\207i\317L\205\234\006>\225W\251\346\352&\220\307k\340\273\332\341|tX\332\330\017\030Q\312\336E\253\321f\252\347C\230\327\23\ 1\224\344\240\211\255\325K;\212\271G \370+\357=\244\202\347\307\f>\327\226`\tz6\263b\272\335\3035\261\264\t\033\251\032\2628\350\02\ 6\246\310~\003\202\202\205J\342\372>\204\220F\2770|\214I\230\323\024\024\350\234\323\305\035\017%\325D0^\322i\276\241z\030m5\0279\2\ 24{\307\f\211i\213\217\003\273\311\274\321\367\313VO\315\217\005 \236\003\216\035\216~&\274y\266g\260\341\002\310\373.@\245\025\352\ 29\277\202I\025\3202\a\r\3144u\016\374\031Q\006\017\303\337#^e\024\326\211;\310Va\245\r\256\254]^\343\304\326\001<\275\3757\032m\22\ 2\345*\373\f\016W\247\262\005\240w\244\307\325\037\"\365\201cQ\343\255\364dd\021\\0:\2454\2741'{\330\021\354\273\361\303\213\302\24\ 4\204\a\003g\276\304\267\240\270^\314c9|\323\"\253L#a\276\004\250\027g\240P\261>L\213\205Z\032r\v\250\006\336\023\036\200\264E\224\\ 374\357\260\242T\241\"H3u\363\335Z\204XvF\003dG\000Gq\366\fj\f{\366`_\340\034\252s\324\365\227\223H\300\n\030\215\213\205|\222\226{\ \315\?=\241\022F\3477*\327Z\f\274H\224\337\037\206Uv\017X\024Q\211[b_\022\a\322\203\2448\360\231bm\211\224:\030\231\0262t\220\322\0\ 34f\376\254\334\376\277\244\035\326H\223\036\227\215\302\a\3260_\2163\236\205\347\rj\3124\n\004\216O6\2745c\243\366\303\216\353Z\32\ 6r3\300\362\306\?\266\356PFa\023\t\016J\026x\313\037\243\337\265\370\256\265\177\235\342\224\343\323\257\032\306\372\255\235D\213\0\ 26E_:^\024\032\225\"\215=\260+\004L\343\252\205\355\243\002B\\\200\000\220\372!J\255\227R\270s\216\332862)=\356j\346\302\222\016@\2\ 41BV\301\334\aH%\320\265\\\016\267\0177\2231\272\201JZ\022\334\?\030\220a3AK\227\030\263F$\276,\341\305\005Z\266\200\025}\r$9ZM\346\ o2%\032\255\251J\316\270\234r\f\240\245\324\016:=jj\333\331{\221\357\021(V\021\365\252P\207\220m\350 \227^\244O>\3273\006\333\235\n\ \277\201V\203\364\357\364\235\314\003~&&\365e$T\314\303*\317\216\300@\357\242\2205[lke\220\rZ\017>\320\227\033\034\366\243\363lf\37\ 1^\341zc\235\344\316x\364\034z\343\b\025\030c\265\254\231\345~#>\350)V2i/\276\355\327\002\020\036\247\235\231\276\277\b\343\215\363t(\306\242kWH]\321R(\\|\201\374bG\346;\355y/\217\032\03\ 1\360\256\354\240\a\271\000}\336\350R\035\371\324\017=c\221\223\"\302\342G\025^\243b\3409\362\026\353\341l\021RJQ\t\300\325o\337ZT\\ 322\363\036$\021%\027\005\222h\021\002\310(T\371\367\272\a\353\241lF\304$7\356\240\317\273\000\246\2332@\337\220gB\233\354\251WEH\2\ 56\\T\362\227\243\306\263\261-\317\244\303\224\201V\232\246\304e\243\0223'\221\331\227\257\024a7\200,\034G;\006D\364\322&j\334\356\\ 260Q\002\311I\002~o}\211\bl\314\034\234h\020gx*\276U\353\235:L.\374\254\245\367\302\247\241\365\034j\326\023\236tV\373M\031\212\032\ \r\310`\252\212\252\307\313rj8~\375\335\0350iy\251`\230\002\222n\206\032\036\240\260D\245\347i\260\265i`9\331\202\370\317\v\270Iu\2\ 34\262\250\344\025\\I]W\211/\334k!\036\032MA\261c\306\316\314\273\305\366\034\306\016\024-\210\207r\253\254jp0\364g\371\332\271\223\ \031l\f\225{\b3+\b\373\fQ\347\355a\370\231\335\006u\342\231\260\226\3011\ae\204\277f\003\005%\235\332Ht\204\231\001v2f\211\031\231\\ 034y\t\220*\016b\312\252\220\201\200\215\021\346U\244\t\b\214\322\211\004A2\016\201\f\0355\342`I\000X\371\234\211\031\202\377@\253\\ 036\265\334\226@C\252qjF\2106,XV\2200\207\207\f\325\3722y2\377KXw\307\2471\326\006zT\r\357\376}H\346\200\005h\344\242Y\237\242\235\\ 035\346\tU\t\300\305/\330\n\336\271_f\351-\202\005\311C\356+\3470\a$\0334\364\330`\224\2036\304\220\361)\035\\L\317z\035\356\2433L\\ 374P\036\207\027g\355#z\234\272\331\024lR\fXz\nA\305|7\216p5\?\2473S\274\215mM\244\032^\224\0058\032\004q\223\360O\024f2\342\235{VJ\ \251\205\0275'2D\313(\235\232ND\326\b\025\354\325\264\253QA\235!\317\322xNOF\317\r\202\\@\332*\000\340\206b\231\231P2\240'\333\310:\ \265\242\005\220>\255b`)\204)\250\fX\367\236\0350I\315d\200`\352z\331b\017\256\242\226\3506\274\311\036\221\277+\352\324I\t\373yiX\\ 216S\333S\355\223\316eXhy\bP\346\333-xDQYw\355i\336bW\006\r\335\244\244\003\270)$8Z8}[\233\022\r\341\2648\265h\003\217\210*,\210\27\ 2\002\324Q\030\240f7\032X6\377\030\265*\320\245\201\265\272\t\222+\277&\245\003\275%\021 6v@\025\375\357\027l(\363\f8\001\206B\201\\ 305\247\002\t\252v\223\314\246\225d\330h\302\221Q0I(\230\220\365\340s\332GR\203\211\221\222\"f$\024\020lA!J\310+\336\260&`{\332\340\ kH\030\236\314\201\301\300\340\263O M\334\?zs\024\200\032\031\032\24356\3314\327ue>,O\017\024\322\340\355V\266\214\322\004:\254\301\ \nF\345\365\230\362\3079s\300\260 0#u\354\323\223\335\325@\3130Q\216\331\221\353\240\317\307<969\003\200\205\023c\344\030\310\022\2\ 40\336&@0>\321\325r{\031\310rjH\016Lc\343#\t\236\2033mk\303b\301VX{5\036V\304\026\347G\fU\356\300z\0172o\031\315yUz\210\346\301\315\ \205\202\205\036\352\006\327\222\215AJn\025yg\302\263\346/@%\t\001\213.\345\312&\204\023[\t`s\022\364\203\245\252V\257k\213\202\201\ y\235\242\242:\210\026z\223\025\315\037\302R\021;\312\v\300\345f\235a\n$\354\262\346\022\321\264\200QNe;\217\005\tS\034\214\334\r\b\ \302\277=\244\372\226\231\035\313OQSF\213\346\316\356\026\226T\251\272\272\275\201\211\032\203bT\221\033\334\241\324\316\243\341\30\ 7G\365\270\020p6s\230<\335\371\371H\324\335\005E\241a\254\225\306m4\031\t@\276H\205\265\350\3265Z\256\316\217\204P\315e\273\200\340\ |\220\025\005\212/\n\005\016\335\203\264\021\004\233B\206$\237\020\213S\333C\334\302#\fG&7\302.\020p%s\211\216\273c\2536\307\353\24\ 3\356P\202\216U,Si\263\bU\002\306Dt\364\357\230\0367\355\\%\201\"\311\326\3010\2234B\200\220S\306\307\330T\373\232\371\267\003c\321\ \bH\2228\022:\334\260\255\266\342\022C\223\vRW\034^\370\351oD}\r\362#\273\035\031\242z\356\371\352\024\205eG7\323V\347\262\320d\031\ \350.\230\224\234q\322\203W\251Y\215o \366\203\300PXo\311\362f\222\332`\360\227\255\347\275W\036\202\016\270l\244}\035\025\274\nx\2\ 50\205\345\351\250\254\244IDG\314\a5\2149\335\244\367\030j)\200\352Q-\3130\030Ir;J%\252H\204fTN\b\033\220\211q\022\020\327\2754\344\ P\315\207lr`\256D\226E\256F\346\316\235\203\324@_\223^\223\336}ZK4w\231\217C\2475G\240\205\311\301P=\261\v \221\244\331&3\351^\245\\ 233\217\265Qa8!\201f\022\\\n\002@\327^\327\233\206\365\312\224\350\030O\254\364\220\203\005w\356\374|7\321\275[\270\372X\v\340\035\\ 203W\036\367M\217D\v\036@\n\347kK\265\301\324\245e\262\306\317\326\\\350\217\ad\033\024\325\250\271=\260\363\340\203\?E\363<\372\22\ 3)\257\243\303I\312\033\n\237w\245\332vS;\267\240\314\315\306\037\377>\3577\274q\220W\310\355\324a\335J\a\347\300\353\351\213\311\0\ 33o\000!\360U\024\201\316b\216\343\261@\355\326n\001h\000T$\271!\362\242Hf9\361\000h\215\2126\302p!\375\201\017\271\373\262n\220\31\ 7\267~\224\311\200\312(\216\310\367\?\206J\003\262\000\344\305m\b\223\027\244z)\304\373\n=\327\273B\252D\231\364\214\217\3157\256\2\ 669/\005\2366\3720\371\316\a\002\241F\232\352e\302$\030\367\231\270\2035\ti\360b\035\331j\3401\"A\035\215\326\211\274[\177\203\347\\ 215/\262\246\026\231^\025\214\264\272\205\237\232a\302\222v\003\244\223\243h\032\350\026AEIHXDD\352@\277#\256D\240\a\"\306\214\241\\ b\216!\036F\217D\234\302\363`\303y\3550\266\as\223y\327V\351\352Y$\320l\250\244\337\002\025\322\321X\3022\265\034\335:\240\332\273\\ 274\332@f1\bh9\030\347\214\023Pf\257\036>V\217\262\f\367\347\234t6\200D!\306\324\310*+\336C\223\362,\003q\223\334\257\177~4<\235\27\ 4\360\345\032\362\225!#\316\377\003\016rQ\"K\230 V\220\210\266\347)\v\227V\321\305\272\324D\374r%\325\217N3yZ~6\021\364\257\306\265\ s\220\373\230\334p\226\252J\211\3262\235x\330\272\030I\b\205K\254\021\277||\035t\372>&\267\275\371\326$/)\332\260\016|]\302h\252k\\ 233\321\310\0220\252\365\234YF\213K\273\213C\244\243f\023\325Uu\306-N\210\311=\356w\260k\333\\z\003\306\243K[\234\033 \221\354\353h\ \272\371L\357\207\254\337\237\034^\306 ]\367*\230r\247\312\227\317\210\310\"p\362\217\252yC\201\203,#$\356;\2417\032\270j;\344\273\ :\214\v^\326\033\321)r\262\243\335\257\231D\227\026\270\033f`\\\223 \243Z,\350\235\025\265Y^t\336sg\205\341\370\250\220DIB\016\237\\ 033s\vqdt\203/!\231\342\244\250\225\322\234\004\354\025C\211\232\025L0u\265a\305\371\220Z\224\306O}\330\244\364\n\025-g\233!\351\251\005&\016+\36\ 5\232yfy!iS2\346\345H\006\242\3054\025\316\232L`\303\213U}\322{\0327VY((U\367\241p\341\302n\373|\314\360v\266\364\374\\\221\246<\03\ 5_\301\002\344\340\024\035\227\310\255qS\326\246\202\332\350[r\026N\360\261e|q\353kc\355\217=:\037\210\341\343\354\344\332\032kg\26\ 11\231\345\265,fQ\353\362X\004\203\250#b2<\025j\340\221\344\303\277jj\375\362\221\363\216\030\245\344\340\221\232\265t\357\303\344\\ 276\322\222\265l=\240\033\036S\006JArGRH\262:7_N&m\n\033\334\361T\353.\206\016\r\333g\024\326O\203U\035\233\f\224\217\025\220>\263o\260\254\223\262|\245\250\017v\3\ 71\331\322\005:q<\235\255F\337_\257m'\031\334:\vl~R\202\372\330%\205\356R\233\367m\331\361\320\335t\340\225X\314v*> r\017\?sYe\207\\ 001<\330O\277Q\242\205.\217s\205\225q)\374\270G\313\217\226\207\300\r\005'5\306dj\304\260\366<\352\252xQY\202\277\f8\225V\337W1f/\2\ 33A\225=DTd_\033\245bU\373{e\352\363i\257\276\374g\346\277[\366\347K\352k\2231p\316\035\361\356V\"\242\203\2477\326Z\373o\312\225\2\ 60\302\277D\225\352\253\325!y}\320o\306\270\250\230(\242b\320\350\024\2078.\b\210Q\306\375+\024\030-\327J\340\366zh]g\256\340%\264\\ 275\366=\355\303rn\263]4\311\v\360\364A\346\002\004\036\274\004\037Y\337\034p\26323]@zT\r\216G\220uf_}\323\277d\224\354D{\256\270\f\ \332\240\357$\356\230\016\006F\\\006\f\207j\205\rq\tPT:f\021\020\201\332\340\005#\263\212\355\236\363&3\346*$\004\244KS\261\037Vr\2\ 10D\206\341\214\260\244\326\253$;\361;\240\334\205\002\001\211\332H\212\003\312\316<\361>+LX\220\254\341\221\327y\365\271\240\250 \\ 000\241U!.\020r!,\210\023\200@\022\266\240B t\022\021\361\353n\372a\327\216C8\305\310A1\263\206`=\327\267\004\350\n\362H22\022}PS\3\ 33\261+A\214a!\352+\235\305\222Z\324n\2649rlX\264\026\331\237hQ\000\214\212c\301\265\223\266\204\277\301;\255\221\207\fv\364\307\02\ 3\323\315h\220sd\216\323\256\026`\304\235\271RE\342\\N\270|\003A\360\255\004\305\304\245r\307\244\204\237C}\003\317\324\b\233\256\0\ 21\265\257\235\207\353\274\303\357\251\27476\004\225\213\241>\237\342v\315\a^)\202LY\027\353c1\212\265\037\006[\025\016\242v\303=a)\ \302\340\236\362a\216\005\327\0172\202J#\223VJ\327\246) I\363Qj8\265\024\024\336=p\232\322v\315\250x\367\333\275h9m\312nG\a\313\235\ z\255\317\215\360f_\033\017:]\?\317Y\272\243\327H\200\\Y\255<\347\274\242)\022\246\204c\321\345>\231\2612L\245$W\222\036g#\231S:ah\\ t\254\2111%\213\035*\370\243` ;!\255\347%\177\273tw\211[A\277\273\3331:\f\251\236^\332\263;\352\264\fdL\302\310\306\276\2534\250\25\ 5&\243\225\024u~\263q\314\364\240I\2229\277'\233>\257\233\276\"\036\260\315=\005Z\223nQ\223|6\263\347s\301_\245:\035@\242\t\302\021\ _4 8#\211\270\355nu\221\256t\357\317\206\327\323/\b\375c\254\254\200\317V\263!\352_\025ck\333\273\345\373Ks\200`\r\361\320\372o\365\ \021#2~A6Pr\254\217\331\fM]\250S))G\342C\037`\323\3033\313\231\025\360[\235&\275\230\340\214\343\352w\202\233\341q\304\224x\230\315\ qS\255g\323\207Cz\311\224N\2703(\325\024,\214\261{\246z\276T\357\006\207\332W\033}\2764\320\330\353,\253\243<#\240\272\351\363\255.\ \372\256z\027\311\321\327\"\3032G&H}\\\315\344\253\221\320\312\303y\362R\316Bw\312C\213r4]K\03153\370\214\337\344\352n]`\?\247\361\\ 234R\267)\2625\212t\377SJ\247\334\260v\344*\v\363~~p\017\333\353\037G|\316B]\345eg\235E\b\365I\264\0179\022\025\030\321@e\026\347*R\ ^\1771\b\357\375\324\373\017\203]~p\365\r\256/p\255\235\035<\203\r\366\347\355\f\224\242@\367\372'4q\342:\240\364\352G\306\3316T\32\ 0@\035\001\337\240\320{>$\357\363H\355\270\343\345\231{\201\335\207\245X\037->\2447\254\220/q\355h\356\262\037L\a\303%7T\033\350\24\ 3BA\"\377]\267\204i\320\341\032\003\267|BW,\224a\254\234\266\365n}m^q\331\215\026\210\261\bga\305\335\353[\355X\253\005\301\027\360\ \006\336_:!\0002u\035A\"J\251\343\331\233Bf\350R\220\213A\304C8\a\\\a\335\250\211\305\321\335\362[\266\353u\237r\237w\253\000\317\2\ 73\341\340/E\006\277%\030\221\226\2446a\260v>\327\343\374_\311\f-\177\307\265\\0\377\025\272\375\232\307\327\034E\200\f`\020\242\24\ 31\251\220\006\376\220\310\267R\364\210r@@\275\300AO\233eo\237\232\250\252\020\225NP\363\245\353]\372\366\377\214\261 \375\230[\347\ \326\371\365\277#\362\017\374\334\017\b\277q\361\304R\316\0344\037_#\353\316\?\372\341\371\331\305\020\323\273\356\374\3748|Y}\216B\ \370WV\025\003|\003\322`y<\303\371@\361v\375>\257\261\236\257\235_g\202\272\341\\\225\214;\260>G\213;'\314\223\205e>\rTp\303W,\374\\ 232\017B\355\034\364\345\n>\275\030\177\243o\323\375\027\217V\343U~`\367\373\242\237i\333\367.\027\267\025(\332o\364\337\340\227zH}\ G\334\373)\363e\206\374\n\240\370O\325\0212\?\?\267\326\267\305\361\307\207P\200q\t\354\351\240\252\365\374\230`B$##\353z\035\177K\\ 315\246\234\331\277\246\252\371\237g\034\216\314\003\341\201-\355\327\253\366\2767\271\355l\305\346N\310=7\270\037\241\bc\r\276\231\ \325_\177\236\026\032\367\254r\307\307\223\331\271\2165-\a\262l\224D\371\034\347\037~\306)|\373\346\367pP\034\301\"\000\214F\006\22\ 6\2651\335 \274zW\024\342\257Q@\302\000\365\3060\314\202 @\210\210)\030\210)%H\b\231\365\302\r$\305B\005\350m-R\025M7,'\240\363 \37\ 0\320{\237\266x\235o\037\3338\t\317\361\353\330\345\351\357\317[\363>\017\201\234\355tt~\177\267\331\227\271\017\034\361vw\247\231\\ 016\325x\2611\000\374\250\227\366\257\336\201y\002@\2018\177Go\232iF\3045\321\356\n_\377z\005\304\366\377\370\322q\223~\357@7\260_\\ 305y},\212\bQ\a\a\347\365w>o\333c\234;<]\226Z\361\177z\302vO,G\301\034\371\251|\246\312\017va\340\367<\?;\0044\200\374\210\217\313\\ 202<\260\370\237<\373\373\037/\345\367\303\242j\354U\037/\312\317\001\353dz\375\341\236\327\263\230e\317\037\\\206\3175\?S\336\360q\ z_u\326u\235{z\255\340\236.\256\003\333>/\313<[xx{\017\275\021M\r<~\357\267\351\362x\362\036O\177L;X>/\317>\257\346\232\362\021\375\ G\346\360\237\"\213\300\216(\244\017\236\024L\312\277\215#n\252\237c\370=\366\353\363A\363\373\032)\001J\004\367|\317\232\324\"\377\ \235>\316\r~\347M\273\275n\033SP\307\243\377\250}\362\250\334\337\037\244\307\325\213\351\221\235t\373\261=\343\343P\375\035\373\26\ 3\370\266\301\372\024<\276\364.\237{\017`\2759\341G\203\327\3142\276\036\377\325\374;\\\017w\344\224\271\364\206\277P\342+\017\313\\ 363y;pn\?\bRT{\276\347_\241\355\375\276+\360\3045\217\215\331\267\247\3572=/N\213\310\020\222v\252|\237\276\261o\267\224\340H\223\0\ 37\302\265\247e\005`PTa=\215\326\265\366B\325]\371*\026-E>\375\237\037\341\0376\260\374\035>&Ey\343\201\315U!\036\311\237\350(\275\\ 367\356\275\341\215\312wB\354\036.u\210<\306D\337r\312(\274'#}\261:\235s\231\353\317\352\030\310\243*@\201c\t\300\301\313\340\221\0\ 06\326\223\262kQ\242\025\204\t9\371G$\0052\200\365\231+\362\241\274\337\021\345\323\331\370z\377\003\267\270\350\270\vh\241\352)U\3\ 35\322\307\322\275}9S\3018\255\356K\021\030\251.B\204\277\030\035\331\233\310\263\030n\002\304@\017\365\363\374\312(\374\221\374\34\ 1A\340\205\275\"\217\301\361\245\315,q\231\a\322\303\033\031\031b`\330q\n1\374[\271\220\204\235|\347\214y\322C\220\2375\271\021\270\ \212f)\036\0232\343\021\200\"s\2320\224\032\320\a\224\301\307\202\036\024\211\210\242\224hH\347c\211\273\005\217\?\001\216\273b\205\ \340\220\f\250\n\201b:\004\332YQ\332\346\356\365\315\237\207\215\030\302/\v\215g\026.\006\265GOB#\246\374\271\3455\r1RU)Z\333\370!*\ \223k\275\234\364\005\324'\362\372\037\000{\324P\226\300k\305\003\320\371-Zh\003\273\303|\327\245\021H(\341#\231VjhA\251\250\006\25\ 1\026\201\000\307\311\217^\330\276\363\364\273=,\215\316M\vH`\217\227y\255\221\026\326\232\330\306\263\261Q\240,Kp\247\214.\r'e\275\ ,#do5\241\343\256d\024$q\234rs\303v\357v\000\364\327\277\203\234P\023\347\274n\036$]\355F\251\aQ\002\022JR\203\b\016\356$\227kI\023\273g\0276\333\311V\264\016H\347\v\ \325c\222\031\a\004\035\326\001\316M\212\230\330_\253\251G\207\352\375\016\341\347IAQ1O\353\340\344\345\352\207\232\037@%\353\274\b\ \237\026\206 \aY\0025c\032\311\220\2359\221q\212\272c\311\351'\237MZ\327\030@\220@P$\020L\3254\224\217\221#\252\002j\230&\212\232\"\ (\242\243d\2411SKU1\022E\020\224\245\004\355\212\241(\n\252\001\233\202cd1\251\000\347\260\034\344CB\372\2274\312\020\034\370\343\2\ 54.\256\316\274\350\232J(\r\250&v\242N\274p\032\355\226eaz,pv\362\345\242\245G\201\264\030\274\331\262\v\b\244\212\276Z\032\221\027\ \352\fC\230v\233\t\347\335\334\304\032\210#\256%\246\vp\276\301z\370\324\220\221\\y\036>0\343ms;\361&\216\300\03404X\305\0030Q\333\\ 241\027X\300\202\371\361\316u!\323\313(\?\a<\237I\313\235\0326\235\257\022S7\b,g1\006\267\\\300tF\247\356\365\200(\240\344\213\357\\ 341\324\3263\2266\3145V\255\020\352\023V\244\311\352\036\347~\t\313\346\a\244w\031\374\262sG\245\302|\263\215i\351oO\304\350\247\00\ 1G\202\340\325Y(W\207&%*\250NF\230\233%\200\265\204\3258\331\326\210\210\242\240\355\214rGl4B\372/\fRD\031\025\214\002\223\202=9\23\ 3\2344\252\262\362\\\230_er\342\032^\240k{EI\311Z_]K4\266\004\205\217\205\240&2\r!-\001&S\0253\223S%\272T\006\021vI\006K$\020C\260$\ \r\352\253\230\353\243z8C\310\237b\251\355\206\tS\221\?]\210\371\252u7M\342h\016\333\362\326\032\207r\273\321\205\264\30124z\244$E\\ 221a\200-\224\242\032(|\342\306P*Oq Y$\021(\223\2559FU\240\214\225\323\223\371\b\243F\027\327\2300\215\356A\370\207\214(N\316\276\0\ 02\257\237x\205)\366\037\a\204M\004\375`}\374\025\300M#\300\021\2521\201ov8\006\206\315\274`C\301\022\211\002\2025T\370\320\232\027\ @yr\263\213\301\343\235qp\305]1\245\2655\241\216\256!\361\363q\031\332\324\226\b\205\372\314W\234H\247\226\021~\n\026\204\240SBy\27\ 7\215\365>E\322\257zQ\017z\006p\037\t\006\2420\330\300\245k\271`\024|\320E\361@S\002\367\bh\300Md\031\240\236\207\355\363\357\236\0\ 03\231\211b\244iH\230\232\211@\314+\306\003\267\205#\347\240\320\034\354\256>l>\204\343\ai\303\256\246\266\033\315\253\313BGJ%\a\34\ 1\vJ\301\024\0360\231\000\024I'\346}gW\277\335\301\355~\033X\344\375\030x\001r\t\021\3301\f\220\244\252q\004;\311\2222\207M\235b\31\ 5.\222\206\bb\202gf\312B\232d9\362\017\334#\316\003\f\023\005\350\035\f\033\250\372\006\021\306\203a\2431#f\362\323a\233,\"P)\2310\\ 023\224\331iHn\031\f\024\v\t\220\233a:5\266\325\265r\253\207\032\347\260\023-\b0\324\312u\030\305\204\0301H\353\200\230\215\243\216\ \326e\202X\327\236t\265\021Dk\016\271\252'\0317\vm\326qS\303\231;\350w\376#\006\3611k\035F\241\311\n\001\343\216i\3113\311:\3505\27\ 2\025^\037\020d\366\301\317S\304|\215/6\305C\210\006\\L(\340\212\221\210\212\251J\227\302\265\030E2A\031\017!\245\265:p\266\031\351\ i\204\336U1\000\340 \204s\364\302\376\030\366w\252\022\b\246\030h\230(\205\252\301_Rh\347\250\?\2524|\326\311{\022\255!\022\tH\230\\ 036\376;\247@\032\344'66\305\n@r\t\356\247\200\334\310'\272T\016`\324+\320%\302\256\204\001\352T\021\023\025\305\240\a\313I\255c\20\ 7\024s\200\266\005\270d\226\fK\027\005\357l\"p\334\r\360B\210\r)\314\304\363\354\203\nu\031\004NMn\005\306\034\304\236\217\211\260.\ ;\001f\264\220\3508\271$\371\033\303\246d\2341beU\304\373\365\231\035\3553,\033\314\"\252\224\257\206&k>\274\316\233Cpj \240\t\245\\ 210J\"]WS\311\340\005\255&\202\230\224h\022\244\3477.u\301\341\335\316\335\017PUSF\215\210\003Ki\311\252G\254\216H\356N`)\323\2355\\ 300T\251\304=\241\"\000\240!\204\266\310i\\\235\355\216\353rC\235&9\243\203A\321\341\f!\243!\315\230f\373XZ\016c\201w\255\320gKs\00\ 6\022\212b\232\212)\212Ls\255\250#\245\203D\233\300\3433\235\216\034G]b\020IF\256y\342\327\023\f\220&J\225\220R\224%:\266\003\266\3\ 05\024\032\355\034\212\247\272\307lfa\211\233\2342GRd\223\353\203\337\367\254Cq\322\f\273]\247V@d%Z\336'c\267\201\313\270{\313\225\\ 025T\026\360\322\304\367\241;\362\234\222\207\222h\251\330\327'@V\357v\261\324\343\207CJ\270\276[\035\270\251\365@4\t\317N\251j+\03\ 0X\016\321\016\357\311\332|(C\360:\261-\240\274[\203\232\006\310!\312Q9\3600Q\275\033\r7\033\343{\327\031\2555\310\017\231\002{ \034\207m\201\r\017M\034suVO\214;9\b\227\255\361\356\ q\236E'8\346\363:\360rbw\334\202\346\256\006\252\022\253\024\f\301\210\033`\246i\240\220lu>y\212\2362\224uF\357\331\273\366\f\312c\\ 023a\365v\301\310\210\\\225\355/=\375\021\350ud\355!\346\032z!\314\203\346\321Q\027\n\035\f\016\331\243\316\316\231\3271\326\367\32\ 7fJq/\246\bm\322\223m\t\227\f\032\200e\351sk\351\356N\357`e$\\\273\326\317<\322\245\3343{1\247|\231bPj\310\344.g3\326\0062@\035\v\0\ 25\237\276\241YP\356p-}-!7\230T\261\016HTP\227\262\240\221@\234\314Q\005\336\206\240J\304\006\337A\006\307!At\302$\300D\004z\\\b\23\ 7\343\341\204g\223\327\211\251c#&r[\303Y\365n6@#rV\304\205\b\021{\340\215\361\246\266'\002\022tc\231@q\0217\306\351\370\257O\257%\2\ 47\352{\345m\002D@\340\030\035]\n\026A\322\257L\f\000\221\261\023F\031\021\232`I\346T\022\"H\204[P\"\314\001$2i3\304\325KZa\305\f\3\ 42\024\350\215\224`\020`\035\020\t\375\032\031\304\353\322\316\027\261\351[\277=\361\357\253^~g\fsu\315\354\271\223zr\360-\324I7p\2\ 05\207\t'\242\211$\025\321i\204\000f\021 \201L\244\233\244\271#\\l\305\365\253\2121\023\250F\022\235\2759\024tE\003\252QWH\033rL9\v\ \f\b\32001\240\241!\000-\330\317G[\273\261\200\032\026Cg\301\0012=\002\003\006\265JP\220\310\b\230\354H\034\230\205\302\232*\0064\0\ 038`\006LEHO\202\205\241U\303\211=x\246\a1t\363\205\243\004\342\343JD\b\v\276\316\274\0240\031\331w\275\b\034\036\330\344\213C\212\\ 213 a\030M\323x\317.X\307/\\o\216\366I: \033\335\030\220$at\270\2560o\214%4m'3`\3530JcB\201\256D\004t\242\245g\201h\364\027\\I\036\\ n\240\220\354\200|)\356\200\231\220\235\305\315I=c;)$H \220\302\347\2121\320i\267\277\n\a\202\005\030\033#\241\261\303\367;\360\247\ \245\035\360\020\210\236\020\2067l Gf\362\001pH!\311\223\n\220\337&\3651\016\360\306\200\222t\v\231\310\337\002\230#2\0232[B\240\31\ 1F\020\363 T\202P\0050\345\000\302\353\246\r\3656{\221-w\332\347\305GL\304\016\264E\"\200\312U>\033#\b\223\001RE\366\022\274\230o\262\344\270\205\002UT\025(H\027\005,@k\204\006a\301up5\214\211=\265[\021*:D\343K\vR\\ 216\240\346\371}\r\351\225\225LJN\365\331\206Ej\225\226\253\026\327\"\253!-p\2118\222\n\226\316\230\002\f9\374\227}\317X\005\036\27\ 0P\232\212\333R\244\024\b\240|\n \270\004\300\b!Y\344\n\340X\267\021\035F>\324\210\241=\035g/\214Y+\216x\342f\320\302\361\355\344p\\ 326\371\035s<\265[\315u#t\305\353C\231\350u\267\324\216\304Y\256\321\363\037P\225\fR\221\275\211\351\a\201fxt\353\316\371\274\260\2\ 44\307\203\031\345;\032}\2077UW`\035\r\0339\346\300\034\000\353\317\004\020\343\fC@\v\323\216\272(\202\351\312y\000h\211]0Du\t\336\\ 261\017U\235\310j\321\017\213\275w{/T\3448 \211\323\221|p\020\220]\\\200\344y\206\373\001\236\2049P\347\201!\207\277\307\003\300\00\ 4}\004\003\?Wr@\366+\324\f\?$\265\024\3245\020\322:4~L\?\376=O\177\260\327I<\332L\264yq\2065\306\033C\260L\312r\253\311psvyM\372\21\ 7:\372\367{\224)\372/\201\"\241\241\311\027,\242\227\216\305\271\375B\257\226\243\360\261\237\207\260\263\350lb\243M\220\276\325#\0\ 01\216z\027\n\025\017%\236n\263\247\301\232a\354\240D\234\322\212\022\345@bE\232\2120\356Y\244\242\264\243\257\301!\301\2754\036\26\ 1\305y7\341Yt\371\023\371\001\241;F\247\\X\250 \312\261\244$\214n\034Of\374\324\215\326\0319`\350L\301\202cZ\357@<\264\310v3\275X\0\ 33\323\273\242\220\222\"\225u\335\346E\204\201\271\fg\273%K\031B\017y\035\035\337/\311\255\rY\302=t\244`\255\?N\334P\343|n\224\?{BB\ _)\2270SArFW\321\320~G\022\320\230\341\310v\305d\"\241lB\327\311\320\032\"\314l\331\230\030\201\322\303\272\001\002w*\"ob\306\257\3\ 30\0168\340o\337\205\263\314a\221$s^[\251\351\206mI5\216\260\030\027g=y\334U\336\322kn\241\257^&\344\325q1\241\256%\317:Z[\363\246/\ \025\227/2XA\231\267#\311\351\371;X\036\271\335\315\035\003G\337\356\323\032(\256qnk\272\243\346\355\242\020\344\202\307\236g\316\2\ 20\312\343{\224\3508\205\324\264\257\261\233\212\333\2730\351\2404N\205\327\022\321\250r\361\023|)\n\205E\347\344\202C\224>HE\345\3\ 43\005{\263b< R6$b\242{l;\0051)\036\322\360\363\\\232#D\360\367\367\344\\q\001 Z\310\360\211\235\371P\276t\324\022\300ma\231\276&\3\ 35\252Ss\203\2207o\34005\264\204\034\307ww}^T\2220\034$+\254\271\025\231\321{\021\364\022-\036\r\337`b\0371\021\003\324C\206\036\26\ 7n\310\237\b4\026f8M\005\016I\340\005\206\314M\260\032$0\225\370C\221\251Z0\205\300\265&\020e\211\256\345\030\325Pf\243\312\337\243\ X\345\202\215]}\367\264\352p\021z\033&mC\001\3306\037Y\t\246\372\317\217\271\037\272\2122HH\201\303\236\323\214\323Ci\266\305\204\3\ 403\340\242g\2117_0&\024I\004Z\365\221T$\\K\210 \306%!c\b!\365\251\345>w\255s\201\256\271\225Df\256\271\244^`\303%b\b\037p\020-\202\ #\255,\206\333c\320Cn\251\215\2625e\216\212m\261E2J\352\340a~dyN\201\325\035b| \024)B)\220\242\"\202\250\002\204*\022\024\003E\257\\ 037\201U\024=_N\274\210\035I\023z:\374|\275O;Y\241\367s\256c\232\246\027\311\tB$\315\272F/\3459t\t\304\021\372\355Y\317r(\233\226B\\ 000\231\367\034o7\306s\261\207\351\021\006\0275mHf\022\337H\"L\2048\371\247#\002\340s\243V\b\f\022`R\341\206\224\036\331\016!\312\0\ 23\032\004\005\311P\250\211m\334\316\325\002\305@\260R\276\027'\022\254\302\254\246\021\"\352GUM\032Q\016\224I\375\234\30748=\230\3\ 73\261\033As\345\243Q\262\331\231\235\355k\303969\000\236r\264Ko\023O\353\216\200z0\004E)U\301\a@S\227\316\241\016\337#CF)t\356\032\ \005\030\243\361()9\3722\352\016E\315\262s*\031\035\2562\270p1\264p\343\272\016\350\006\331\264\023'E\355\300R\022D+\2150\f\344\204\ \265)\305\025\220RD\3509\257\223\245\234qm\215m\030\261}|U\373\345G\026G\r\020\275\322c\222\312\341\017\021\004\030\344\210\005n\32\ 0\344}\213\271\320\332\211\300\256\305\031\341Vc2\244\272+\362/#^W\211\303\202*D\343FKc\025y\222B\220/;8\302!\252\246T\316\303\030\\ 253\212>t\306\026\275'3\001^\230\205\213\253\230\261\023p\351|\352\262\342b\357z\305\360\363\266T-\304*\334\0200[m\035\263m\034\020\ l\204(!\321Xj\252\2600\365\264\212\035\035j\200\262\321\234m\210&\301\341@\214h\356v>\327\255F\244XH\321o\311On\247\216&dYa\"bC[/\0\ 00\036\340\f\200\a\275\244.\221\310\247\303\021\356j\017d\345\250\310\313\t\242B\330\315\032\f\363p\242!(\217\rl\024@\004\t\311+\27\ 0\034\323tI\323\r\333h\344\335\212\266 \027\252h4o\233|)\020Q\371\347\260\f@T\200\031\250n\000\315(\231`0\310\202\302\200n,2\343\27\ 4\344y\tq\325=\257\263\262\354{\034\301\r1\"8\347\002\021\244-\023xpq&\343\274L8\256[\273\321I\vBb`\260\002\213B\334\000\250\006\33\ 0A\204\034\rJ\202C\202%\250\005x\267\016\353\261\3024`\251\212\333&\035\335\260{\017\200\361A\374\251)\344\003\332\037k\355\0370\01\ 6\031}\234\221\tH\207\236\307\255g\001\360q\316\023\222I\250\363=T\361|\243\317h\364\223!\fK\274\347\265\016T\v\224\362\201\232|\34\ 6\v\302\301\343C\02119\217g\316\206\370N\330\031\310>\371\032i\323\327\347\313\270a\224E\034\240\375\220}\263Q.BdR\361\017=0\240\03\ 7\221\t\374\265\252\205|\323#\354\214\215'\322\372\254\314\243\005\371p%\363\030\276\277a\327\027\361\351\211\v\334N5\220\331\2745\\ 025.X\362\310h\231\334\r\244::\366>t\315\217<\271E\\\3439{l\321\261\rO\016\325`\204\"D\212(\tC\365\220P\304\306c\316\3\ 22\"A\245\333j\t\264\rX:\003\021\332\032\362n&Y\301\304\356a\030p\374\036\260\370\322B\a\2209\336\263AM\221\220$\022\272\252\365+c\\ 244\323\343\351\034@S\310}\214N\376\254\320p.M\016\375\330^\362\000g B\325Uh4\352\003\246&~4\033\201\210\365&\312\206SQ\v\3319\005\\ 271\366>\225\v\020\203\233,\320\204\266\300\035A\022@\242b\003\330iL\340\236,\235^)\322\360\250B\204s1\003\233-N\247\246\202\223\24\ 4\031LQ\250\"\330\206\213\312\332m0\311u\243,\317I{\"\276\252\036h\?\216\002\224\354\a@Nm\032\240N\271\203IT\241E\006\341\333\0177m\ \0306\370\235\005\006\026,\231\231N\300\370\360\361\307p\177\031\331\260\354C`r\340\022\032\220C\027\262!\326\353\221\210\230=\257\\ 005\362\330{Y\235Rb\001\320\032\320S@\247\200\357\216N\244\032\032\025\343\361\f\375`P\r\002\024\2424\005\006\240\265\337\220;\233\\ 351N\203\325/{\210\366\037\274\222\fJ\222\002r;\270`\230\361/V\a%\227\266\340\020d\241\035p8\342T\321K}\?{A\252\202\232\004\326\234\ Q\324+\271\032W2\306\330e\211\3557\277\270:\0171\367\227\202x\311C\317b\022\021]\261{\320\adN\321\313BZ\006\360vE\205\350\250l\363q\ 7\351A\305\213\025\306\030\004j\210\n\344\001\374\211\344r!Dya>.R\016W\260`\276\343\343\036RO\\{m~3Y$\331\264zG\r\340\215;\335\r\22\ 5\025\245a\"\305 =Rk{\316=\220\233\332L\270\246C\326\342[\313+QcF#G\246C\021G$\204\220\353\215t\002\356\t$\a49\026\006\017#\22\ 0\206\310\230i\215\025\0048\a\277\265\252\b\251)\032M\006\273H\033\331\335\320\032\037/\035\215\bU\024D\255PQ$ A!B\264\251L\301HSJ\\ 325+\023J4\001ERW~t\001HD\024(V\330\022\202\321\252\200*#M.j\002\232B\202\232\210\n\033dy\325%#\026\307\234<\245\340\363+\344\304C\\ 221\314\332\362\211\273k\240\271E\221\214\346\210\362)\342\212\312\341\005AT\207i\315\213HI\254IRA\336=\a\335\207\314\276\344=\260\\ 320\237\225d\224\020\324\220\004\201J\264\222\305\001H\323\"E4\204T\b\322\204\212\321\004C\022\0012\312B\205\024D\255\005LI>\002\37\ 0\025\024\035>I\361\366\036\2368\266\337\205\\\317\b\300\216c\243\337\313\301\020\250\361\003N\3245\2749\346\022\nL0\304iJiO\250\20\ 7\331\232\322q\2366\225\353\006F\275W\320\347\210\303\334\033\024>\310(A\333\263\316\366\336Dv\027\323\344\222(\a\227\227\225D\362\\ 017l\343]\275\331\204\034\022\023a\307\313\002I\031\341\300\366\0331\264\204i \210E\0020\200B\0010\221\321I\237E\244R\224\370\301\2\ 32\343F\022t'3\000\363\203!\351\317\201M\356g\357\243\235\273\340\032\002\337\310\357\335.\227k\2445\330\225\204,\\ 225I\303{\226%\351\030\022Q\211*\017\340O\341H!\234\214RA0M\266\345@d\n\322\241$\211)\220sR\031\034TC\262\004\356P\"j\204\231\b\310\ 0\033\026\034\346\025rA\222\264T\002g\th\222P%65Uc\314\037&;\351\266\001 -\026S\0050\303)D8\016\020\004\016\322\r! \206\031\210P\23\ 2\201\020\230\b(R\202i\r\313\242\341\203\2074\3152\230HH\317\266`9\206\216\236\030K\027\315\215:(,)\a9\323\000\3140\211\030\030\246\ \314\200L\000\276\305K(\344\027;\225%\b\233s\000\324d\034\211\207s\351\271k!$$HH\004^\206\204\311f\022\tIn\020\206\022\204PI\022R\0\ 01J\224\302H\016Y@\326\230\261\332\270\005\304\240zM q\362\332\311\243F\030\306\224z\261\210R\232\262\324e\320_\003\302\316$\354\26\ 0\226\315\345\001\246{\200m\303\345/\334w\203\035\271\310I(\344\347\256\365\371z\224\330\b\273B\347\251\002\302\353<$\030\305x\315X\ \b@K\263\006'=P\234\006\222\243\345\243\264\275y\n9L\321B\032\020\315,\030\210F\264\206h\317N\003\022\302p\223\242'\n\302i\035j]\215\201\\ 347*\017\316\037\252`>0\322\024\201\371 \311\363\247,\\\222\206 \222\nF\315\206\203$\"\331d&\357Yp\225\373\221\205\021\315I\305\345\ \275\032\016 \264\270n\3160\343\364\235\270i\v3#,\313\016S\363\214~\021\365\251\3164\274\313\250\310\aD\bxK\2357\240w\"n)i\245\334\\ 344\024\214H\263&\246\272\316\211\344\247\2602'^\251\202\244\212[\203\004,'\227\306/\020\031\276\331\360n\240\334.\200\353\033(\036\ J\022\200\320\035\311\257=\341\262X\255(\361\035pY\004\363\302\357_|\247\230(\037p%\363\217\?\317X\231&f\230\a\313\302\264X\336\003\ \301\351\362\303[\302\257\250\260\034\315_\003\215:!\267\006L\024\244\204NM\301\266\022H\241\3277\231^\212\326t\000\034:\256N\023\0\ 20\205\244_\031W\304\304\03045\206a~8Uka(9)\344;`\370y9,\221\021\0030\321@\024\304\023\025DH\221(P$\300\224\003BR\020@\225G08M\004B\ \305\004\260%\000\312\0374\216\376\2547% {\311\003\210\003\310\205\352\200k:\004\221\250\000TFz\232\256G\260\337Hd7iN\323\220\221Xf\ u6\000\371M\000\320\257\304t\341\300\373\302P(\006\217A\3456\243\3574\017\"\207\224\024FA5\230\bp\300Cy\275=m\270\243sQ\250\017F\30\ 3DCh\006\211\223{\204.\b\206\334\023\321\273z\234\301\b\022z\2612\022\336&\243D\336\226\001A\226A\275oN\213r\021\243\003V\217\305\0\ 06\0171\305.\2714\271$\361\313\216\270\314(\255\212LRh5\027\0021\3134J]m\332pA\022\205\vAF\361\321\243v\276\273\016N;\252\036k\276\\ 251\0214w\350\267\a\262\035\231`\326\005\024Dg\336\3011\313\036\251\261p+g9\r\242\231(h\003\220-R\023\230\242r3\354\356G\216a\001\t\ \366\234\313G\355q\202\365\031S\304\224_\021<\304|\303\335\360\306\214\373>\275\005[\300@\357\230\372\002{%)\0044H&\0201%!H\243\350\ }\337S\270G@\314>6\367\336\000@\237\036-!\302v\235\341\210\236\3740G\316\000\310(P\350\252<\253\224\264\247\260\036\304\2532\003H16\ >&\203\270@\310\017@\002Z\324\0339R\321$\a\303\n\f\001\340\323\274%\026\336\334\214\275\261\022\236\230\342\336\367\210\266\207\202\ D\225)\022\270\364X\005\2734y\b2\021\235\375\337\003:\333j\334PZZ\257T\3138A\367\271U\217\273\346\236a\003\344\0340N\341\300\023\34\ 5\301\304\211\0319\355)\274\260\361((\214,5\213\370u\372c\032@8|\211n\363Y2`\345\2407\035=\217\307\345\356\323\371\332\304\374R\371\ \301\356s\265\243\vqG\032#\002\216\222aj\r\222\032!\340\205\310\r}9\2427(\033\227*\016\220\016A\314'\f\216\300\236\311#\220\365\204\ \343\241\234\332\215K\323\216\237\213||\204\016|\320\370Bx&\262\331\217\356\262'\fDQ\370 f\034I\002\242\006\017\235\300\324k\223\00\ 3\222={\237@\000{\037\003\221\020\373\3050\22335\346;\365\203G\264\317\310\364\f\224\230\232\365,C~B\322\001\267`\r/~ }\t\364 {\377\ \211+\362\336\305\001\350$0G\272\372kzIP\365\202d\200\242\243\325\374\260\036`\016p7\347\306\234g70k\346\020\t\333\216\t\0344\006\3\ 32\230\304`!`\3272L\360\222.8\rc\r\034\220G3\331\360\225\022U>\315\214\3249G`\016\253\342\0354l\3540l\2427<\006n\b\222\002\022_\302\ \226E\000\301G\250\356\207\006\326\202\020^\004\006\030\236,\204*\016\260C\206P\271\021\037\306\326\264\t\313\001\017\277\vK\3722\3\ 72\222=H\023\313\316\005\361\363\300\244\324'\020\016\246\370p1\267\f\"7Xbj\034\303\313P.\354\221b\a\222\034 H\225\324\271\003\367\\ 344\016]r\374{5\023\222r,\204\211\003\221\024\271\250\035\a\214\311\rN[\032\n\b\002C\302\b\f)\346\240N\353Fq\016&\212:\313fR\025\02\ 4\325T\322D\225AI\026\261\0214x\223\254e5w\031\344r\202\"\224\211\272\262\2244\215\314\274\341\230\350\2154\021\241\304\035lh\311DA\ 1>\t\321DBr\034\224\0014\317k\023\330\316\344i\230\344\343\266\\\a%\353\2065\242H\200\250\246\245\231\000\210 \240\"Jh\211\b\232\25\ 2b\226\202\206#\2064A\017V\220&\n\001\243\267\244\307wYG\264\234\214*\362\037\321R\341P\320dE\326P#\275\026\237\322\200\344\365r;DI\ \346\003^\241\323:\316\tiD-f\255\t\250J\n\327W\230&a\200kZ\255!\230\347.f\267\255j\246|\344\360\035\020\301\021\035\364<\202\006$\n\ Db\021\240t\313%sF`\323\336\325\332\243\221\326\200\317\003\033\035\035u\324s1Aj\036]G9\030\"\016l\365\227ln\213R\235\035\320\366a\372:\310%\2\ 74\002\360\bN\322\017\241\325\027\271\323j\235\a\307\b\312G\255\225\nA\243d\233\300\221\364N\373\350\036\366\023\346P\006$\t\020\22\ 1I\033\037F\211\201H7\200\242v\b$\370\204Dw\233\247,\205Q'\r\271\313B\345T-\271\016H\347\312\350\034P\314\036i\210\221\252\005\257(\ \227\027\313\312\236\313+\357\240@\036a\205~\023\300k\346!\370:\2208\t\344\304\2203+\210\016\220\036TRC\003S\3347\246\035S%\274\312\ \005\022\322@h\002\301\322tG\340\a\306-\343qm\0169J@\247B\032\247\266ZC\036\273\vA\241\000\214L\2230\2442\2207\211\336\243f\327\264\ \256\022\210H\314\223\247\203KH\221\304z \251\256\305!A\221\016\200V\301\263P\364\003\210\037;\301\352\242rl5\336\2449>}\332\r\265\\ t@oO\222x\211\345\346:R\322\a\356\314\211\345.\204&j\267\255\a\n\264\363\261\264\240qz^\212\252\222X\245\344ibBI(\032(\240\002p$\35\ 2:\316\362\275\243\300\003\337-\b\344\257\230gP@\353 tN\261\2704L%\\\330\207\2309\312\032\210\001\253\206\003U\2404\320h\343r\330\3\ 15\r\032\314$\263E1PL\016\235W\225$\320s\030v\315\266B\v\301$m\033\006\344YB\t\327\346\320p\215\216\312\032\313\203\323\320<\244\00\ 6\276\?^\031\240cB\036\?\240:\203A\024\275yL\341\256\347\266\024\376\026\241\234\b$\321\333\177\347\f\205S\020\314\016\3130\336\305\0217p\r\270\036\305p\020\312\231\340\344\317\237e\035\371\247\034p\362dF\213\334\2068 I\360`\ I\020A\203\006\370Oin*R\220\003(\304N:\241@Q\000c\b3\210*Y\206{!\202805\021\254\034\237_!\223\003\230\320\t\253\020@\023\260\020\34\ 6z\363\335\333\241\036^\310dA\004\217r\a\226\205\036QD\331\217J\203\035s\357\"F\320\204\220\321\200\031\003\245\a\212@\373N\256v;\3\ 11\263\325\205\026d\210\242\t\004u\t\317\211r,\337\270\be(\222,\371\246uw\244+\"$\3043\204\030\324\3044\323Q&[\216\273\321x\035s\27\ 6o\255\233Y`\203\341\276\3539a\?\0228c\212\0034`z\"=\310\263\302\216\303\201\021\033\0239\327>0O\200\303p\211\033rC\312p\310\216\31\ 5\233\245\004J\210\256\312\302\001\332\206\351\3310c%\005AG\333\345pl\213>\344<\213\017\tr\036\346\214\017_\f\343\2239\207!\"\r\226\ ;9\264\207O\300=\230\304-\311\004\314\215\344\267i\3703\006#\203\261\004l\201\275 6i\372.JN}qxp\3028\020\205\222\330\nV\035\036\242\ M<\221tf\005`F\220\363\331\253uL% \341\345\256}\227d\r\362\242\222:\356\245\211Z<\272,\022\t\003\243\027\344V\247@{\253ap\206\212\0\ 00\300\331\312\240\rU\315\237)+\341\316\366\350\034Z\2722\023\312\220\302\327T\332\210\235e\a\023t2O\225\203\264.\205i\316J}G\220\3\ 67\307\f\267\304\371J\264B\272\214\3337\206\373\333}A\305\242%\200u\242\330\212X\004Z\211\032\321\321\2358l\311P\203C\246z\201P9\00\ 0B@\323\034\254\353\242\262\362\223\364\320Ab\253=\271\371sX\231\035\314-S\035r\245K\r\016\032\276\313F\247\303\360z\254\366\245G{w\ \305y\236|q\257\245h\223\301*r^\370v\345\307$E\033V\233\353\223\362.Hl5\344t\035\264\221\333\216\017\243\332D\330\307$\271\332\211k\ +Qz\301\356\353\216ebq\211\310n\375`\337F\030\032\256#\rX\016\351\262\340 \246e\313 6\210\004\301\222\213%\020e8d\023E\017iLPw$\205\ j(\022I\023\001\000\270H\201EA\261,\347\326\352r\275#\262\356D\360\350\316\354s1\357!\255c\222c`c\200\005\0057\316`<*\236\225\a\214\ \311yO\315\302\351\021$\301n\251\200E\375l>h\304\363\300\362\355v\223J(\204\b\262\261r\253\323\320\326\212\345\366\236\0304\022\302\ 4R\251\222\311\320\235\243\270h\210\"\3573\210\330\320\346p\003e\302\373\354\362`+\246HH\350\341bK\205\026\245\f\323\b\221\005\024\\ r\323@g:\310-\326V\ahpu\211\206K\204Rcd\035\027\200{\212\200t\344\321/;\036\274\?\216D\311\032\a\267\201Tc\223\317GF\236\276\3600e\\ 242!\200\245\222\a\201\200\351\370s\200\367\a\216\220\3754\360\363\274-\206\034a\333\261\243AEBE\004\204\224j@\370H'y\027\300\016\2\ 00\347,\216\313\021\364\201)Z\020{\021PM\002P\232}/i\334\?A\331\351\306\034\235F\236\357j\320\271-u\320\272mm\324~\277^\032\307\v\3\ 33\243\006\263\327\023\246g\212\035X\300C\023\021\023\001;v\343\244\213\270\303)y\234\206\231\"\027\233Z\316sY\302oyY\264\235\346\0\ 16\305\202\325O\022PD\232&\267\\\206\265\204\230C\336\260A\020\206\261A\324::\330\"hz3\224\221\274\356p\033\0220\377\306S\330\336R\\ 247\214\250|\004!*\005\342e\023\320\"f2\333B=\341]\242\330K\033hd\264\264^H\253\247\225\200\342\21187&\373\301\352Q\002\341#\2624\0\ 24$\220$\313 S5\r\005T\005\034\241\333X\302h\350\2040\325\031\230\321\220\300\003\215\201p;\351\365 }9d\322\030\342\027GU\n<\001\34\ 3`Y\201\350\037>\264\244'\353g\355\327xN\261\204\024\277\b\\\204\t\006\307\021$\217L9\363A\017\274\f*\207\351\273\324L\211\032g\025\ Ei\303\204\f\324\373\321\030\3548\022\270l\376]]\2018\221zC6\322\341\320\251\242\034\\qfdr\257\021\212!\030\025f\017\025U>7\003 \35\ 1\017'\035\372q\307-\004\225\370,I\252\250\210\210\343\241\234\021\305\r\330[\263\216\220\310I\030\357V($\201@\330\002\243\262\241\\ 215(,\240yn\f\311\254\335\213\"\00085\2414oT\316\v,\v\002\272\342w5\265Dg;\025$\352\b@$ag\004\310R\220\224\021\"\252\360g\343\"T\30\ 0A\225\225,\016\004\215\306\357\254\032h\331\205\001^e\032\241a\241d$dq\303\301\0274.\260\360)0\306\003\265\276\261kC|F[\023\\tl\26\ 6\242\211\030\022'\232M\2321D(\246\030\215\002I\200d\206l\002V%6\350M^H\023r\212\020\256\000\223$\002\246D]\021\035+6h\206\306\255\\ t\305xh\212An\bn:\204-\242\302D\3424\270\022,\2068A\n1\2421\232(\034,J\n\021\026@p\243R<\004F\316\246\263s \201;\f\022'a@\262m\001\\ b5% \002\326\222\244 Z\320\260\036\215\bb\201\226\005\?\254\270\223\031\276\261V\207%\215\n\004\004\273\356j\272#\317Z\314\r\016\21\ 0w9\f\020G8k\235s\315\357eV\236E\t#D\221$F\220[\245\254C\b\330UyT\212\322\023\223G\f*\227\263F\201\335\302\003\2644bI\345b\341\302\\ vM\212\033a\226$Y\026]\232\f\266\034\032\v\224\360\324_*re*\334`\243\001\336n\252\243\006\226\221 \235\3274\024\204o\030\267p\210\0\ 26\314\35308W\263}\316\210\221\270E F\202\212 H\222\210\264\201\202\202\276iW6\020nh\336_\fp\001\347\220\344\206\245\363\032`\342\2\ 30\244`\023\316\230\2210\026\251\002]64\200R\364HE\262\271\241^(7h7\233\231\221\212]\rq\227\f\335\241\a\212fm\321\2038\332\331i&\21\ 3:\202\250\242\233(2\003\024\037\016b\322\241\255-\233\002\357BV\251Q\034A\025\307\004\006\302`M\b\276;\365\037t\247\267\360\031\33\ 48\300\354)\244\250\032 )\fYC\274Rl\"@\375\322J\003\304%2\004iC\201:\311\rv\r\355\022'\241\021\336\353\004N\236\313\341\3531\231E\2\ 018i=\a\215&[$\344\272\251\220\333\2278K\243\204\035\214tu\316\235p\032H7\254)\n5\037E\2519\\7{;y\363s\274\273\346\362;f1B\t(\216\0\ 16h\205\022F\225\371\227\240\206\002\315\032i\230&\210\271\312\030\270*\305J\2124\b\002MeTH\322\r\322l\225\021\016t\351\225\025\n\2\ 06\206J\233\272\210\206\244\226Z\355A\242T\flD\335\223\335\243\030;\"67#<{\263Y\231\3235\254\325\316\334u\2678l\245\264\346DP\006\2\ 57\016\301\321\016\307W\261\316\361\353\257^\335\250\234\361\305\3511\262\343I\364\223\341@D\244\351\025g\355\216U)*$\314{i\304bk\3\ 22L#\333\024\037\231\032&\024\360\205\300\246\224\217<\000\343\272\252\217\262X\200D\240A\034\003\224<\243\212Z\200\260\300\313\240\ Q\354\264\230\220\\\303vV\306\"%f\255\016\2201\030sOu7\354T\362\347\005\215\221*`\0324\300\235\264\305\352\300\313B\300P\232I\342\2\ 45!\317Jq\325%\302.\330'l\216Q\037O\035\363\365\266&>\267\350\264}\216\232\000cF\202\201\351\271\245A\201\210\240$\275\332\033=\026\ \322\202\000\334P\275\354\240\220\006%#\r\"I\\\306GYS\027 \344E\316`\2560\034\341\316sh5\315\223\221T\211HA\315\226\202\200\256\026\ (\305A\242\245\213\3025\016\353\005\215\321\233\035\270bb\213\253m`\315\a7c\262\256\351\272\330\355\267X\305\2678b)\341\265\022\324\ v\321\022u\212\210\343\233\023\023\226N\006Y\206X\346\264kQg\035\3049N .\244\235.\236^$Arj\2163\017\247\233N\313f\030\2357\323\263\\ "\206\340\005\245\n\024\002\242\245h\003\020\366\203\213\303\245H\300\200\372\030fX\225@\351\324\232\030\220\314\206\251((R\225\n\n\ h\241\301h\321ZJ\006\264:\242\251\023A\241\"\246\240\202\\\347\036\f\225K,`q\307|2S\266Ce\324\275\234A@0\031\217hC\237{\352\250x#\3\ 20\244H\313Q\222\000\367jI\250b\211\244\005\030\222\000\332\b[\0025up\202B\340\202AI\022\210\253\254t0\231 z\tJ\322\a\263\224\327m\\ 267\by\311\242`:o\257 \a\201%{\016\017o\235\023\0044PTQJ\301HH>b~/o\357\245\035\363\240e\204\252\255R1C\001<\324\244|H\355\320\343Q\ *\001\303\215p\336\253]\342h\300\203w}\354\324\0217:\037\343c{\r\303\244\003\277\357\304\027Z\031\\,\352\240\023 \234\340\226\360\0\ 23\265S4]\206\330Bb\a\335\301d\025w\bjH\201\224\001\256\023\345Eh\263\256\311\241\232\n\"\2766\257\336~\213\343\362\213\203\365&\20\ 0\271\370T\236/\266\223\206#\275\000\371\304\000\324\357=2'\"\331\3479\206\021\337`;p\347BX\214@\352\035\2412\236\250\204\036\277i\\ 223@SUG\217+\245\nAz\201\323\022\024#J\352\263\034\232\023P;\331\344~T\036\367I\353\016\220\036TC\260@\365=\261O\264\024\200\360\37\ 4\353Y\272#Q{\310:\232\244\201\314Mf!p\243j\177C\301\256n \350\237\203\223\334\232\a\275\2354&\341\217\317\363>\250\250\207\353\303\ $)\242b\212\250\252\001rD\366\022/\317\333\365\260\005\a\333\264\351\r\005\336\201\255\357\366\342\204\203\332\246\340\006'\nd9i:J(\ \225\235\251\210=d\271\300\311E!\214\033\004k\023\0231Y\035V)\316\244\251\236\017c\2708w\"{A=N\fb@\324\f\021\371\275@\222 \206!\240\ w\274=!\344\274\304\035\317\207[\336\b\247\224\200\035#\006\340`MyV\205\212\263\263Ny\256\271\316\240:\2003\024\365u\314PQ%\267z7\0\ 33Cp!\222\205\254wf\263=c7\231\025WeZ|\354\026\305\t\320u:\035\264\363\034\212\212\0308\204\006\202\356_0\220a\341\220\344/\217\001\ \307\034\r\232\306\2330\312\303\024\3712\206t\357\032\271\307\000\265\205=\374\234\036\241\312<\202j\256s'\230\303\246\364\352\347N\ \303\240\361\342s\327\227\241\317C\240A5D4T\004P\313\323\361\340\232\374\320\220(\323]\002\236\214\245`z&\213\250\357\267\244\264\2\ 40!HR %\024\200d\236\005\272\234$U\r\330\002\310\265\235\"&!=\304=e\353\302\234\336|\353\366\?\311\3777\006\004\346\020\243\307#\31\ 3\000\366Tw\370K\337z\026\t\036UA\222(6\321\264\002\026t\300l~W\b\300\264\360N\030\037\226j\244I\020\310Fe\v0\?\030\260\234\244\351\ \340\372\361\311\016F\"\b$\020\035\245-\023\243$i\255]\236&(R\300\220d5s78\346\2540\016\200\270@\343\204$\3369\2749\260\253[\326o6\\ 031t\314A\372za\302$\302}2%\tC\255\032/\233\341=\304\3458\216>+\030\006\211J)\b\212S\005\221!\235\350Q\016(\t\364P\337X\330\356\277\ \343\177C\370\177\324\257\331\177t\375\343/\002\265e\227\223\213Bj\303\242\356i\240I\372\276_@\026\274\376C\035\351{\346\225\370)\3\ 55(J1\"\217\341\335\345\202\022j]\303\"S\243JBV\267<^\2350\235\231Y\216&\035:dEIC\370spw\300\373u\207NSG\0320\266\367t\343\301=te\3\ 34\304\307\207\363\332\211\2075\320\307\233\265\253d\311\a\035\360mf\327\251a\337x\225\367q2\b\257\v8\355\032\336\324{HU\247\016\33\ 0Y\231\230w\226\021\035\202\"\314t@\0247\320b\204\304\270\034t\242\225\016 \252jf@$\223Q\301\343c\r\215\336\f:J\220T\220J\3676)%QA\\ 215R0t\251\266\004\231V\305F*\245\200&\3042+J%\340p\300\265$\030\033\220\310,\004%\375\3440\004\202\207E\252\231\bp\342\030:P\254)D\ \306\333.\030\"\021 \301\016Sj\347\226\271\373\310c%\n\371\272\214\b\016 \201px\t0\206\265,\301t\342\204\204\0030RCn=\205>\361\363W\ \254\2574\360\257\0202`\022t(\244\302\215\320(D\035(\b\200\320@\203\003\304\204\322\b\202\374\314\311\246\305\031\005\240\004\301&\\ 345\202X\204A0\310\223\363\227\f\220+\022k\206\017\227\303yA\2517\353MD\003@\342\021\314\301\003\022\342\2350@3E\343t\t\024AG(\277\\ ?w\217XA>\340\247\251o\311\360b\200\036\026\bD\027\305\271\245\314\313e\347\301@\204\201\r\2473 \v \022*\235\223\311\fO\304\356\033\ ,\360\b\211\377\361w$S\205\t\004\211(\177\360"; ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/help/help.c0000664000175100017510000007100715265101745017640 0ustar00runnerrunner/*********************************************************************************************************************************** Help Command ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/help/help.h" #include "common/compress/bz2/decompress.h" #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/fdWrite.h" #include "common/io/io.h" #include "common/memContext.h" #include "common/type/pack.h" #include "config/config.intern.h" #include "config/parse.h" #include "version.h" // Include auto-generated help data #include "command/help/help.auto.c.inc" /*********************************************************************************************************************************** Define the console width - use a fixed width of 80 since this should be safe on virtually all consoles ***********************************************************************************************************************************/ #define CONSOLE_WIDTH 80 /*********************************************************************************************************************************** Helper function to split a string into a string list based on a delimiter and max size per item. In other words each item in the list will be no longer than size even if multiple delimiters are skipped. This is useful for breaking up text on spaces, for example. ***********************************************************************************************************************************/ static StringList * helpRenderSplitSize(const String *const string, const char *const delimiter, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_PARAM(STRINGZ, delimiter); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(string != NULL); ASSERT(delimiter != NULL); ASSERT(size > 0); // Create the list StringList *const this = strLstNew(); // Base points to the beginning of the string that is being searched const char *stringBase = strZ(string); // Match points to the next delimiter match that has been found const char *stringMatchLast = NULL; const char *stringMatch = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { do { // Find a delimiter match stringMatch = strstr(stringMatchLast == NULL ? stringBase : stringMatchLast, delimiter); // If a match was found then add the string if (stringMatch != NULL) { if ((size_t)(stringMatch - stringBase) >= size) { if (stringMatchLast != NULL) stringMatch = stringMatchLast - strlen(delimiter); strLstAddZSub(this, stringBase, (size_t)(stringMatch - stringBase)); stringBase = stringMatch + strlen(delimiter); stringMatchLast = NULL; } else stringMatchLast = stringMatch + strlen(delimiter); } // Else make whatever is left the last string else { if (stringMatchLast != NULL && strlen(stringBase) - strlen(delimiter) >= size) { strLstAddZSub(this, stringBase, (size_t)((stringMatchLast - strlen(delimiter)) - stringBase)); stringBase = stringMatchLast; } strLstAddZ(this, stringBase); } } while (stringMatch != NULL); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN String * helpRenderText( const String *const text, const bool internal, const bool beta, const size_t indent, const bool indentFirst, const size_t length) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, text); FUNCTION_TEST_PARAM(BOOL, internal); FUNCTION_TEST_PARAM(BOOL, beta); FUNCTION_TEST_PARAM(SIZE, indent); FUNCTION_TEST_PARAM(BOOL, indentFirst); FUNCTION_TEST_PARAM(SIZE, length); FUNCTION_TEST_END(); ASSERT(text != NULL); ASSERT(length > 0); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { // Split the text into paragraphs const StringList *const lineList = strLstNewSplitZ( strNewFmt( "%s%s", strZ(text), beta ? "\n\nFOR BETA TESTING ONLY. DO NOT USE IN PRODUCTION." : (internal ? "\n\nFOR INTERNAL USE ONLY. DO NOT USE IN PRODUCTION." : "")), "\n"); // Iterate through each paragraph and split the lines according to the line length for (unsigned int lineIdx = 0; lineIdx < strLstSize(lineList); lineIdx++) { // Add LF if there is already content if (strSize(result) != 0) strCat(result, LF_STR); // Split the paragraph into lines that don't exceed the line length const StringList *const partList = helpRenderSplitSize(strLstGet(lineList, lineIdx), " ", length - indent); for (unsigned int partIdx = 0; partIdx < strLstSize(partList); partIdx++) { // Indent when required if (partIdx != 0 || indentFirst) { if (partIdx != 0) strCat(result, LF_STR); if (strSize(strLstGet(partList, partIdx))) strCatFmt(result, "%*s", (int)indent, ""); } // Add the line strCat(result, strLstGet(partList, partIdx)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Helper functions for helpRender() to output values as strings ***********************************************************************************************************************************/ static String * helpRenderValueIdx(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); String *result = NULL; if (cfgOptionIdxSource(optionId, optionIdx) != cfgSourceDefault) { result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { const Variant *const value = cfgOptionIdxVar(optionId, optionIdx); ASSERT(value != NULL); switch (varType(value)) { case varTypeKeyValue: { const KeyValue *const optionKv = varKv(value); const VariantList *const keyList = kvKeyList(optionKv); for (unsigned int keyIdx = 0; keyIdx < varLstSize(keyList); keyIdx++) { if (keyIdx != 0) strCatZ(result, ", "); strCatFmt( result, "%s=%s", strZ(varStr(varLstGet(keyList, keyIdx))), strZ(varStrForce(kvGet(optionKv, varLstGet(keyList, keyIdx))))); } break; } case varTypeVariantList: { const VariantList *const list = varVarLst(value); for (unsigned int listIdx = 0; listIdx < varLstSize(list); listIdx++) { if (listIdx != 0) strCatZ(result, ", "); strCatFmt(result, "%s", strZ(varStr(varLstGet(list, listIdx)))); } break; } default: strCat(result, cfgOptionIdxDisplay(optionId, optionIdx)); break; } } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN(STRING, result); } static String * helpRenderValue(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); String *result = helpRenderValueIdx(optionId, 0); if (cfgOptionGroup(optionId)) { MEM_CONTEXT_TEMP_BEGIN() { for (unsigned int optionIdx = 1; optionIdx < cfgOptionGroupIdxTotal(cfgOptionGroupId(optionId)); optionIdx++) { const String *const value = helpRenderValueIdx(optionId, optionIdx); if (!strEq(result, value)) { MEM_CONTEXT_PRIOR_BEGIN() { strFree(result); result = strNewZ(""); } MEM_CONTEXT_PRIOR_END(); break; } } } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Helper function for helpRender() to output defaults ***********************************************************************************************************************************/ static String * helpRenderDefault(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); String *result = strDup(cfgOptionIdxDefaultValue(optionId, 0)); if (cfgOptionGroup(optionId)) { MEM_CONTEXT_TEMP_BEGIN() { for (unsigned int optionIdx = 1; optionIdx < cfgOptionGroupIdxTotal(cfgOptionGroupId(optionId)); optionIdx++) { const String *const defaultValue = cfgOptionIdxDefaultValue(optionId, optionIdx); if (!strEq(result, defaultValue)) { MEM_CONTEXT_PRIOR_BEGIN() { strFree(result); result = strNewZ(""); } MEM_CONTEXT_PRIOR_END(); break; } } } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Determine if the first character of a summary should be lower-case ***********************************************************************************************************************************/ static String * helpRenderSummary(const String *const summary) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, summary); FUNCTION_TEST_END(); ASSERT(summary != NULL); // Strip final period off summary String *const result = strCatN(strNew(), summary, strSize(summary) - 1); // Lower-case first letter if first word does not appear to be an acronym or proper name unsigned int totalLetter = 0; unsigned int totalCapital = 0; for (unsigned int resultIdx = 0; resultIdx < strSize(result); resultIdx++) { const char resultChar = strZ(result)[resultIdx]; if (resultChar == ' ') break; if (isalpha(resultChar)) totalLetter++; if (isupper(resultChar)) totalCapital++; } if (totalCapital == 1 && totalCapital != totalLetter) strFirstLower(result); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Render help to a string ***********************************************************************************************************************************/ // Stored unpacked command data typedef struct HelpCommandData { bool internal; // Is the command internal? const String *summary; // Short summary of the command const String *description; // Full description of the command } HelpCommandData; // Stored unpacked option data typedef struct HelpOptionData { bool internal; // Is the option internal? const String *section; // eg. general, command const String *summary; // Short summary of the option const String *description; // Full description of the option StringList *deprecatedNames; // Deprecated names for the option } HelpOptionData; static String * helpRender(void) { FUNCTION_TEST_VOID(); String *const result = strCatZ(strNew(), PROJECT_NAME " " PROJECT_VERSION); // Display version only if (!cfgCommandHelp() && ((cfgCommand() == cfgCmdHelp && cfgOptionBool(cfgOptVersion) && !cfgOptionBool(cfgOptHelp)) || cfgCommand() == cfgCmdVersion)) { if (cfgCommand() == cfgCmdVersion && cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_VERSION_OUTPUT_NUM) FUNCTION_TEST_RETURN(STRING, strCatFmt(strTrunc(result), "%d", PROJECT_VERSION_NUM)); else { strCatChr(result, '\n'); FUNCTION_TEST_RETURN(STRING, result); } } MEM_CONTEXT_TEMP_BEGIN() { // Set a small buffer size to minimize memory usage ioBufferSizeSet(8192); // Read pack from compressed buffer IoRead *const helpRead = ioBufferReadNew(BUF(helpData, sizeof(helpData))); ioFilterGroupAdd(ioReadFilterGroup(helpRead), bz2DecompressNew(false)); ioReadOpen(helpRead); PackRead *const pckHelp = pckReadNewIo(helpRead); // Unpack command data HelpCommandData *const commandData = memNew(sizeof(HelpCommandData) * CFG_COMMAND_TOTAL); pckReadArrayBeginP(pckHelp); for (ConfigCommand commandId = 0; commandId < CFG_COMMAND_TOTAL; commandId++) { commandData[commandId] = (HelpCommandData) { .internal = pckReadBoolP(pckHelp), .summary = pckReadStrP(pckHelp), .description = pckReadStrP(pckHelp), }; } pckReadArrayEndP(pckHelp); // Message for more help when it is available const String *more = NULL; // Display general help if (!cfgCommandHelp()) { strCatZ( result, " - General help\n" "\n" "Usage:\n" " " PROJECT_BIN " [options] [command]\n" "\n" "Commands:\n"); // Find size of longest command name size_t commandSizeMax = 0; for (ConfigCommand commandId = 0; commandId < CFG_COMMAND_TOTAL; commandId++) { if (commandData[commandId].internal) continue; if (strlen(cfgParseCommandName(commandId)) > commandSizeMax) commandSizeMax = strlen(cfgParseCommandName(commandId)); } // Output help for each command for (ConfigCommand commandId = 0; commandId < CFG_COMMAND_TOTAL; commandId++) { if (commandData[commandId].internal) continue; strCatFmt( result, " %s%*s%s\n", cfgParseCommandName(commandId), (int)(commandSizeMax - strlen(cfgParseCommandName(commandId)) + 2), "", strZ( helpRenderText( helpRenderSummary(commandData[commandId].summary), false, false, commandSizeMax + 6, false, CONSOLE_WIDTH))); } // Construct message for more help more = strNewZ("[command]"); } else { const ConfigCommand commandId = cfgCommand(); const char *const commandName = cfgParseCommandName(commandId); // Unpack option data HelpOptionData *const optionData = memNew(sizeof(HelpOptionData) * CFG_OPTION_TOTAL); pckReadArrayBeginP(pckHelp); for (ConfigOption optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++) { optionData[optionId] = (HelpOptionData) { .internal = pckReadBoolP(pckHelp), .section = pckReadStrP(pckHelp, .defaultValue = STRDEF("general")), .summary = pckReadStrP(pckHelp), .description = pckReadStrP(pckHelp), }; // Unpack deprecated names if (!pckReadNullP(pckHelp)) { optionData[optionId].deprecatedNames = strLstNew(); pckReadArrayBeginP(pckHelp); while (pckReadNext(pckHelp)) strLstAdd(optionData[optionId].deprecatedNames, pckReadStrP(pckHelp)); pckReadArrayEndP(pckHelp); } // Unpack command overrides if (!pckReadNullP(pckHelp)) { pckReadArrayBeginP(pckHelp); while (pckReadNext(pckHelp)) { // Get command override id const ConfigCommand commandIdArray = pckReadId(pckHelp) - 1; // Unpack override data pckReadObjBeginP(pckHelp, .id = commandIdArray + 1); const bool internal = pckReadBoolP(pckHelp, .defaultValue = optionData[optionId].internal); const String *const summary = pckReadStrP(pckHelp, .defaultValue = optionData[optionId].summary); const String *const description = pckReadStrP(pckHelp, .defaultValue = optionData[optionId].description); pckReadObjEndP(pckHelp); // Only use overrides for the current command if (commandId == commandIdArray) { optionData[optionId].internal = internal; optionData[optionId].section = NULL; optionData[optionId].summary = summary; optionData[optionId].description = description; } } pckReadArrayEndP(pckHelp); } } pckReadArrayEndP(pckHelp); // Output command part of title strCatFmt(result, " - '%s' command", commandName); // If no additional params then this is command help if (strLstEmpty(cfgCommandParam())) { // Output command summary and description. Add a warning for internal commands. CHECK( AssertError, commandData[commandId].summary != NULL && commandData[commandId].description != NULL, "command help missing"); strCatFmt( result, " help\n" "\n" "%s\n" "\n" "%s\n", strZ(helpRenderText(commandData[commandId].summary, false, false, 0, true, CONSOLE_WIDTH)), strZ( helpRenderText( commandData[commandId].description, commandData[commandId].internal, false, 0, true, CONSOLE_WIDTH))); // Construct key/value of sections and options KeyValue *const optionKv = kvNew(); size_t optionSizeMax = 0; for (unsigned int optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++) { if (cfgParseOptionValid(commandId, cfgCmdRoleMain, optionId) && !optionData[optionId].internal) { const String *section = optionData[optionId].section; if (section == NULL || (!strEqZ(section, "general") && !strEqZ(section, "log") && !strEqZ(section, "maintainer") && !strEqZ(section, "repository") && !strEqZ(section, "stanza"))) { section = strNewZ("command"); } kvAdd(optionKv, VARSTR(section), VARUINT(optionId)); if (strlen(cfgParseOptionName(optionId)) > optionSizeMax) optionSizeMax = strlen(cfgParseOptionName(optionId)); } } // Output sections const StringList *const sectionList = strLstSort(strLstNewVarLst(kvKeyList(optionKv)), sortOrderAsc); for (unsigned int sectionIdx = 0; sectionIdx < strLstSize(sectionList); sectionIdx++) { const String *const section = strLstGet(sectionList, sectionIdx); strCatFmt(result, "\n%s Options:\n\n", strZ(strFirstUpper(strDup(section)))); // Output options const VariantList *const optionList = kvGetList(optionKv, VARSTR(section)); for (unsigned int optionIdx = 0; optionIdx < varLstSize(optionList); optionIdx++) { const ConfigOption optionId = varUInt(varLstGet(optionList, optionIdx)); String *const summary = helpRenderSummary(optionData[optionId].summary); // Output current and default values if they exist const String *const defaultValue = helpRenderDefault(optionId); const String *const value = helpRenderValue(optionId); if (value != NULL || defaultValue != NULL) { strCatZ(summary, " ["); if (value != NULL) strCatFmt(summary, "current=%s", cfgParseOptionSecure(optionId) ? "" : strZ(value)); if (defaultValue != NULL) { if (value != NULL) strCatZ(summary, ", "); strCatFmt(summary, "default=%s", strZ(defaultValue)); } strCatZ(summary, "]"); } // Output option help strCatFmt( result, " --%s%*s%s\n", cfgParseOptionName(optionId), (int)(optionSizeMax - strlen(cfgParseOptionName(optionId)) + 2), "", strZ(helpRenderText(summary, false, false, optionSizeMax + 6, false, CONSOLE_WIDTH))); } } // Construct message for more help ASSERT(optionSizeMax > 0); more = strNewFmt("%s [option]", commandName); } // Else option help for the specified command else { // Make sure only one option was specified if (strLstSize(cfgCommandParam()) > 1) THROW(ParamInvalidError, "only one option allowed for option help"); // Ensure the option is valid const String *const optionName = strLstGet(cfgCommandParam(), 0); const CfgParseOptionResult option = cfgParseOptionP(optionName, .ignoreMissingIndex = true); // Error when option is not found or is invalid for the current command if (!option.found || !cfgParseOptionValid(cfgCommand(), cfgCmdRoleMain, option.id)) THROW_FMT(OptionInvalidError, "option '%s' is not valid for command '%s'", strZ(optionName), commandName); // Output option summary and description. Add a warning for internal and beta options. CHECK( AssertError, optionData[option.id].summary != NULL && optionData[option.id].description != NULL, "option help missing"); strCatFmt( result, " - '%s' option help\n" "\n" "%s\n" "\n" "%s\n", cfgParseOptionName(option.id), strZ(helpRenderText(optionData[option.id].summary, false, false, 0, true, CONSOLE_WIDTH)), strZ( helpRenderText( optionData[option.id].description, optionData[option.id].internal, option.beta, 0, true, CONSOLE_WIDTH))); // Output current and default values if they exist const String *const defaultValue = helpRenderDefault(option.id); const String *const value = helpRenderValue(option.id); if (value != NULL || defaultValue != NULL) { strCat(result, LF_STR); if (value != NULL) { strCatZ(result, "current:"); if (cfgParseOptionSecure(option.id)) strCatZ(result, " \n"); else if (!strEqZ(value, "")) strCatFmt(result, " %s\n", strZ(value)); else { const unsigned int groupId = cfgOptionGroupId(option.id); strCatChr(result, '\n'); for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(groupId); optionIdx++) { const String *const value = helpRenderValueIdx(option.id, optionIdx); if (value != NULL) strCatFmt(result, " %s: %s\n", cfgOptionGroupName(groupId, optionIdx), strZ(value)); } } } if (defaultValue != NULL) { strCatZ(result, "default:"); if (!strEqZ(defaultValue, "")) strCatFmt(result, " %s\n", strZ(defaultValue)); else { const unsigned int groupId = cfgOptionGroupId(option.id); strCatChr(result, '\n'); for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(groupId); optionIdx++) { const String *const defaultValue = cfgOptionIdxDefaultValue(option.id, optionIdx); if (defaultValue != NULL) strCatFmt(result, " %s: %s\n", cfgOptionGroupName(groupId, optionIdx), strZ(defaultValue)); } } } } // Output alternate name (call it deprecated so the user will know not to use it) if (optionData[option.id].deprecatedNames != NULL) { strCatFmt( result, "\ndeprecated name%s: %s\n", strLstSize(optionData[option.id].deprecatedNames) > 1 ? "s" : "", strZ(strLstJoin(optionData[option.id].deprecatedNames, ", "))); } } } // If there is more help available output a message to let the user know if (more != NULL) strCatFmt(result, "\nUse '" PROJECT_BIN " help %s' for more information.\n", strZ(more)); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void cmdHelp(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { ioFdWriteOneStr(STDOUT_FILENO, helpRender()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/help/help.h0000664000175100017510000000154415265101745017644 0ustar00runnerrunner/*********************************************************************************************************************************** Help Command ***********************************************************************************************************************************/ #ifndef COMMAND_HELP_HELP_H #define COMMAND_HELP_HELP_H #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Render help and output to stdout FN_EXTERN void cmdHelp(void); // Render help text FN_EXTERN String *helpRenderText(const String *text, bool internal, bool beta, size_t indent, bool indentFirst, size_t length); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/help/meson.build0000664000175100017510000000215315265101745020702 0ustar00runnerrunner#################################################################################################################################### # Generate help #################################################################################################################################### # Generate help.auto.c.inc, unless this is a distribution tarball that ships the file. The generator writes into the build path so # the file lands where the output is declared, else it would be missing there and ninja would run the generator on every build. if not dist help_auto_c_inc = custom_target( 'help.auto.c.inc', output: 'help.auto.c.inc', depend_files: [ '../../../build/config.yaml', '../../../doc/xml/reference.xml', build_code_lib, ], command: [ build_code, 'help', '--repo-path=@CURRENT_SOURCE_DIR@/../../..', '--build-path=@BUILD_ROOT@', ], ) else # The shipped file is included by help.c via the implicit source include path, so it is not a compiled source help_auto_c_inc = [] endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/info/0000775000175100017510000000000015265101745016542 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/info/info.c0000664000175100017510000027160115265101745017650 0ustar00runnerrunner/*********************************************************************************************************************************** Info Command ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/archive/common.h" #include "command/info/info.h" #include "command/lock.h" #include "common/crypto/common.h" #include "common/debug.h" #include "common/io/fdWrite.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/type/json.h" #include "config/config.h" #include "info/info.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "info/infoPg.h" #include "info/manifest/manifest.h" #include "postgres/interface.h" #include "storage/helper.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ // Naming convention: _KEY__VAR. If the key exists in multiple sections, then _ is omitted. VARIANT_STRDEF_STATIC(ARCHIVE_KEY_MIN_VAR, "min"); VARIANT_STRDEF_STATIC(ARCHIVE_KEY_MAX_VAR, "max"); VARIANT_STRDEF_STATIC(BACKREST_KEY_FORMAT_VAR, "format"); VARIANT_STRDEF_STATIC(BACKREST_KEY_VERSION_VAR, "version"); VARIANT_STRDEF_STATIC(BACKUP_KEY_ANNOTATION_VAR, "annotation"); VARIANT_STRDEF_STATIC(BACKUP_KEY_BACKREST_VAR, "backrest"); VARIANT_STRDEF_STATIC(BACKUP_KEY_ERROR_VAR, "error"); VARIANT_STRDEF_STATIC(BACKUP_KEY_ERROR_LIST_VAR, "error-list"); VARIANT_STRDEF_STATIC(BACKUP_KEY_DATABASE_REF_VAR, "database-ref"); VARIANT_STRDEF_STATIC(BACKUP_KEY_INFO_VAR, "info"); VARIANT_STRDEF_STATIC(BACKUP_KEY_LABEL_VAR, "label"); VARIANT_STRDEF_STATIC(BACKUP_KEY_LINK_VAR, "link"); VARIANT_STRDEF_STATIC(BACKUP_KEY_LSN_VAR, "lsn"); VARIANT_STRDEF_STATIC(BACKUP_KEY_PRIOR_VAR, "prior"); VARIANT_STRDEF_STATIC(BACKUP_KEY_REFERENCE_VAR, "reference"); VARIANT_STRDEF_STATIC(BACKUP_KEY_REFERENCE_FULL_VAR, "reference-full"); VARIANT_STRDEF_STATIC(BACKUP_KEY_REFERENCE_DIFF_VAR, "reference-diff"); VARIANT_STRDEF_STATIC(BACKUP_KEY_REFERENCE_INCR_VAR, "reference-incr"); VARIANT_STRDEF_STATIC(BACKUP_KEY_TABLESPACE_VAR, "tablespace"); VARIANT_STRDEF_STATIC(BACKUP_KEY_TIMESTAMP_VAR, "timestamp"); VARIANT_STRDEF_STATIC(BACKUP_KEY_TYPE_VAR, "type"); VARIANT_STRDEF_STATIC(DB_KEY_ID_VAR, "id"); VARIANT_STRDEF_STATIC(DB_KEY_SYSTEM_ID_VAR, "system-id"); VARIANT_STRDEF_STATIC(DB_KEY_VERSION_VAR, "version"); VARIANT_STRDEF_STATIC(INFO_KEY_REPOSITORY_VAR, "repository"); VARIANT_STRDEF_STATIC(KEY_ARCHIVE_VAR, "archive"); VARIANT_STRDEF_STATIC(KEY_CIPHER_VAR, "cipher"); VARIANT_STRDEF_STATIC(KEY_DATABASE_VAR, "database"); VARIANT_STRDEF_STATIC(KEY_DELTA_VAR, "delta"); VARIANT_STRDEF_STATIC(KEY_DELTA_MAP_VAR, "delta-map"); VARIANT_STRDEF_STATIC(KEY_DESTINATION_VAR, "destination"); VARIANT_STRDEF_STATIC(KEY_NAME_VAR, "name"); VARIANT_STRDEF_STATIC(KEY_OID_VAR, "oid"); VARIANT_STRDEF_STATIC(KEY_REPO_KEY_VAR, "repo-key"); VARIANT_STRDEF_STATIC(KEY_SIZE_VAR, "size"); VARIANT_STRDEF_STATIC(KEY_SIZE_MAP_VAR, "size-map"); VARIANT_STRDEF_STATIC(KEY_START_VAR, "start"); VARIANT_STRDEF_STATIC(KEY_STOP_VAR, "stop"); VARIANT_STRDEF_STATIC(REPO_KEY_KEY_VAR, "key"); VARIANT_STRDEF_STATIC(STANZA_KEY_BACKUP_VAR, "backup"); VARIANT_STRDEF_STATIC(STANZA_KEY_REPO_VAR, "repo"); VARIANT_STRDEF_STATIC(STANZA_KEY_STATUS_VAR, "status"); VARIANT_STRDEF_STATIC(STANZA_KEY_DB_VAR, "db"); VARIANT_STRDEF_STATIC(STATUS_KEY_CODE_VAR, "code"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_VAR, "lock"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_BACKUP_VAR, "backup"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_HELD_VAR, "held"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_PERCENT_COMPLETE_VAR, "pct-cplt"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_RESTORE_VAR, "restore"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_SIZE_COMPLETE_VAR, "size-cplt"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_SIZE_VAR, "size"); VARIANT_STRDEF_STATIC(STATUS_KEY_MESSAGE_VAR, "message"); #define INFO_STANZA_STATUS_OK "ok" #define INFO_STANZA_STATUS_ERROR "error" #define INFO_STANZA_MIXED "mixed" #define INFO_STANZA_STATUS_CODE_OK 0 STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_OK_STR, "ok"); #define INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH 1 STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_MISSING_STANZA_PATH_STR, "missing stanza path"); #define INFO_STANZA_STATUS_CODE_NO_BACKUP 2 STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_NO_BACKUP_STR, "no valid backups"); #define INFO_STANZA_STATUS_CODE_MISSING_STANZA_DATA 3 STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_MISSING_STANZA_DATA_STR, "missing stanza data"); #define INFO_STANZA_STATUS_CODE_MIXED 4 // If the cipher or status of the stanza is different across repos, then the overall cipher or status message is mixed STRING_STATIC(INFO_STANZA_MESSAGE_MIXED_STR, "different across repos"); #define INFO_STANZA_STATUS_CODE_PG_MISMATCH 5 STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_PG_MISMATCH_STR, "database mismatch across repos"); #define INFO_STANZA_STATUS_CODE_BACKUP_MISSING 6 STRING_STATIC(INFO_STANZA_STATUS_CODE_BACKUP_MISSING_STR, "requested backup not found"); #define INFO_STANZA_STATUS_CODE_OTHER 99 #define INFO_STANZA_STATUS_MESSAGE_OTHER "other" STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_OTHER_STR, INFO_STANZA_STATUS_MESSAGE_OTHER); STRING_STATIC(INFO_STANZA_INVALID_STR, "[invalid]"); #define INFO_STANZA_STATUS_MESSAGE_LOCK_BACKUP "backup/expire running" #define INFO_STANZA_STATUS_MESSAGE_LOCK_RESTORE "restore running" /*********************************************************************************************************************************** Data types and structures ***********************************************************************************************************************************/ // Information for a lockfile of a stanza typedef struct InfoStanzaLock { bool held; // Is lock held on the system where info command is run? uint64_t sizeComplete; // Completed size of the backup/restore in bytes uint64_t size; // Total size of the backup/restore in bytes } InfoStanzaLock; #define FUNCTION_LOG_INFO_STANZA_LOCK_TYPE \ InfoStanzaLock * #define FUNCTION_LOG_INFO_STANZA_LOCK_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoStanzaLock", buffer, bufferSize) // Repository information for a stanza typedef struct InfoRepoData { unsigned int key; // User-defined repo key CipherType cipher; // Encryption type int stanzaStatus; // Status code of the stanza on this repo unsigned int backupIdx; // Index of the next backup that may be a candidate for sorting InfoBackup *backupInfo; // Contents of the backup.info file of the stanza on this repo InfoArchive *archiveInfo; // Contents of the archive.info file of the stanza on this repo Manifest *manifest; // Contents of manifest if backup requested and is on this repo String *error; // Formatted error InfoStanzaLock backupLock; // Info for backup lock on this repo } InfoRepoData; #define FUNCTION_LOG_INFO_REPO_DATA_TYPE \ InfoRepoData * #define FUNCTION_LOG_INFO_REPO_DATA_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoRepoData", buffer, bufferSize) // Stanza with repository list of information for each repository typedef struct InfoStanzaRepo { const String *name; // Name of the stanza uint64_t currentPgSystemId; // Current postgres system id for the stanza unsigned int currentPgVersion; // Current postgres version for the stanza InfoStanzaLock restoreLock; // Info for restore lock InfoRepoData *repoList; // List of configured repositories } InfoStanzaRepo; #define FUNCTION_LOG_INFO_STANZA_REPO_TYPE \ InfoStanzaRepo * #define FUNCTION_LOG_INFO_STANZA_REPO_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoStanzaRepo", buffer, bufferSize) // Group all databases with the same system-id and version together regardless of db-id or repo typedef struct DbGroup { uint64_t systemId; // Postgres database system id const String *version; // Postgres database version bool current; // Is this the current postgres database? String *archiveMin; // Minimum WAL found for this database over all repositories String *archiveMax; // Maximum WAL found for this database over all repositories VariantList *backupList; // List of backups found for this database over all repositories } DbGroup; #define FUNCTION_LOG_DB_GROUP_TYPE \ DbGroup * #define FUNCTION_LOG_DB_GROUP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "DbGroup", buffer, bufferSize) /*********************************************************************************************************************************** Helper function for reporting errors ***********************************************************************************************************************************/ static void infoStanzaErrorAdd(InfoRepoData *const repoList, const ErrorType *const type, const String *const message) { repoList->stanzaStatus = INFO_STANZA_STATUS_CODE_OTHER; repoList->error = strNewFmt("[%s] %s", errorTypeName(type), strZ(message)); // Free the info objects for this stanza since we cannot process it infoBackupFree(repoList->backupInfo); infoArchiveFree(repoList->archiveInfo); manifestFree(repoList->manifest); repoList->backupInfo = NULL; repoList->archiveInfo = NULL; repoList->manifest = NULL; } /*********************************************************************************************************************************** Add lock information to the target key-value ***********************************************************************************************************************************/ static void stanzaStatusLockAdd(KeyValue *targetKv, const Variant *const lockKey, const InfoStanzaLock *lock) { KeyValue *const lockKv = kvPutKv(targetKv, lockKey); kvPut(lockKv, STATUS_KEY_LOCK_HELD_VAR, VARBOOL(lock->held)); if (lock->size != 0) { kvPut(lockKv, STATUS_KEY_LOCK_SIZE_COMPLETE_VAR, VARUINT64(lock->sizeComplete)); kvPut(lockKv, STATUS_KEY_LOCK_SIZE_VAR, VARUINT64(lock->size)); if (cfgOptionSeq(cfgOptOutput) != CFGOPTVAL_OUTPUT_JSON) kvPut(lockKv, STATUS_KEY_LOCK_PERCENT_COMPLETE_VAR, VARUINT(cvtPctToUInt(lock->sizeComplete, lock->size))); } } /*********************************************************************************************************************************** Add per-repo backup lock information to the target key-value ***********************************************************************************************************************************/ static void stanzaStatusBackupLockAdd( KeyValue *const targetKv, const InfoRepoData *const repoList, const unsigned int repoIdxMin, const unsigned int repoIdxMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, targetKv); FUNCTION_TEST_PARAM(INFO_REPO_DATA, repoList); FUNCTION_TEST_PARAM(UINT, repoIdxMin); FUNCTION_TEST_PARAM(UINT, repoIdxMax); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(targetKv != NULL); ASSERT(repoList != NULL); ASSERT(repoIdxMin <= repoIdxMax); KeyValue *const lockKv = kvPutKv(targetKv, STATUS_KEY_LOCK_BACKUP_VAR); bool held = false; MEM_CONTEXT_TEMP_BEGIN() { // Report backup progress per repo VariantList *const repoLockList = varLstNew(); uint64_t sizeTotal = 0; uint64_t sizeCompleteTotal = 0; for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { const InfoStanzaLock *const lock = &repoList[repoIdx].backupLock; if (lock->held) { held = true; if (lock->size != 0) { Variant *const repoLockInfo = varNewKv(kvNew()); kvPut(varKv(repoLockInfo), REPO_KEY_KEY_VAR, VARUINT(repoList[repoIdx].key)); kvPut(varKv(repoLockInfo), STATUS_KEY_LOCK_SIZE_VAR, VARUINT64(lock->size)); kvPut(varKv(repoLockInfo), STATUS_KEY_LOCK_SIZE_COMPLETE_VAR, VARUINT64(lock->sizeComplete)); varLstAdd(repoLockList, repoLockInfo); sizeTotal += lock->size; sizeCompleteTotal += lock->sizeComplete; } } } // Report backup progress aggregated across all repos kvPut(lockKv, STATUS_KEY_LOCK_HELD_VAR, VARBOOL(held)); if (!varLstEmpty(repoLockList)) { // Add summed size/size-cplt for backwards compatibility kvPut(lockKv, STATUS_KEY_LOCK_SIZE_COMPLETE_VAR, VARUINT64(sizeCompleteTotal)); kvPut(lockKv, STATUS_KEY_LOCK_SIZE_VAR, VARUINT64(sizeTotal)); // Add pct-cplt for text output only if (cfgOptionSeq(cfgOptOutput) != CFGOPTVAL_OUTPUT_JSON) kvPut(lockKv, STATUS_KEY_LOCK_PERCENT_COMPLETE_VAR, VARUINT(cvtPctToUInt(sizeCompleteTotal, sizeTotal))); // Add per-repo detail kvPut(lockKv, STANZA_KEY_REPO_VAR, varNewVarLst(repoLockList)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Set the overall error status code and message for the stanza to the code and message passed ***********************************************************************************************************************************/ static void stanzaStatus( const int code, const InfoStanzaRepo *const stanzaData, const Variant *const stanzaInfo, const unsigned int repoIdxMin, const unsigned int repoIdxMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, code); FUNCTION_TEST_PARAM(INFO_STANZA_REPO, stanzaData); FUNCTION_TEST_PARAM(VARIANT, stanzaInfo); FUNCTION_TEST_PARAM(UINT, repoIdxMin); FUNCTION_TEST_PARAM(UINT, repoIdxMax); FUNCTION_TEST_END(); ASSERT((code >= 0 && code <= 6) || code == 99); ASSERT(stanzaData != NULL); ASSERT(stanzaInfo != NULL); KeyValue *const statusKv = kvPutKv(varKv(stanzaInfo), STANZA_KEY_STATUS_VAR); kvPut(statusKv, STATUS_KEY_CODE_VAR, VARINT(code)); switch (code) { case INFO_STANZA_STATUS_CODE_OK: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_OK_STR)); break; case INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_MISSING_STANZA_PATH_STR)); break; case INFO_STANZA_STATUS_CODE_MISSING_STANZA_DATA: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_MISSING_STANZA_DATA_STR)); break; case INFO_STANZA_STATUS_CODE_NO_BACKUP: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_NO_BACKUP_STR)); break; case INFO_STANZA_STATUS_CODE_MIXED: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_MESSAGE_MIXED_STR)); break; case INFO_STANZA_STATUS_CODE_PG_MISMATCH: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_PG_MISMATCH_STR)); break; case INFO_STANZA_STATUS_CODE_BACKUP_MISSING: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_CODE_BACKUP_MISSING_STR)); break; case INFO_STANZA_STATUS_CODE_OTHER: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_OTHER_STR)); break; } // Construct a specific lock part KeyValue *const lockKv = kvPutKv(statusKv, STATUS_KEY_LOCK_VAR); stanzaStatusBackupLockAdd(lockKv, stanzaData->repoList, repoIdxMin, repoIdxMax); stanzaStatusLockAdd(lockKv, STATUS_KEY_LOCK_RESTORE_VAR, &stanzaData->restoreLock); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Set the error status code and message for the stanza on the repo to the code and message passed ***********************************************************************************************************************************/ static void repoStanzaStatus(const int code, const Variant *const repoStanzaInfo, const InfoRepoData *const repoData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, code); FUNCTION_TEST_PARAM(VARIANT, repoStanzaInfo); FUNCTION_TEST_PARAM(INFO_REPO_DATA, repoData); FUNCTION_TEST_END(); ASSERT((code >= 0 && code <= 3) || code == 99 || code == 6); ASSERT(repoStanzaInfo != NULL); KeyValue *const statusKv = kvPutKv(varKv(repoStanzaInfo), STANZA_KEY_STATUS_VAR); kvPut(statusKv, STATUS_KEY_CODE_VAR, VARINT(code)); switch (code) { case INFO_STANZA_STATUS_CODE_OK: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_OK_STR)); break; case INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_MISSING_STANZA_PATH_STR)); break; case INFO_STANZA_STATUS_CODE_MISSING_STANZA_DATA: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_MISSING_STANZA_DATA_STR)); break; case INFO_STANZA_STATUS_CODE_NO_BACKUP: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_NO_BACKUP_STR)); break; case INFO_STANZA_STATUS_CODE_BACKUP_MISSING: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_CODE_BACKUP_MISSING_STR)); break; case INFO_STANZA_STATUS_CODE_OTHER: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(repoData->error)); break; } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Set the data for the archive section of the stanza for the database info from the backup.info file ***********************************************************************************************************************************/ static void archiveDbList( const String *const stanza, const InfoPgData *const pgData, VariantList *const archiveSection, const InfoArchive *const info, const bool currentDb, const unsigned int repoIdx, const unsigned int repoKey) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, stanza); FUNCTION_TEST_PARAM_P(INFO_PG_DATA, pgData); FUNCTION_TEST_PARAM(VARIANT_LIST, archiveSection); FUNCTION_TEST_PARAM(BOOL, currentDb); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_PARAM(UINT, repoKey); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(stanza != NULL); ASSERT(pgData != NULL); ASSERT(archiveSection != NULL); // With multiple DB versions, the backup.info history-id may not be the same as archive.info history-id, so the archive path // must be built by retrieving the archive id given the db version and system id of the backup.info file. If there is no match, // an error will be thrown. const String *const archiveId = infoArchiveIdHistoryMatch(info, pgData->id, pgData->version, pgData->systemId); String *const archivePath = strNewFmt(STORAGE_PATH_ARCHIVE "/%s/%s", strZ(stanza), strZ(archiveId)); const String *archiveStart = NULL; const String *archiveStop = NULL; Variant *const archiveInfo = varNewKv(kvNew()); const Storage *const storageRepo = storageRepoIdx(repoIdx); // Get a list of WAL directories in the archive repo from oldest to newest, if any exist const StringList *const walDir = strLstSort( storageListP(storageRepo, archivePath, .expression = WAL_SEGMENT_DIR_REGEXP_STR), sortOrderAsc); if (!strLstEmpty(walDir)) { // Not every WAL dir has WAL files so check each for (unsigned int idx = 0; idx < strLstSize(walDir); idx++) { // Get a list of all WAL in this WAL dir and sort the list from oldest to newest to get the oldest starting WAL archived // for this db const StringList *const list = strLstSort( storageListP( storageRepo, strNewFmt("%s/%s", strZ(archivePath), strZ(strLstGet(walDir, idx))), .expression = WAL_SEGMENT_FILE_REGEXP_STR), sortOrderAsc); // If wal segments are found, get the oldest one as the archive start if (!strLstEmpty(list)) { archiveStart = strSubN(strLstGet(list, 0), 0, 24); break; } } // Iterate through the directory list in reverse processing newest first. Cast comparison to an int for readability. for (unsigned int idx = strLstSize(walDir) - 1; (int)idx >= 0; idx--) { // Get a list of all WAL in this WAL dir and sort the list from newest to oldest to get the newest ending WAL archived // for this db const StringList *const list = strLstSort( storageListP( storageRepo, strNewFmt("%s/%s", strZ(archivePath), strZ(strLstGet(walDir, idx))), .expression = WAL_SEGMENT_FILE_REGEXP_STR), sortOrderDesc); // If wal segments are found, get the newest one as the archive stop if (!strLstEmpty(list)) { archiveStop = strSubN(strLstGet(list, 0), 0, 24); break; } } } // If there is an archive or the database is the current database then store it if (currentDb || archiveStart != NULL) { // Add empty database section to archiveInfo and then fill in database id from the backup.info KeyValue *const databaseInfo = kvPutKv(varKv(archiveInfo), KEY_DATABASE_VAR); kvPut(databaseInfo, DB_KEY_ID_VAR, VARUINT(pgData->id)); kvPut(databaseInfo, KEY_REPO_KEY_VAR, VARUINT(repoKey)); kvPut(varKv(archiveInfo), DB_KEY_ID_VAR, VARSTR(archiveId)); kvPut(varKv(archiveInfo), ARCHIVE_KEY_MIN_VAR, (archiveStart != NULL ? VARSTR(archiveStart) : (Variant *)NULL)); kvPut(varKv(archiveInfo), ARCHIVE_KEY_MAX_VAR, (archiveStop != NULL ? VARSTR(archiveStop) : (Variant *)NULL)); varLstAdd(archiveSection, archiveInfo); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Add the backup data to the backup section ***********************************************************************************************************************************/ static void backupListAdd( VariantList *backupSection, InfoBackupData *backupData, const String *backupLabel, InfoRepoData *repoData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT_LIST, backupSection); // The section to add the backup data to FUNCTION_TEST_PARAM_P(INFO_BACKUP_DATA, backupData); // The data for the backup FUNCTION_TEST_PARAM(STRING, backupLabel); // Backup label to filter if requested by the user FUNCTION_TEST_PARAM(INFO_REPO_DATA, repoData); // The repo data where this backup is located FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(backupSection != NULL); ASSERT(backupData != NULL); ASSERT(repoData != NULL); Variant *const backupInfo = varNewKv(kvNew()); // Flags used to decide what data to add const bool outputJson = cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_OUTPUT_JSON; // main keys kvPut(varKv(backupInfo), BACKUP_KEY_LABEL_VAR, VARSTR(backupData->backupLabel)); kvPut(varKv(backupInfo), BACKUP_KEY_TYPE_VAR, VARSTR(strNewStrId(backupData->backupType))); kvPut( varKv(backupInfo), BACKUP_KEY_PRIOR_VAR, (backupData->backupPrior != NULL ? VARSTR(backupData->backupPrior) : NULL)); kvPut( varKv(backupInfo), BACKUP_KEY_REFERENCE_VAR, (backupData->backupReference != NULL ? varNewVarLst(varLstNewStrLst(backupData->backupReference)) : NULL)); // Display complete reference list only for json output or --set text. Otherwise keep track of full, diff and incremental to // display a summary. if (backupData->backupReference != NULL && backupLabel == NULL && !outputJson) { StringList *const fullList = strLstNew(); StringList *const diffList = strLstNew(); StringList *const incrList = strLstNew(); for (unsigned int backupIdx = 0; backupIdx < strLstSize(backupData->backupReference); backupIdx++) { const String *const backupReferenceLabel = strLstGet(backupData->backupReference, backupIdx); if (regExpMatchOne(backupRegExpP(.full = true), backupReferenceLabel)) strLstAdd(fullList, backupReferenceLabel); if (regExpMatchOne(backupRegExpP(.differential = true), backupReferenceLabel)) strLstAdd(diffList, backupReferenceLabel); if (regExpMatchOne(backupRegExpP(.incremental = true), backupReferenceLabel)) strLstAdd(incrList, backupReferenceLabel); } // The reference list will always contain at least 1 full kvPut(varKv(backupInfo), BACKUP_KEY_REFERENCE_FULL_VAR, varNewVarLst(varLstNewStrLst(fullList))); kvPut( varKv(backupInfo), BACKUP_KEY_REFERENCE_DIFF_VAR, strLstSize(diffList) > 0 ? varNewVarLst(varLstNewStrLst(diffList)) : NULL); kvPut( varKv(backupInfo), BACKUP_KEY_REFERENCE_INCR_VAR, strLstSize(incrList) > 0 ? varNewVarLst(varLstNewStrLst(incrList)) : NULL); } // archive section KeyValue *const archiveInfo = kvPutKv(varKv(backupInfo), KEY_ARCHIVE_VAR); kvPut( archiveInfo, KEY_START_VAR, (backupData->backupArchiveStart != NULL ? VARSTR(backupData->backupArchiveStart) : NULL)); kvPut( archiveInfo, KEY_STOP_VAR, (backupData->backupArchiveStop != NULL ? VARSTR(backupData->backupArchiveStop) : NULL)); // backrest section KeyValue *const backrestInfo = kvPutKv(varKv(backupInfo), BACKUP_KEY_BACKREST_VAR); kvPut(backrestInfo, BACKREST_KEY_FORMAT_VAR, VARUINT(backupData->backrestFormat)); kvPut(backrestInfo, BACKREST_KEY_VERSION_VAR, VARSTR(backupData->backrestVersion)); // database section KeyValue *const dbInfo = kvPutKv(varKv(backupInfo), KEY_DATABASE_VAR); kvPut(dbInfo, DB_KEY_ID_VAR, VARUINT(backupData->backupPgId)); kvPut(dbInfo, KEY_REPO_KEY_VAR, VARUINT(repoData->key)); // info section KeyValue *const infoInfo = kvPutKv(varKv(backupInfo), BACKUP_KEY_INFO_VAR); kvPut(infoInfo, KEY_SIZE_VAR, VARUINT64(backupData->backupInfoSize)); kvPut(infoInfo, KEY_DELTA_VAR, VARUINT64(backupData->backupInfoSizeDelta)); // info:repository section KeyValue *const repoInfo = kvPutKv(infoInfo, INFO_KEY_REPOSITORY_VAR); if (backupData->backupInfoRepoSizeMap == NULL) kvPut(repoInfo, KEY_SIZE_VAR, VARUINT64(backupData->backupInfoRepoSize)); kvPut(repoInfo, KEY_DELTA_VAR, VARUINT64(backupData->backupInfoRepoSizeDelta)); if (outputJson && backupData->backupInfoRepoSizeMap != NULL) { kvPut(repoInfo, KEY_SIZE_MAP_VAR, backupData->backupInfoRepoSizeMap); kvPut(repoInfo, KEY_DELTA_MAP_VAR, backupData->backupInfoRepoSizeMapDelta); } // timestamp section KeyValue *const timeInfo = kvPutKv(varKv(backupInfo), BACKUP_KEY_TIMESTAMP_VAR); // time_t is generally a signed int so cast it to uint64 since it can never be negative (before 1970) in our system kvPut(timeInfo, KEY_START_VAR, VARUINT64((uint64_t)backupData->backupTimestampStart)); kvPut(timeInfo, KEY_STOP_VAR, VARUINT64((uint64_t)backupData->backupTimestampStop)); // Report errors only if the error status is known if (backupData->backupError != NULL) kvPut(varKv(backupInfo), BACKUP_KEY_ERROR_VAR, backupData->backupError); // Add start/stop backup lsn info to json output or --set text if ((outputJson || backupLabel != NULL) && backupData->backupLsnStart != NULL && backupData->backupLsnStop != NULL) { KeyValue *const lsnInfo = kvPutKv(varKv(backupInfo), BACKUP_KEY_LSN_VAR); kvPut(lsnInfo, KEY_START_VAR, VARSTR(backupData->backupLsnStart)); kvPut(lsnInfo, KEY_STOP_VAR, VARSTR(backupData->backupLsnStop)); } // Add annotations to json output or --set text if ((outputJson || backupLabel != NULL) && backupData->backupAnnotation != NULL) kvPut(varKv(backupInfo), BACKUP_KEY_ANNOTATION_VAR, backupData->backupAnnotation); // If a backup label was specified and this is that label, then get the data from the loaded manifest if (backupLabel != NULL) { // Get the list of databases in this backup VariantList *const databaseSection = varLstNew(); for (unsigned int dbIdx = 0; dbIdx < manifestDbTotal(repoData->manifest); dbIdx++) { const ManifestDb *const db = manifestDb(repoData->manifest, dbIdx); // Do not display template databases if (!pgDbIsTemplate(db->name)) { Variant *const database = varNewKv(kvNew()); kvPut(varKv(database), KEY_NAME_VAR, VARSTR(db->name)); kvPut(varKv(database), KEY_OID_VAR, VARUINT64(db->id)); varLstAdd(databaseSection, database); } } // Add the database section even if none found kvPut(varKv(backupInfo), BACKUP_KEY_DATABASE_REF_VAR, varNewVarLst(databaseSection)); // Get symlinks and tablespaces VariantList *const linkSection = varLstNew(); VariantList *const tablespaceSection = varLstNew(); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(repoData->manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(repoData->manifest, targetIdx); Variant *const link = varNewKv(kvNew()); Variant *const tablespace = varNewKv(kvNew()); if (target->type == manifestTargetTypeLink) { if (target->tablespaceName != NULL) { kvPut(varKv(tablespace), KEY_NAME_VAR, VARSTR(target->tablespaceName)); kvPut(varKv(tablespace), KEY_DESTINATION_VAR, VARSTR(target->path)); kvPut(varKv(tablespace), KEY_OID_VAR, VARUINT64(target->tablespaceId)); varLstAdd(tablespaceSection, tablespace); } else if (target->file != NULL) { kvPut(varKv(link), KEY_NAME_VAR, varNewStr(target->file)); kvPut( varKv(link), KEY_DESTINATION_VAR, varNewStr(strNewFmt("%s/%s", strZ(target->path), strZ(target->file)))); varLstAdd(linkSection, link); } else { kvPut(varKv(link), KEY_NAME_VAR, VARSTR(manifestPathPg(target->name))); kvPut(varKv(link), KEY_DESTINATION_VAR, VARSTR(target->path)); varLstAdd(linkSection, link); } } } kvPut(varKv(backupInfo), BACKUP_KEY_LINK_VAR, (!varLstEmpty(linkSection) ? varNewVarLst(linkSection) : NULL)); kvPut( varKv(backupInfo), BACKUP_KEY_TABLESPACE_VAR, (!varLstEmpty(tablespaceSection) ? varNewVarLst(tablespaceSection) : NULL)); // Get the list of files with an error VariantList *const checksumPageErrorList = varLstNew(); for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(repoData->manifest); fileIdx++) { const ManifestFile file = manifestFile(repoData->manifest, fileIdx); if (file.checksumPageError) varLstAdd(checksumPageErrorList, varNewStr(manifestPathPg(file.name))); } if (!varLstEmpty(checksumPageErrorList)) { kvPut(varKv(backupInfo), BACKUP_KEY_ERROR_LIST_VAR, varNewVarLst(checksumPageErrorList)); // It is possible that backup-error is not set in backup.info but there are errors in manifest because backup-error was // added in a later version than manifest errors. However, it should not be possible for backup-error to be present but // false if there are errors in the manifest. In production this condition will be ignored and error set to true. ASSERT( kvGet(varKv(backupInfo), BACKUP_KEY_ERROR_VAR) == NULL || varBool(kvGet(varKv(backupInfo), BACKUP_KEY_ERROR_VAR))); kvPut(varKv(backupInfo), BACKUP_KEY_ERROR_VAR, BOOL_TRUE_VAR); } manifestFree(repoData->manifest); repoData->manifest = NULL; } varLstAdd(backupSection, backupInfo); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** For each current backup in the backup.info file of the stanza, set the data for the backup section ***********************************************************************************************************************************/ static void backupList( VariantList *const backupSection, InfoStanzaRepo *const stanzaData, const String *const backupLabel, const unsigned int repoIdxMin, const unsigned int repoIdxMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT_LIST, backupSection); // The section to add the backup data to FUNCTION_TEST_PARAM(INFO_STANZA_REPO, stanzaData); // The data for the stanza FUNCTION_TEST_PARAM(STRING, backupLabel); // Backup label to filter if requested by the user FUNCTION_TEST_PARAM(UINT, repoIdxMin); // The start index of the repo array to begin checking FUNCTION_TEST_PARAM(UINT, repoIdxMax); // The index of the last repo to check FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(backupSection != NULL); ASSERT(stanzaData != NULL); unsigned int backupNextRepoIdx = 0; unsigned int backupTotal = 0; unsigned int backupTotalProcessed = 0; // Get the number of backups to be processed for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { const InfoRepoData *const repoData = &stanzaData->repoList[repoIdx]; if (repoData->backupInfo != NULL && infoBackupDataTotal(repoData->backupInfo) > 0) backupTotal += infoBackupDataTotal(repoData->backupInfo); } // Process any backups while (backupTotalProcessed < backupTotal) { time_t backupNextTime = 0; // Backups are sorted for each repo, so iterate over the lists to create a single list ordered by backup-timestamp-stop for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { const InfoRepoData *const repoData = &stanzaData->repoList[repoIdx]; // If there are current backups on this repo for this stanza and the end of this backup list has not been reached // determine the next backup for display if (repoData->backupInfo != NULL && infoBackupDataTotal(repoData->backupInfo) > 0 && repoData->backupIdx < infoBackupDataTotal(repoData->backupInfo)) { InfoBackupData backupData = infoBackupData(repoData->backupInfo, repoData->backupIdx); // See if this backup should be next in the list, ordering from oldest to newest if (backupNextTime == 0 || backupData.backupTimestampStop < backupNextTime) { backupNextTime = backupData.backupTimestampStop; backupNextRepoIdx = repoIdx; } } } InfoRepoData *const repoData = &stanzaData->repoList[backupNextRepoIdx]; InfoBackupData backupData = infoBackupData(repoData->backupInfo, repoData->backupIdx); repoData->backupIdx++; backupTotalProcessed++; // Don't add the backup data to the backup section if a backup label was specified but this is not it if (backupLabel != NULL && !strEq(backupData.backupLabel, backupLabel)) continue; // Add the backup data to the backup section if (!cfgOptionTest(cfgOptType) || cfgOptionStrId(cfgOptType) == backupData.backupType) backupListAdd(backupSection, &backupData, backupLabel, repoData); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Set the stanza data for each stanza found in the repo ***********************************************************************************************************************************/ static VariantList * stanzaInfoList( List *const stanzaRepoList, const String *const backupLabel, const unsigned int repoIdxMin, const unsigned int repoIdxMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, stanzaRepoList); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_PARAM(UINT, repoIdxMin); FUNCTION_TEST_PARAM(UINT, repoIdxMax); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(stanzaRepoList != NULL); // Is full output requested? const bool outputFull = cfgOptionSeq(cfgOptDetailLevel) == CFGOPTVAL_DETAIL_LEVEL_FULL; VariantList *const result = varLstNew(); // Sort the list of stanzas lstSort(stanzaRepoList, sortOrderAsc); // Process each stanza requested for (unsigned int idx = 0; idx < lstSize(stanzaRepoList); idx++) { InfoStanzaRepo *const stanzaData = lstGet(stanzaRepoList, idx); // Create the stanzaInfo and section variables Variant *const stanzaInfo = varNewKv(kvNew()); VariantList *const dbSection = varLstNew(); VariantList *const backupSection = varLstNew(); VariantList *const archiveSection = varLstNew(); VariantList *const repoSection = varLstNew(); int stanzaStatusCode = -1; uint64_t stanzaCipherType = cipherTypeNone; // Set the stanza name and initialize the overall stanza variables kvPut(varKv(stanzaInfo), KEY_NAME_VAR, VARSTR(stanzaData->name)); // Get the stanza for each requested repo for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { InfoRepoData *const repoData = &stanzaData->repoList[repoIdx]; // When full output is not requested (progress mode), skip collecting detailed information and only update status code if (!outputFull) { if (repoIdx == repoIdxMin) stanzaStatusCode = repoData->stanzaStatus; else { stanzaStatusCode = stanzaStatusCode != repoData->stanzaStatus ? INFO_STANZA_STATUS_CODE_MIXED : repoData->stanzaStatus; } continue; } Variant *const repoInfo = varNewKv(kvNew()); kvPut(varKv(repoInfo), REPO_KEY_KEY_VAR, VARUINT(repoData->key)); kvPut(varKv(repoInfo), KEY_CIPHER_VAR, VARSTR(strNewStrId(repoData->cipher))); // If the stanza on this repo has the default status of ok but the backupInfo was not read, then the stanza exists on // other repos but not this one if (repoData->stanzaStatus == INFO_STANZA_STATUS_CODE_OK && repoData->backupInfo == NULL) repoData->stanzaStatus = INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH; TRY_BEGIN() { // If the backup.info file has been read, then get the backup and archive information on this repo if (repoData->backupInfo != NULL) { // If the backup.info file exists, get the database history information (oldest to newest) and corresponding // archive for (unsigned int pgIdx = infoPgDataTotal(infoBackupPg(repoData->backupInfo)) - 1; (int)pgIdx >= 0; pgIdx--) { const InfoPgData pgData = infoPgData(infoBackupPg(repoData->backupInfo), pgIdx); Variant *const pgInfo = varNewKv(kvNew()); kvPut(varKv(pgInfo), DB_KEY_ID_VAR, VARUINT(pgData.id)); kvPut(varKv(pgInfo), DB_KEY_SYSTEM_ID_VAR, VARUINT64(pgData.systemId)); kvPut(varKv(pgInfo), DB_KEY_VERSION_VAR, VARSTR(pgVersionToStr(pgData.version))); kvPut(varKv(pgInfo), KEY_REPO_KEY_VAR, VARUINT(repoData->key)); varLstAdd(dbSection, pgInfo); // Get the archive info for the DB from the archive.info file archiveDbList( stanzaData->name, &pgData, archiveSection, repoData->archiveInfo, (pgIdx == 0 ? true : false), repoIdx, repoData->key); } // Set stanza status if the current db sections do not match across repos const InfoPgData backupInfoCurrentPg = infoPgData( infoBackupPg(repoData->backupInfo), infoPgDataCurrentId(infoBackupPg(repoData->backupInfo))); // The current PG system and version must match across repos for the stanza, if not, a failure may have occurred // during an upgrade or the repo may have been disabled during the stanza upgrade to protect from error // propagation if (stanzaData->currentPgVersion != backupInfoCurrentPg.version || stanzaData->currentPgSystemId != backupInfoCurrentPg.systemId) { stanzaStatusCode = INFO_STANZA_STATUS_CODE_PG_MISMATCH; } } } CATCH_ANY() { infoStanzaErrorAdd(repoData, errorType(), STR(errorMessage())); } TRY_END(); // If there are no current backups on this repo then set status to no backup if (repoData->stanzaStatus == INFO_STANZA_STATUS_CODE_OK && infoBackupDataTotal(repoData->backupInfo) == 0) repoData->stanzaStatus = INFO_STANZA_STATUS_CODE_NO_BACKUP; // Track the status over all repos if the status for the stanza has not already been determined if (stanzaStatusCode != INFO_STANZA_STATUS_CODE_PG_MISMATCH) { if (repoIdx == repoIdxMin) stanzaStatusCode = repoData->stanzaStatus; else { stanzaStatusCode = stanzaStatusCode != repoData->stanzaStatus ? INFO_STANZA_STATUS_CODE_MIXED : repoData->stanzaStatus; } } // Track cipher type over all repos if (repoIdx == repoIdxMin) stanzaCipherType = repoData->cipher; else stanzaCipherType = stanzaCipherType != repoData->cipher ? INFO_STANZA_STATUS_CODE_MIXED : repoData->cipher; // Add the status of the stanza on the repo to the repo section, and the repo to the repo array repoStanzaStatus(repoData->stanzaStatus, repoInfo, repoData); varLstAdd(repoSection, repoInfo); // Add the database history, backup, archive and repo arrays to the stanza info kvPut(varKv(stanzaInfo), STANZA_KEY_DB_VAR, varNewVarLst(dbSection)); kvPut(varKv(stanzaInfo), KEY_ARCHIVE_VAR, varNewVarLst(archiveSection)); kvPut(varKv(stanzaInfo), STANZA_KEY_REPO_VAR, varNewVarLst(repoSection)); } // Collect backup and cipher data if full output is requested if (outputFull) { // Get a sorted list of the data for all existing backups for this stanza over all repos backupList(backupSection, stanzaData, backupLabel, repoIdxMin, repoIdxMax); kvPut(varKv(stanzaInfo), STANZA_KEY_BACKUP_VAR, varNewVarLst(backupSection)); // Set the overall cipher type if (stanzaCipherType != INFO_STANZA_STATUS_CODE_MIXED) kvPut(varKv(stanzaInfo), KEY_CIPHER_VAR, VARSTR(strNewStrId(stanzaCipherType))); else kvPut(varKv(stanzaInfo), KEY_CIPHER_VAR, VARSTRDEF(INFO_STANZA_MIXED)); } // Set the overall stanza status and gather progress information stanzaStatus(stanzaStatusCode, stanzaData, stanzaInfo, repoIdxMin, repoIdxMax); varLstAdd(result, stanzaInfo); } FUNCTION_TEST_RETURN(VARIANT_LIST, result); } /*********************************************************************************************************************************** Format the text output for archive and backups for a database group of a stanza ***********************************************************************************************************************************/ // Helper to format date/time with timezone offset static String * formatTextBackupDateTime(const time_t epoch) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TIME, epoch); FUNCTION_TEST_END(); // Construct date/time String *const result = strNew(); struct tm timePart; char timeBuffer[20]; strftime(timeBuffer, sizeof(timeBuffer), "%Y-%m-%d %H:%M:%S", localtime_r(&epoch, &timePart)); strCatZ(result, timeBuffer); // Add timezone offset. Since this is not directly available in Posix-compliant APIs, use the difference between gmtime_r() and // localtime_r to determine the offset. Handle minute offsets when present. struct tm timePartGm; gmtime_r(&epoch, &timePartGm); timePart.tm_isdst = -1; timePartGm.tm_isdst = -1; time_t offset = mktime(&timePart) - mktime(&timePartGm); if (offset >= 0) strCatChr(result, '+'); else { offset *= -1; strCatChr(result, '-'); } const unsigned int minute = (unsigned int)(offset / 60); const unsigned int hour = minute / 60; strCatFmt(result, "%02u", hour); if (minute % 60 != 0) strCatFmt(result, ":%02u", minute - (hour * 60)); FUNCTION_TEST_RETURN(STRING, result); } static void formatTextBackup(const DbGroup *const dbGroup, String *const resultStr) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(DB_GROUP, dbGroup); FUNCTION_TEST_PARAM(STRING, resultStr); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(dbGroup != NULL); strCatFmt(resultStr, "\n wal archive min/max (%s): ", strZ(dbGroup->version)); // Get the archive min/max if there are any archives for the database if (dbGroup->archiveMin != NULL) strCatFmt(resultStr, "%s/%s\n", strZ(dbGroup->archiveMin), strZ(dbGroup->archiveMax)); else strCatZ(resultStr, "none present\n"); for (unsigned int backupIdx = 0; backupIdx < varLstSize(dbGroup->backupList); backupIdx++) { const KeyValue *const backupInfo = varKv(varLstGet(dbGroup->backupList, backupIdx)); strCatFmt( resultStr, "\n %s backup: %s\n", strZ(varStr(kvGet(backupInfo, BACKUP_KEY_TYPE_VAR))), strZ(varStr(kvGet(backupInfo, BACKUP_KEY_LABEL_VAR)))); // Get and format backup start/stop time const KeyValue *const timestampInfo = varKv(kvGet(backupInfo, BACKUP_KEY_TIMESTAMP_VAR)); strCatFmt( resultStr, " timestamp start/stop: %s / %s\n", strZ(formatTextBackupDateTime((time_t)varUInt64(kvGet(timestampInfo, KEY_START_VAR)))), strZ(formatTextBackupDateTime((time_t)varUInt64(kvGet(timestampInfo, KEY_STOP_VAR))))); strCatZ(resultStr, " wal start/stop: "); // Get and format WAL start/stop const KeyValue *const archiveBackupInfo = varKv(kvGet(backupInfo, KEY_ARCHIVE_VAR)); if (kvGet(archiveBackupInfo, KEY_START_VAR) != NULL && kvGet(archiveBackupInfo, KEY_STOP_VAR) != NULL) { strCatFmt( resultStr, "%s / %s\n", strZ(varStr(kvGet(archiveBackupInfo, KEY_START_VAR))), strZ(varStr(kvGet(archiveBackupInfo, KEY_STOP_VAR)))); } else strCatZ(resultStr, "n/a\n"); const KeyValue *const lsnInfo = varKv(kvGet(backupInfo, BACKUP_KEY_LSN_VAR)); if (lsnInfo != NULL) { strCatFmt( resultStr, " lsn start/stop: %s / %s\n", strZ(varStr(kvGet(lsnInfo, KEY_START_VAR))), strZ(varStr(kvGet(lsnInfo, KEY_STOP_VAR)))); } const KeyValue *const info = varKv(kvGet(backupInfo, BACKUP_KEY_INFO_VAR)); strCatFmt( resultStr, " database size: %s, database backup size: %s\n", strZ(strSizeFormat(varUInt64Force(kvGet(info, KEY_SIZE_VAR)))), strZ(strSizeFormat(varUInt64Force(kvGet(info, KEY_DELTA_VAR))))); const KeyValue *const repoInfo = varKv(kvGet(info, INFO_KEY_REPOSITORY_VAR)); const Variant *const repoSizeInfo = kvGet(repoInfo, KEY_SIZE_VAR); strCatFmt(resultStr, " repo%u: ", varUInt(kvGet(varKv(kvGet(backupInfo, KEY_DATABASE_VAR)), KEY_REPO_KEY_VAR))); if (repoSizeInfo != NULL) strCatFmt(resultStr, "backup set size: %s, ", strZ(strSizeFormat(varUInt64Force(kvGet(repoInfo, KEY_SIZE_VAR))))); strCatFmt(resultStr, "backup size: %s\n", strZ(strSizeFormat(varUInt64Force(kvGet(repoInfo, KEY_DELTA_VAR))))); if (kvGet(backupInfo, BACKUP_KEY_REFERENCE_VAR) != NULL) { if (kvGet(backupInfo, BACKUP_KEY_REFERENCE_FULL_VAR) != NULL) { const String *const diffListSizeText = kvGet(backupInfo, BACKUP_KEY_REFERENCE_DIFF_VAR) != NULL ? strNewFmt(", %u diff", varLstSize(varVarLst(kvGet(backupInfo, BACKUP_KEY_REFERENCE_DIFF_VAR)))) : EMPTY_STR; const String *const incrListSizeText = kvGet(backupInfo, BACKUP_KEY_REFERENCE_INCR_VAR) != NULL ? strNewFmt(", %u incr", varLstSize(varVarLst(kvGet(backupInfo, BACKUP_KEY_REFERENCE_INCR_VAR)))) : EMPTY_STR; strCatFmt( resultStr, " backup reference total: %u full%s%s\n", varLstSize(varVarLst(kvGet(backupInfo, BACKUP_KEY_REFERENCE_FULL_VAR))), strZ(diffListSizeText), strZ(incrListSizeText)); } else { const StringList *const referenceList = strLstNewVarLst(varVarLst(kvGet(backupInfo, BACKUP_KEY_REFERENCE_VAR))); strCatFmt(resultStr, " backup reference list: %s\n", strZ(strLstJoin(referenceList, ", "))); } } if (kvGet(backupInfo, BACKUP_KEY_DATABASE_REF_VAR) != NULL) { const VariantList *const dbSection = kvGetList(backupInfo, BACKUP_KEY_DATABASE_REF_VAR); strCatZ(resultStr, " database list:"); if (varLstEmpty(dbSection)) strCatZ(resultStr, " none\n"); else { for (unsigned int dbIdx = 0; dbIdx < varLstSize(dbSection); dbIdx++) { const KeyValue *const db = varKv(varLstGet(dbSection, dbIdx)); strCatFmt( resultStr, " %s (%s)", strZ(varStr(kvGet(db, KEY_NAME_VAR))), strZ(varStrForce(kvGet(db, KEY_OID_VAR)))); if (dbIdx != varLstSize(dbSection) - 1) strCatZ(resultStr, ","); } strCat(resultStr, LF_STR); } } if (kvGet(backupInfo, BACKUP_KEY_LINK_VAR) != NULL) { const VariantList *const linkSection = kvGetList(backupInfo, BACKUP_KEY_LINK_VAR); strCatZ(resultStr, " symlinks:\n"); for (unsigned int linkIdx = 0; linkIdx < varLstSize(linkSection); linkIdx++) { const KeyValue *const link = varKv(varLstGet(linkSection, linkIdx)); strCatFmt( resultStr, " %s => %s", strZ(varStr(kvGet(link, KEY_NAME_VAR))), strZ(varStr(kvGet(link, KEY_DESTINATION_VAR)))); if (linkIdx != varLstSize(linkSection) - 1) strCat(resultStr, LF_STR); } strCat(resultStr, LF_STR); } if (kvGet(backupInfo, BACKUP_KEY_TABLESPACE_VAR) != NULL) { const VariantList *const tablespaceSection = kvGetList(backupInfo, BACKUP_KEY_TABLESPACE_VAR); strCatZ(resultStr, " tablespaces:\n"); for (unsigned int tblIdx = 0; tblIdx < varLstSize(tablespaceSection); tblIdx++) { const KeyValue *const tablespace = varKv(varLstGet(tablespaceSection, tblIdx)); strCatFmt( resultStr, " %s (%s) => %s", strZ(varStr(kvGet(tablespace, KEY_NAME_VAR))), strZ(varStrForce(kvGet(tablespace, KEY_OID_VAR))), strZ(varStr(kvGet(tablespace, KEY_DESTINATION_VAR)))); if (tblIdx != varLstSize(tablespaceSection) - 1) strCat(resultStr, LF_STR); } strCat(resultStr, LF_STR); } // If errors were detected during the backup if (kvGet(backupInfo, BACKUP_KEY_ERROR_VAR) != NULL && varBool(kvGet(backupInfo, BACKUP_KEY_ERROR_VAR))) { // Output error list if present if (kvGet(backupInfo, BACKUP_KEY_ERROR_LIST_VAR) != NULL) { const StringList *const checksumPageErrorList = strLstNewVarLst( varVarLst(kvGet(backupInfo, BACKUP_KEY_ERROR_LIST_VAR))); strCatFmt(resultStr, " error list: %s\n", strZ(strLstJoin(checksumPageErrorList, ", "))); } // Else output a general message else strCatZ(resultStr, " error(s) detected during backup\n"); } // Annotations metadata if (kvGet(backupInfo, BACKUP_KEY_ANNOTATION_VAR) != NULL) { const KeyValue *const annotationKv = varKv(kvGet(backupInfo, BACKUP_KEY_ANNOTATION_VAR)); const StringList *const annotationKeyList = strLstNewVarLst(kvKeyList(annotationKv)); String *const annotationStr = strNew(); for (unsigned int keyIdx = 0; keyIdx < strLstSize(annotationKeyList); keyIdx++) { const String *const key = strLstGet(annotationKeyList, keyIdx); const String *const value = varStr(kvGet(annotationKv, VARSTR(key))); ASSERT(value != NULL); strCatFmt(annotationStr, " %s: %s\n", strZ(key), strZ(value)); } strCatFmt(resultStr, " annotation(s)\n%s", strZ(annotationStr)); } } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Format the text output for each database of the stanza ***********************************************************************************************************************************/ static void formatTextDb( const KeyValue *const stanzaInfo, String *const resultStr, const String *const currentPgVersion, const uint64_t currentPgSystemId, const String *const backupLabel) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, stanzaInfo); FUNCTION_TEST_PARAM(STRING, resultStr); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_PARAM(STRING, currentPgVersion); FUNCTION_TEST_PARAM(UINT64, currentPgSystemId); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(stanzaInfo != NULL); ASSERT(currentPgVersion != NULL); const VariantList *const dbSection = kvGetList(stanzaInfo, STANZA_KEY_DB_VAR); const VariantList *const archiveSection = kvGetList(stanzaInfo, KEY_ARCHIVE_VAR); const VariantList *const backupSection = kvGetList(stanzaInfo, STANZA_KEY_BACKUP_VAR); List *const dbGroupList = lstNewP(sizeof(DbGroup)); // For each database update the corresponding archive info for (unsigned int dbIdx = 0; dbIdx < varLstSize(dbSection); dbIdx++) { const KeyValue *const pgInfo = varKv(varLstGet(dbSection, dbIdx)); const uint64_t dbSysId = varUInt64(kvGet(pgInfo, DB_KEY_SYSTEM_ID_VAR)); const String *const dbVersion = varStr(kvGet(pgInfo, DB_KEY_VERSION_VAR)); const unsigned int dbId = varUInt(kvGet(pgInfo, DB_KEY_ID_VAR)); const unsigned int dbRepoKey = varUInt(kvGet(pgInfo, KEY_REPO_KEY_VAR)); DbGroup *dbGroup = NULL; for (unsigned int dbGrpIdx = 0; dbGrpIdx < lstSize(dbGroupList); dbGrpIdx++) { DbGroup *const dbGroupInfo = lstGet(dbGroupList, dbGrpIdx); if (dbGroupInfo->systemId == dbSysId && strEq(dbGroupInfo->version, dbVersion)) { dbGroup = dbGroupInfo; break; } } // If the group was not found, then add it if (dbGroup == NULL) { const DbGroup dbGroupInfo = { .systemId = dbSysId, .version = dbVersion, .current = (currentPgSystemId == dbSysId && strEq(currentPgVersion, dbVersion)), .archiveMin = NULL, .archiveMax = NULL, .backupList = varLstNew(), }; lstAdd(dbGroupList, &dbGroupInfo); dbGroup = lstGetLast(dbGroupList); } // For each archive of this stanza, update the archive min/max for this database group if necessary for (unsigned int archiveIdx = 0; archiveIdx < varLstSize(archiveSection); archiveIdx++) { const KeyValue *const archiveInfo = varKv(varLstGet(archiveSection, archiveIdx)); const KeyValue *const archiveDbInfo = varKv(kvGet(archiveInfo, KEY_DATABASE_VAR)); const unsigned int archiveDbId = varUInt(kvGet(archiveDbInfo, DB_KEY_ID_VAR)); const unsigned int archiveRepoKey = varUInt(kvGet(archiveDbInfo, KEY_REPO_KEY_VAR)); // If there are archives and the min is less than that for this database group, then update the group if (archiveDbId == dbId && archiveRepoKey == dbRepoKey && varStr(kvGet(archiveInfo, ARCHIVE_KEY_MIN_VAR)) != NULL) { // Although archives should continue to increment over system-id/version with different db-ids, there may be cases // where an archived WAL may exist on both, and if the archive id on a later db is less than a prior instance of // the same PG, then ensure it is updated as the min. Any need to error should not be handled in the info command. if (dbGroup->archiveMin == NULL || strCmp(dbGroup->archiveMin, varStr(kvGet(archiveInfo, ARCHIVE_KEY_MIN_VAR))) > 0) dbGroup->archiveMin = varStrForce(kvGet(archiveInfo, ARCHIVE_KEY_MIN_VAR)); if (dbGroup->archiveMax == NULL || strCmp(dbGroup->archiveMax, varStr(kvGet(archiveInfo, ARCHIVE_KEY_MAX_VAR))) < 0) dbGroup->archiveMax = varStrForce(kvGet(archiveInfo, ARCHIVE_KEY_MAX_VAR)); } } } unsigned int backupDbGrpIdxMin = 0; unsigned int backupDbGrpIdxMax = lstSize(dbGroupList); // For every backup (oldest to newest) for the stanza, add it to the database group based on system-id and version for (unsigned int backupIdx = 0; backupIdx < varLstSize(backupSection); backupIdx++) { const KeyValue *const backupInfo = varKv(varLstGet(backupSection, backupIdx)); const KeyValue *const backupDbInfo = varKv(kvGet(backupInfo, KEY_DATABASE_VAR)); const unsigned int backupDbId = varUInt(kvGet(backupDbInfo, DB_KEY_ID_VAR)); const unsigned int backupRepoKey = varUInt(kvGet(backupDbInfo, KEY_REPO_KEY_VAR)); // Find the database group this backup belongs to and add it for (unsigned int dbIdx = 0; dbIdx < varLstSize(dbSection); dbIdx++) { const KeyValue *const pgInfo = varKv(varLstGet(dbSection, dbIdx)); const unsigned int dbId = varUInt(kvGet(pgInfo, DB_KEY_ID_VAR)); const unsigned int dbRepoKey = varUInt(kvGet(pgInfo, KEY_REPO_KEY_VAR)); if (backupDbId == dbId && backupRepoKey == dbRepoKey) { for (unsigned int dbGrpIdx = 0; dbGrpIdx < lstSize(dbGroupList); dbGrpIdx++) { const DbGroup *const dbGroupInfo = lstGet(dbGroupList, dbGrpIdx); if (dbGroupInfo->systemId == varUInt64(kvGet(pgInfo, DB_KEY_SYSTEM_ID_VAR)) && strEq(dbGroupInfo->version, varStr(kvGet(pgInfo, DB_KEY_VERSION_VAR)))) { varLstAdd(dbGroupInfo->backupList, varLstGet(backupSection, backupIdx)); // If we're only looking for one backup, then narrow the db group iterators if (backupLabel != NULL) { backupDbGrpIdxMin = dbGrpIdx; backupDbGrpIdxMax = dbGrpIdx + 1; } dbGrpIdx = lstSize(dbGroupList); } } dbIdx = varLstSize(dbSection); } } } String *const resultCurrent = strCatZ(strNew(), "\n db (current)"); bool displayCurrent = false; for (unsigned int dbGrpIdx = backupDbGrpIdxMin; dbGrpIdx < backupDbGrpIdxMax; dbGrpIdx++) { const DbGroup *const dbGroupInfo = lstGet(dbGroupList, dbGrpIdx); // Sort the results based on current or prior and only show the prior if it has archives or backups if (dbGroupInfo->current) { formatTextBackup(dbGroupInfo, resultCurrent); displayCurrent = true; } else if (dbGroupInfo->archiveMin != NULL || !varLstEmpty(dbGroupInfo->backupList)) { strCatZ(resultStr, "\n db (prior)"); formatTextBackup(dbGroupInfo, resultStr); } } // Add the current results to the end if necessary (e.g. current not displayed if a specified backup label is only in prior) if (displayCurrent == true) strCat(resultStr, resultCurrent); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Get the lock info of the specified lock type for the stanza ***********************************************************************************************************************************/ static void infoUpdateStanzaLock( InfoStanzaLock *const stanzaLock, const String *const stanzaName, const unsigned int repoIdx, const LockType lockType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO_STANZA_LOCK, stanzaLock); FUNCTION_TEST_PARAM(STRING, stanzaName); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_PARAM(ENUM, lockType); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(stanzaLock != NULL); ASSERT(stanzaName != NULL); // If there is a valid lock for this stanza then backup/expire/restore must be running const LockReadResult lockResult = cmdLockRead(lockType, stanzaName, repoIdx); if (lockResult.status == lockReadStatusValid) { stanzaLock->held = true; if (lockResult.data.size != NULL) { ASSERT(lockResult.data.size != NULL); stanzaLock->sizeComplete += varUInt64(lockResult.data.sizeComplete); stanzaLock->size += varUInt64(lockResult.data.size); } } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Get the backup and archive info files on the specified repo for the stanza ***********************************************************************************************************************************/ static void infoUpdateStanza( const Storage *const storage, InfoStanzaRepo *const stanzaRepo, const unsigned int repoIdx, const bool stanzaExists, const String *const backupLabel) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE, storage); FUNCTION_TEST_PARAM(INFO_STANZA_REPO, stanzaRepo); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_PARAM(BOOL, stanzaExists); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(storage != NULL); ASSERT(stanzaRepo != NULL); // If the stanza exists, attempt to get the info files if (stanzaExists) { volatile int stanzaStatus = INFO_STANZA_STATUS_CODE_OK; TRY_BEGIN() { // If full output is requested read info and manifest files const bool outputFull = cfgOptionSeq(cfgOptDetailLevel) == CFGOPTVAL_DETAIL_LEVEL_FULL; if (outputFull) { // Catch certain errors TRY_BEGIN() { // Attempt to load the backup info file stanzaRepo->repoList[repoIdx].backupInfo = infoBackupLoadFile( storage, strNewFmt(STORAGE_PATH_BACKUP "/%s/%s", strZ(stanzaRepo->name), INFO_BACKUP_FILE), cfgCipherSpecMainIdx(repoIdx)); } CATCH(FileMissingError) { // If there is no backup.info then set the status to indicate missing stanzaStatus = INFO_STANZA_STATUS_CODE_MISSING_STANZA_DATA; } CATCH(CryptoError) { // If a reason for the error is due to a an encryption error, add a hint THROW_FMT( CryptoError, "%s\n" "HINT: use option --stanza if encryption settings are different for the stanza than the global settings.", errorMessage()); } TRY_END(); // If backup.info was found, then get the archive.info file, which must exist if backup.info exists, else the failed // load will throw an error which will be trapped and recorded if (stanzaRepo->repoList[repoIdx].backupInfo != NULL) { stanzaRepo->repoList[repoIdx].archiveInfo = infoArchiveLoadFile( storage, strNewFmt(STORAGE_PATH_ARCHIVE "/%s/%s", strZ(stanzaRepo->name), INFO_ARCHIVE_FILE), cfgCipherSpecMainIdx(repoIdx)); // If a specific backup exists on this repo then attempt to load the manifest if (backupLabel != NULL) { stanzaRepo->repoList[repoIdx].manifest = manifestLoadFile( storage, strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)), infoBackupCipherSpec(stanzaRepo->repoList[repoIdx].backupInfo)); } } } // Read the lock files if backup.info is present. Exception: when only progress is requested, backup.info is skipped for // performance, so the lock files are read unconditionally -- though they may be outdated in this case. if (stanzaRepo->repoList[repoIdx].backupInfo != NULL || !outputFull) { infoUpdateStanzaLock(&stanzaRepo->repoList[repoIdx].backupLock, stanzaRepo->name, repoIdx, lockTypeBackup); infoUpdateStanzaLock(&stanzaRepo->restoreLock, stanzaRepo->name, repoIdx, lockTypeRestore); } stanzaRepo->repoList[repoIdx].stanzaStatus = stanzaStatus; } CATCH_ANY() { infoStanzaErrorAdd(&stanzaRepo->repoList[repoIdx], errorType(), STR(errorMessage())); } TRY_END(); } else stanzaRepo->repoList[repoIdx].stanzaStatus = INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH; // If the backup.info and therefore archive.info exist, and the currentPg has not been set for the stanza, then set it if (stanzaRepo->currentPgVersion == 0 && stanzaRepo->repoList[repoIdx].backupInfo != NULL) { InfoPgData backupInfoCurrentPg = infoPgData( infoBackupPg(stanzaRepo->repoList[repoIdx].backupInfo), infoPgDataCurrentId(infoBackupPg(stanzaRepo->repoList[repoIdx].backupInfo))); stanzaRepo->currentPgVersion = backupInfoCurrentPg.version; stanzaRepo->currentPgSystemId = backupInfoCurrentPg.systemId; } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Render the information for the stanza based on the command parameters ***********************************************************************************************************************************/ static String * infoRender(void) { FUNCTION_LOG_VOID(logLevelDebug); String *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get stanza if specified const String *const stanza = cfgOptionStrNull(cfgOptStanza); // Initialize the list of stanzas on all repos List *const stanzaRepoList = lstNewP(sizeof(InfoStanzaRepo), .sortOrder = sortOrderAsc, .comparator = lstComparatorStr); // Get the backup label if specified const String *const backupLabel = cfgOptionStrNull(cfgOptSet); bool backupFound = false; // If only progress info is requested then details about a specific backup may not be requested if (backupLabel != NULL && cfgOptionSeq(cfgOptDetailLevel) == CFGOPTVAL_DETAIL_LEVEL_PROGRESS) { THROW_FMT(OptionInvalidError, "option '%s' cannot be used with option '%s' = '%s'", cfgOptionName(cfgOptSet), cfgOptionName(cfgOptDetailLevel), CFGOPTVAL_DETAIL_LEVEL_PROGRESS_Z); } // Initialize the repo index unsigned int repoIdxMin = 0; const unsigned int repoTotal = cfgOptionGroupIdxTotal(cfgOptGrpRepo); unsigned int repoIdxMax = repoTotal - 1; // If the repo was specified then set index to the array location and max to loop only once if (cfgOptionTest(cfgOptRepo)) { repoIdxMin = cfgOptionGroupIdxDefault(cfgOptGrpRepo); repoIdxMax = repoIdxMin; } // Initialize error reporting at the repo level InfoRepoData *const repoErrorList = memNew(repoTotal * sizeof(InfoRepoData)); bool repoError = false; for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { // Initialize the error list on this repo repoErrorList[repoIdx] = (InfoRepoData) { .key = cfgOptionGroupIdxToKey(cfgOptGrpRepo, repoIdx), .cipher = cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), .error = NULL, }; // Initialize backup label indicator const String *backupExistsOnRepo = NULL; // Catch any repo errors TRY_BEGIN() { // Get the repo storage in case it is remote and encryption settings need to be pulled down const Storage *const storageRepo = storageRepoIdx(repoIdx); // If a backup set was specified, see if the manifest exists if (backupLabel != NULL) { // If the backup exists on this repo, set the global indicator that we found it on at least one repo and // set the exists label for later loading of the manifest if (storageExistsP(storageRepo, strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)))) { backupFound = true; backupExistsOnRepo = backupLabel; } } // Get a list of stanzas in the backup directory StringList *stanzaNameList = strLstSort( storageListP(storageRepo, STORAGE_PATH_BACKUP_STR), sortOrderAsc); // All stanzas will be "found" if they are in the storage list bool stanzaExists = true; if (stanza != NULL) { // If a specific stanza was requested and it is not on this repo, then stanzaExists flag will be reset to false if (strLstEmpty(stanzaNameList) || !strLstExists(stanzaNameList, stanza)) stanzaExists = false; // Narrow the list to only the requested stanza strLstFree(stanzaNameList); stanzaNameList = strLstNew(); strLstAdd(stanzaNameList, stanza); } // Process each stanza for (unsigned int stanzaIdx = 0; stanzaIdx < strLstSize(stanzaNameList); stanzaIdx++) { const String *const stanzaName = strLstGet(stanzaNameList, stanzaIdx); // Get the stanza if it is already in the list InfoStanzaRepo *const stanzaRepo = lstFind(stanzaRepoList, &stanzaName); // If the stanza was already added to the array, then update this repo for the stanza, else the stanza has not // yet been added to the list, so add it if (stanzaRepo != NULL) infoUpdateStanza(storageRepo, stanzaRepo, repoIdx, stanzaExists, backupExistsOnRepo); else { InfoStanzaRepo stanzaRepo = { .name = stanzaName, .currentPgVersion = 0, .currentPgSystemId = 0, .repoList = memNew(repoTotal * sizeof(InfoRepoData)), }; // Initialize all the repos for (unsigned int repoListIdx = 0; repoListIdx < repoTotal; repoListIdx++) { stanzaRepo.repoList[repoListIdx] = (InfoRepoData) { .key = cfgOptionGroupIdxToKey(cfgOptGrpRepo, repoListIdx), .cipher = cfgOptionIdxStrId(cfgOptRepoCipherType, repoListIdx), .error = NULL, }; } // Update the info for this repo infoUpdateStanza(storageRepo, &stanzaRepo, repoIdx, stanzaExists, backupExistsOnRepo); lstAdd(stanzaRepoList, &stanzaRepo); } } } CATCH_ANY() { // At this point, stanza-level errors were caught and stored in the stanza structure, any errors caught here are due // to higher level problems (e.g. repo inaccessible, invalid permissions on the repo, etc) and will be reported // later if there are valid stanzas. If there are no valid stanzas after the loop exits, then these errors will be // reported with a stanza named "[invalid]". infoStanzaErrorAdd(&repoErrorList[repoIdx], errorType(), STR(errorMessage())); repoError = true; } TRY_END(); } VariantList *infoList = varLstNew(); String *resultStr = strNew(); // Record any repository-level errors with each stanza -- if there are no stanzas and one was not requested, then create an // "[invalid]" one for reporting if (repoError) { if (lstEmpty(stanzaRepoList)) { const InfoStanzaRepo stanzaRepo = { .name = stanza != NULL ? stanza : INFO_STANZA_INVALID_STR, .currentPgVersion = 0, .currentPgSystemId = 0, .repoList = repoErrorList, }; lstAdd(stanzaRepoList, &stanzaRepo); } else { // For each stanza report the repos that were in error for (unsigned int idx = 0; idx < lstSize(stanzaRepoList); idx++) { const InfoStanzaRepo *const stanzaData = lstGet(stanzaRepoList, idx); for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { if (repoErrorList[repoIdx].error != NULL) stanzaData->repoList[repoIdx] = repoErrorList[repoIdx]; } } } } // If a backup label was requested but it was not found on any repo, report the error here rather than individually to avoid // listing each repo as "requested backup not found" if (backupLabel != NULL && !backupFound) { // Get the stanza record and update each repo to indicate backup not found where there is not already an error status // so that errors on other repositories will be displayed and not overwritten const InfoStanzaRepo *const stanzaRepo = lstFind(stanzaRepoList, &stanza); for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { if (stanzaRepo->repoList[repoIdx].stanzaStatus == INFO_STANZA_STATUS_CODE_OK) { stanzaRepo->repoList[repoIdx].stanzaStatus = INFO_STANZA_STATUS_CODE_BACKUP_MISSING; infoBackupFree(stanzaRepo->repoList[repoIdx].backupInfo); stanzaRepo->repoList[repoIdx].backupInfo = NULL; } } } // If the backup storage exists, then search for and process any stanzas if (!lstEmpty(stanzaRepoList)) infoList = stanzaInfoList(stanzaRepoList, backupLabel, repoIdxMin, repoIdxMax); // Format text output if (cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_OUTPUT_TEXT) { // Process any stanza directories if (!varLstEmpty(infoList)) { // Is full output requested? const bool outputFull = cfgOptionSeq(cfgOptDetailLevel) == CFGOPTVAL_DETAIL_LEVEL_FULL; for (unsigned int stanzaIdx = 0; stanzaIdx < varLstSize(infoList); stanzaIdx++) { const KeyValue *const stanzaInfo = varKv(varLstGet(infoList, stanzaIdx)); const String *const stanzaName = varStr(kvGet(stanzaInfo, KEY_NAME_VAR)); // Add a carriage return between stanzas if (stanzaIdx > 0) strCatZ(resultStr, "\n"); // Stanza name and status strCatFmt(resultStr, "stanza: %s\n status: ", strZ(stanzaName)); // If an error has occurred, provide the information that is available and move onto next stanza const KeyValue *const stanzaStatus = varKv(kvGet(stanzaInfo, STANZA_KEY_STATUS_VAR)); const int statusCode = varInt(kvGet(stanzaStatus, STATUS_KEY_CODE_VAR)); // Get the backup lock info const KeyValue *const lockKv = varKv(kvGet(stanzaStatus, STATUS_KEY_LOCK_VAR)); const KeyValue *const backupLockKv = varKv(kvGet(lockKv, STATUS_KEY_LOCK_BACKUP_VAR)); const bool backupLockHeld = varBool(kvGet(backupLockKv, STATUS_KEY_LOCK_HELD_VAR)); const Variant *const backupPercentComplete = kvGet(backupLockKv, STATUS_KEY_LOCK_PERCENT_COMPLETE_VAR); const String *const backupPercentCompleteStr = backupPercentComplete != NULL ? strNewFmt(" - %s complete", strZ(strNewPct(varUInt(backupPercentComplete), 10000))) : EMPTY_STR; // Get the restore lock info const KeyValue *const restoreLockKv = varKv(kvGet(lockKv, STATUS_KEY_LOCK_RESTORE_VAR)); const bool restoreLockHeld = varBool(kvGet(restoreLockKv, STATUS_KEY_LOCK_HELD_VAR)); const Variant *const restorePercentComplete = kvGet(restoreLockKv, STATUS_KEY_LOCK_PERCENT_COMPLETE_VAR); const String *const restorePercentCompleteStr = restorePercentComplete != NULL ? strNewFmt(" - %s complete", strZ(strNewPct(varUInt(restorePercentComplete), 10000))) : EMPTY_STR; // Build stanza status const bool errorStatus = statusCode != INFO_STANZA_STATUS_CODE_OK && (outputFull == false || statusCode != INFO_STANZA_STATUS_CODE_MIXED); const bool progressStatus = backupLockHeld == true || restoreLockHeld == true; const String *const statusLabelStr = statusCode == INFO_STANZA_STATUS_CODE_OK ? strNewZ(INFO_STANZA_STATUS_OK) : errorStatus == true ? strNewZ(INFO_STANZA_STATUS_ERROR) : strNewZ(INFO_STANZA_MIXED); const String *const statusErrorStr = errorStatus ? varStr(kvGet(stanzaStatus, STATUS_KEY_MESSAGE_VAR)) : EMPTY_STR; const String *const progressStr = backupLockHeld == true && restoreLockHeld == true ? strNewFmt( INFO_STANZA_STATUS_MESSAGE_LOCK_BACKUP "%s, " INFO_STANZA_STATUS_MESSAGE_LOCK_RESTORE "%s", strZ(backupPercentCompleteStr), strZ(restorePercentCompleteStr)) : backupLockHeld == true ? strNewFmt(INFO_STANZA_STATUS_MESSAGE_LOCK_BACKUP "%s", strZ(backupPercentCompleteStr)) : restoreLockHeld == true ? strNewFmt(INFO_STANZA_STATUS_MESSAGE_LOCK_RESTORE "%s", strZ(restorePercentCompleteStr)) : EMPTY_STR; if (progressStatus) { // Status: error (message, progress) if (errorStatus) { strCatFmt(resultStr, "%s (%s, %s)\n", strZ(statusLabelStr), strZ(statusErrorStr), strZ(progressStr)); } // Status: ok/mixed (progress) else strCatFmt(resultStr, "%s (%s)\n", strZ(statusLabelStr), strZ(progressStr)); } else { // Status: error (message) if (errorStatus) { strCatFmt(resultStr, "%s (%s)\n", strZ(statusLabelStr), strZ(statusErrorStr)); } // Status: ok/mixed else strCatFmt(resultStr, "%s\n", strZ(statusLabelStr)); } // Output per-repo backup progress when multiple backups are running const Variant *const backupRepoVar = kvGet(backupLockKv, STANZA_KEY_REPO_VAR); const VariantList *const backupRepoList = backupRepoVar != NULL ? varVarLst(backupRepoVar) : NULL; if (backupRepoList != NULL && varLstSize(backupRepoList) > 1) { for (unsigned int repoLockIdx = 0; repoLockIdx < varLstSize(backupRepoList); repoLockIdx++) { const KeyValue *const repoLockKv = varKv(varLstGet(backupRepoList, repoLockIdx)); strCatFmt( resultStr, " repo%u backup: %s complete\n", varUInt(kvGet(repoLockKv, REPO_KEY_KEY_VAR)), strZ(strNewPct( cvtPctToUInt( varUInt64(kvGet(repoLockKv, STATUS_KEY_LOCK_SIZE_COMPLETE_VAR)), varUInt64(kvGet(repoLockKv, STATUS_KEY_LOCK_SIZE_VAR))), 10000))); } } // Output the status per repo if (outputFull && (statusCode == INFO_STANZA_STATUS_CODE_MIXED || statusCode == INFO_STANZA_STATUS_CODE_PG_MISMATCH || statusCode == INFO_STANZA_STATUS_CODE_OTHER)) { const VariantList *const repoSection = kvGetList(stanzaInfo, STANZA_KEY_REPO_VAR); const bool multiRepo = varLstSize(repoSection) > 1; const char *const formatSpacer = multiRepo ? " " : " "; for (unsigned int repoIdx = 0; repoIdx < varLstSize(repoSection); repoIdx++) { const KeyValue *const repoInfo = varKv(varLstGet(repoSection, repoIdx)); const KeyValue *const repoStatus = varKv(kvGet(repoInfo, STANZA_KEY_STATUS_VAR)); // If more than one repo configured, then add the repo status per repo if (multiRepo) strCatFmt(resultStr, " repo%u: ", varUInt(kvGet(repoInfo, REPO_KEY_KEY_VAR))); if (varInt(kvGet(repoStatus, STATUS_KEY_CODE_VAR)) == INFO_STANZA_STATUS_CODE_OK) strCatZ(resultStr, INFO_STANZA_STATUS_OK "\n"); else { if (varInt(kvGet(repoStatus, STATUS_KEY_CODE_VAR)) == INFO_STANZA_STATUS_CODE_OTHER) { const StringList *const repoError = strLstNewSplit( varStr(kvGet(repoStatus, STATUS_KEY_MESSAGE_VAR)), STRDEF("\n")); strCatFmt( resultStr, "%s%s%s\n", multiRepo ? INFO_STANZA_STATUS_ERROR " (" INFO_STANZA_STATUS_MESSAGE_OTHER ")\n" : "", formatSpacer, strZ(strLstJoin(repoError, zNewFmt("\n%s", formatSpacer)))); } else { strCatFmt( resultStr, INFO_STANZA_STATUS_ERROR " (%s)\n", strZ(varStr(kvGet(repoStatus, STATUS_KEY_MESSAGE_VAR)))); } } } } // Add cipher type if the stanza is found on at least one repo if (outputFull && statusCode != INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH) { strCatFmt(resultStr, " cipher: %s\n", strZ(varStr(kvGet(stanzaInfo, KEY_CIPHER_VAR)))); // If the cipher is mixed across repos for this stanza then display the per-repo cipher type if (strEq(varStr(kvGet(stanzaInfo, KEY_CIPHER_VAR)), STRDEF(INFO_STANZA_MIXED))) { const VariantList *const repoSection = kvGetList(stanzaInfo, STANZA_KEY_REPO_VAR); for (unsigned int repoIdx = 0; repoIdx < varLstSize(repoSection); repoIdx++) { const KeyValue *const repoInfo = varKv(varLstGet(repoSection, repoIdx)); strCatFmt( resultStr, " repo%u: %s\n", varUInt(kvGet(repoInfo, REPO_KEY_KEY_VAR)), strZ(varStr(kvGet(repoInfo, KEY_CIPHER_VAR)))); } } } // Get the current database for this stanza if (outputFull && !varLstEmpty(kvGetList(stanzaInfo, STANZA_KEY_DB_VAR))) { const InfoStanzaRepo *const stanzaRepo = lstFind(stanzaRepoList, &stanzaName); formatTextDb( stanzaInfo, resultStr, pgVersionToStr(stanzaRepo->currentPgVersion), stanzaRepo->currentPgSystemId, backupLabel); } } } else resultStr = strNewZ("No stanzas exist in the repository.\n"); } // Format json output else { ASSERT(cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_OUTPUT_JSON); resultStr = jsonFromVar(varNewVarLst(infoList)); } MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(resultStr); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void cmdInfo(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { ioFdWriteOneStr(STDOUT_FILENO, infoRender()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/info/info.h0000664000175100017510000000125615265101745017652 0ustar00runnerrunner/*********************************************************************************************************************************** Info Command ***********************************************************************************************************************************/ #ifndef COMMAND_INFO_INFO_H #define COMMAND_INFO_INFO_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Render info and output to stdout FN_EXTERN void cmdInfo(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/local/0000775000175100017510000000000015265101745016701 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/local/local.c0000664000175100017510000000334115265101745020140 0ustar00runnerrunner/*********************************************************************************************************************************** Local Command ***********************************************************************************************************************************/ #include #include "command/archive/get/protocol.h" #include "command/archive/push/protocol.h" #include "command/backup/protocol.h" #include "command/local/local.h" #include "command/restore/protocol.h" #include "command/verify/protocol.h" #include "common/debug.h" #include "common/log.h" #include "config/config.intern.h" #include "config/protocol.h" #include "protocol/helper.h" #include "protocol/server.h" /*********************************************************************************************************************************** Command handlers ***********************************************************************************************************************************/ static const ProtocolServerHandler commandLocalHandler[] = { PROTOCOL_SERVER_HANDLER_ARCHIVE_GET_LIST PROTOCOL_SERVER_HANDLER_ARCHIVE_PUSH_LIST PROTOCOL_SERVER_HANDLER_BACKUP_LIST PROTOCOL_SERVER_HANDLER_RESTORE_LIST PROTOCOL_SERVER_HANDLER_VERIFY_LIST }; static const List *const commandLocalHandlerList = LSTDEF(commandLocalHandler); /**********************************************************************************************************************************/ FN_EXTERN void cmdLocal(ProtocolServer *const server) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { protocolServerProcess(server, cfgCommandJobRetry(), commandLocalHandlerList); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/local/local.h0000664000175100017510000000132115265101745020141 0ustar00runnerrunner/*********************************************************************************************************************************** Local Command ***********************************************************************************************************************************/ #ifndef COMMAND_LOCAL_LOCAL_H #define COMMAND_LOCAL_LOCAL_H #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Local command FN_EXTERN void cmdLocal(ProtocolServer *server); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/lock.c0000664000175100017510000001740115265101745016706 0ustar00runnerrunner/*********************************************************************************************************************************** Command Lock Handler ***********************************************************************************************************************************/ #include #include "command/lock.h" #include "common/debug.h" #include "common/log.h" #include "config/config.h" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct CmdLockLocal { bool held; // Is the lock being held? } cmdLockLocal; /*********************************************************************************************************************************** Lock type names ***********************************************************************************************************************************/ static const char *const lockTypeName[] = { "archive", // lockTypeArchive "backup", // lockTypeBackup "restore", // lockTypeRestore }; /*********************************************************************************************************************************** Generate lock file name ***********************************************************************************************************************************/ static String * cmdLockFileName(const String *const stanza, const LockType lockType, const unsigned int repoKey) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, stanza); FUNCTION_TEST_PARAM(ENUM, lockType); FUNCTION_TEST_PARAM(UINT, repoKey); FUNCTION_TEST_END(); ASSERT(stanza != NULL); ASSERT(lockType < lockTypeAll); ASSERT(repoKey > 0); FUNCTION_TEST_RETURN(STRING, strNewFmt("%s-%s-%u" LOCK_FILE_EXT, strZ(stanza), lockTypeName[lockType], repoKey)); } /**********************************************************************************************************************************/ FN_EXTERN StringList * cmdLockList(void) { FUNCTION_TEST_VOID(); ASSERT(cfgLockType() != lockTypeNone); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { const LockType lockType = cfgLockType(); const LockType lockMin = lockType == lockTypeAll ? lockTypeArchive : lockType; const LockType lockMax = lockType == lockTypeAll ? (lockTypeAll - 1) : lockType; const bool repoSet = cfgOptionTest(cfgOptRepo); const unsigned int repoMin = repoSet ? cfgOptionGroupIdxDefault(cfgOptGrpRepo) : 0; const unsigned int repoMax = repoSet ? repoMin : cfgOptionGroupIdxTotal(cfgOptGrpRepo) - 1; for (LockType lockIdx = lockMin; lockIdx <= lockMax; lockIdx++) { for (unsigned int repoIdx = repoMin; repoIdx <= repoMax; repoIdx++) { strLstAdd( result, cmdLockFileName(cfgOptionStr(cfgOptStanza), lockIdx, cfgOptionGroupIdxToKey(cfgOptGrpRepo, repoIdx))); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN bool cmdLockAcquire(const LockAcquireParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BOOL, param.returnOnNoLock); FUNCTION_LOG_END(); bool result = true; // Don't allow return on no lock when locking more than one file. This makes cleanup difficult and there are no known use cases. ASSERT(!param.returnOnNoLock || cfgLockType() != lockTypeAll); // Don't allow another lock if one is already held if (cmdLockLocal.held) THROW(AssertError, "lock is already held by this process"); // Lock files MEM_CONTEXT_TEMP_BEGIN() { const StringList *const lockList = cfgOptionTest(cfgOptLock) ? strLstNewVarLst(cfgOptionLst(cfgOptLock)) : cmdLockList(); for (unsigned int lockListIdx = 0; lockListIdx < strLstSize(lockList); lockListIdx++) result = lockAcquire(strLstGet(lockList, lockListIdx), param); } MEM_CONTEXT_TEMP_END(); // Set lock held flag cmdLockLocal.held = result; FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN bool cmdLockOwn(void) { FUNCTION_LOG_VOID(logLevelDebug); ASSERT(cfgLockType() != lockTypeNone); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { const StringList *const lockList = cmdLockList(); // A valid lock held by our own exec id means our own async process is running and holding the lock for (unsigned int lockListIdx = 0; lockListIdx < strLstSize(lockList); lockListIdx++) { const LockReadResult lockResult = lockReadP(strLstGet(lockList, lockListIdx)); if (lockResult.status == lockReadStatusValid && strEq(lockResult.data.execId, lockExecId())) { result = true; break; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN void cmdLockWrite(const LockWriteParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(VARIANT, param.percentComplete); FUNCTION_LOG_PARAM(VARIANT, param.sizeComplete); FUNCTION_LOG_PARAM(VARIANT, param.size); FUNCTION_LOG_END(); String *const lockFileName = cmdLockFileName( cfgOptionStr(cfgOptStanza), cfgLockType(), cfgOptionGroupIdxToKey(cfgOptGrpRepo, cfgOptionGroupIdxDefault(cfgOptGrpRepo))); lockWrite(lockFileName, param); strFree(lockFileName); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN LockReadResult cmdLockRead(const LockType lockType, const String *const stanza, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(ENUM, lockType); FUNCTION_LOG_PARAM(STRING, stanza); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(lockType == lockTypeBackup || lockType == lockTypeRestore); ASSERT(stanza != NULL); FUNCTION_AUDIT_STRUCT(); LockReadResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { const String *const lockFileName = cmdLockFileName(stanza, lockType, cfgOptionGroupIdxToKey(cfgOptGrpRepo, repoIdx)); MEM_CONTEXT_PRIOR_BEGIN() { result = lockReadP(lockFileName); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN bool cmdLockRelease(const LockReleaseParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BOOL, param.returnOnNoLock); FUNCTION_LOG_END(); bool result = true; // Release lock(s) if (cmdLockLocal.held) { result = lockRelease(param); cmdLockLocal.held = false; } // Else error when requested else if (!param.returnOnNoLock) THROW(AssertError, "no lock is held by this process"); FUNCTION_LOG_RETURN(BOOL, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/lock.h0000664000175100017510000000440215265101745016710 0ustar00runnerrunner/*********************************************************************************************************************************** Command Lock Handler ***********************************************************************************************************************************/ #ifndef COMMAND_LOCK_H #define COMMAND_LOCK_H #include "common/lock.h" #include "config/config.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Get list of required locks FN_EXTERN StringList *cmdLockList(void); // Acquire a command lock. This will involve locking one or more files on disk depending on the lock type. Most operations only // acquire a single lock type (archive or backup), but the stanza commands all need to lock both. #define cmdLockAcquireP(...) \ cmdLockAcquire((LockAcquireParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool cmdLockAcquire(LockAcquireParam param); // Write data to command lock file #define cmdLockWriteP(...) \ cmdLockWrite((LockWriteParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void cmdLockWrite(LockWriteParam param); // Read a command lock file held by another process to get information about what the process is doing FN_EXTERN LockReadResult cmdLockRead(LockType lockType, const String *stanza, unsigned int repoKey); // Is a command lock held by this process's own exec id, i.e. is our own async process running? Async archive-get uses this to // decide whether to wait for its own async process or fetch the segment synchronously (when the lock is free or held by a foreign // process). FN_EXTERN bool cmdLockOwn(void); // Release command lock(s) #define cmdLockReleaseP(...) \ cmdLockRelease((LockReleaseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool cmdLockRelease(LockReleaseParam param); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/manifest/0000775000175100017510000000000015265101745017415 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/manifest/manifest.c0000664000175100017510000005710115265101745021373 0ustar00runnerrunner/*********************************************************************************************************************************** Manifest Command ***********************************************************************************************************************************/ #include #include #include "command/backup/blockMap.h" #include "command/manifest/manifest.h" #include "command/restore/blockChecksum.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/io/fdWrite.h" #include "common/io/io.h" #include "common/log.h" #include "common/type/json.h" #include "config/config.h" #include "info/manifest/manifest.h" #include "storage/helper.h" /*********************************************************************************************************************************** Block map render ***********************************************************************************************************************************/ typedef struct ManifestBlockDeltaReference { unsigned int reference; // Reference List *blockList; // List of blocks in the block map for the reference } ManifestBlockDeltaReference; // Reads that must be performed in order to extract blocks typedef struct ManifestBlockDeltaRead { unsigned int reference; // Reference to read from uint64_t bundleId; // Bundle to read from uint64_t offset; // Offset to begin read from uint64_t size; // Size of the read List *superBlockList; // Super block list } ManifestBlockDeltaRead; // Super blocks to be extracted from the read typedef struct ManifestBlockDeltaSuperBlock { uint64_t superBlockSize; // Super block size uint64_t size; // Stored super block size (with compression, etc.) List *blockList; // Block list } ManifestBlockDeltaSuperBlock; // Blocks to be extracted from the super block typedef struct ManifestBlockDeltaBlock { uint64_t no; // Block number in the super block uint64_t offset; // Offset into original file uint8_t checksum[XX_HASH_SIZE_MAX]; // Checksum of the block } ManifestBlockDeltaBlock; static List * cmdManifestBlockDelta( const BlockMap *const blockMap, const size_t blockSize, const size_t checksumSize, const Buffer *const blockChecksum) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BLOCK_MAP, blockMap); FUNCTION_TEST_PARAM(SIZE, blockSize); FUNCTION_TEST_PARAM(SIZE, checksumSize); FUNCTION_TEST_PARAM(BUFFER, blockChecksum); FUNCTION_TEST_END(); ASSERT(blockMap != NULL); ASSERT(blockSize > 0); List *const result = lstNewP(sizeof(ManifestBlockDeltaRead)); MEM_CONTEXT_TEMP_BEGIN() { // Build list of references and for each reference the list of blocks for that reference const unsigned int blockChecksumSize = blockChecksum == NULL ? 0 : (unsigned int)(bufUsed(blockChecksum) / checksumSize); List *const referenceList = lstNewP(sizeof(ManifestBlockDeltaReference), .comparator = lstComparatorUInt); for (unsigned int blockMapIdx = 0; blockMapIdx < blockMapSize(blockMap); blockMapIdx++) { const BlockMapItem *const blockMapItem = blockMapGet(blockMap, blockMapIdx); // The block must be updated if it is beyond the blocks that exist in the block checksum list or when the checksum // stored in the repository is different from the block checksum list if (blockMapIdx >= blockChecksumSize || !bufEq( BUF(blockMapItem->checksum, checksumSize), BUF(bufPtrConst(blockChecksum) + blockMapIdx * checksumSize, checksumSize))) { const unsigned int reference = blockMapItem->reference; ManifestBlockDeltaReference *const referenceData = lstFind(referenceList, &reference); // If the reference has not been added if (referenceData == NULL) { ManifestBlockDeltaReference *referenceData = lstAdd( referenceList, &(ManifestBlockDeltaReference){.reference = reference, .blockList = lstNewP(sizeof(unsigned int))}); lstAdd(referenceData->blockList, &blockMapIdx); } // Else add the new block else lstAdd(referenceData->blockList, &blockMapIdx); } } // Sort the reference list ascending. This is an arbitrary choice as the order does not matter. lstSort(referenceList, sortOrderAsc); // Build delta for (unsigned int referenceIdx = 0; referenceIdx < lstSize(referenceList); referenceIdx++) { const ManifestBlockDeltaReference *const referenceData = (const ManifestBlockDeltaReference *)lstGet( referenceList, referenceIdx); ManifestBlockDeltaRead *blockDeltaRead = NULL; ManifestBlockDeltaSuperBlock *blockDeltaSuperBlock = NULL; const BlockMapItem *blockMapItemPrior = NULL; for (unsigned int blockIdx = 0; blockIdx < lstSize(referenceData->blockList); blockIdx++) { const unsigned int blockMapIdx = *(unsigned int *)lstGet(referenceData->blockList, blockIdx); const BlockMapItem *const blockMapItem = blockMapGet(blockMap, blockMapIdx); // Add read when it has changed if (blockMapItemPrior == NULL || (blockMapItemPrior->offset != blockMapItem->offset && blockMapItemPrior->offset + blockMapItemPrior->size != blockMapItem->offset)) { MEM_CONTEXT_OBJ_BEGIN(result) { ManifestBlockDeltaRead blockDeltaReadNew = { .reference = blockMapItem->reference, .bundleId = blockMapItem->bundleId, .offset = blockMapItem->offset, .superBlockList = lstNewP(sizeof(ManifestBlockDeltaSuperBlock)), }; blockDeltaRead = lstAdd(result, &blockDeltaReadNew); } MEM_CONTEXT_OBJ_END(); } // Add super block when it has changed if (blockMapItemPrior == NULL || blockMapItemPrior->offset != blockMapItem->offset) { MEM_CONTEXT_OBJ_BEGIN(blockDeltaRead->superBlockList) { ManifestBlockDeltaSuperBlock blockDeltaSuperBlockNew = { .superBlockSize = blockMapItem->superBlockSize, .size = blockMapItem->size, .blockList = lstNewP(sizeof(ManifestBlockDeltaBlock)), }; blockDeltaSuperBlock = lstAdd(blockDeltaRead->superBlockList, &blockDeltaSuperBlockNew); blockDeltaRead->size += blockMapItem->size; } MEM_CONTEXT_OBJ_END(); } // Add block ManifestBlockDeltaBlock blockDeltaBlockNew = { .no = blockMapItem->block, .offset = blockMapIdx * blockSize, }; memcpy( blockDeltaBlockNew.checksum, blockMapItem->checksum, SIZE_OF_STRUCT_MEMBER(ManifestBlockDeltaBlock, checksum)); lstAdd(blockDeltaSuperBlock->blockList, &blockDeltaBlockNew); // Set prior item for comparison on the next loop blockMapItemPrior = blockMapItem; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(LIST, result); } /*********************************************************************************************************************************** Block map render ***********************************************************************************************************************************/ static String * cmdManifestBlockDeltaRender(const Manifest *const manifest, const ManifestFile *const file, const bool json) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(MANIFEST_FILE, file); FUNCTION_LOG_PARAM(BOOL, json); FUNCTION_LOG_END(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { // Load block checksums for the file if the pg option is explicitly set const Buffer *blockChecksum = NULL; const Buffer *checksum = NULL; if (cfgOptionSource(cfgOptPg) != cfgSourceDefault) { IoRead *const read = storageReadIo(storageNewReadP(storagePg(), manifestPathPg(file->name))); ioFilterGroupAdd(ioReadFilterGroup(read), cryptoHashNew(hashTypeSha1)); ioFilterGroupAdd(ioReadFilterGroup(read), blockChecksumNew(file->blockIncrSize, file->blockIncrChecksumSize)); ioReadDrain(read); checksum = pckReadBinP(ioFilterGroupResultP(ioReadFilterGroup(read), CRYPTO_HASH_FILTER_TYPE)); blockChecksum = pckReadBinP(ioFilterGroupResultP(ioReadFilterGroup(read), BLOCK_CHECKSUM_FILTER_TYPE)); } // If the file is up-to-date if (checksum != NULL && bufEq(checksum, BUF(file->checksumSha1, HASH_TYPE_SHA1_SIZE))) { if (json) strCatZ(result, "null"); else strCatZ(result, " file is up-to-date\n"); } // Else render the block delta else { StorageRead *const read = storageNewReadP( storageRepo(), backupFileRepoPathP( file->reference != NULL ? file->reference : manifestData(manifest)->backupLabel, .manifestName = file->name, .bundleId = file->bundleId, .compressType = manifestData(manifest)->backupOptionCompressType, .blockIncr = true), .offset = file->bundleOffset + file->sizeRepo - file->blockIncrMapSize, .limit = VARUINT64(file->blockIncrMapSize)); if (cipherSpecType(manifestCipherSpec(manifest)) != cipherTypeNone) { ioFilterGroupAdd( ioReadFilterGroup(storageReadIo(read)), cipherBlockNewP(cipherModeDecrypt, manifestCipherSpec(manifest), .header = cipherBlockHeaderNone)); } ioReadOpen(storageReadIo(read)); List *const blockDelta = cmdManifestBlockDelta( blockMapNewRead(storageReadIo(read), file->blockIncrSize, file->blockIncrChecksumSize), file->blockIncrSize, file->blockIncrChecksumSize, blockChecksum); unsigned int referenceRead = 0; uint64_t referenceReadSize = 0; unsigned int referenceSuperBlock = 0; uint64_t referenceSuperBlockSize = 0; unsigned int referenceBlock = 0; unsigned int totalRead = 0; uint64_t totalReadSize = 0; unsigned int totalSuperBlock = 0; uint64_t totalSuperBlockSize = 0; unsigned int totalBlock = 0; bool first = true; if (json) strCatChr(result, '['); else strCatChr(result, '\n'); for (unsigned int readIdx = 0; readIdx < lstSize(blockDelta); readIdx++) { const ManifestBlockDeltaRead *const read = lstGet(blockDelta, readIdx); referenceRead++; referenceReadSize += read->size; referenceSuperBlock += lstSize(read->superBlockList); for (unsigned int superBlockIdx = 0; superBlockIdx < lstSize(read->superBlockList); superBlockIdx++) { const ManifestBlockDeltaSuperBlock *const superBlock = lstGet(read->superBlockList, superBlockIdx); referenceSuperBlockSize += superBlock->superBlockSize; referenceBlock += lstSize(superBlock->blockList); } if (readIdx == lstSize(blockDelta) - 1 || read->reference != ((const ManifestBlockDeltaRead *)lstGet(blockDelta, readIdx + 1))->reference) { if (json) { if (first) first = false; else strCatChr(result, ','); strCatFmt( result, "{\"reference\":%u,\"read\":{\"total\":%u,\"size\":%" PRIu64 "},\"superBlock\":{\"total\":%u" ",\"size\":%" PRIu64 "},\"block\":{\"total\":%u}}", read->reference, referenceRead, referenceReadSize, referenceSuperBlock, referenceSuperBlockSize, referenceBlock); } else { const String *const reference = strSub( backupFileRepoPathP( strLstGet(manifestReferenceList(manifest), read->reference), .manifestName = file->name, .bundleId = read->bundleId, .compressType = manifestData(manifest)->backupOptionCompressType, .blockIncr = true), sizeof(STORAGE_REPO_BACKUP)); strCatFmt( result, " reference: %s, read: %u/%s, superBlock: %u/%s, block: %u/%s\n", strZ(reference), referenceRead, strZ(strSizeFormat(referenceReadSize)), referenceSuperBlock, strZ(strSizeFormat(referenceSuperBlockSize)), referenceBlock, strZ(strSizeFormat(referenceBlock * file->blockIncrSize))); } totalRead += referenceRead; totalReadSize += referenceReadSize; totalSuperBlock += referenceSuperBlock; totalSuperBlockSize += referenceSuperBlockSize; totalBlock += referenceBlock; referenceRead = 0; referenceReadSize = 0; referenceSuperBlock = 0; referenceSuperBlockSize = 0; referenceBlock = 0; } } if (json) strCatChr(result, ']'); else { strCatFmt( result, " total read: %u/%s, superBlock: %u/%s, block: %u/%s\n", totalRead, strZ(strSizeFormat(totalReadSize)), totalSuperBlock, strZ(strSizeFormat(totalSuperBlockSize)), totalBlock, strZ(strSizeFormat(totalBlock * file->blockIncrSize))); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } /*********************************************************************************************************************************** Manifest file render ***********************************************************************************************************************************/ static String * cmdManifestFileRender(const Manifest *const manifest, const ManifestFile *const file, const bool blockDelta, const bool json) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(MANIFEST_FILE, file); FUNCTION_LOG_PARAM(BOOL, blockDelta); FUNCTION_LOG_PARAM(BOOL, json); FUNCTION_LOG_END(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { if (json) { strCatFmt(result, "{\"name\":%s", strZ(jsonFromVar(VARSTR(file->name)))); if (file->reference != NULL) strCatFmt(result, ",\"reference\":%s", strZ(jsonFromVar(VARSTR(file->reference)))); strCatFmt(result, ",\"size\":%" PRIu64, file->size); strCatFmt( result, ",\"checksum\":\"%s\"", strZ(strNewEncode(encodingHex, BUF(file->checksumSha1, HASH_TYPE_SHA1_SIZE)))); strCatFmt(result, ",\"repo\":{\"size\":%" PRIu64 "}", file->sizeRepo); if (file->bundleId != 0) strCatFmt(result, ",\"bundle\":{\"id\":%" PRIu64 ",\"offset\":%" PRIu64 "}", file->bundleId, file->bundleOffset); if (file->blockIncrSize != 0) { strCatFmt( result, ",\"block\":{\"size\":%zu,\"map\":{\"size\":%" PRIu64, file->blockIncrSize, file->blockIncrMapSize); if (blockDelta) strCatFmt(result, ",\"delta\":%s", strZ(cmdManifestBlockDeltaRender(manifest, file, json))); strCatFmt(result, "},\"checksum\":{\"size\":%zu}}", file->blockIncrChecksumSize); } strCatChr(result, '}'); } else { strCatFmt(result, " - %s\n", strZ(file->name)); if (file->reference != NULL) strCatFmt(result, " reference: %s\n", strZ(file->reference)); strCatFmt( result, " size: %s, repo %s\n", strZ(strSizeFormat(file->size)), strZ(strSizeFormat(file->sizeRepo))); strCatFmt( result, " checksum: %s\n", strZ(strNewEncode(encodingHex, BUF(file->checksumSha1, HASH_TYPE_SHA1_SIZE)))); if (file->bundleId != 0) strCatFmt(result, " bundle: %" PRIu64 "\n", file->bundleId); if (file->blockIncrSize != 0) { strCatFmt( result, " block: size %s, map size %s, checksum size %s\n", strZ(strSizeFormat(file->blockIncrSize)), strZ(strSizeFormat(file->blockIncrMapSize)), strZ(strSizeFormat(file->blockIncrChecksumSize))); if (blockDelta) { strCatZ(result, " block delta:"); strCat(result, cmdManifestBlockDeltaRender(manifest, file, json)); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } /*********************************************************************************************************************************** Manifest render ***********************************************************************************************************************************/ static String * cmdManifestRender(void) { FUNCTION_LOG_VOID(logLevelDebug); String *const result = strNew(); // Set dry run to make sure this command never writes anything storageHelperDryRunInit(true); MEM_CONTEXT_TEMP_BEGIN() { // Get the repo storage in case it is remote and encryption settings need to be pulled down storageRepo(); // Load backup.info and cipher const InfoBackup *const infoBackup = infoBackupLoadFile(storageRepo(), INFO_BACKUP_PATH_FILE_STR, cfgCipherSpecMain()); // Load manifest const Manifest *const manifest = manifestLoadFile( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(cfgOptionStr(cfgOptSet))), infoBackupCipherSpec(infoBackup)); // Manifest info const ManifestData *const data = manifestData(manifest); const bool json = cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_OUTPUT_JSON; if (json) { strCatFmt(result, "{\"label\":\"%s\"", strZ(data->backupLabel)); strCatFmt(result, ",\"reference\":[%s]", strZ(strLstJoinQuote(manifestReferenceList(manifest), ",", "\""))); strCatFmt(result, ",\"type\":\"%s\"", zNewStrId(data->backupType)); strCatFmt( result, ",\"time\":{\"start\":%" PRId64 ",\"copy\":%" PRId64 ",\"stop\":%" PRId64 "}", (int64_t)data->backupTimestampStart, (int64_t)data->backupTimestampCopyStart, (int64_t)data->backupTimestampStop); strCatFmt( result, ",\"bundle\":{\"bundle\":%s,\"raw\":%s}", cvtBoolToConstZ(data->bundle), cvtBoolToConstZ(data->bundleRaw)); strCatFmt(result, ",\"block\":{\"block\":%s}", cvtBoolToConstZ(data->blockIncr)); } else { strCatFmt(result, "label: %s\n", strZ(data->backupLabel)); strCatFmt(result, "reference: %s\n", strZ(strLstJoin(manifestReferenceList(manifest), ", "))); strCatFmt(result, "type: %s\n", zNewStrId(data->backupType)); int64_t duration = (int64_t)(data->backupTimestampStop - data->backupTimestampStart); strCatFmt( result, "time: start: %s, stop: %s, duration: %" PRId64 ":%02" PRId64 ":%02" PRId64 "\n", strZ(strNewTimeP("%Y-%m-%d %H:%M:%S", data->backupTimestampStart)), strZ(strNewTimeP("%Y-%m-%d %H:%M:%S", data->backupTimestampStop)), duration / 3600, duration % 3600 / 60, duration % 60); strCatFmt(result, "bundle: %s\n", cvtBoolToConstZ(data->bundle)); strCatFmt(result, "block: %s\n", cvtBoolToConstZ(data->blockIncr)); strCatChr(result, '\n'); } // File list if (json) strCatZ(result, ",\"fileList\":["); else strCatZ(result, "file list:\n"); const String *const filter = cfgOptionStrNull(cfgOptFilter); // Single file if (filter != NULL) { const ManifestFile file = manifestFileFind(manifest, filter); strCat(result, cmdManifestFileRender(manifest, &file, true, json)); } // Multiple files else { bool first = true; for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFile file = manifestFile(manifest, fileIdx); // Filter on reference if (cfgOptionTest(cfgOptReference)) { if (strEqZ(cfgOptionStr(cfgOptReference), "latest")) { if (file.reference != NULL) continue; } else if (!strEq(cfgOptionStr(cfgOptReference), file.reference)) continue; } if (!first) { if (json) strCatChr(result, ','); else strCatChr(result, '\n'); } else first = false; strCat(result, cmdManifestFileRender(manifest, &file, false, json)); } } if (json) strCatZ(result, "]}"); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void cmdManifest(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { ioFdWriteOneStr(STDOUT_FILENO, cmdManifestRender()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/manifest/manifest.h0000664000175100017510000000126315265101745021376 0ustar00runnerrunner/*********************************************************************************************************************************** Manifest Command ***********************************************************************************************************************************/ #ifndef COMMAND_MANIFEST_MANIFEST_H #define COMMAND_MANIFEST_MANIFEST_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Manifest info FN_EXTERN void cmdManifest(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/remote/0000775000175100017510000000000015265101745017102 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/remote/remote.c0000664000175100017510000001074015265101745020543 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Command ***********************************************************************************************************************************/ #include #include #include "command/backup/blockIncr.h" #include "command/backup/pageChecksum.h" #include "command/control/common.h" #include "command/lock.h" #include "command/remote/remote.h" #include "command/restore/blockChecksum.h" #include "common/crypto/cipherBlock.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/format/cipherBlockFormat.h" #include "common/io/filter/sink.h" #include "common/io/filter/size.h" #include "common/log.h" #include "config/config.h" #include "config/protocol.h" #include "db/protocol.h" #include "protocol/helper.h" #include "protocol/server.h" #include "storage/remote/protocol.h" /*********************************************************************************************************************************** Command handlers ***********************************************************************************************************************************/ static const ProtocolServerHandler commandRemoteHandler[] = { PROTOCOL_SERVER_HANDLER_DB_LIST PROTOCOL_SERVER_HANDLER_OPTION_LIST PROTOCOL_SERVER_HANDLER_STORAGE_REMOTE_LIST }; static const List *const commandRemoteHandlerList = LSTDEF(commandRemoteHandler); /*********************************************************************************************************************************** Filter handlers ***********************************************************************************************************************************/ static const StorageRemoteFilterHandler storageRemoteFilterHandler[] = { {.type = BLOCK_CHECKSUM_FILTER_TYPE, .handlerParam = blockChecksumNewPack}, {.type = BLOCK_INCR_FILTER_TYPE, .handlerParam = blockIncrNewPack}, {.type = CIPHER_BLOCK_FILTER_TYPE, .handlerParam = cipherBlockNewPack}, {.type = CIPHER_BLOCK_FORMAT_FILTER_TYPE, .handlerParam = cipherBlockFormatNewPack}, {.type = CIPHER_BLOCK_FORMAT_HEADER_FILTER_TYPE, .handlerParam = cipherBlockFormatHeaderNewPack}, {.type = CRYPTO_HASH_FILTER_TYPE, .handlerParam = cryptoHashNewPack}, {.type = PAGE_CHECKSUM_FILTER_TYPE, .handlerParam = pageChecksumNewPack}, {.type = SINK_FILTER_TYPE, .handlerNoParam = ioSinkNew}, {.type = SIZE_FILTER_TYPE, .handlerNoParam = ioSizeNew}, }; static const List *const storageRemoteFilterHandlerList = LSTDEF(storageRemoteFilterHandler); /**********************************************************************************************************************************/ FN_EXTERN void cmdRemote(ProtocolServer *const server) { FUNCTION_LOG_VOID(logLevelDebug); // Set filter handlers storageRemoteFilterHandlerSet(storageRemoteFilterHandlerList); MEM_CONTEXT_TEMP_BEGIN() { // Acquire a lock if this command needs one. We'll use the noop that is always sent from the client right after the // handshake to return an error. We can't take a lock earlier than this because we want the error to go back through the // protocol layer. volatile bool success = false; TRY_BEGIN() { // Get the command. No need to check parameters since we know this is the first noop. CHECK(FormatError, protocolServerRequest(server).id == PROTOCOL_COMMAND_NOOP, "noop expected"); // Only try the lock if this is process 0, i.e. the remote started from the main process if (cfgOptionUInt(cfgOptProcess) == 0) { // Acquire a lock if this command requires a lock if (cfgLockRemoteRequired()) { // Make sure the local host is not stopped lockStopTest(); // Acquire the lock cmdLockAcquireP(); } } // Notify the client of success protocolServerResponseP(server); success = true; } CATCH_ANY() { protocolServerError(server, errorCode(), STR(errorMessage()), STR(errorStackTrace())); } TRY_END(); // If not successful we'll just exit if (success) protocolServerProcess(server, NULL, commandRemoteHandlerList); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/remote/remote.h0000664000175100017510000000133015265101745020543 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Command ***********************************************************************************************************************************/ #ifndef COMMAND_REMOTE_REMOTE_H #define COMMAND_REMOTE_REMOTE_H #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Remote command FN_EXTERN void cmdRemote(ProtocolServer *server); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/repo/0000775000175100017510000000000015265101745016554 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/repo/common.c0000664000175100017510000000525415265101745020216 0ustar00runnerrunner/*********************************************************************************************************************************** Common Functions and Definitions for Repo Commands ***********************************************************************************************************************************/ #include #include "command/repo/common.h" #include "common/debug.h" #include "config/config.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN String * repoPathIsValid(const String *path) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_END(); ASSERT(path != NULL); String *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Make sure there are no occurrences of // if (strstr(strZ(path), "//") != NULL) THROW_FMT(ParamInvalidError, "path '%s' cannot contain //", strZ(path)); // Make sure the path does not end with a slash if (strEndsWith(path, FSLASH_STR)) path = strPath(path); // Validate absolute paths if (strBeginsWith(path, FSLASH_STR)) { // Get the repo path using repo storage in case it is remotely configured const String *const repoPath = storagePathP(storageRepo(), NULL); // If the path is exactly equal to the repo path then the relative path is empty if (strEq(path, repoPath)) { MEM_CONTEXT_PRIOR_BEGIN() { result = strNew(); } MEM_CONTEXT_PRIOR_END(); } // Else check that the file path begins with the repo path else { if (!strEq(repoPath, FSLASH_STR) && !strBeginsWith(path, strNewFmt("%s/", strZ(repoPath)))) { THROW_FMT( ParamInvalidError, "absolute path '%s' is not in base path '%s'", strZ(path), strZ(cfgOptionDisplay(cfgOptRepoPath))); } MEM_CONTEXT_PRIOR_BEGIN() { // Get the relative part of the path/file result = strSub(path, strEq(repoPath, FSLASH_STR) ? 1 : strSize(repoPath) + 1); } MEM_CONTEXT_PRIOR_END(); } } else { MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(path); } MEM_CONTEXT_PRIOR_END(); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/repo/common.h0000664000175100017510000000154415265101745020221 0ustar00runnerrunner/*********************************************************************************************************************************** Repo Command Common ***********************************************************************************************************************************/ #ifndef COMMAND_REPO_COMMON_H #define COMMAND_REPO_COMMON_H #include "common/type/string.h" /*********************************************************************************************************************************** Functions and Definitions ***********************************************************************************************************************************/ // Path cannot contain //. Strip trailing /. Absolute path must fall under repo path. Throw error or return validated relative path. FN_EXTERN String *repoPathIsValid(const String *path); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/repo/get.c0000664000175100017510000002247715265101745017513 0ustar00runnerrunner/*********************************************************************************************************************************** Repository Get Command ***********************************************************************************************************************************/ #include #include #include "command/repo/common.h" #include "command/repo/get.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/format/cipherBlockFormat.h" #include "common/io/fdWrite.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "storage/helper.h" #include "info/info.h" #include "info/infoArchive.h" #include "info/infoBackup.h" /*********************************************************************************************************************************** Write source file to destination IO ***********************************************************************************************************************************/ static int storageGetProcess(IoWrite *const destination) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, destination); FUNCTION_LOG_END(); // Get source file if (strLstSize(cfgCommandParam()) != 1) THROW(ParamRequiredError, "source file required"); const String *file = strLstGet(cfgCommandParam(), 0); // Assume the file is missing int result = 1; MEM_CONTEXT_TEMP_BEGIN() { // Is path valid for repo? file = repoPathIsValid(file); // Create new file read IoRead *const source = storageReadIo( storageNewReadP(storageRepo(), file, .ignoreMissing = cfgOptionBool(cfgOptIgnoreMissing))); // Is the file an info file, i.e. one that has a header in front of its content? bool fileIsInfo = false; // Add decryption if needed if (!cfgOptionBool(cfgOptRaw)) { const CipherType repoCipherType = cfgOptionStrId(cfgOptRepoCipherType); if (repoCipherType != cipherTypeNone) { // Determine the passphrase using the following pattern: // // REPO / (main passphrase) // / archive / (main passphrase) // / archive / stanza / (archive passphrase) // / backup / (main passphrase) // / backup / stanza / (manifest passphrase) // / backup / stanza / set / (backup passphrase) // / backup / stanza / backup.history / (manifest passphrase) // // Nothing should be stored at the top level of the repo except the backup/archive paths. The backup/archive paths // should contain only stanza paths. // ----------------------------------------------------------------------------------------------------------------- const CipherSpec *cipherSpec = NULL; const CipherSpecMap *cipherSpecMap = NULL; const StringList *const filePathSplitLst = strLstNewSplit(file, FSLASH_STR); // At a minimum the path must contain archive/backup, a stanza, and a file if (strLstSize(filePathSplitLst) > 2) { const String *const stanza = strLstGet(filePathSplitLst, 1); // If stanza option is specified then it must match the given file path if (cfgOptionStrNull(cfgOptStanza) != NULL && !strEq(stanza, cfgOptionStr(cfgOptStanza))) { THROW_FMT( OptionInvalidValueError, "stanza name '%s' given in option doesn't match the given path", strZ(cfgOptionDisplay(cfgOptStanza))); } // Archive path if (strEq(strLstGet(filePathSplitLst, 0), STORAGE_PATH_ARCHIVE_STR)) { // Find the archive keys. WAL stores the id of the key it is encrypted with in the header. if (!strEndsWithZ(file, INFO_ARCHIVE_FILE) && !strEndsWithZ(file, INFO_ARCHIVE_FILE INFO_COPY_EXT)) { const InfoArchive *const info = infoArchiveLoadFile( storageRepo(), strNewFmt(STORAGE_PATH_ARCHIVE "/%s/%s", strZ(stanza), INFO_ARCHIVE_FILE), cfgCipherSpecMain()); cipherSpecMap = infoArchiveCipherSpecMap(info); } // Else the file is the archive info, which the repo passphrase opens else { cipherSpec = cfgCipherSpecMain(); fileIsInfo = true; } } // Backup path if (strEq(strLstGet(filePathSplitLst, 0), STORAGE_PATH_BACKUP_STR)) { cipherSpec = cfgCipherSpecMain(); if (!strEndsWithZ(file, INFO_BACKUP_FILE) && !strEndsWithZ(file, INFO_BACKUP_FILE INFO_COPY_EXT)) { // Find the manifest passphrase const InfoBackup *const info = infoBackupLoadFile( storageRepo(), strNewFmt(STORAGE_PATH_BACKUP "/%s/%s", strZ(stanza), INFO_BACKUP_FILE), cfgCipherSpecMain()); const CipherSpec *const cipherSpecManifest = infoBackupCipherSpec(info); // Find the backup passphrase if not a manifest if (!strEq(strLstGet(filePathSplitLst, 2), STRDEF(BACKUP_PATH_HISTORY)) && !strEndsWithZ(file, BACKUP_MANIFEST_FILE) && !strEndsWithZ(file, BACKUP_MANIFEST_FILE INFO_COPY_EXT)) { const Manifest *const manifest = manifestLoadFile( storageRepo(), strNewFmt( STORAGE_PATH_BACKUP "/%s/%s/%s", strZ(stanza), strZ(strLstGet(filePathSplitLst, 2)), BACKUP_MANIFEST_FILE), cipherSpecManifest); cipherSpec = manifestCipherSpec(manifest); } // Else use the manifest passphrase else cipherSpec = cipherSpecManifest; } // Else the file is the backup info, which the repo passphrase opens else fileIsInfo = true; } } // Error when unable to determine cipher passphrase if (cipherSpec == NULL && cipherSpecMap == NULL) THROW_FMT(OptionInvalidValueError, "unable to determine cipher passphrase for '%s'", strZ(file)); // Add the decryption filter. An info file is read through the filter that reads its header and WAL through the // filter that selects the key by the id in the header. if (fileIsInfo) cipherBlockFormatFilterGroupReadAdd(ioReadFilterGroup(source), cipherSpec); else if (cipherSpecMap != NULL) cipherBlockFormatFilterGroupReadAddMap(ioReadFilterGroup(source), cipherSpecMap); else { ASSERT(cipherSpecType(cipherSpec) != cipherTypeNone); cipherBlockFilterGroupAdd(ioReadFilterGroup(source), cipherModeDecrypt, cipherSpec); } } } // Open source if (ioReadOpen(source)) { // Open the destination file now that we know the source exists and is readable ioWriteOpen(destination); // Copy data from source to destination ioCopyP(source, destination); // Close the source and destination ioReadClose(source); ioWriteClose(destination); // Source file exists result = 0; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INT, result); } /**********************************************************************************************************************************/ FN_EXTERN int cmdStorageGet(void) { FUNCTION_LOG_VOID(logLevelDebug); // Assume the file is missing int result = 1; MEM_CONTEXT_TEMP_BEGIN() { TRY_BEGIN() { result = storageGetProcess(ioFdWriteNew(STRDEF("stdout"), STDOUT_FILENO, cfgOptionUInt64(cfgOptIoTimeout))); } // Ignore write errors because it's possible (even likely) that this output is being piped to something like head which will // exit when it gets what it needs and leave us writing to a broken pipe. It would be better to just ignore the broken pipe // error but currently we don't store system error codes. CATCH(FileWriteError) { } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/repo/get.h0000664000175100017510000000127715265101745017513 0ustar00runnerrunner/*********************************************************************************************************************************** Repository Get Command ***********************************************************************************************************************************/ #ifndef COMMAND_STORAGE_GET_H #define COMMAND_STORAGE_GET_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Get a file from the repository FN_EXTERN int cmdStorageGet(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/repo/ls.c0000664000175100017510000001352415265101745017343 0ustar00runnerrunner/*********************************************************************************************************************************** Repository List Command ***********************************************************************************************************************************/ #include #include #include "command/repo/common.h" #include "command/repo/ls.h" #include "common/debug.h" #include "common/io/fdWrite.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/type/json.h" #include "common/type/string.h" #include "config/config.h" #include "storage/helper.h" /*********************************************************************************************************************************** Render storage list ***********************************************************************************************************************************/ static void storageListRenderInfo(const StorageInfo *const info, IoWrite *const write, const bool json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_PARAM(IO_WRITE, write); FUNCTION_TEST_PARAM(BOOL, json); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(info != NULL); ASSERT(write != NULL); // Render in json if (json) { ioWriteStr(write, jsonFromVar(VARSTR(info->name))); ioWrite(write, BUFSTRDEF(":{\"type\":\"")); switch (info->type) { case storageTypeFile: ioWrite(write, BUFSTRDEF("file\"")); break; case storageTypeLink: ioWrite(write, BUFSTRDEF("link\"")); break; case storageTypePath: ioWrite(write, BUFSTRDEF("path\"")); break; case storageTypeSpecial: ioWrite(write, BUFSTRDEF("special\"")); break; } if (info->type == storageTypeFile) { ioWriteStr(write, strNewFmt(",\"size\":%" PRIu64, info->size)); ioWriteStr(write, strNewFmt(",\"time\":%" PRId64, (int64_t)info->timeModified)); if (info->versionId != NULL) ioWriteStr(write, strNewFmt(",\"version\":%s", strZ(jsonFromVar(VARSTR(info->versionId))))); } if (info->type == storageTypeLink) ioWriteStr(write, strNewFmt(",\"destination\":%s", strZ(jsonFromVar(VARSTR(info->linkDestination))))); ioWrite(write, BRACER_BUF); } // Render in text else { ioWrite(write, BUFSTR(info->name)); ioWrite(write, LF_BUF); } FUNCTION_TEST_RETURN_VOID(); } static void storageListRender(IoWrite *const write) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); // Get path const String *path = NULL; // Get and validate if path is valid for repo if (strLstSize(cfgCommandParam()) == 1) path = repoPathIsValid(strLstGet(cfgCommandParam(), 0)); else if (strLstSize(cfgCommandParam()) > 1) THROW(ParamInvalidError, "only one path may be specified"); // Get options const bool json = cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_OUTPUT_JSON; const String *const expression = cfgOptionStrNull(cfgOptFilter); RegExp *const regExp = expression == NULL ? NULL : regExpNew(expression); ioWriteOpen(write); if (json) ioWrite(write, BRACEL_BUF); // Check if this is a file StorageInfo info = storageInfoP(storageRepo(), path, .ignoreMissing = true); if (info.exists && info.type == storageTypeFile) { if (regExp == NULL || regExpMatch(regExp, storagePathP(storageRepo(), path))) { info.name = DOT_STR; storageListRenderInfo(&info, write, json); } } // Else try to list the path else { // The path will always be reported as existing so we don't get different results from storage that does not support paths bool first = true; if (json && (regExp == NULL || regExpMatch(regExp, DOT_STR))) { storageListRenderInfo(&(StorageInfo){.type = storageTypePath, .name = DOT_STR}, write, json); first = false; } // List content of the path StorageIterator *const storageItr = storageNewItrP( storageRepo(), path, .sortOrder = cfgOptionSeq(cfgOptSort), .expression = expression, .recurse = cfgOptionBool(cfgOptRecurse)); while (storageItrMore(storageItr)) { const StorageInfo info = storageItrNext(storageItr); // Add separator character if (!first && json) ioWrite(write, COMMA_BUF); else first = false; storageListRenderInfo(&info, write, json); } } if (json) { ioWrite(write, BRACER_BUF); ioWrite(write, LF_BUF); } ioWriteClose(write); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cmdStorageList(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { TRY_BEGIN() { storageListRender(ioFdWriteNew(STRDEF("stdout"), STDOUT_FILENO, cfgOptionUInt64(cfgOptIoTimeout))); } // Ignore write errors because it's possible (even likely) that this output is being piped to something like head which will // exit when it gets what it needs and leave us writing to a broken pipe. It would be better to just ignore the broken pipe // error but currently we don't store system error codes. CATCH(FileWriteError) { } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/repo/ls.h0000664000175100017510000000130215265101745017337 0ustar00runnerrunner/*********************************************************************************************************************************** Repository List Command ***********************************************************************************************************************************/ #ifndef COMMAND_STORAGE_LIST_H #define COMMAND_STORAGE_LIST_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // List files in the repository FN_EXTERN void cmdStorageList(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/repo/rm.c0000664000175100017510000000346315265101745017344 0ustar00runnerrunner/*********************************************************************************************************************************** Repository Remove Command ***********************************************************************************************************************************/ #include #include "command/repo/common.h" #include "command/repo/rm.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdStorageRemove(void) { FUNCTION_LOG_VOID(logLevelDebug); // Get path const String *path = NULL; if (strLstSize(cfgCommandParam()) == 1) path = strLstGet(cfgCommandParam(), 0); else if (strLstSize(cfgCommandParam()) > 1) THROW(ParamInvalidError, "only one path may be specified"); MEM_CONTEXT_TEMP_BEGIN() { // Is path valid for repo? if (path != NULL) path = repoPathIsValid(path); // Check if this is a file const StorageInfo info = storageInfoP(storageRepo(), path, .ignoreMissing = true); if (info.exists && info.type == storageTypeFile) { storageRemoveP(storageRepoWrite(), path); } // Else try to remove a path else { const bool recurse = cfgOptionBool(cfgOptRecurse); if (!recurse && !strLstEmpty(storageListP(storageRepo(), path))) THROW(OptionInvalidError, CFGOPT_RECURSE " option must be used to delete non-empty path"); storagePathRemoveP(storageRepoWrite(), path, .recurse = recurse); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/repo/rm.h0000664000175100017510000000132415265101745017343 0ustar00runnerrunner/*********************************************************************************************************************************** Repository Remove Command ***********************************************************************************************************************************/ #ifndef COMMAND_STORAGE_REMOVE_H #define COMMAND_STORAGE_REMOVE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Remove paths/files from the repository FN_EXTERN void cmdStorageRemove(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/0000775000175100017510000000000015265101745017272 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/blockChecksum.c0000664000175100017510000001555015265101745022221 0ustar00runnerrunner/*********************************************************************************************************************************** Block Hash List ***********************************************************************************************************************************/ #include #include "command/restore/blockChecksum.h" #include "common/crypto/common.h" #include "common/crypto/xxhash.h" #include "common/debug.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct BlockChecksum { size_t blockSize; // Block size for checksums size_t checksumSize; // Checksum size size_t blockCurrent; // Size of current block IoFilter *checksum; // Checksum of current block List *list; // List of checksums } BlockChecksum; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_BLOCK_CHECKSUM_TYPE \ BlockChecksum * #define FUNCTION_LOG_BLOCK_CHECKSUM_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "BlockChecksum", buffer, bufferSize) /*********************************************************************************************************************************** Generate block checksum list ***********************************************************************************************************************************/ static void blockChecksumProcess(THIS_VOID, const Buffer *const input) { THIS(BlockChecksum); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BLOCK_CHECKSUM, this); FUNCTION_LOG_PARAM(BUFFER, input); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(input != NULL); size_t inputOffset = 0; // Loop until input is consumed while (inputOffset != bufUsed(input)) { // Create checksum object if needed if (this->checksum == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->checksum = xxHashNew(this->checksumSize); this->blockCurrent = 0; } MEM_CONTEXT_OBJ_END(); } // Calculate how much data to checksum and perform checksum const size_t blockRemains = this->blockSize - this->blockCurrent; const size_t inputRemains = bufUsed(input) - inputOffset; const size_t blockChecksumSize = blockRemains < inputRemains ? blockRemains : inputRemains; ioFilterProcessIn(this->checksum, BUF(bufPtrConst(input) + inputOffset, blockChecksumSize)); // Update amount of data checksummed inputOffset += blockChecksumSize; this->blockCurrent += blockChecksumSize; // If the block size has been reached then output the checksum if (this->blockCurrent == this->blockSize) { MEM_CONTEXT_TEMP_BEGIN() { lstAdd(this->list, bufPtrConst(pckReadBinP(pckReadNew(ioFilterResult(this->checksum))))); ioFilterFree(this->checksum); this->checksum = NULL; } MEM_CONTEXT_TEMP_END(); } } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get a binary representation of the checksum list ***********************************************************************************************************************************/ static Pack * blockChecksumResult(THIS_VOID) { THIS(BlockChecksum); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BLOCK_CHECKSUM, this); FUNCTION_LOG_END(); ASSERT(this != NULL); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); // If there is a remainder in the checksum if (this->checksum) lstAdd(this->list, bufPtrConst(pckReadBinP(pckReadNew(ioFilterResult(this->checksum))))); pckWriteBinP(packWrite, BUF(lstGet(this->list, 0), lstSize(this->list) * this->checksumSize)); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * blockChecksumNew(const size_t blockSize, const size_t checksumSize) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SIZE, blockSize); FUNCTION_LOG_PARAM(SIZE, checksumSize); FUNCTION_LOG_END(); ASSERT(blockSize != 0); ASSERT(checksumSize != 0); // Allocate memory to hold process state OBJ_NEW_BEGIN(BlockChecksum, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (BlockChecksum) { .blockSize = blockSize, .checksumSize = checksumSize, .list = lstNewP(checksumSize), }; } OBJ_NEW_END(); // Create param list Pack *paramList; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteU64P(packWrite, blockSize); pckWriteU64P(packWrite, checksumSize); pckWriteEndP(packWrite); paramList = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP(BLOCK_CHECKSUM_FILTER_TYPE, this, paramList, .in = blockChecksumProcess, .result = blockChecksumResult)); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * blockChecksumNewPack(const Pack *const paramList) { IoFilter *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackRead *const paramListPack = pckReadNew(paramList); const size_t blockSize = (size_t)pckReadU64P(paramListPack); const size_t checksumSize = (size_t)pckReadU64P(paramListPack); result = ioFilterMove(blockChecksumNew(blockSize, checksumSize), memContextPrior()); } MEM_CONTEXT_TEMP_END(); return result; } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/blockChecksum.h0000664000175100017510000000257015265101745022224 0ustar00runnerrunner/*********************************************************************************************************************************** Block Hash List Build a list of hashes based on a block size. This is used to compare the contents of a file to block map to determine what needs to be updated. ***********************************************************************************************************************************/ #ifndef COMMAND_RESTORE_BLOCK_CHECKSUM_H #define COMMAND_RESTORE_BLOCK_CHECKSUM_H #include "common/io/filter/filter.h" #include "common/type/stringId.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define BLOCK_CHECKSUM_FILTER_TYPE STRID5("blk-chk", 0x2d03dad820) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *blockChecksumNew(size_t blockSize, size_t checksumSize); FN_EXTERN IoFilter *blockChecksumNewPack(const Pack *paramList); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/blockDelta.c0000664000175100017510000003346115265101745021511 0ustar00runnerrunner/*********************************************************************************************************************************** Block Restore ***********************************************************************************************************************************/ #include #include "command/backup/blockIncr.h" #include "command/restore/blockDelta.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/io/limitRead.h" #include "common/log.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct BlockDeltaSuperBlock { uint64_t superBlockSize; // Super block size uint64_t size; // Stored size of superblock (with compression, etc.) List *blockList; // Block list } BlockDeltaSuperBlock; typedef struct BlockDeltaBlock { uint64_t no; // Block number in the super block uint64_t offset; // Offset into original file uint8_t checksum[XX_HASH_SIZE_MAX]; // Checksum of the block } BlockDeltaBlock; struct BlockDelta { BlockDeltaPub pub; // Publicly accessible variables size_t blockSize; // Block size size_t checksumSize; // Checksum size const CipherSpec *cipherSpecBackup; // Cipher spec CompressType compressType; // Compress type const BlockDeltaSuperBlock *superBlockData; // Current super block data unsigned int superBlockIdx; // Current super block index IoRead *limitRead; // Limit read for current super block const BlockDeltaBlock *blockData; // Current block data unsigned int blockIdx; // Current block index unsigned int blockTotal; // Block total for super block unsigned int blockFindIdx; // Index of the block to find in the super block BlockDeltaWrite write; // Block/offset to be returned for write }; /**********************************************************************************************************************************/ typedef struct BlockDeltaReference { unsigned int reference; // Reference List *blockList; // List of blocks in the block map for the reference } BlockDeltaReference; FN_EXTERN BlockDelta * blockDeltaNew( const BlockMap *const blockMap, const size_t blockSize, const size_t checksumSize, const Buffer *const blockChecksum, const CipherSpec *const cipherSpecBackup, const CompressType compressType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BLOCK_MAP, blockMap); FUNCTION_TEST_PARAM(SIZE, blockSize); FUNCTION_TEST_PARAM(SIZE, checksumSize); FUNCTION_TEST_PARAM(BUFFER, blockChecksum); FUNCTION_TEST_PARAM(CIPHER_SPEC, cipherSpecBackup); FUNCTION_TEST_PARAM(ENUM, compressType); FUNCTION_TEST_END(); ASSERT(blockMap != NULL); ASSERT(blockSize > 0); ASSERT(cipherSpecBackup != NULL); OBJ_NEW_BEGIN(BlockDelta, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (BlockDelta) { .pub = { .readList = lstNewP(sizeof(BlockDeltaRead)), }, .blockSize = blockSize, .checksumSize = checksumSize, .cipherSpecBackup = cipherSpecDupP(cipherSpecBackup), .compressType = compressType, .write = { .block = bufNew(blockSize), } }; MEM_CONTEXT_TEMP_BEGIN() { // Build list of references and for each reference the list of blocks for that reference const unsigned int blockChecksumSize = blockChecksum == NULL ? 0 : (unsigned int)(bufUsed(blockChecksum) / this->checksumSize); List *const referenceList = lstNewP(sizeof(BlockDeltaReference), .comparator = lstComparatorUInt); for (unsigned int blockMapIdx = 0; blockMapIdx < blockMapSize(blockMap); blockMapIdx++) { const BlockMapItem *const blockMapItem = blockMapGet(blockMap, blockMapIdx); // The block must be updated if it is beyond the blocks that exist in the block checksum list or when the checksum // stored in the repository is different from the block checksum list if (blockMapIdx >= blockChecksumSize || !bufEq( BUF(blockMapItem->checksum, this->checksumSize), BUF(bufPtrConst(blockChecksum) + blockMapIdx * this->checksumSize, this->checksumSize))) { const unsigned int reference = blockMapItem->reference; BlockDeltaReference *const referenceData = lstFind(referenceList, &reference); // If the reference has not been added if (referenceData == NULL) { const BlockDeltaReference *const referenceData = lstAdd( referenceList, &(BlockDeltaReference){.reference = reference, .blockList = lstNewP(sizeof(unsigned int))}); lstAdd(referenceData->blockList, &blockMapIdx); } // Else add the new block else lstAdd(referenceData->blockList, &blockMapIdx); } } // Sort the reference list descending. This is an arbitrary choice as the order does not matter. lstSort(referenceList, sortOrderDesc); // Build delta for (unsigned int referenceIdx = 0; referenceIdx < lstSize(referenceList); referenceIdx++) { const BlockDeltaReference *const referenceData = (const BlockDeltaReference *)lstGet(referenceList, referenceIdx); BlockDeltaRead *blockDeltaRead = NULL; const BlockDeltaSuperBlock *blockDeltaSuperBlock = NULL; const BlockMapItem *blockMapItemPrior = NULL; for (unsigned int blockIdx = 0; blockIdx < lstSize(referenceData->blockList); blockIdx++) { const unsigned int blockMapIdx = *(unsigned int *)lstGet(referenceData->blockList, blockIdx); const BlockMapItem *const blockMapItem = blockMapGet(blockMap, blockMapIdx); // Add read when it has changed if (blockMapItemPrior == NULL || (blockMapItemPrior->offset != blockMapItem->offset && blockMapItemPrior->offset + blockMapItemPrior->size != blockMapItem->offset)) { MEM_CONTEXT_OBJ_BEGIN(this->pub.readList) { const BlockDeltaRead blockDeltaReadNew = { .reference = blockMapItem->reference, .bundleId = blockMapItem->bundleId, .offset = blockMapItem->offset, .superBlockList = lstNewP(sizeof(BlockDeltaSuperBlock)), }; blockDeltaRead = lstAdd(this->pub.readList, &blockDeltaReadNew); } MEM_CONTEXT_OBJ_END(); } // Add super block when it has changed if (blockMapItemPrior == NULL || blockMapItemPrior->offset != blockMapItem->offset) { MEM_CONTEXT_OBJ_BEGIN(blockDeltaRead->superBlockList) { const BlockDeltaSuperBlock blockDeltaSuperBlockNew = { .superBlockSize = blockMapItem->superBlockSize, .size = blockMapItem->size, .blockList = lstNewP(sizeof(BlockDeltaBlock)), }; blockDeltaSuperBlock = lstAdd(blockDeltaRead->superBlockList, &blockDeltaSuperBlockNew); blockDeltaRead->size += blockMapItem->size; } MEM_CONTEXT_OBJ_END(); } // Add block BlockDeltaBlock blockDeltaBlockNew = { .no = blockMapItem->block, .offset = blockMapIdx * blockSize, }; memcpy(blockDeltaBlockNew.checksum, blockMapItem->checksum, SIZE_OF_STRUCT_MEMBER(BlockDeltaBlock, checksum)); lstAdd(blockDeltaSuperBlock->blockList, &blockDeltaBlockNew); // Set prior item for comparison on the next loop blockMapItemPrior = blockMapItem; } } } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(BLOCK_DELTA, this); } /**********************************************************************************************************************************/ FN_EXTERN const BlockDeltaWrite * blockDeltaNext(BlockDelta *const this, const BlockDeltaRead *const readDelta, IoRead *const readIo) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BLOCK_DELTA, this); FUNCTION_TEST_PARAM_P(VOID, readDelta); FUNCTION_TEST_PARAM(IO_READ, readIo); FUNCTION_TEST_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); ASSERT(readDelta != NULL); ASSERT(readIo != NULL); BlockDeltaWrite *result = NULL; // Iterate super blocks while (this->superBlockIdx < lstSize(readDelta->superBlockList)) { // If the super block read has not begun yet if (this->superBlockData == NULL) { // Free prior limit read and create limit read for current super block ioReadFree(this->limitRead); this->superBlockData = lstGet(readDelta->superBlockList, this->superBlockIdx); MEM_CONTEXT_OBJ_BEGIN(this) { this->limitRead = ioLimitReadNew(readIo, this->superBlockData->size); } MEM_CONTEXT_OBJ_END(); if (cipherSpecType(this->cipherSpecBackup) != cipherTypeNone) { ioFilterGroupAdd( ioReadFilterGroup(this->limitRead), cipherBlockNewP( cipherModeDecrypt, this->cipherSpecBackup, .header = cipherBlockHeaderNone)); } if (this->compressType != compressTypeNone) ioFilterGroupAdd(ioReadFilterGroup(this->limitRead), decompressFilterP(this->compressType, .raw = true)); ioReadOpen(this->limitRead); // Set block info this->blockIdx = 0; this->blockFindIdx = 0; this->blockTotal = (unsigned int)(this->superBlockData->superBlockSize / this->blockSize) + (this->superBlockData->superBlockSize % this->blockSize == 0 ? 0 : 1); this->blockData = lstGet(this->superBlockData->blockList, this->blockFindIdx); } // Find required blocks in the super block while (this->blockIdx < this->blockTotal) { // Clear buffer and read block bufUsedZero(this->write.block); bufLimitClear(this->write.block); ioRead(this->limitRead, this->write.block); // If the block matches the block we are expecting if (this->blockIdx == this->blockData->no) { ASSERT(result == NULL); this->write.offset = this->blockData->offset; result = &this->write; this->blockFindIdx++; // Get the next block if there are any more to read if (this->blockFindIdx < lstSize(this->superBlockData->blockList)) this->blockData = lstGet(this->superBlockData->blockList, this->blockFindIdx); } // Increment the block to read in the super block this->blockIdx++; // Break if there is a result if (result != NULL) break; } // Break if there is a result if (result != NULL) break; // Check that no bytes remain to be written. It is possible that some bytes remain in the super block, however, since we may // have gotten all the bytes we needed but just missed reading something important, e.g. an end of file marker. If we do not // read the remaining bytes then the next read will start too early. ioReadFlushP(this->limitRead, .errorOnBytes = true); this->superBlockData = NULL; this->superBlockIdx++; } // If no result then the super blocks have been read. Reset for the next read. if (result == NULL) { ASSERT(this->superBlockIdx == lstSize(readDelta->superBlockList)); ASSERT(this->blockIdx == this->blockTotal); this->superBlockData = NULL; this->superBlockIdx = 0; } FUNCTION_TEST_RETURN_TYPE_P(BlockDeltaWrite, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/blockDelta.h0000664000175100017510000001071515265101745021513 0ustar00runnerrunner/*********************************************************************************************************************************** Block Restore Calculate and return the blocks required to restore a file using an optional block checksum list. The block checksum list is optional because the file to restore may not exist so all the blocks will need to be restored. ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_BLOCKDELTA_H #define COMMAND_BACKUP_BLOCKDELTA_H #include "command/backup/blockMap.h" #include "common/compress/helper.h" #include "common/crypto/spec.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct BlockDelta BlockDelta; // Reads that must be performed in order to extract blocks typedef struct BlockDeltaRead { unsigned int reference; // Reference to read from uint64_t bundleId; // Bundle to read from uint64_t offset; // Offset to begin read from uint64_t size; // Size of the read List *superBlockList; // Super block list } BlockDeltaRead; // Writes that need to be performed to restore the file typedef struct BlockDeltaWrite { uint64_t offset; // Offset for the write Buffer *block; // Block to write } BlockDeltaWrite; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN BlockDelta *blockDeltaNew( const BlockMap *blockMap, size_t blockSize, size_t checksumSize, const Buffer *blockChecksum, const CipherSpec *cipherSpecBackup, const CompressType compressType); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Get the next write for the restore FN_EXTERN const BlockDeltaWrite *blockDeltaNext(BlockDelta *this, const BlockDeltaRead *readDelta, IoRead *readIo); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct BlockDeltaPub { List *readList; // Read list } BlockDeltaPub; // Get read info FN_INLINE_ALWAYS const BlockDeltaRead * blockDeltaReadGet(const BlockDelta *const this, const unsigned int readIdx) { return (BlockDeltaRead *)lstGet(THIS_PUB(BlockDelta)->readList, readIdx); } // Read list size FN_INLINE_ALWAYS unsigned int blockDeltaReadSize(const BlockDelta *const this) { return lstSize(THIS_PUB(BlockDelta)->readList); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void blockDeltaFree(BlockMap *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_BLOCK_DELTA_TYPE \ BlockDelta * #define FUNCTION_LOG_BLOCK_DELTA_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "BlockDelta", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/clean.c.inc0000664000175100017510000010067615265101745021302 0ustar00runnerrunner/*********************************************************************************************************************************** Check ownership of items in the manifest ***********************************************************************************************************************************/ // Helper to determine what the user/group of a path/file/link should be static const String * restoreManifestOwnerReplace(const String *const owner, const String *const ownerDefaultRoot) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, owner); FUNCTION_TEST_PARAM(STRING, ownerDefaultRoot); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, userRoot() ? (owner == NULL ? ownerDefaultRoot : owner) : NULL); } // Helper to get list of owners from a file/link/path list #define RESTORE_MANIFEST_OWNER_GET(type, deref) \ for (unsigned int itemIdx = 0; itemIdx < manifest##type##Total(manifest); itemIdx++) \ { \ const Manifest##type item = deref manifest##type(manifest, itemIdx); \ \ if (item.user == NULL) \ userNull = true; \ else \ strLstAddIfMissing(userList, item.user); \ \ if (item.group == NULL) \ groupNull = true; \ else \ strLstAddIfMissing(groupList, item.group); \ } // Helper to warn when an owner is missing and must be remapped #define RESTORE_MANIFEST_OWNER_WARN(type) \ do \ { \ if (type##Null) \ LOG_WARN("unknown " #type " in backup manifest mapped to current " #type); \ \ for (unsigned int ownerIdx = 0; ownerIdx < strLstSize(type##List); ownerIdx++) \ { \ const String *const owner = strLstGet(type##List, ownerIdx); \ \ if (type##Name() == NULL || !strEq(type##Name(), owner)) \ LOG_WARN_FMT("unknown " #type " '%s' in backup manifest mapped to current " #type, strZ(owner)); \ } \ } \ while (0) static void restoreManifestOwner(const Manifest *const manifest, const String **const rootReplaceUser, const String **const rootReplaceGroup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM_P(VOID, rootReplaceUser); FUNCTION_LOG_PARAM_P(VOID, rootReplaceGroup); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(manifest != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Build a list of users and groups in the manifest // ------------------------------------------------------------------------------------------------------------------------- bool userNull = false; StringList *const userList = strLstNew(); bool groupNull = false; StringList *const groupList = strLstNew(); RESTORE_MANIFEST_OWNER_GET(File, ); RESTORE_MANIFEST_OWNER_GET(Link, *(const ManifestLink *)); RESTORE_MANIFEST_OWNER_GET(Path, *(const ManifestPath *)); // Update users and groups in the manifest (this can only be done as root) // ------------------------------------------------------------------------------------------------------------------------- if (userRoot()) { // Get user/group info from data directory to use for invalid user/groups StorageInfo pathInfo = storageInfoP(storagePg(), manifestTargetBase(manifest)->path, .ignoreMissing = true); // If user/group is null then set it to root if (pathInfo.user == NULL) // {vm_covered} pathInfo.user = userName(); // {vm_covered} if (pathInfo.group == NULL) // {vm_covered} pathInfo.group = groupName(); // {vm_covered} if (userNull || groupNull) { if (userNull) LOG_WARN_FMT("unknown user in backup manifest mapped to '%s'", strZ(pathInfo.user)); if (groupNull) LOG_WARN_FMT("unknown group in backup manifest mapped to '%s'", strZ(pathInfo.group)); MEM_CONTEXT_PRIOR_BEGIN() { *rootReplaceUser = strDup(pathInfo.user); *rootReplaceGroup = strDup(pathInfo.group); } MEM_CONTEXT_PRIOR_END(); } } // Else set owners to NULL. This means we won't make any attempt to update ownership and will just leave it as written by // the current user/group. If there are existing files that are not owned by the current user/group then we will attempt to // update them, which will generally cause an error, though some systems allow updates to the group ownership. // ------------------------------------------------------------------------------------------------------------------------- else { RESTORE_MANIFEST_OWNER_WARN(user); RESTORE_MANIFEST_OWNER_WARN(group); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Clean the data directory of any paths/files/links that are not in the manifest and create missing links/paths ***********************************************************************************************************************************/ typedef struct RestoreCleanCallbackData { const Manifest *manifest; // Manifest to compare against const ManifestTarget *target; // Current target being compared const String *targetName; // Name to use when finding files/paths/links const String *targetPath; // Path of target currently being compared bool basePath; // Is this the base path? bool exists; // Does the target path exist? bool delta; // Is this a delta restore? StringList *fileIgnore; // Files to ignore during clean const String *rootReplaceUser; // User to replace invalid users when root const String *rootReplaceGroup; // Group to replace invalid group when root } RestoreCleanCallbackData; // Helper to update ownership on a file/link/path static void restoreCleanOwnership( const String *const pgPath, const String *manifestUserName, const String *const rootReplaceUser, const String *manifestGroupName, const String *const rootReplaceGroup, const uid_t actualUserId, const gid_t actualGroupId, const bool new) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, pgPath); FUNCTION_TEST_PARAM(STRING, manifestUserName); FUNCTION_TEST_PARAM(STRING, manifestGroupName); FUNCTION_TEST_PARAM(UINT, actualUserId); FUNCTION_TEST_PARAM(UINT, actualGroupId); FUNCTION_TEST_PARAM(BOOL, new); FUNCTION_TEST_END(); ASSERT(pgPath != NULL); // Get the expected user id uid_t expectedUserId = userId(); manifestUserName = restoreManifestOwnerReplace(manifestUserName, rootReplaceUser); if (manifestUserName != NULL) { const uid_t manifestUserId = userIdFromName(manifestUserName); if (manifestUserId != (uid_t)-1) expectedUserId = manifestUserId; } // Get the expected group id gid_t expectedGroupId = groupId(); manifestGroupName = restoreManifestOwnerReplace(manifestGroupName, rootReplaceGroup); if (manifestGroupName != NULL) { const uid_t manifestGroupId = groupIdFromName(manifestGroupName); if (manifestGroupId != (uid_t)-1) expectedGroupId = manifestGroupId; } // Update ownership if not as expected if (actualUserId != expectedUserId || actualGroupId != expectedGroupId) { // If this is a newly created file/link/path then there's no need to log updated permissions if (!new) LOG_DETAIL_FMT("update ownership for '%s'", strZ(pgPath)); THROW_ON_SYS_ERROR_FMT( lchown(strZ(pgPath), expectedUserId, expectedGroupId) == -1, FileOwnerError, "unable to set ownership for '%s'", strZ(pgPath)); } FUNCTION_TEST_RETURN_VOID(); } // Helper to update mode on a file/path static void restoreCleanMode(const String *const pgPath, const mode_t manifestMode, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, pgPath); FUNCTION_TEST_PARAM(MODE, manifestMode); FUNCTION_TEST_PARAM(INFO, info); FUNCTION_TEST_END(); ASSERT(pgPath != NULL); ASSERT(info != NULL); // Update mode if not as expected if (manifestMode != info->mode) { LOG_DETAIL_FMT("update mode for '%s' to %04o", strZ(pgPath), manifestMode); THROW_ON_SYS_ERROR_FMT( chmod(strZ(pgPath), manifestMode) == -1, FileModeError, "unable to set mode for '%s'", strZ(pgPath)); } FUNCTION_TEST_RETURN_VOID(); } // Recurse paths static void restoreCleanBuildRecurse(StorageIterator *const storageItr, const RestoreCleanCallbackData *const cleanData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_ITERATOR, storageItr); FUNCTION_TEST_PARAM_P(VOID, cleanData); FUNCTION_TEST_END(); ASSERT(storageItr != NULL); ASSERT(cleanData != NULL); MEM_CONTEXT_TEMP_RESET_BEGIN() { while (storageItrMore(storageItr)) { const StorageInfo info = storageItrNext(storageItr); // Don't include backup.manifest or recovery.conf (when preserved) in the comparison or empty directory check if (cleanData->basePath && info.type == storageTypeFile && strLstExists(cleanData->fileIgnore, info.name)) continue; // If this is not a delta then error because the directory is expected to be empty. Ignore the . path. if (!cleanData->delta) { THROW_FMT( PathNotEmptyError, "unable to restore to path '%s' because it contains files\n" "HINT: try using --delta if this is what you intended.", strZ(cleanData->targetPath)); } // Construct the name used to find this file/link/path in the manifest const String *const manifestName = strNewFmt("%s/%s", strZ(cleanData->targetName), strZ(info.name)); // Construct the path of this file/link/path in the PostgreSQL data directory const String *const pgPath = strNewFmt("%s/%s", strZ(cleanData->targetPath), strZ(info.name)); switch (info.type) { case storageTypeFile: { if (manifestFileExists(cleanData->manifest, manifestName) && manifestLinkFindDefault(cleanData->manifest, manifestName, NULL) == NULL) { const ManifestFile manifestFile = manifestFileFind(cleanData->manifest, manifestName); restoreCleanOwnership( pgPath, manifestFile.user, cleanData->rootReplaceUser, manifestFile.group, cleanData->rootReplaceGroup, info.userId, info.groupId, false); restoreCleanMode(pgPath, manifestFile.mode, &info); } else { LOG_DETAIL_FMT("remove invalid file '%s'", strZ(pgPath)); storageRemoveP(storageLocalWrite(), pgPath, .errorOnMissing = true); } break; } case storageTypeLink: { const ManifestLink *const manifestLink = manifestLinkFindDefault(cleanData->manifest, manifestName, NULL); if (manifestLink != NULL) { if (!strEq(manifestLink->destination, info.linkDestination)) { LOG_DETAIL_FMT("remove link '%s' because destination changed", strZ(pgPath)); storageRemoveP(storageLocalWrite(), pgPath, .errorOnMissing = true); } else { restoreCleanOwnership( pgPath, manifestLink->user, cleanData->rootReplaceUser, manifestLink->group, cleanData->rootReplaceGroup, info.userId, info.groupId, false); } } else { LOG_DETAIL_FMT("remove invalid link '%s'", strZ(pgPath)); storageRemoveP(storageLocalWrite(), pgPath, .errorOnMissing = true); } break; } case storageTypePath: { const ManifestPath *const manifestPath = manifestPathFindDefault(cleanData->manifest, manifestName, NULL); if (manifestPath != NULL && manifestLinkFindDefault(cleanData->manifest, manifestName, NULL) == NULL) { // Check ownership/permissions restoreCleanOwnership( pgPath, manifestPath->user, cleanData->rootReplaceUser, manifestPath->group, cleanData->rootReplaceGroup, info.userId, info.groupId, false); restoreCleanMode(pgPath, manifestPath->mode, &info); // Recurse into the path RestoreCleanCallbackData cleanDataSub = *cleanData; cleanDataSub.targetName = strNewFmt("%s/%s", strZ(cleanData->targetName), strZ(info.name)); cleanDataSub.targetPath = strNewFmt("%s/%s", strZ(cleanData->targetPath), strZ(info.name)); cleanDataSub.basePath = false; restoreCleanBuildRecurse( storageNewItrP( storageLocalWrite(), cleanDataSub.targetPath, .errorOnMissing = true, .sortOrder = sortOrderAsc), &cleanDataSub); } else { LOG_DETAIL_FMT("remove invalid path '%s'", strZ(pgPath)); storagePathRemoveP(storageLocalWrite(), pgPath, .errorOnMissing = true, .recurse = true); } break; } // Special file types cannot exist in the manifest so just delete them case storageTypeSpecial: LOG_DETAIL_FMT("remove special file '%s'", strZ(pgPath)); storageRemoveP(storageLocalWrite(), pgPath, .errorOnMissing = true); break; } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } static void restoreCleanBuild(const Manifest *const manifest, const String *const rootReplaceUser, const String *const rootReplaceGroup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(STRING, rootReplaceUser); FUNCTION_LOG_PARAM(STRING, rootReplaceGroup); FUNCTION_LOG_END(); ASSERT(manifest != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Is this a delta restore? const bool delta = cfgOptionBool(cfgOptDelta) || cfgOptionBool(cfgOptForce); // Allocate data for each target RestoreCleanCallbackData *const cleanDataList = memNew(sizeof(RestoreCleanCallbackData) * manifestTargetTotal(manifest)); // Step 1: Check permissions and validity (is the directory empty without delta?) if the target directory exists // ------------------------------------------------------------------------------------------------------------------------- StringList *const pathChecked = strLstNew(); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { RestoreCleanCallbackData *const cleanData = &cleanDataList[targetIdx]; *cleanData = (RestoreCleanCallbackData) { .manifest = manifest, .target = manifestTarget(manifest, targetIdx), .delta = delta, .fileIgnore = strLstNew(), .rootReplaceUser = rootReplaceUser, .rootReplaceGroup = rootReplaceGroup, }; cleanData->targetName = cleanData->target->name; cleanData->targetPath = manifestTargetPath(manifest, cleanData->target); cleanData->basePath = strEq(cleanData->targetName, MANIFEST_TARGET_PGDATA_STR); // Ignore backup.manifest while cleaning since it may exist from an prior incomplete restore strLstAdd(cleanData->fileIgnore, BACKUP_MANIFEST_FILE_STR); // Also ignore recovery files when recovery type = preserve if (cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_PRESERVE) { // If recovery GUCs then three files must be preserved if (manifestData(manifest)->pgVersion >= PG_VERSION_RECOVERY_GUC) { strLstAdd(cleanData->fileIgnore, PG_FILE_POSTGRESQLAUTOCONF_STR); strLstAdd(cleanData->fileIgnore, PG_FILE_RECOVERYSIGNAL_STR); strLstAdd(cleanData->fileIgnore, PG_FILE_STANDBYSIGNAL_STR); } // Else just recovery.conf else strLstAdd(cleanData->fileIgnore, PG_FILE_RECOVERYCONF_STR); } // If this is a tablespace append the tablespace identifier if (cleanData->target->type == manifestTargetTypeLink && cleanData->target->tablespaceId != 0) { const String *const tablespaceId = pgTablespaceId( manifestData(manifest)->pgVersion, manifestData(manifest)->pgCatalogVersion); cleanData->targetName = strNewFmt("%s/%s", strZ(cleanData->targetName), strZ(tablespaceId)); cleanData->targetPath = strNewFmt("%s/%s", strZ(cleanData->targetPath), strZ(tablespaceId)); } strLstSort(cleanData->fileIgnore, sortOrderAsc); // Check that the path exists. If not, there's no need to do any cleaning and we'll attempt to create it later. Don't // log check for the same path twice. There can be multiple links to files in the same path, but logging it more than // once makes the logs noisy and looks like a bug. if (!strLstExists(pathChecked, cleanData->targetPath)) LOG_DETAIL_FMT("check '%s' exists", strZ(cleanData->targetPath)); const StorageInfo info = storageInfoP(storageLocal(), cleanData->targetPath, .ignoreMissing = true, .followLink = true); strLstAdd(pathChecked, cleanData->targetPath); if (info.exists) { // Make sure our uid will be able to write to this directory if (!userRoot() && userId() != info.userId) { THROW_FMT( PathOpenError, "unable to restore to path '%s' not owned by current user", strZ(cleanData->targetPath)); } if ((info.mode & 0700) != 0700) { THROW_FMT( PathOpenError, "unable to restore to path '%s' without rwx permissions", strZ(cleanData->targetPath)); } // If not a delta restore then check that the directories are empty, or if a file link, that the file doesn't exist if (!cleanData->delta) { if (cleanData->target->file == NULL) { restoreCleanBuildRecurse( storageNewItrP(storageLocal(), cleanData->targetPath, .errorOnMissing = true), cleanData); } else { const String *const file = strNewFmt("%s/%s", strZ(cleanData->targetPath), strZ(cleanData->target->file)); if (storageExistsP(storageLocal(), file)) { THROW_FMT( FileExistsError, "unable to restore file '%s' because it already exists\n" "HINT: try using --delta if this is what you intended.", strZ(file)); } } // Now that we know there are no files in this target enable delta for processing in step 2 cleanData->delta = true; } // The target directory exists and is valid and will need to be cleaned cleanData->exists = true; } } // Skip the tablespace_map file when present so PostgreSQL does not rewrite links in pg_tblspc. The tablespace links will be // created after paths are cleaned. if (manifestFileExists(manifest, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_FILE_TABLESPACEMAP))) { LOG_DETAIL_FMT("skip '" PG_FILE_TABLESPACEMAP "' -- tablespace links will be created based on mappings"); manifestFileRemove(manifest, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_FILE_TABLESPACEMAP)); } // Skip postgresql.auto.conf if preserve is set and the PostgreSQL version supports recovery GUCs if (manifestData(manifest)->pgVersion >= PG_VERSION_RECOVERY_GUC && cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_PRESERVE && manifestFileExists(manifest, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_FILE_POSTGRESQLAUTOCONF))) { LOG_DETAIL_FMT("skip '" PG_FILE_POSTGRESQLAUTOCONF "' -- recovery type is preserve"); manifestFileRemove(manifest, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_FILE_POSTGRESQLAUTOCONF)); } // Step 2: Clean target directories // ------------------------------------------------------------------------------------------------------------------------- // Delete the pg_control file (if it exists) so the cluster cannot be started if restore does not complete. Sync the path so // the file does not return, zombie-like, in the case of a host crash. if (storageExistsP(storagePg(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL))) { LOG_DETAIL_FMT( "remove '" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "' so cluster will not start if restore does not complete"); storageRemoveP(storagePgWrite(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)); storagePathSyncP(storagePgWrite(), PG_PATH_GLOBAL_STR); } for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const RestoreCleanCallbackData *const cleanData = &cleanDataList[targetIdx]; // Only clean if the target exists if (cleanData->exists) { // Don't clean file links. It doesn't matter whether the file exists or not since we know it is in the manifest. if (cleanData->target->file == NULL) { // Only log when doing a delta restore because otherwise the targets should be empty. We'll still run the clean // to fix permissions/ownership on the target paths. if (delta) LOG_INFO_FMT("remove invalid files/links/paths from '%s'", strZ(cleanData->targetPath)); // Check target ownership/permissions const ManifestPath *const manifestPath = manifestPathFind(cleanData->manifest, cleanData->targetName); const StorageInfo info = storageInfoP(storageLocal(), cleanData->targetPath, .followLink = true); restoreCleanOwnership( cleanData->targetPath, manifestPath->user, rootReplaceUser, manifestPath->group, rootReplaceGroup, info.userId, info.groupId, false); restoreCleanMode(cleanData->targetPath, manifestPath->mode, &info); // Clean the target restoreCleanBuildRecurse( storageNewItrP( storageLocalWrite(), cleanData->targetPath, .errorOnMissing = true, .sortOrder = sortOrderAsc), cleanData); } } // If the target does not exist we'll attempt to create it else { const ManifestPath *path = NULL; // There is no path information for a file link so we'll need to use the data directory if (cleanData->target->file != NULL) { path = manifestPathFind(manifest, MANIFEST_TARGET_PGDATA_STR); } // Else grab the info for the path that matches the link name else path = manifestPathFind(manifest, cleanData->target->name); storagePathCreateP(storageLocalWrite(), cleanData->targetPath, .mode = path->mode); restoreCleanOwnership( cleanData->targetPath, path->user, rootReplaceUser, path->group, rootReplaceGroup, userId(), groupId(), true); } } // Step 3: Create missing paths and path links // ------------------------------------------------------------------------------------------------------------------------- for (unsigned int pathIdx = 0; pathIdx < manifestPathTotal(manifest); pathIdx++) { const ManifestPath *const path = manifestPath(manifest, pathIdx); // Skip the pg_tblspc path because it only maps to the manifest. We should remove this in a future release but not much // can be done about it for now. if (strEq(path->name, MANIFEST_TARGET_PGTBLSPC_STR)) continue; // If this path has been mapped as a link then create a link. The path has already been created as part of target // creation (or it might have already existed). const ManifestLink *const link = manifestLinkFindDefault( manifest, strBeginsWith(path->name, MANIFEST_TARGET_PGTBLSPC_STR) ? strNewFmt(MANIFEST_TARGET_PGDATA "/%s", strZ(path->name)) : path->name, NULL); if (link != NULL) { const String *const pgPath = storagePathP(storagePg(), manifestPathPg(link->name)); const StorageInfo linkInfo = storageInfoP(storagePg(), pgPath, .ignoreMissing = true); // Create the link if it is missing. If it exists it should already have the correct ownership and destination. if (!linkInfo.exists) { LOG_DETAIL_FMT("create symlink '%s' to '%s'", strZ(pgPath), strZ(link->destination)); storageLinkCreateP(storagePgWrite(), link->destination, pgPath); restoreCleanOwnership( pgPath, link->user, rootReplaceUser, link->group, rootReplaceGroup, userId(), groupId(), true); } } // Create the path normally else { const String *const pgPath = storagePathP(storagePg(), manifestPathPg(path->name)); const StorageInfo pathInfo = storageInfoP(storagePg(), pgPath, .ignoreMissing = true); // Create the path if it is missing. If it exists it should already have the correct ownership and mode. if (!pathInfo.exists) { LOG_DETAIL_FMT("create path '%s'", strZ(pgPath)); storagePathCreateP(storagePgWrite(), pgPath, .mode = path->mode, .noParentCreate = true, .errorOnExists = true); restoreCleanOwnership( storagePathP(storagePg(), pgPath), path->user, rootReplaceUser, path->group, rootReplaceGroup, userId(), groupId(), true); } } } // Step 4: Create file links. These don't get created during path creation because they do not have a matching path entry. // ------------------------------------------------------------------------------------------------------------------------- for (unsigned int linkIdx = 0; linkIdx < manifestLinkTotal(manifest); linkIdx++) { const ManifestLink *const link = manifestLink(manifest, linkIdx); const String *const pgPath = storagePathP(storagePg(), manifestPathPg(link->name)); const StorageInfo linkInfo = storageInfoP(storagePg(), pgPath, .ignoreMissing = true); // Create the link if it is missing. If it exists it should already have the correct ownership and destination. if (!linkInfo.exists) { LOG_DETAIL_FMT("create symlink '%s' to '%s'", strZ(pgPath), strZ(link->destination)); storageLinkCreateP(storagePgWrite(), link->destination, pgPath); restoreCleanOwnership( pgPath, link->user, rootReplaceUser, link->group, rootReplaceGroup, userId(), groupId(), true); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/config.c.inc0000664000175100017510000004474415265101745021470 0ustar00runnerrunner/*********************************************************************************************************************************** Recovery constants ***********************************************************************************************************************************/ #define RESTORE_COMMAND "restore_command" STRING_STATIC(RESTORE_COMMAND_STR, RESTORE_COMMAND); #define RECOVERY_TARGET "recovery_target" #define RECOVERY_TARGET_LSN "recovery_target_lsn" #define RECOVERY_TARGET_NAME "recovery_target_name" #define RECOVERY_TARGET_TIME "recovery_target_time" #define RECOVERY_TARGET_XID "recovery_target_xid" #define RECOVERY_TARGET_ACTION "recovery_target_action" #define RECOVERY_TARGET_INCLUSIVE "recovery_target_inclusive" #define RECOVERY_TARGET_TIMELINE "recovery_target_timeline" #define RECOVERY_TARGET_TIMELINE_CURRENT "current" #define STANDBY_MODE "standby_mode" STRING_STATIC(STANDBY_MODE_STR, STANDBY_MODE); #define ARCHIVE_MODE "archive_mode" /*********************************************************************************************************************************** Generate the recovery file ***********************************************************************************************************************************/ // Helper to generate recovery options static KeyValue * restoreRecoveryOption(const unsigned int pgVersion) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_END(); KeyValue *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { result = kvNew(); StringList *recoveryOptionKey = strLstNew(); if (cfgOptionTest(cfgOptRecoveryOption)) { const KeyValue *const recoveryOption = cfgOptionKv(cfgOptRecoveryOption); recoveryOptionKey = strLstSort(strLstNewVarLst(kvKeyList(recoveryOption)), sortOrderAsc); for (unsigned int keyIdx = 0; keyIdx < strLstSize(recoveryOptionKey); keyIdx++) { // Get the key and value String *const key = strLstGet(recoveryOptionKey, keyIdx); const String *const value = varStr(kvGet(recoveryOption, VARSTR(key))); // Replace - in key with _. Since we use - users naturally will as well. strReplaceChr(key, '-', '_'); kvPut(result, VARSTR(key), VARSTR(value)); } strLstSort(recoveryOptionKey, sortOrderAsc); } // If archive-mode is not preserve if (cfgOptionSeq(cfgOptArchiveMode) != CFGOPTVAL_ARCHIVE_MODE_PRESERVE) { if (pgVersion < PG_VERSION_12) { THROW_FMT( OptionInvalidError, "option '" CFGOPT_ARCHIVE_MODE "' is not supported on " PG_NAME " < " PG_VERSION_12_Z "\n" "HINT: 'archive_mode' should be manually set to 'off' in postgresql.conf."); } // The only other valid option is off ASSERT(cfgOptionSeq(cfgOptArchiveMode) == CFGOPTVAL_ARCHIVE_MODE_OFF); // If archive-mode=off then set archive_mode=off kvPut(result, VARSTRDEF(ARCHIVE_MODE), VARSTRDEF(CFGOPTVAL_ARCHIVE_MODE_OFF_Z)); } // Write restore_command if (!strLstExists(recoveryOptionKey, RESTORE_COMMAND_STR)) { // Null out options that it does not make sense to pass from the restore command to archive-get. All of these have // reasonable defaults so there is no danger of an error -- they just might not be optimal. In any case, it seems better // than, for example, passing --process-max=32 to archive-get because it was specified for restore. KeyValue *const optionReplace = kvNew(); kvPut(optionReplace, VARSTRDEF(CFGOPT_EXEC_ID), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_JOB_RETRY), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_JOB_RETRY_INTERVAL), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_CONSOLE), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_FILE), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_STDERR), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_SUBPROCESS), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_TIMESTAMP), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_PROCESS_MAX), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_CMD), NULL); kvPut( result, VARSTRDEF(RESTORE_COMMAND), VARSTR( strNewFmt( "%s %s %%f \"%%p\"", strZ(cfgOptionStr(cfgOptCmd)), strZ(strLstJoin(cfgExecParam(cfgCmdArchiveGet, cfgCmdRoleMain, optionReplace, true, true), " "))))); } // If recovery type is immediate if (cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_IMMEDIATE) { kvPut(result, VARSTRDEF(RECOVERY_TARGET), VARSTRDEF(CFGOPTVAL_RESTORE_TYPE_IMMEDIATE_Z)); } // Else recovery type is standby else if (cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_STANDBY) { // Write standby_mode for PostgreSQL versions that support it if (pgVersion < PG_VERSION_RECOVERY_GUC) kvPut(result, VARSTR(STANDBY_MODE_STR), VARSTRDEF("on")); } // Else recovery type is not default so write target options else if (cfgOptionSeq(cfgOptType) != CFGOPTVAL_RESTORE_TYPE_DEFAULT) { // Write the recovery target kvPut( result, VARSTR(strNewFmt(RECOVERY_TARGET "_%s", strZ(cfgOptionDisplay(cfgOptType)))), VARSTR(cfgOptionStr(cfgOptTarget))); // Write recovery_target_inclusive if (cfgOptionTest(cfgOptTargetExclusive) && cfgOptionBool(cfgOptTargetExclusive)) kvPut(result, VARSTRDEF(RECOVERY_TARGET_INCLUSIVE), VARSTR(FALSE_STR)); } // Write recovery_target_action if (cfgOptionTest(cfgOptTargetAction)) { const StringId targetAction = cfgOptionStrId(cfgOptTargetAction); if (targetAction != CFGOPTVAL_TARGET_ACTION_PAUSE) { kvPut(result, VARSTRDEF(RECOVERY_TARGET_ACTION), VARSTR(strNewStrId(targetAction))); } } // Write recovery_target_timeline if set if (cfgOptionTest(cfgOptTargetTimeline)) { // Do not set current when PostgreSQL < 12 since this is the default and if current is explicitly set it acts as latest if (pgVersion >= PG_VERSION_12 || !strEqZ(cfgOptionStr(cfgOptTargetTimeline), RECOVERY_TARGET_TIMELINE_CURRENT)) kvPut(result, VARSTRDEF(RECOVERY_TARGET_TIMELINE), VARSTR(cfgOptionStr(cfgOptTargetTimeline))); } // Else explicitly set target timeline to "current" when type=immediate and PostgreSQL >= 12. We do this because // type=immediate means there won't be any actual attempt to change timelines, but if we leave the target timeline as the // default of "latest" then PostgreSQL might fail to restore because it can't reach the "latest" timeline in the repository // from this backup. // // This is really a PostgreSQL bug and will hopefully be addressed there, but we'll handle it here for older versions, at // least until they aren't really seen in the wild any longer. // // PostgreSQL < 12 defaults to "current" (but does not accept "current" as a parameter) so no need set it explicitly. else if (cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_IMMEDIATE && pgVersion >= PG_VERSION_12) kvPut(result, VARSTRDEF(RECOVERY_TARGET_TIMELINE), VARSTRDEF(RECOVERY_TARGET_TIMELINE_CURRENT)); // Move to prior context kvMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(KEY_VALUE, result); } // Helper to convert recovery options to text format static String * restoreRecoveryConf(const unsigned int pgVersion, const String *const restoreLabel) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(STRING, restoreLabel); FUNCTION_LOG_END(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { strCatFmt(result, "# Recovery settings generated by " PROJECT_NAME " restore on %s\n", strZ(restoreLabel)); // Output all recovery options const KeyValue *const optionKv = restoreRecoveryOption(pgVersion); const VariantList *const optionKeyList = kvKeyList(optionKv); for (unsigned int optionKeyIdx = 0; optionKeyIdx < varLstSize(optionKeyList); optionKeyIdx++) { const Variant *const optionKey = varLstGet(optionKeyList, optionKeyIdx); strCatFmt(result, "%s = '%s'\n", strZ(varStr(optionKey)), strZ(varStr(kvGet(optionKv, optionKey)))); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } // Helper to write recovery options into recovery.conf static void restoreRecoveryWriteConf(const StorageInfo *const fileInfo, const unsigned int pgVersion, const String *const restoreLabel) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_INFO, fileInfo); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(STRING, restoreLabel); FUNCTION_LOG_END(); // Only write recovery.conf if recovery type != none if (cfgOptionSeq(cfgOptType) != CFGOPTVAL_RESTORE_TYPE_NONE) { MEM_CONTEXT_TEMP_BEGIN() { LOG_INFO_FMT("write %s", strZ(storagePathP(storagePg(), PG_FILE_RECOVERYCONF_STR))); // Write recovery.conf storagePutP( storageNewWriteP( storagePgWrite(), PG_FILE_RECOVERYCONF_STR, .noCreatePath = true, .modeFile = fileInfo->mode, .noAtomic = true, .noSyncPath = true, .user = fileInfo->user, .group = fileInfo->group), BUFSTR(restoreRecoveryConf(pgVersion, restoreLabel))); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } // Helper to write recovery options into postgresql.auto.conf static void restoreRecoveryWriteAutoConf(const StorageInfo *const fileInfo, const unsigned int pgVersion, const String *const restoreLabel) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_INFO, fileInfo); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(STRING, restoreLabel); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { String *const content = strNew(); // Load postgresql.auto.conf so we can preserve the existing contents const Buffer *const autoConf = storageGetP( storageNewReadP(storagePg(), PG_FILE_POSTGRESQLAUTOCONF_STR, .ignoreMissing = true)); // It is unusual for the file not to exist, but we'll continue processing by creating a blank file if (autoConf == NULL) { LOG_WARN(PG_FILE_POSTGRESQLAUTOCONF " does not exist -- creating to contain recovery settings"); } // Else the file does exist so comment out old recovery options that could interfere with the current recovery. Don't // comment out *all* recovery options because some should only be commented out if there is a new option to replace it, e.g. // primary_conninfo. If the option shouldn't be commented out all the time then it won't ever be commented out -- this may // not be ideal but it is what was decided. PostgreSQL will use the last value set so this is safe as long as the option // does not have dependencies on other options. else { // Generate a regexp that will match on all current recovery_target settings RegExp *const recoveryExp = regExpNew( STRDEF( "^[\t ]*(" RECOVERY_TARGET "|" RECOVERY_TARGET_ACTION "|" RECOVERY_TARGET_INCLUSIVE "|" RECOVERY_TARGET_LSN "|" RECOVERY_TARGET_NAME "|" RECOVERY_TARGET_TIME "|" RECOVERY_TARGET_TIMELINE "|" RECOVERY_TARGET_XID ")[\t ]*=")); // Check each line for recovery settings const StringList *const contentList = strLstNewSplit(strNewBuf(autoConf), LF_STR); for (unsigned int contentIdx = 0; contentIdx < strLstSize(contentList); contentIdx++) { if (contentIdx != 0) strCat(content, LF_STR); const String *const line = strLstGet(contentList, contentIdx); if (regExpMatch(recoveryExp, line)) strCatFmt(content, "# Removed by " PROJECT_NAME " restore on %s # ", strZ(restoreLabel)); strCat(content, line); } // If settings will be appended then format the file so a blank line will be between old and new settings if (cfgOptionSeq(cfgOptType) != CFGOPTVAL_RESTORE_TYPE_NONE) { strTrim(content); strCatZ(content, "\n\n"); } } // If recovery was requested then write the recovery options if (cfgOptionSeq(cfgOptType) != CFGOPTVAL_RESTORE_TYPE_NONE) { // If the user specified standby_mode as a recovery option then error. It's tempting to just set type=standby in this // case but since config parsing has already happened the target options could be in an invalid state. if (cfgOptionTest(cfgOptRecoveryOption)) { const KeyValue *const recoveryOption = cfgOptionKv(cfgOptRecoveryOption); const StringList *const recoveryOptionKey = strLstNewVarLst(kvKeyList(recoveryOption)); for (unsigned int keyIdx = 0; keyIdx < strLstSize(recoveryOptionKey); keyIdx++) { // Get the key and value String *const key = strLstGet(recoveryOptionKey, keyIdx); // Replace - in key with _. Since we use - users naturally will as well. strReplaceChr(key, '-', '_'); if (strEq(key, STANDBY_MODE_STR)) { THROW_FMT( OptionInvalidError, "'" STANDBY_MODE "' setting is not valid for " PG_NAME " >= %s\n" "HINT: use --" CFGOPT_TYPE "=" CFGOPTVAL_RESTORE_TYPE_STANDBY_Z " instead of --" CFGOPT_RECOVERY_OPTION "=" STANDBY_MODE "=on.", strZ(pgVersionToStr(PG_VERSION_RECOVERY_GUC))); } } } strCat(content, restoreRecoveryConf(pgVersion, restoreLabel)); } LOG_INFO_FMT( "write %s%s", autoConf == NULL ? "" : "updated ", strZ(storagePathP(storagePg(), PG_FILE_POSTGRESQLAUTOCONF_STR))); // Write postgresql.auto.conf storagePutP( storageNewWriteP( storagePgWrite(), PG_FILE_POSTGRESQLAUTOCONF_STR, .noCreatePath = true, .modeFile = fileInfo->mode, .noAtomic = true, .noSyncPath = true, .user = fileInfo->user, .group = fileInfo->group), BUFSTR(content)); // The standby.signal file is required for standby mode if (cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_STANDBY) { storagePutP( storageNewWriteP( storagePgWrite(), PG_FILE_STANDBYSIGNAL_STR, .noCreatePath = true, .modeFile = fileInfo->mode, .noAtomic = true, .noSyncPath = true, .user = fileInfo->user, .group = fileInfo->group), NULL); } // Else the recovery.signal file is required for targeted recovery. Skip writing this file if the backup was offline and // recovery type is none since PostgreSQL will error in this case when wal_level=minimal. else if (cfgOptionSeq(cfgOptType) != CFGOPTVAL_RESTORE_TYPE_NONE) { storagePutP( storageNewWriteP( storagePgWrite(), PG_FILE_RECOVERYSIGNAL_STR, .noCreatePath = true, .modeFile = fileInfo->mode, .noAtomic = true, .noSyncPath = true, .user = fileInfo->user, .group = fileInfo->group), NULL); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } static void restoreRecoveryWrite(const Manifest *const manifest, const StorageInfo *const fileInfo) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(STORAGE_INFO, fileInfo); FUNCTION_LOG_END(); // Get PostgreSQL version to write recovery for const unsigned int pgVersion = manifestData(manifest)->pgVersion; MEM_CONTEXT_TEMP_BEGIN() { // If recovery type is preserve then leave recovery file as it is if (cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_PRESERVE) { // Determine which file recovery settings will be written to const String *const recoveryFile = pgVersion >= PG_VERSION_RECOVERY_GUC ? PG_FILE_POSTGRESQLAUTOCONF_STR : PG_FILE_RECOVERYCONF_STR; if (!storageExistsP(storagePg(), recoveryFile)) { LOG_WARN_FMT( "recovery type is " CFGOPTVAL_RESTORE_TYPE_PRESERVE_Z " but recovery file does not exist at '%s'", strZ(storagePathP(storagePg(), recoveryFile))); } } // Else write recovery file else { // Generate a label used to identify this restore in the recovery file const String *const restoreLabel = strNewTimeP("%Y-%m-%d %H:%M:%S", time(NULL)); // Write recovery file based on PostgreSQL version if (pgVersion >= PG_VERSION_RECOVERY_GUC) restoreRecoveryWriteAutoConf(fileInfo, pgVersion, restoreLabel); else restoreRecoveryWriteConf(fileInfo, pgVersion, restoreLabel); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/file.c.inc0000664000175100017510000005140215265101745021127 0ustar00runnerrunner/*********************************************************************************************************************************** Copy a file from the backup to the specified destination ***********************************************************************************************************************************/ typedef struct RestoreFile { const String *name; // File to restore const Buffer *checksum; // Expected checksum uint64_t size; // Expected size time_t timeModified; // Original modification time mode_t mode; // Original mode bool zero; // Should the file be zeroed? const String *user; // Original user const String *group; // Original group uint64_t offset; // Offset into repo file where pg file is located const Variant *limit; // Limit for read in the repo file uint64_t blockIncrMapSize; // Block incremental map size (0 if not incremental) size_t blockIncrSize; // Block incremental size (when map size > 0) size_t blockIncrChecksumSize; // Checksum size (when map size > 0) const String *manifestFile; // Manifest file const Buffer *blockChecksum; // Checksums for block incremental restore, set in restoreFile() } RestoreFile; typedef struct RestoreFileResult { const String *manifestFile; // Manifest file RestoreResult result; // Restore result (e.g. preserve, copy) uint64_t blockIncrDeltaSize; // Size restored by block incremental delta } RestoreFileResult; static List * restoreFile( const String *const repoFile, const unsigned int repoIdx, const CompressType repoFileCompressType, const time_t copyTimeBegin, const bool delta, const bool deltaForce, const bool bundleRaw, const CipherSpec *const cipherSpecBackup, const StringList *const referenceList, List *const fileList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, repoFile); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_PARAM(ENUM, repoFileCompressType); FUNCTION_LOG_PARAM(TIME, copyTimeBegin); FUNCTION_LOG_PARAM(BOOL, delta); FUNCTION_LOG_PARAM(BOOL, deltaForce); FUNCTION_LOG_PARAM(BOOL, bundleRaw); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpecBackup); FUNCTION_LOG_PARAM(STRING_LIST, referenceList); // List of references (for block incremental) FUNCTION_LOG_PARAM(LIST, fileList); // List of files to restore FUNCTION_LOG_END(); ASSERT(repoFile != NULL); // Restore file results List *const result = lstNewP(sizeof(RestoreFileResult)); MEM_CONTEXT_TEMP_BEGIN() { // Check files to determine which ones need to be restored StorageReadMulti *const repoFileRead = storageNewReadMultiP(storageRepoIdx(repoIdx)); for (unsigned int fileIdx = 0; fileIdx < lstSize(fileList); fileIdx++) { // Use a per-file mem context to reduce memory usage MEM_CONTEXT_TEMP_BEGIN() { RestoreFile *const file = lstGet(fileList, fileIdx); ASSERT(file->name != NULL); ASSERT(file->limit == NULL || varType(file->limit) == varTypeUInt64); RestoreFileResult *const fileResult = lstAdd( result, &(RestoreFileResult){.manifestFile = file->manifestFile, .result = restoreResultCopy}); // Perform delta if requested. Delta zero-length files to avoid overwriting the file if the timestamp is correct. if (delta && !file->zero) { // Perform delta if the file exists StorageInfo info = storageInfoP(storagePg(), file->name, .ignoreMissing = true, .followLink = true); if (info.exists) { // If force then use size/timestamp delta if (deltaForce) { // Make sure that timestamp/size are equal and the timestamp is before the copy start time of the backup if (info.size == file->size && info.timeModified == file->timeModified && info.timeModified < copyTimeBegin) { fileResult->result = restoreResultPreserve; } } // Else use size and checksum else { // Only continue delta if the file size is as expected or larger (for normal files) or if block // incremental and the file to delta is not zero-length. Block incremental can potentially use almost // any portion of an existing file, but of course zero-length files do not have anything to reuse. if (info.size >= file->size || (file->blockIncrMapSize != 0 && info.size != 0)) { const char *const fileName = strZ(storagePathP(storagePg(), file->name)); // If the file was extended since the backup, then truncate it to the size it was during the backup // as it might have only been appended to with the earlier portion being unchanged (we will verify // this using the checksum below) if (info.size > file->size) { // Open file for write IoWrite *const pgWriteTruncate = storageWriteIo( storageNewWriteP( storagePgWrite(), file->name, .noAtomic = true, .noCreatePath = true, .noSyncPath = true, .noTruncate = true)); ioWriteOpen(pgWriteTruncate); // Truncate to original size THROW_ON_SYS_ERROR_FMT( ftruncate(ioWriteFd(pgWriteTruncate), (off_t)file->size) == -1, FileWriteError, "unable to truncate file '%s'", fileName); // Close file ioWriteClose(pgWriteTruncate); // Update info info = storageInfoP(storagePg(), file->name, .followLink = true); // For block incremental it is very important that the file be exactly the expected size, so // make sure the truncate worked as expected CHECK_FMT( FileWriteError, info.size == file->size, "unable to truncate '%s' to %" PRIu64 " bytes", strZ(file->name), file->size); } // Generate checksum for the file if size is not zero IoRead *read = NULL; if (file->size != 0) { read = storageReadIo(storageNewReadP(storagePg(), file->name)); // Calculate checksum only when size matches if (info.size == file->size) ioFilterGroupAdd(ioReadFilterGroup(read), cryptoHashNew(hashTypeSha1)); // Generate block checksum list if block incremental if (file->blockIncrMapSize != 0) { ioFilterGroupAdd( ioReadFilterGroup(read), blockChecksumNew(file->blockIncrSize, file->blockIncrChecksumSize)); } ioReadDrain(read); } // If size/checksum is the same (or file is zero size) then no need to copy the file if (file->size == 0 || (info.size == file->size && bufEq( file->checksum, pckReadBinP(ioFilterGroupResultP(ioReadFilterGroup(read), CRYPTO_HASH_FILTER_TYPE))))) { // If the checksum/size are now the same but the time is not, then set the time back to the // backup time. This helps with unit testing, but also presents a pristine version of the // database after restore. if (info.timeModified != file->timeModified) { const struct utimbuf uTimeBuf = { .actime = file->timeModified, .modtime = file->timeModified, }; THROW_ON_SYS_ERROR_FMT( utime(fileName, &uTimeBuf) == -1, FileInfoError, "unable to set time for '%s'", fileName); } fileResult->result = restoreResultPreserve; } // If block incremental and not preserving the file, store the block checksum list for later use in // reconstructing the pg file if (file->blockIncrMapSize != 0 && fileResult->result != restoreResultPreserve) { PackRead *const blockChecksumResult = ioFilterGroupResultP( ioReadFilterGroup(read), BLOCK_CHECKSUM_FILTER_TYPE); MEM_CONTEXT_OBJ_BEGIN(fileList) { file->blockChecksum = pckReadBinP(blockChecksumResult); } MEM_CONTEXT_OBJ_END(); } } } } } // Create zeroed and zero-length files if (fileResult->result == restoreResultCopy && (file->size == 0 || file->zero)) { // Create destination file StorageWrite *const pgFileWrite = storageNewWriteP( storagePgWrite(), file->name, .modeFile = file->mode, .user = file->user, .group = file->group, .timeModified = file->timeModified, .noAtomic = true, .noCreatePath = true, .noSyncPath = true); ioWriteOpen(storageWriteIo(pgFileWrite)); // Truncate the file to specified length (note in this case the file will grow, not shrink) if (file->zero) { THROW_ON_SYS_ERROR_FMT( ftruncate(ioWriteFd(storageWriteIo(pgFileWrite)), (off_t)file->size) == -1, FileWriteError, "unable to truncate '%s'", strZ(file->name)); } ioWriteClose(storageWriteIo(pgFileWrite)); // Report the file as zeroed or zero-length fileResult->result = restoreResultZero; } // If the file will be copied add it to the read list if (fileResult->result == restoreResultCopy) { storageReadMultiAddP( repoFileRead, repoFile, .compressible = repoFileCompressType == compressTypeNone && cipherSpecType(cipherSpecBackup) == cipherTypeNone, .offset = file->offset, .limit = file->limit); } } MEM_CONTEXT_TEMP_END(); } // Copy files from repository to database ioReadOpen(storageReadMultiIo(repoFileRead)); for (unsigned int fileIdx = 0; fileIdx < lstSize(fileList); fileIdx++) { // Copy file from repository to database RestoreFileResult *const fileResult = lstGet(result, fileIdx); if (fileResult->result == restoreResultCopy) { // Use a per-file mem context to reduce memory usage MEM_CONTEXT_TEMP_BEGIN() { const RestoreFile *const file = lstGet(fileList, fileIdx); // Create pg file StorageWrite *const pgFileWrite = storageNewWriteP( storagePgWrite(), file->name, .modeFile = file->mode, .user = file->user, .group = file->group, .timeModified = file->timeModified, .noAtomic = true, .noCreatePath = true, .noSyncPath = true, .noTruncate = file->blockChecksum != NULL); // If block incremental file const Buffer *checksum = NULL; if (file->blockIncrMapSize != 0) { ASSERT(referenceList != NULL); // Read block map. This will be compared to the block checksum list already created to determine which // blocks need to be fetched from the repository. If we got here there must be at least one block to fetch. IoRead *const blockMapRead = ioLimitReadNew(storageReadMultiIo(repoFileRead), varUInt64(file->limit)); if (cipherSpecType(cipherSpecBackup) != cipherTypeNone) { ioFilterGroupAdd( ioReadFilterGroup(blockMapRead), cipherBlockNewP( cipherModeDecrypt, cipherSpecBackup, .header = cipherBlockHeaderNone)); } ioReadOpen(blockMapRead); const BlockMap *const blockMap = blockMapNewRead( blockMapRead, file->blockIncrSize, file->blockIncrChecksumSize); ioReadFree(blockMapRead); // Open file to write ioWriteOpen(storageWriteIo(pgFileWrite)); // Apply delta to file BlockDelta *const blockDelta = blockDeltaNew( blockMap, file->blockIncrSize, file->blockIncrChecksumSize, file->blockChecksum, cipherSpecBackup, repoFileCompressType); for (unsigned int readIdx = 0; readIdx < blockDeltaReadSize(blockDelta); readIdx++) { const BlockDeltaRead *const read = blockDeltaReadGet(blockDelta, readIdx); // Open the super block list for read. Using one read for all super blocks is cheaper than reading from // the file multiple times, which is especially noticeable on object stores. StorageRead *const superBlockRead = storageNewReadP( storageRepoIdx(repoIdx), backupFileRepoPathP( strLstGet(referenceList, read->reference), .manifestName = file->manifestFile, .bundleId = read->bundleId, .blockIncr = true), .offset = read->offset, .limit = VARUINT64(read->size)); ioReadOpen(storageReadIo(superBlockRead)); // Write updated blocks to the file const BlockDeltaWrite *deltaWrite = blockDeltaNext(blockDelta, read, storageReadIo(superBlockRead)); while (deltaWrite != NULL) { // Seek to the block offset. It is possible we are already at the correct position but it is easier // and safer to let ioWriteSeek() figure it out. ioWriteSeek(storageWriteIo(pgFileWrite), deltaWrite->offset); // Write block ioWrite(storageWriteIo(pgFileWrite), deltaWrite->block); fileResult->blockIncrDeltaSize += bufUsed(deltaWrite->block); deltaWrite = blockDeltaNext(blockDelta, read, storageReadIo(superBlockRead)); } storageReadFree(superBlockRead); } // Close the file to complete the update ioWriteClose(storageWriteIo(pgFileWrite)); // Calculate checksum. In theory this is not needed because the file should always be reconstructed // correctly. However, it seems better to check and the pages should still be buffered making the operation // very fast. IoRead *const read = storageReadIo(storageNewReadP(storagePg(), file->name)); ioFilterGroupAdd(ioReadFilterGroup(read), cryptoHashNew(hashTypeSha1)); ioReadDrain(read); checksum = pckReadBinP(ioFilterGroupResultP(ioReadFilterGroup(read), CRYPTO_HASH_FILTER_TYPE)); } // Else normal file else { IoFilterGroup *const filterGroup = ioWriteFilterGroup(storageWriteIo(pgFileWrite)); // Add decryption filter if (cipherSpecType(cipherSpecBackup) != cipherTypeNone) { ioFilterGroupAdd( filterGroup, cipherBlockNewP( cipherModeDecrypt, cipherSpecBackup, .header = bundleRaw ? cipherBlockHeaderNone : cipherBlockHeaderMagic)); } // Add decompression filter if (repoFileCompressType != compressTypeNone) ioFilterGroupAdd(filterGroup, decompressFilterP(repoFileCompressType, .raw = bundleRaw)); // Add sha1 filter ioFilterGroupAdd(filterGroup, cryptoHashNew(hashTypeSha1)); // Add size filter ioFilterGroupAdd(filterGroup, ioSizeNew()); // Copy file ioWriteOpen(storageWriteIo(pgFileWrite)); ioCopyP(storageReadMultiIo(repoFileRead), storageWriteIo(pgFileWrite), .limit = file->limit); ioWriteClose(storageWriteIo(pgFileWrite)); // Get checksum result checksum = pckReadBinP(ioFilterGroupResultP(filterGroup, CRYPTO_HASH_FILTER_TYPE)); } // Validate checksum if (!bufEq(file->checksum, checksum)) { THROW_FMT( ChecksumError, "error restoring '%s': actual checksum '%s' does not match expected checksum '%s'", strZ(file->name), strZ(strNewEncode(encodingHex, checksum)), strZ(strNewEncode(encodingHex, file->checksum))); } } MEM_CONTEXT_TEMP_END(); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(LIST, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/process.c.inc0000664000175100017510000005013415265101745021667 0ustar00runnerrunner/*********************************************************************************************************************************** Generate a list of queues that determine the order of file processing ***********************************************************************************************************************************/ // Comparator to order ManifestFile objects by size then name static const Manifest *restoreProcessQueueComparatorManifest = NULL; static int restoreProcessQueueComparator(const void *const item1, const void *const item2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, item1); FUNCTION_TEST_PARAM_P(VOID, item2); FUNCTION_TEST_END(); ASSERT(item1 != NULL); ASSERT(item2 != NULL); // Unpack files const ManifestFile file1 = manifestFileUnpack(restoreProcessQueueComparatorManifest, *(const ManifestFilePack *const *)item1); const ManifestFile file2 = manifestFileUnpack(restoreProcessQueueComparatorManifest, *(const ManifestFilePack *const *)item2); // Zero length files should be ordered at the end if (file1.size == 0) { if (file2.size != 0) FUNCTION_TEST_RETURN(INT, 1); } else if (file2.size == 0) FUNCTION_TEST_RETURN(INT, -1); // If the bundle id differs that is enough to determine order if (file1.bundleId < file2.bundleId) FUNCTION_TEST_RETURN(INT, -1); else if (file1.bundleId > file2.bundleId) FUNCTION_TEST_RETURN(INT, 1); // If the bundle ids are 0 if (file1.bundleId == 0) { // If the size differs then that's enough to determine order if (file1.size < file2.size) FUNCTION_TEST_RETURN(INT, 1); else if (file1.size > file2.size) FUNCTION_TEST_RETURN(INT, -1); // If size is the same then use name to generate a deterministic ordering (names must be unique) ASSERT(!strEq(file1.name, file2.name)); FUNCTION_TEST_RETURN(INT, strCmp(file2.name, file1.name)); } // If the reference differs that is enough to determine order if (file1.reference == NULL) { if (file2.reference != NULL) FUNCTION_TEST_RETURN(INT, 1); } else if (file2.reference == NULL) FUNCTION_TEST_RETURN(INT, -1); else { const int backupLabelCmp = strCmp(file2.reference, file1.reference) * -1; if (backupLabelCmp != 0) FUNCTION_TEST_RETURN(INT, backupLabelCmp); } // Finally order by bundle offset ASSERT(file1.bundleOffset != file2.bundleOffset); if (file1.bundleOffset < file2.bundleOffset) FUNCTION_TEST_RETURN(INT, -1); FUNCTION_TEST_RETURN(INT, 1); } static uint64_t restoreProcessQueue(const Manifest *const manifest, List **const queueList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM_P(LIST, queueList); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(manifest != NULL); uint64_t result = 0; MEM_CONTEXT_TEMP_BEGIN() { // Create list of process queues (use void * instead of List * to avoid Coverity false positive) *queueList = lstNewP(sizeof(void *)); // Generate the list of processing queues (there is always at least one) StringList *const targetList = strLstNew(); strLstAddZ(targetList, MANIFEST_TARGET_PGDATA "/"); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); if (target->tablespaceId != 0) strLstAddFmt(targetList, "%s/", strZ(target->name)); } // Generate the processing queues MEM_CONTEXT_BEGIN(lstMemContext(*queueList)) { for (unsigned int targetIdx = 0; targetIdx < strLstSize(targetList); targetIdx++) { List *const queue = lstNewP(sizeof(ManifestFile *), .comparator = restoreProcessQueueComparator); lstAdd(*queueList, &queue); } } MEM_CONTEXT_END(); // Now put all files into the processing queues for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFilePack *const filePack = manifestFilePackGet(manifest, fileIdx); const ManifestFile file = manifestFileUnpack(manifest, filePack); // Find the target that contains this file unsigned int targetIdx = 0; do { // A target should always be found CHECK(FormatError, targetIdx < strLstSize(targetList), "backup target not found"); if (strBeginsWith(file.name, strLstGet(targetList, targetIdx))) break; targetIdx++; } while (1); // Add file to queue lstAdd(*(List **)lstGet(*queueList, targetIdx), &filePack); // Add size to total result += file.size; } // Sort the queues restoreProcessQueueComparatorManifest = manifest; for (unsigned int targetIdx = 0; targetIdx < strLstSize(targetList); targetIdx++) lstSort(*(List **)lstGet(*queueList, targetIdx), sortOrderAsc); // Move process queues to prior context lstMove(*queueList, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT64, result); } /*********************************************************************************************************************************** Log the results of a job and throw errors ***********************************************************************************************************************************/ // Helper function to determine if a file should be zeroed static bool restoreFileZeroed(const String *const manifestName, RegExp *const zeroExp) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, manifestName); FUNCTION_TEST_PARAM(REGEXP, zeroExp); FUNCTION_TEST_END(); ASSERT(manifestName != NULL); FUNCTION_TEST_RETURN( BOOL, zeroExp == NULL ? false : regExpMatch(zeroExp, manifestName) && !strEndsWith(manifestName, STRDEF("/" PG_FILE_PGVERSION))); } // Helper function to construct the absolute pg path for any file. Add a temp extension to pg_control so a partially restored // cluster cannot be started. static String * restoreFilePgPath(const Manifest *const manifest, const String *const manifestName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, manifest); FUNCTION_TEST_PARAM(STRING, manifestName); FUNCTION_TEST_END(); ASSERT(manifest != NULL); ASSERT(manifestName != NULL); String *const pathPg = manifestPathPg(manifestName); String *const result = strNewFmt( "%s/%s%s", strZ(manifestTargetBase(manifest)->path), strZ(pathPg), strEqZ(manifestName, MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL) ? "." STORAGE_FILE_TEMP_EXT : ""); strFree(pathPg); FUNCTION_TEST_RETURN(STRING, result); } static uint64_t restoreJobResult( const Manifest *const manifest, ProtocolParallelJob *const job, RegExp *const zeroExp, const uint64_t sizeTotal, uint64_t sizeRestored, unsigned int *const checksumErrorFileTotal, unsigned int *const currentPercentComplete) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, job); FUNCTION_LOG_PARAM(REGEXP, zeroExp); FUNCTION_LOG_PARAM(UINT64, sizeTotal); FUNCTION_LOG_PARAM(UINT64, sizeRestored); FUNCTION_LOG_PARAM_P(UINT, checksumErrorFileTotal); FUNCTION_LOG_PARAM_P(UINT, currentPercentComplete); FUNCTION_LOG_END(); ASSERT(manifest != NULL); // The job was successful if (protocolParallelJobErrorCode(job) == 0) { MEM_CONTEXT_TEMP_BEGIN() { PackRead *const jobResult = protocolParallelJobResult(job); unsigned int percentComplete = 0; while (!pckReadNullP(jobResult)) { const ManifestFile file = manifestFileFind(manifest, pckReadStrP(jobResult)); const bool zeroed = restoreFileZeroed(file.name, zeroExp); const RestoreResult result = (RestoreResult)pckReadU32P(jobResult); const uint64_t blockIncrDeltaSize = pckReadU64P(jobResult); String *const log = strCatZ(strNew(), "restore"); if (file.checksumPage && file.checksumPageError && result == restoreResultCopy) { const String *const message = strNewFmt( "invalid page checksum(s) found in file %s", strZ(restoreFilePgPath(manifest, file.name))); if (cfgOptionBool(cfgOptChecksumPageError)) THROW(ChecksumError, strZ(message)); LOG_WARN(strZ(message)); (*checksumErrorFileTotal)++; } // Note if file was zeroed (i.e. selective restore) if (zeroed) strCatZ(log, " zeroed"); // Add filename strCatFmt(log, " file %s", strZ(restoreFilePgPath(manifest, file.name))); // If preserved add details to explain why it was not copied or zeroed if (result == restoreResultPreserve) { strCatZ(log, " - "); // On force we match on size and modification time if (cfgOptionBool(cfgOptForce)) { strCatFmt( log, "exists and matches size %" PRIu64 " and modification time %" PRIu64, file.size, (uint64_t)file.timestamp); } // Else a checksum delta or file is zero-length else { strCatZ(log, "exists and "); // No need to copy zero-length files if (file.size == 0) { strCatZ(log, "is zero size"); } // The file matched the manifest checksum so did not need to be copied else strCatZ(log, "matches backup"); } } // Add bundle info strCatZ(log, " ("); if (file.bundleId != 0) { ASSERT(varUInt64(protocolParallelJobKey(job)) == file.bundleId); strCatZ(log, "bundle "); if (file.reference != NULL) strCatFmt(log, "%s/", strZ(file.reference)); strCatFmt(log, "%" PRIu64 "/%" PRIu64 ", ", file.bundleId, file.bundleOffset); } // Add block incremental delta size, i.e. amount of the file that block incremental updated if (file.blockIncrMapSize != 0 && result != restoreResultPreserve) { strCatZ(log, "bi "); if (blockIncrDeltaSize != file.size) strCatFmt(log, "%s/", strZ(strSizeFormat(blockIncrDeltaSize))); } // Add size and percent complete sizeRestored += file.size; // Store percentComplete as an integer (used to update progress in the lock file) percentComplete = cvtPctToUInt(sizeRestored, sizeTotal); strCatFmt(log, "%s, %s)", strZ(strSizeFormat(file.size)), strZ(strNewPct(sizeRestored, sizeTotal))); // If not zero-length add the checksum if (file.size != 0 && !zeroed) strCatFmt(log, " checksum %s", strZ(strNewEncode(encodingHex, BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE)))); LOG_DETAIL_PID(protocolParallelJobProcessId(job), strZ(log)); } // Update currentPercentComplete and lock file when the change is significant enough if (percentComplete - *currentPercentComplete > 10) { *currentPercentComplete = percentComplete; cmdLockWriteP( .percentComplete = VARUINT(*currentPercentComplete), .sizeComplete = VARUINT64(sizeRestored), .size = VARUINT64(sizeTotal)); } } MEM_CONTEXT_TEMP_END(); // Free the job protocolParallelJobFree(job); } // Else the job errored else THROW_CODE(protocolParallelJobErrorCode(job), strZ(protocolParallelJobErrorMessage(job))); FUNCTION_LOG_RETURN(UINT64, sizeRestored); } /*********************************************************************************************************************************** Return new restore jobs as requested ***********************************************************************************************************************************/ typedef struct RestoreJobData { unsigned int repoIdx; // Internal repo idx Manifest *manifest; // Backup manifest List *queueList; // List of processing queues RegExp *zeroExp; // Identify files that should be sparse zeroed const CipherSpec *cipherSpecBackup; // Cipher spec used to decrypt files in the backup const String *rootReplaceUser; // User to replace invalid users when root const String *rootReplaceGroup; // Group to replace invalid group when root } RestoreJobData; // Helper to calculate the next queue to scan based on the client index static int restoreJobQueueNext(const unsigned int clientIdx, int queueIdx, const unsigned int queueTotal) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, clientIdx); FUNCTION_TEST_PARAM(INT, queueIdx); FUNCTION_TEST_PARAM(UINT, queueTotal); FUNCTION_TEST_END(); // Move (forward or back) to the next queue queueIdx += clientIdx % 2 ? -1 : 1; // Deal with wrapping on either end if (queueIdx < 0) FUNCTION_TEST_RETURN(INT, (int)queueTotal - 1); else if (queueIdx == (int)queueTotal) FUNCTION_TEST_RETURN(INT, 0); FUNCTION_TEST_RETURN(INT, queueIdx); } // Callback to fetch restore jobs for the parallel executor static ProtocolParallelJob * restoreJobCallback(void *const data, const unsigned int clientIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(UINT, clientIdx); FUNCTION_TEST_END(); ASSERT(data != NULL); ProtocolParallelJob *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get a new job if there are any left RestoreJobData *const jobData = data; // Determine where to begin scanning the queue (we'll stop when we get back here) PackWrite *param = NULL; int queueIdx = (int)(clientIdx % lstSize(jobData->queueList)); const int queueEnd = queueIdx; // Create restore job do { List *const queue = *(List **)lstGet(jobData->queueList, (unsigned int)queueIdx); bool fileAdded = false; const String *fileName = NULL; uint64_t bundleId = 0; const String *reference = NULL; while (!lstEmpty(queue)) { const ManifestFile file = manifestFileUnpack(jobData->manifest, *(ManifestFilePack **)lstGet(queue, 0)); // Break if bundled files have already been added and 1) the bundleId has changed or 2) the reference has changed if (fileAdded && (bundleId != file.bundleId || !strEq(reference, file.reference))) break; // Add common parameters before first file if (param == NULL) { param = protocolPackNew(); if (file.bundleId != 0) { bundleId = file.bundleId; reference = file.reference; } else fileName = file.name; pckWriteStrP( param, backupFileRepoPathP( file.reference != NULL ? file.reference : manifestData(jobData->manifest)->backupLabel, .manifestName = file.name, .bundleId = file.bundleId, .compressType = manifestData(jobData->manifest)->backupOptionCompressType, .blockIncr = file.blockIncrMapSize != 0)); pckWriteU32P(param, jobData->repoIdx); pckWriteU32P(param, manifestData(jobData->manifest)->backupOptionCompressType); pckWriteTimeP(param, manifestData(jobData->manifest)->backupTimestampCopyStart); pckWriteBoolP(param, cfgOptionBool(cfgOptDelta)); pckWriteBoolP(param, cfgOptionBool(cfgOptDelta) && cfgOptionBool(cfgOptForce)); pckWriteBoolP(param, file.bundleId != 0 && manifestData(jobData->manifest)->bundleRaw); cipherSpecPack(param, jobData->cipherSpecBackup); pckWriteStrLstP(param, manifestReferenceList(jobData->manifest)); fileAdded = true; } pckWriteStrP(param, restoreFilePgPath(jobData->manifest, file.name)); pckWriteBinP(param, BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE)); pckWriteU64P(param, file.size); pckWriteTimeP(param, file.timestamp); pckWriteModeP(param, file.mode); pckWriteBoolP(param, restoreFileZeroed(file.name, jobData->zeroExp)); pckWriteStrP(param, restoreManifestOwnerReplace(file.user, jobData->rootReplaceUser)); pckWriteStrP(param, restoreManifestOwnerReplace(file.group, jobData->rootReplaceGroup)); // If block incremental then modify offset and size to where the map is stored since we need to read that first. if (file.blockIncrMapSize != 0) { pckWriteBoolP(param, true); pckWriteU64P(param, file.bundleOffset + file.sizeRepo - file.blockIncrMapSize); pckWriteU64P(param, file.blockIncrMapSize); } // Else write bundle offset/size else if (file.bundleId != 0) { pckWriteBoolP(param, true); pckWriteU64P(param, file.bundleOffset); pckWriteU64P(param, file.sizeRepo); } // Else restore as a whole file else pckWriteBoolP(param, false); // Block incremental pckWriteU64P(param, file.blockIncrMapSize); if (file.blockIncrMapSize != 0) { pckWriteU64P(param, file.blockIncrSize); pckWriteU64P(param, file.blockIncrChecksumSize); } pckWriteStrP(param, file.name); // Remove job from the queue lstRemoveIdx(queue, 0); // Break if the file is not bundled if (bundleId == 0) break; } if (fileAdded) { // Assign job to result MEM_CONTEXT_PRIOR_BEGIN() { result = protocolParallelJobNew( bundleId != 0 ? VARUINT64(bundleId) : VARSTR(fileName), PROTOCOL_COMMAND_RESTORE_FILE, param); } MEM_CONTEXT_PRIOR_END(); break; } queueIdx = restoreJobQueueNext(clientIdx, queueIdx, lstSize(jobData->queueList)); } while (queueIdx != queueEnd); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/protocol.c0000664000175100017510000000735115265101745021305 0ustar00runnerrunner/*********************************************************************************************************************************** Restore Protocol Handler ***********************************************************************************************************************************/ #include #include #include #include "command/backup/blockMap.h" #include "command/backup/common.h" #include "command/restore/blockChecksum.h" #include "command/restore/blockDelta.h" #include "command/restore/protocol.h" #include "common/crypto/cipherBlock.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/filter/size.h" #include "common/io/io.h" #include "common/io/limitRead.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "storage/helper.h" #include "command/restore/file.c.inc" /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * restoreFileProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Restore file const String *const repoFile = pckReadStrP(param); const unsigned int repoIdx = pckReadU32P(param); const CompressType repoFileCompressType = (CompressType)pckReadU32P(param); const time_t copyTimeBegin = pckReadTimeP(param); const bool delta = pckReadBoolP(param); const bool deltaForce = pckReadBoolP(param); const bool bundleRaw = pckReadBoolP(param); const CipherSpec *const cipherSpecBackup = cipherSpecNewPack(param); const StringList *const referenceList = pckReadStrLstP(param); // Build the file list List *const fileList = lstNewP(sizeof(RestoreFile)); while (!pckReadNullP(param)) { RestoreFile file = {.name = pckReadStrP(param)}; file.checksum = pckReadBinP(param); file.size = pckReadU64P(param); file.timeModified = pckReadTimeP(param); file.mode = pckReadModeP(param); file.zero = pckReadBoolP(param); file.user = pckReadStrP(param); file.group = pckReadStrP(param); if (pckReadBoolP(param)) { file.offset = pckReadU64P(param); file.limit = varNewUInt64(pckReadU64P(param)); } // Block incremental file.blockIncrMapSize = pckReadU64P(param); if (file.blockIncrMapSize != 0) { file.blockIncrSize = (size_t)pckReadU64P(param); file.blockIncrChecksumSize = (size_t)pckReadU64P(param); } file.manifestFile = pckReadStrP(param); lstAdd(fileList, &file); } // Restore files const List *const resultList = restoreFile( repoFile, repoIdx, repoFileCompressType, copyTimeBegin, delta, deltaForce, bundleRaw, cipherSpecBackup, referenceList, fileList); // Return result PackWrite *const data = protocolServerResultData(result); for (unsigned int resultIdx = 0; resultIdx < lstSize(resultList); resultIdx++) { const RestoreFileResult *const fileResult = lstGet(resultList, resultIdx); pckWriteStrP(data, fileResult->manifestFile); pckWriteU32P(data, fileResult->result); pckWriteU64P(data, fileResult->blockIncrDeltaSize); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/protocol.h0000664000175100017510000000347715265101745021317 0ustar00runnerrunner/*********************************************************************************************************************************** Restore Protocol Handler ***********************************************************************************************************************************/ #ifndef COMMAND_RESTORE_PROTOCOL_H #define COMMAND_RESTORE_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Restore file types ***********************************************************************************************************************************/ typedef enum { restoreResultPreserve, restoreResultZero, restoreResultCopy, } RestoreResult; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process protocol requests FN_EXTERN ProtocolServerResult *restoreFileProtocol(PackRead *param); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_RESTORE_FILE STRID5("rs-f", 0x36e720) #define PROTOCOL_SERVER_HANDLER_RESTORE_LIST \ {.command = PROTOCOL_COMMAND_RESTORE_FILE, .process = restoreFileProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/remap.c.inc0000664000175100017510000002750715265101745021325 0ustar00runnerrunner/*********************************************************************************************************************************** Remap the manifest based on mappings provided by the user ***********************************************************************************************************************************/ static void restoreManifestMap(Manifest *const manifest) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_END(); ASSERT(manifest != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Remap the data directory // ------------------------------------------------------------------------------------------------------------------------- const String *const pgPath = cfgOptionStr(cfgOptPgPath); const ManifestTarget *const targetBase = manifestTargetBase(manifest); if (!strEq(targetBase->path, pgPath)) { LOG_INFO_FMT("remap data directory to '%s'", strZ(pgPath)); manifestTargetUpdate(manifest, targetBase->name, pgPath, NULL); } // Remap tablespaces // ------------------------------------------------------------------------------------------------------------------------- const KeyValue *const tablespaceMap = cfgOptionKvNull(cfgOptTablespaceMap); const String *const tablespaceMapAllPath = cfgOptionStrNull(cfgOptTablespaceMapAll); if (tablespaceMap != NULL || tablespaceMapAllPath != NULL) { StringList *const tablespaceRemapped = strLstNew(); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); // Is this a tablespace? if (target->tablespaceId != 0) { const String *tablespacePath = NULL; // Check for an individual mapping for this tablespace if (tablespaceMap != NULL) { // Attempt to get the tablespace by name const String *const tablespacePathByName = varStr(kvGet(tablespaceMap, VARSTR(target->tablespaceName))); if (tablespacePathByName != NULL) strLstAdd(tablespaceRemapped, target->tablespaceName); // Attempt to get the tablespace by id const String *const tablespacePathById = varStr( kvGet(tablespaceMap, VARSTR(varStrForce(VARUINT(target->tablespaceId))))); if (tablespacePathById != NULL) strLstAdd(tablespaceRemapped, varStrForce(VARUINT(target->tablespaceId))); // Error when both are set but the paths are different if (tablespacePathByName != NULL && tablespacePathById != NULL && ! strEq(tablespacePathByName, tablespacePathById)) { THROW_FMT( TablespaceMapError, "tablespace remapped by name '%s' and id %u with different paths", strZ(target->tablespaceName), target->tablespaceId); } // Else set the path by name else if (tablespacePathByName != NULL) { tablespacePath = tablespacePathByName; } // Else set the path by id else if (tablespacePathById != NULL) tablespacePath = tablespacePathById; } // If not individual mapping check if all tablespaces are being remapped if (tablespacePath == NULL && tablespaceMapAllPath != NULL) tablespacePath = strNewFmt("%s/%s", strZ(tablespaceMapAllPath), strZ(target->tablespaceName)); // Remap tablespace if a mapping was found if (tablespacePath != NULL) { LOG_INFO_FMT("map tablespace '%s' to '%s'", strZ(target->name), strZ(tablespacePath)); manifestTargetUpdate(manifest, target->name, tablespacePath, NULL); manifestLinkUpdate(manifest, strNewFmt(MANIFEST_TARGET_PGDATA "/%s", strZ(target->name)), tablespacePath); } } } // Error on invalid tablespaces if (tablespaceMap != NULL) { const VariantList *const tablespaceMapList = kvKeyList(tablespaceMap); strLstSort(tablespaceRemapped, sortOrderAsc); for (unsigned int tablespaceMapIdx = 0; tablespaceMapIdx < varLstSize(tablespaceMapList); tablespaceMapIdx++) { const String *const tablespace = varStr(varLstGet(tablespaceMapList, tablespaceMapIdx)); if (!strLstExists(tablespaceRemapped, tablespace)) THROW_FMT(TablespaceMapError, "unable to remap invalid tablespace '%s'", strZ(tablespace)); } } } // Remap links // ------------------------------------------------------------------------------------------------------------------------- const KeyValue *const linkMap = cfgOptionKvNull(cfgOptLinkMap); if (linkMap != NULL) { const StringList *const linkMapList = strLstSort(strLstNewVarLst(kvKeyList(linkMap)), sortOrderAsc); for (unsigned int linkMapIdx = 0; linkMapIdx < strLstSize(linkMapList); linkMapIdx++) { const String *const link = strLstGet(linkMapList, linkMapIdx); const String *const linkPath = varStr(kvGet(linkMap, VARSTR(link))); const String *const manifestName = strNewFmt(MANIFEST_TARGET_PGDATA "/%s", strZ(link)); // Attempt to find the link target ManifestTarget target = {0}; if (manifestTargetFindDefault(manifest, manifestName, NULL) != NULL) target = *manifestTargetFind(manifest, manifestName); // If the target was not found then check if the link is a valid file or path bool create = false; if (target.name == NULL) { // Is the specified link a file or a path? Error if they both match. const bool pathExists = manifestPathFindDefault(manifest, manifestName, NULL) != NULL; const bool fileExists = manifestFileExists(manifest, manifestName); CHECK(FormatError, !pathExists || !fileExists, "link may not be both file and path"); target = (ManifestTarget){.name = manifestName, .path = linkPath, .type = manifestTargetTypeLink}; // If a file if (fileExists) { // File needs to be set so the file/path is updated later but set it to something invalid just in case it // it does not get updated due to a regression target.file = DOT_STR; } // Else error if not a path else if (!pathExists) { THROW_FMT( LinkMapError, "unable to map link '%s'\n" "HINT: Does the link reference a valid backup path or file?", strZ(link)); } // Add the link. Copy user/group from the base data directory. const ManifestPath *const pathBase = manifestPathFind(manifest, MANIFEST_TARGET_PGDATA_STR); const ManifestLink manifestLink = { .name = manifestName, .destination = linkPath, .group = pathBase->group, .user = pathBase->user, }; manifestLinkAdd(manifest, &manifestLink); create = true; } // Else update target to new path else target.path = linkPath; // The target must be a link since pg_data/ was prepended and pgdata is the only allowed path CHECK(FormatError, target.type == manifestTargetTypeLink, "target must be a link"); // Error if the target is a tablespace if (target.tablespaceId != 0) { THROW_FMT( LinkMapError, "unable to remap tablespace '%s'\n" "HINT: use '" CFGOPT_TABLESPACE_MAP "' option to remap tablespaces.", strZ(link)); } LOG_INFO_FMT("%slink '%s' to '%s'", create ? "" : "map ", strZ(link), strZ(target.path)); // If the link was not created update to the new destination if (!create) manifestLinkUpdate(manifest, target.name, target.path); // If the link is a file separate the file name from the path if (target.file != NULL) { // The link destination must have at least one path component in addition to the file part. So '..' would // not be a valid destination but '../file' or '/file' is. if (strSize(strPath(target.path)) == 0) { THROW_FMT( LinkMapError, "'%s' is not long enough to be the destination for file link '%s'", strZ(target.path), strZ(link)); } target.file = strBase(target.path); target.path = strPath(target.path); } // Create a new target or update the existing target file/path if (create) manifestTargetAdd(manifest, &target); else manifestTargetUpdate(manifest, target.name, target.path, target.file); } } // If all links are not being restored then check for links that were not remapped and remove them if (!cfgOptionBool(cfgOptLinkAll)) { unsigned int targetIdx = 0; while (targetIdx < manifestTargetTotal(manifest)) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); // Is this a non-tablespace link? if (target->type == manifestTargetTypeLink && target->tablespaceId == 0) { const String *const link = strSub(target->name, strSize(MANIFEST_TARGET_PGDATA_STR) + 1); // If the link was not remapped then remove it if (linkMap == NULL || kvGet(linkMap, VARSTR(link)) == NULL) { if (target->file != NULL) LOG_WARN_FMT("file link '%s' will be restored as a file at the same location", strZ(link)); else { LOG_WARN_FMT( "contents of directory link '%s' will be restored in a directory at the same location", strZ(link)); } manifestLinkRemove(manifest, target->name); manifestTargetRemove(manifest, target->name); continue; } } targetIdx++; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/restore.c0000664000175100017510000003145415265101745021130 0ustar00runnerrunner/*********************************************************************************************************************************** Restore Command ***********************************************************************************************************************************/ #include #include #include "command/lock.h" #include "command/restore/protocol.h" #include "command/restore/restore.h" #include "command/restore/timeline.h" #include "common/regExp.h" #include "common/user.h" #include "config/config.h" #include "config/exec.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "info/manifest/manifest.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "protocol/parallel.h" #include "storage/helper.h" #include "clean.c.inc" #include "config.c.inc" #include "process.c.inc" #include "remap.c.inc" #include "select.c.inc" #include "validate.c.inc" /**********************************************************************************************************************************/ FN_EXTERN void cmdRestore(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { // Get information for the current user userInit(); // PostgreSQL must be local pgIsLocalVerify(); // Validate restore path restorePathValidate(); // Remove stanza archive spool path so existing files do not interfere with the new cluster. For instance, old archive-push // acknowledgements could cause a new cluster to skip archiving. This should not happen if a new timeline is selected but // better to be safe. Missing stanza spool paths are ignored. storagePathRemoveP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_STR, .recurse = true); // Get the backup set const RestoreBackupData backupData = restoreBackupSet(); // Load manifest RestoreJobData jobData = {.repoIdx = backupData.repoIdx}; jobData.manifest = manifestLoadFile( storageRepoIdx(backupData.repoIdx), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupData.backupSet)), backupData.cipherSpecManifest); // Verify that the selected timeline is valid for the backup -- including current and latest timelines if (manifestData(jobData.manifest)->backupOptionOnline) { const ManifestData *const data = manifestData(jobData.manifest); const InfoArchive *const archiveInfo = infoArchiveLoadFile( storageRepoIdx(backupData.repoIdx), INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMainIdx(backupData.repoIdx)); timelineVerify( storageRepoIdx(backupData.repoIdx), strNewFmt("%s-%u", strZ(pgVersionToStr(data->pgVersion)), data->pgId), data->pgVersion, cvtZToUIntBase(strZ(strSubN(data->archiveStart, 0, 8)), 16), pgLsnFromStr(data->lsnStart), cfgOptionStrNull(cfgOptTargetTimeline), cfgOptionSeq(cfgOptType), infoArchiveCipherSpecMap(archiveInfo)); } // Remotes (if any) are no longer needed since the rest of the repository reads will be done by the local processes protocolFree(); // If the backup was made offline and no restore type was specified then set to none. This prevents recovery.signal from // being generated by default for PostgreSQL >= 12. Offline backups created with wal_level=minimal will error in this case // so this should be a good default. However, if the user explicitly sets type and it does not equal none then they will get // an error if wal_level=minimal. if (!manifestData(jobData.manifest)->backupOptionOnline && cfgOptionSource(cfgOptType) == cfgSourceDefault) cfgOptionSet(cfgOptType, cfgSourceParam, VARUINT64(CFGOPTVAL_RESTORE_TYPE_NONE_STRID)); // Validate manifest. Don't use strict mode because we'd rather ignore problems that won't affect a restore. manifestValidate(jobData.manifest, false); // Get the cipher spec used to decrypt files in the backup jobData.cipherSpecBackup = manifestCipherSpec(jobData.manifest); // Validate the manifest restoreManifestValidate(jobData.manifest, backupData.backupSet); // Log the backup set to restore. If the backup was online then append the time recovery will start from. String *const message = strCatFmt( strNew(), "%s: restore backup set %s", cfgOptionGroupName(cfgOptGrpRepo, backupData.repoIdx), strZ(backupData.backupSet)); if (manifestData(jobData.manifest)->backupOptionOnline) { strCatZ(message, ", recovery will start at "); strCatTimeP(message, "%Y-%m-%d %H:%M:%S", manifestData(jobData.manifest)->backupTimestampStart); } LOG_INFO(strZ(message)); // Map manifest restoreManifestMap(jobData.manifest); // Check that links are sane manifestLinkCheck(jobData.manifest); // Update ownership restoreManifestOwner(jobData.manifest, &jobData.rootReplaceUser, &jobData.rootReplaceGroup); // Generate the selective restore expression const String *const expression = restoreSelectiveExpression(jobData.manifest); jobData.zeroExp = expression == NULL ? NULL : regExpNew(expression); // Clean the data directory and build path/link structure restoreCleanBuild(jobData.manifest, jobData.rootReplaceUser, jobData.rootReplaceGroup); // Generate processing queues const uint64_t sizeTotal = restoreProcessQueue(jobData.manifest, &jobData.queueList); // Save manifest to the data directory so we can restart a delta restore even if the PG_VERSION file is missing manifestSave(jobData.manifest, storageWriteIo(storageNewWriteP(storagePgWrite(), BACKUP_MANIFEST_FILE_STR))); // Create the parallel executor ProtocolParallel *const parallelExec = protocolParallelNew( cfgOptionUInt64(cfgOptProtocolTimeout) / 2, restoreJobCallback, &jobData); for (unsigned int processIdx = 1; processIdx <= cfgOptionUInt(cfgOptProcessMax); processIdx++) protocolParallelClientAdd(parallelExec, protocolLocalGet(protocolStorageTypeRepo, 0, processIdx)); // Process jobs uint64_t sizeRestored = 0; // Initialize percent complete and bytes completed/total unsigned int currentPercentComplete = 0; cmdLockWriteP( .percentComplete = VARUINT(currentPercentComplete), .sizeComplete = VARUINT64(sizeRestored), .size = VARUINT64(sizeTotal)); unsigned int checksumErrorFileTotal = 0; MEM_CONTEXT_TEMP_RESET_BEGIN() { do { const unsigned int completed = protocolParallelProcess(parallelExec); for (unsigned int jobIdx = 0; jobIdx < completed; jobIdx++) { sizeRestored = restoreJobResult( jobData.manifest, protocolParallelResult(parallelExec), jobData.zeroExp, sizeTotal, sizeRestored, &checksumErrorFileTotal, ¤tPercentComplete); } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } while (!protocolParallelDone(parallelExec)); } MEM_CONTEXT_TEMP_END(); // Write recovery settings. Use the data directory to set permissions and ownership for recovery files. StorageInfo fileInfo = storageInfoP(storagePg(), NULL); fileInfo.user = restoreManifestOwnerReplace(fileInfo.user, jobData.rootReplaceUser); fileInfo.group = restoreManifestOwnerReplace(fileInfo.group, jobData.rootReplaceGroup); fileInfo.mode = fileInfo.mode & (S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); restoreRecoveryWrite(jobData.manifest, &fileInfo); // Remove backup.manifest storageRemoveP(storagePgWrite(), BACKUP_MANIFEST_FILE_STR); // Sync file link paths. These need to be synced separately because they are not linked from the data directory. StringList *const pathSynced = strLstNew(); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(jobData.manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(jobData.manifest, targetIdx); if (target->type == manifestTargetTypeLink && target->file != NULL) { const String *const pgPath = manifestTargetPath(jobData.manifest, target); // Don't sync the same path twice. There can be multiple links to files in the same path, but syncing it more than // once makes the logs noisy and looks like a bug even though it doesn't hurt anything or realistically affect // performance. if (strLstExists(pathSynced, pgPath)) continue; else strLstAdd(pathSynced, pgPath); // Sync the path LOG_DETAIL_FMT("sync path '%s'", strZ(pgPath)); storagePathSyncP(storageLocalWrite(), pgPath); } } // Sync paths in the data directory for (unsigned int pathIdx = 0; pathIdx < manifestPathTotal(jobData.manifest); pathIdx++) { const String *const manifestName = manifestPath(jobData.manifest, pathIdx)->name; // Skip the pg_tblspc path because it only maps to the manifest. We should remove this in a future release but not much // can be done about it for now. if (strEqZ(manifestName, MANIFEST_TARGET_PGTBLSPC)) continue; // We'll sync global after pg_control is written if (strEq(manifestName, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL))) continue; const String *const pgPath = storagePathP(storagePg(), manifestPathPg(manifestName)); LOG_DETAIL_FMT("sync path '%s'", strZ(pgPath)); storagePathSyncP(storagePgWrite(), pgPath); } // Rename pg_control to remove the temp extension. This is done last to prevent a partially restored (or unsynced) cluster // from being started. if (storageExistsP(storagePg(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "." STORAGE_FILE_TEMP_EXT))) { // Invalidate the checkpoint in pg_control so the cluster cannot be started without backup_label if (manifestData(jobData.manifest)->backupOptionOnline) { Buffer *const pgControlBuffer = storageGetP( storageNewReadP(storagePg(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "." STORAGE_FILE_TEMP_EXT))); const ManifestFile pgControlFile = manifestFileFind( jobData.manifest, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)); pgControlCheckpointInvalidate(pgControlBuffer, cfgOptionStrNull(cfgOptPgVersionForce)); storagePutP( storageNewWriteP( storagePgWrite(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "." STORAGE_FILE_TEMP_EXT), .timeModified = pgControlFile.timestamp, .noAtomic = true, .noCreatePath = true, .noSyncPath = true), pgControlBuffer); } LOG_INFO( "restore " PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL " (performed last to ensure aborted restores cannot be started)"); storageMoveP( storagePgWrite(), storageNewReadP(storagePg(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "." STORAGE_FILE_TEMP_EXT)), storageNewWriteP(storagePgWrite(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL), .noSyncPath = true)); } else LOG_WARN("backup does not contain '" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "' -- cluster will not start"); // Sync global path to persist pg_control rename. If this fails or if the system crashes before it can complete, then // pg_control should retain the temp extension, which will prevent the cluster from being started. LOG_DETAIL_FMT("sync path '%s'", strZ(storagePathP(storagePg(), PG_PATH_GLOBAL_STR))); storagePathSyncP(storagePgWrite(), PG_PATH_GLOBAL_STR); // Restore info LOG_INFO_FMT( "restore size = %s, file total = %u", strZ(strSizeFormat(sizeRestored)), manifestFileTotal(jobData.manifest)); if (checksumErrorFileTotal > 0) { LOG_WARN_FMT( CFGCMD_RESTORE " command encountered page checksum error(s) in %u file(s), check the log file for details", checksumErrorFileTotal); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/restore.h0000664000175100017510000000126015265101745021125 0ustar00runnerrunner/*********************************************************************************************************************************** Restore Command ***********************************************************************************************************************************/ #ifndef COMMAND_RESTORE_RESTORE_H #define COMMAND_RESTORE_RESTORE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Restore a backup FN_EXTERN void cmdRestore(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/select.c.inc0000664000175100017510000004521215265101745021471 0ustar00runnerrunner/*********************************************************************************************************************************** Get the backup set to restore ***********************************************************************************************************************************/ typedef struct RestoreBackupData { unsigned int repoIdx; // Internal repo index const CipherSpec *cipherSpecManifest; // Cipher spec to open the manifest, none when not encrypted const String *backupSet; // Backup set to restore } RestoreBackupData; #define FUNCTION_LOG_RESTORE_BACKUP_DATA_TYPE \ RestoreBackupData #define FUNCTION_LOG_RESTORE_BACKUP_DATA_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "RestoreBackupData", buffer, bufferSize) // Helper function for restoreBackupSet static RestoreBackupData restoreBackupData(const String *const backupLabel, const unsigned int repoIdx, const CipherSpec *const cipherSpecManifest) { ASSERT(backupLabel != NULL); RestoreBackupData restoreBackup = {0}; MEM_CONTEXT_PRIOR_BEGIN() { restoreBackup.backupSet = strDup(backupLabel); restoreBackup.repoIdx = repoIdx; restoreBackup.cipherSpecManifest = cipherSpecDupP(cipherSpecManifest); } MEM_CONTEXT_PRIOR_END(); return restoreBackup; } static RestoreBackupData restoreBackupSet(void) { FUNCTION_LOG_VOID(logLevelDebug); FUNCTION_AUDIT_STRUCT(); RestoreBackupData result = {0}; MEM_CONTEXT_TEMP_BEGIN() { // Initialize the repo index unsigned int repoIdxMin = 0; unsigned int repoIdxMax = cfgOptionGroupIdxTotal(cfgOptGrpRepo) - 1; // If the repo was specified then set index to the array location and max to loop only once if (cfgOptionTest(cfgOptRepo)) { repoIdxMin = cfgOptionGroupIdxDefault(cfgOptGrpRepo); repoIdxMax = repoIdxMin; } // If the set option was not provided by the user but a target was set, then we will need to search for a backup set that // satisfies the target condition, else we will use the backup provided const String *backupSetRequested = NULL; const unsigned int targetType = cfgOptionSeq(cfgOptType); union { time_t time; uint64_t lsn; } target = {0}; if (cfgOptionSource(cfgOptSet) == cfgSourceDefault) { if (targetType == CFGOPTVAL_RESTORE_TYPE_TIME) { TRY_BEGIN() { target.time = cvtZToTime(strZ(cfgOptionStr(cfgOptTarget))); } CATCH_ANY() { THROW_FMT( FormatError, "automatic backup set selection cannot be performed with provided time '%s'\n" "HINT: time format must be YYYY-MM-DD HH:MM:SS with optional msec and optional timezone (+/- HH or HHMM or" " HH:MM) - if timezone is omitted, local time is assumed (for UTC use +00)", strZ(cfgOptionStr(cfgOptTarget))); } TRY_END(); } else if (targetType == CFGOPTVAL_RESTORE_TYPE_LSN) target.lsn = pgLsnFromStr(cfgOptionStr(cfgOptTarget)); } else backupSetRequested = cfgOptionStr(cfgOptSet); // Search through the repo list for a backup set to use for recovery for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { // Get the repo storage in case it is remote and encryption settings need to be pulled down storageRepoIdx(repoIdx); const InfoBackup *infoBackup = NULL; // Attempt to load backup.info TRY_BEGIN() { infoBackup = infoBackupLoadFile(storageRepoIdx(repoIdx), INFO_BACKUP_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); } CATCH_ANY() { LOG_WARN_FMT("%s: [%s] %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorTypeName(errorType()), errorMessage()); } TRY_END(); // If unable to load the backup info file, then move on to next repo if (infoBackup == NULL) continue; if (infoBackupDataTotal(infoBackup) == 0) { LOG_WARN_FMT( "%s: [%s] no backup sets to restore", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorTypeName(&BackupSetInvalidError)); continue; } // If a backup set was not specified, then see if a target was requested if (backupSetRequested == NULL) { // Get the latest backup const InfoBackupData latestBackup = infoBackupData(infoBackup, infoBackupDataTotal(infoBackup) - 1); // If a target was requested, attempt to determine the backup set if (targetType == CFGOPTVAL_RESTORE_TYPE_TIME || targetType == CFGOPTVAL_RESTORE_TYPE_LSN) { bool found = false; // Search current backups from newest to oldest for (unsigned int keyIdx = infoBackupDataTotal(infoBackup) - 1; (int)keyIdx >= 0; keyIdx--) { // Get the backup data const InfoBackupData backupData = infoBackupData(infoBackup, keyIdx); // If target is lsn and no backupLsnStop exists, exit this repo and log that backup may be manually selected if (targetType == CFGOPTVAL_RESTORE_TYPE_LSN && !backupData.backupLsnStop) { LOG_WARN_FMT( "%s reached backup from prior version missing required LSN info before finding a match -- backup" " auto-select has been disabled for this repo\n" "HINT: you may specify a backup to restore using the --set option.", cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); break; } // If the end of the backup is valid for the target, then select this backup if ((targetType == CFGOPTVAL_RESTORE_TYPE_TIME && backupData.backupTimestampStop < target.time) || (targetType == CFGOPTVAL_RESTORE_TYPE_LSN && pgLsnFromStr(backupData.backupLsnStop) <= target.lsn)) { found = true; result = restoreBackupData(backupData.backupLabel, repoIdx, infoBackupCipherSpec(infoBackup)); break; } } // If a backup was found on this repo matching the criteria for time then exit if (found) break; } // Else use backup set found else { // Is this backup part of the latest pg history? const InfoPgData backupInfoPg = infoPgData( infoBackupPg(infoBackup), infoPgDataCurrentId(infoBackupPg(infoBackup))); if (latestBackup.backupPgId < backupInfoPg.id) { THROW_FMT( BackupSetInvalidError, "the latest backup set found '%s' is from a prior version of " PG_NAME "\n" "HINT: was a backup created after the stanza-upgrade?\n" "HINT: specify --" CFGOPT_SET " or --" CFGOPT_TYPE "=time/lsn to restore from a prior version of" " " PG_NAME ".", strZ(latestBackup.backupLabel)); } result = restoreBackupData(latestBackup.backupLabel, repoIdx, infoBackupCipherSpec(infoBackup)); break; } } // Otherwise check to see if the specified backup set is on this repo else { for (unsigned int backupIdx = 0; backupIdx < infoBackupDataTotal(infoBackup); backupIdx++) { if (strEq(infoBackupData(infoBackup, backupIdx).backupLabel, backupSetRequested)) { result = restoreBackupData(backupSetRequested, repoIdx, infoBackupCipherSpec(infoBackup)); break; } } // If the backup set is found, then exit, else continue to next repo if (result.backupSet != NULL) break; } } // Still no backup set to use after checking all the repos required to be checked? if (result.backupSet == NULL) { if (backupSetRequested != NULL) THROW_FMT(BackupSetInvalidError, "backup set %s is not valid", strZ(backupSetRequested)); else if (targetType == CFGOPTVAL_RESTORE_TYPE_TIME || targetType == CFGOPTVAL_RESTORE_TYPE_LSN) { THROW_FMT( BackupSetInvalidError, "unable to find backup set with %s '%s'", targetType == CFGOPTVAL_RESTORE_TYPE_LSN ? "lsn less than or equal to" : "stop time less than", strZ(cfgOptionDisplay(cfgOptTarget))); } else THROW(BackupSetInvalidError, "no backup set found to restore"); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /*********************************************************************************************************************************** Generate the expression to zero files that are not needed for selective restore ***********************************************************************************************************************************/ static String * restoreSelectiveExpression(const Manifest *const manifest) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_END(); ASSERT(manifest != NULL); String *result = NULL; // Continue if databases to include or exclude have been specified if (cfgOptionTest(cfgOptDbExclude) || cfgOptionTest(cfgOptDbInclude)) { MEM_CONTEXT_TEMP_BEGIN() { // Generate base expression RegExp *const baseRegExp = regExpNew(STRDEF("^" MANIFEST_TARGET_PGDATA "/" PG_PATH_BASE "/[0-9]+/" PG_FILE_PGVERSION)); // Generate tablespace expression const String *const tablespaceId = pgTablespaceId( manifestData(manifest)->pgVersion, manifestData(manifest)->pgCatalogVersion); RegExp *const tablespaceRegExp = regExpNew( strNewFmt("^" MANIFEST_TARGET_PGTBLSPC "/[0-9]+/%s/[0-9]+/" PG_FILE_PGVERSION, strZ(tablespaceId))); // Generate a list of databases in base or in a tablespace and get all standard system databases, even in cases where // users have recreated them StringList *systemDbIdList = strLstNew(); StringList *dbList = strLstNew(); for (unsigned int systemDbIdx = 0; systemDbIdx < manifestDbTotal(manifest); systemDbIdx++) { const ManifestDb *const systemDb = manifestDb(manifest, systemDbIdx); if (pgDbIsSystem(systemDb->name) || pgDbIsSystemId(systemDb->id)) { // Build the system id list and add to the dbList for logging and checking const String *const systemDbId = varStrForce(VARUINT(systemDb->id)); strLstAdd(systemDbIdList, systemDbId); strLstAdd(dbList, systemDbId); } } for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const String *const fileName = manifestFileNameGet(manifest, fileIdx); if (regExpMatch(baseRegExp, fileName) || regExpMatch(tablespaceRegExp, fileName)) { const String *const dbId = strBase(strPath(fileName)); // In the highly unlikely event that a system database was somehow added after the backup began, it will only be // found in the file list and not the manifest db section, so add it to the system database list if (pgDbIsSystemId(cvtZToUInt(strZ(dbId)))) strLstAddIfMissing(systemDbIdList, dbId); strLstAddIfMissing(dbList, dbId); } } strLstSort(dbList, sortOrderAsc); // If no databases were found then this backup is not a valid cluster if (strLstEmpty(dbList)) THROW(FormatError, "no databases found for selective restore\nHINT: is this a valid cluster?"); // Log databases found LOG_DETAIL_FMT("databases found for selective restore (%s)", strZ(strLstJoin(dbList, ", "))); // Generate list with ids of databases to exclude StringList *const excludeDbIdList = strLstNew(); const StringList *const excludeList = strLstNewVarLst(cfgOptionLst(cfgOptDbExclude)); for (unsigned int excludeIdx = 0; excludeIdx < strLstSize(excludeList); excludeIdx++) { const String *excludeDb = strLstGet(excludeList, excludeIdx); // If the db to exclude is not in the list as an id then search by name if (!strLstExists(dbList, excludeDb)) { const ManifestDb *const db = manifestDbFindDefault(manifest, excludeDb, NULL); if (db == NULL || !strLstExists(dbList, varStrForce(VARUINT(db->id)))) THROW_FMT(DbMissingError, "database to exclude '%s' does not exist", strZ(excludeDb)); // Set the exclude db to the id if the name mapping was successful excludeDb = varStrForce(VARUINT(db->id)); } // Add to exclude list strLstAdd(excludeDbIdList, excludeDb); } // Remove included databases from the list const StringList *const includeList = strLstNewVarLst(cfgOptionLst(cfgOptDbInclude)); for (unsigned int includeIdx = 0; includeIdx < strLstSize(includeList); includeIdx++) { const String *includeDb = strLstGet(includeList, includeIdx); // If the db to include is not in the list as an id then search by name if (!strLstExists(dbList, includeDb)) { const ManifestDb *const db = manifestDbFindDefault(manifest, includeDb, NULL); if (db == NULL || !strLstExists(dbList, varStrForce(VARUINT(db->id)))) THROW_FMT(DbMissingError, "database to include '%s' does not exist", strZ(includeDb)); // Set the include db to the id if the name mapping was successful includeDb = varStrForce(VARUINT(db->id)); } // Error if the db is a system db if (strLstExists(systemDbIdList, includeDb)) THROW(DbInvalidError, "system databases (template0, postgres, etc.) are included by default"); // Error if the db id is in the exclude list if (strLstExists(excludeDbIdList, includeDb)) THROW_FMT(DbInvalidError, "database to include '%s' is in the exclude list", strZ(includeDb)); // Remove from list of DBs to zero strLstRemove(dbList, includeDb); } // Only exclude specified db in case no db to include has been provided if (strLstEmpty(includeList)) { dbList = strLstDup(excludeDbIdList); } // Else, remove the system databases from list of DBs to zero unless they are excluded explicitly else { strLstSort(systemDbIdList, sortOrderAsc); strLstSort(excludeDbIdList, sortOrderAsc); systemDbIdList = strLstMergeAnti(systemDbIdList, excludeDbIdList); dbList = strLstMergeAnti(dbList, systemDbIdList); } // Build regular expression to identify files that will be zeroed String *expression = NULL; if (!strLstEmpty(dbList)) { LOG_DETAIL_FMT("databases excluded (zeroed) from selective restore (%s)", strZ(strLstJoin(dbList, ", "))); // Generate the expression from the list of databases to be zeroed. Only user created databases can be zeroed, never // system databases. for (unsigned int dbIdx = 0; dbIdx < strLstSize(dbList); dbIdx++) { const String *const db = strLstGet(dbList, dbIdx); // Create expression string or append | if (expression == NULL) expression = strNew(); else strCatZ(expression, "|"); // Filter files in base directory strCatFmt(expression, "(^" MANIFEST_TARGET_PGDATA "/" PG_PATH_BASE "/%s/)", strZ(db)); // Filter files in tablespace directories for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); if (target->tablespaceId != 0) strCatFmt(expression, "|(^%s/%s/%s/)", strZ(target->name), strZ(tablespaceId), strZ(db)); } } } // If all user databases have been selected then nothing to do if (expression == NULL) { LOG_INFO_FMT("nothing to filter - all user databases have been selected"); } // Else return the expression else { MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(expression); } MEM_CONTEXT_PRIOR_END(); } } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/timeline.c0000664000175100017510000003115715265101745021253 0ustar00runnerrunner/*********************************************************************************************************************************** Timeline Management ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/restore/timeline.h" #include "common/format/cipherBlockFormat.h" #include "common/log.h" #include "config/config.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/helper.h" /*********************************************************************************************************************************** Parse history file ***********************************************************************************************************************************/ typedef struct HistoryItem { unsigned int timeline; // Parent timeline uint64_t lsn; // Boundary lsn } HistoryItem; static List * historyParse(const String *const history) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, history); FUNCTION_LOG_END(); List *const result = lstNewP(sizeof(HistoryItem)); MEM_CONTEXT_TEMP_BEGIN() { const StringList *const historyList = strLstNewSplitZ(history, "\n"); for (unsigned int historyIdx = 0; historyIdx < strLstSize(historyList); historyIdx++) { const String *const line = strTrim(strLstGet(historyList, historyIdx)); // Skip empty lines if (strSize(line) == 0) continue; // Skip comments. PostgreSQL does not write comments into history files but it does allow for them. if (strBeginsWithZ(line, "#")) continue; // Split line const StringList *const split = strLstNewSplitZ(line, "\t"); if (strLstSize(split) < 2) THROW_FMT(FormatError, "invalid history line format '%s'", strZ(strLstGet(historyList, historyIdx))); const HistoryItem historyItem = { .timeline = cvtZToUInt(strZ(strLstGet(split, 0))), .lsn = pgLsnFromStr(strLstGet(split, 1)), }; ASSERT(historyItem.lsn > 0); ASSERT(historyItem.timeline > 0); lstAdd(result, &historyItem); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(LIST, result); } /*********************************************************************************************************************************** Load history file ***********************************************************************************************************************************/ static List * historyLoad( const Storage *const storageRepo, const String *const archiveId, const unsigned int timeline, const CipherSpecMap *const cipherSpecArchive) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storageRepo); FUNCTION_LOG_PARAM(STRING, archiveId); FUNCTION_LOG_PARAM(UINT, timeline); FUNCTION_LOG_PARAM(CIPHER_SPEC_MAP, cipherSpecArchive); FUNCTION_LOG_END(); List *result; MEM_CONTEXT_TEMP_BEGIN() { const String *const historyFile = strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%08X.history", strZ(archiveId), timeline); StorageRead *const storageRead = storageNewReadP(storageRepo, historyFile); cipherBlockFormatFilterGroupReadAddMap(ioReadFilterGroup(storageReadIo(storageRead)), cipherSpecArchive); const Buffer *const history = storageGetP(storageRead); TRY_BEGIN() { result = lstMove(historyParse(strNewBuf(history)), memContextPrior()); } CATCH_ANY() { THROW_FMT( FormatError, "invalid timeline '%X' at '%s': %s", timeline, strZ(storagePathP(storageRepo, historyFile)), errorMessage()); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(LIST, result); } /*********************************************************************************************************************************** Find latest timeline ***********************************************************************************************************************************/ static unsigned int timelineLatest(const Storage *const storageRepo, const String *const archiveId, const unsigned int timelineCurrent) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storageRepo); FUNCTION_LOG_PARAM(STRING, archiveId); FUNCTION_LOG_PARAM(UINT, timelineCurrent); FUNCTION_LOG_END(); ASSERT(storageRepo != NULL); unsigned int result = timelineCurrent; MEM_CONTEXT_TEMP_BEGIN() { const StringList *const historyList = strLstSort( storageListP( storageRepo, strNewFmt(STORAGE_REPO_ARCHIVE "/%s", strZ(archiveId)), .expression = STRDEF("[0-F]{8}\\.history")), sortOrderDesc); if (!strLstEmpty(historyList)) { const unsigned int timelineLatest = cvtZToUIntBase(strZ(strSubN(strLstGet(historyList, 0), 0, 8)), 16); if (timelineLatest > timelineCurrent) result = timelineLatest; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } /**********************************************************************************************************************************/ STRING_STATIC(TIMELINE_CURRENT_STR, "current"); STRING_STATIC(TIMELINE_LATEST_STR, "latest"); FN_EXTERN void timelineVerify( const Storage *const storageRepo, const String *const archiveId, const unsigned int pgVersion, const unsigned int timelineBackup, const uint64_t lsnBackup, const String *timelineTargetStr, const unsigned int recoveryType, const CipherSpecMap *const cipherSpecArchive) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storageRepo); FUNCTION_LOG_PARAM(STRING, archiveId); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT, timelineBackup); FUNCTION_LOG_PARAM(UINT64, lsnBackup); FUNCTION_LOG_PARAM(STRING, timelineTargetStr); FUNCTION_LOG_PARAM(UINT, recoveryType); FUNCTION_LOG_PARAM(CIPHER_SPEC_MAP, cipherSpecArchive); FUNCTION_LOG_END(); ASSERT(storageRepo != NULL); ASSERT(archiveId != NULL); ASSERT(pgVersion != 0); ASSERT(timelineBackup > 0); ASSERT(lsnBackup > 0); // If timeline target is not specified then set based on the default by PostgreSQL version. Force timeline to current when the // recovery type is immediate since no timeline switch will be needed but PostgreSQL will error if there is a problem with the // latest timeline. if (timelineTargetStr == NULL) { if (pgVersion <= PG_VERSION_11 || recoveryType == CFGOPTVAL_RESTORE_TYPE_IMMEDIATE) timelineTargetStr = TIMELINE_CURRENT_STR; else timelineTargetStr = TIMELINE_LATEST_STR; } // If target timeline is not current then verify that the timeline is valid based on backup timeline and lsn. Following the // current/backup timeline is guaranteed as long as the WAL is available -- no history file required. if (!strEq(timelineTargetStr, TIMELINE_CURRENT_STR)) { MEM_CONTEXT_TEMP_BEGIN() { // Determine timeline target in order to load the history file. If target timeline is latest scan the history files to // find the most recent. If a target timeline is a number then parse it. unsigned int timelineTarget; if (strEq(timelineTargetStr, TIMELINE_LATEST_STR)) timelineTarget = timelineLatest(storageRepo, archiveId, timelineBackup); else { TRY_BEGIN() { if (strBeginsWithZ(timelineTargetStr, "0x")) timelineTarget = cvtZToUIntBase(strZ(timelineTargetStr) + 2, 16); else timelineTarget = cvtZToUInt(strZ(timelineTargetStr)); } CATCH_ANY() { THROW_FMT(DbMismatchError, "invalid target timeline '%s'", strZ(timelineTargetStr)); } TRY_END(); } // Only proceed if target timeline is not the backup timeline if (timelineTarget != timelineBackup) { // Search through the history for the target timeline to make sure it includes the backup timeline. This follows // the logic in PostgreSQL's readTimeLineHistory() and tliOfPointInHistory() but is a bit simplified for our usage. const List *const historyList = historyLoad(storageRepo, archiveId, timelineTarget, cipherSpecArchive); unsigned int timelineFound = 0; for (unsigned int historyIdx = 0; historyIdx < lstSize(historyList); historyIdx++) { const HistoryItem *const historyItem = lstGet(historyList, historyIdx); // If backup timeline exists in the target timeline's history before the fork LSN then restore can proceed if (historyItem->timeline == timelineBackup && lsnBackup < historyItem->lsn) { timelineFound = timelineBackup; break; } } // Error when the timeline was not found or not found in the expected range if (timelineFound == 0) { uint64_t lsnFound = 0; // Check if timeline exists but did not fork off when expected for (unsigned int historyIdx = 0; historyIdx < lstSize(historyList); historyIdx++) { const HistoryItem *const historyItem = lstGet(historyList, historyIdx); if (historyItem->timeline == timelineBackup) { timelineFound = timelineBackup; lsnFound = historyItem->lsn; } } // The timeline was found but it forked off earlier than the backup LSN. This covers the common case where a // standby is promoted by accident, which creates a new timeline, and then the user tries to restore a backup // from the original primary that was made after the new timeline was created. Since PostgreSQL >= 12 will // always try to follow the latest timeline this restore will fail since there is no path to the target // timeline. if (timelineFound != 0) { ASSERT(lsnFound != 0); THROW_FMT( DbMismatchError, "target timeline %X forked from backup timeline %X at %s which is before backup lsn of %s\n" "HINT: was the target timeline created by accidentally promoting a standby?\n" "HINT: was the target timeline created by testing a restore without --archive-mode=off?\n" "HINT: was the backup made after the target timeline was created?", timelineTarget, timelineBackup, strZ(pgLsnToStr(lsnFound)), strZ(pgLsnToStr(lsnBackup))); } // Else the backup timeline was not found at all. This less common case occurs when, for example, a restore from // timeline 4 results in timeline 7 being created on promotion because 5 and 6 have already been created by // prior promotions and then the user tries to restore a backup from timeline 5 with a target of timeline 7. // There is no relationship between timeline 5 and 7 in this case at any LSN range. else { THROW_FMT( DbMismatchError, "backup timeline %X, lsn %s is not in the history of target timeline %X\n" "HINT: was the target timeline created by promoting from a timeline < latest?", timelineBackup, strZ(pgLsnToStr(lsnBackup)), timelineTarget); } } } } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/timeline.h0000664000175100017510000000200715265101745021250 0ustar00runnerrunner/*********************************************************************************************************************************** Timeline Management ***********************************************************************************************************************************/ #ifndef COMMAND_RESTORE_TIMELINE_H #define COMMAND_RESTORE_TIMELINE_H #include "common/format/cipherSpecMap.h" #include "storage/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Verify that target timeline is valid for a backup FN_EXTERN void timelineVerify( const Storage *storageRepo, const String *archiveId, unsigned int pgVersion, unsigned int timelineBackup, uint64_t lsnBackup, const String *timelineTargetStr, unsigned int recoveryType, const CipherSpecMap *cipherSpecArchive); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/restore/validate.c.inc0000664000175100017510000000646015265101745022005 0ustar00runnerrunner/*********************************************************************************************************************************** Validate restore path ***********************************************************************************************************************************/ static void restorePathValidate(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { // PostgreSQL must not be running if (storageExistsP(storagePg(), PG_FILE_POSTMTRPID_STR)) { THROW_FMT( PgRunningError, "unable to restore while PostgreSQL is running\n" "HINT: presence of '" PG_FILE_POSTMTRPID "' in '%s' indicates PostgreSQL is running.\n" "HINT: remove '" PG_FILE_POSTMTRPID "' only if PostgreSQL is not running.", strZ(cfgOptionDisplay(cfgOptPgPath))); } // If the restore will be destructive attempt to verify that PGDATA looks like a valid PostgreSQL directory if ((cfgOptionBool(cfgOptDelta) || cfgOptionBool(cfgOptForce)) && !storageExistsP(storagePg(), PG_FILE_PGVERSION_STR) && !storageExistsP(storagePg(), BACKUP_MANIFEST_FILE_STR)) { LOG_WARN_FMT( "--delta or --force specified but unable to find '" PG_FILE_PGVERSION "' or '" BACKUP_MANIFEST_FILE "' in '%s' to" " confirm that this is a valid $PGDATA directory. --delta and --force have been disabled and if any files exist" " in the destination directories the restore will be aborted.", strZ(cfgOptionDisplay(cfgOptPgPath))); // Disable delta and force so restore will fail if the directories are not empty cfgOptionSet(cfgOptDelta, cfgSourceDefault, BOOL_FALSE_VAR); cfgOptionSet(cfgOptForce, cfgSourceDefault, BOOL_FALSE_VAR); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Validate the manifest ***********************************************************************************************************************************/ static void restoreManifestValidate(const Manifest *const manifest, const String *const backupSet) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(STRING, backupSet); FUNCTION_LOG_END(); ASSERT(manifest != NULL); ASSERT(backupSet != NULL); MEM_CONTEXT_TEMP_BEGIN() { // If there are no files in the manifest then something has gone horribly wrong CHECK(FormatError, manifestFileTotal(manifest) > 0, "manifest missing files"); // Sanity check to ensure the manifest has not been moved to a new directory const ManifestData *const data = manifestData(manifest); if (!strEq(data->backupLabel, backupSet)) { THROW_FMT( FormatError, "requested backup '%s' and manifest label '%s' do not match\n" "HINT: this indicates some sort of corruption (at the very least paths have been renamed).", strZ(backupSet), strZ(data->backupLabel)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/server/0000775000175100017510000000000015265101745017115 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/server/ping.c0000664000175100017510000000346415265101745020225 0ustar00runnerrunner/*********************************************************************************************************************************** Server Ping Command ***********************************************************************************************************************************/ #include #include "command/server/ping.h" #include "common/debug.h" #include "common/io/socket/client.h" #include "common/io/tls/client.h" #include "config/config.h" #include "protocol/client.h" #include "protocol/helper.h" FN_EXTERN void cmdServerPing(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { // Check for user-specified host const String *host = cfgOptionStr(cfgOptTlsServerAddress); const StringList *const commandParam = cfgCommandParam(); if (strLstSize(commandParam) == 1) host = strLstGet(commandParam, 0); else if (strLstSize(commandParam) > 1) THROW(ParamInvalidError, "extra parameters found"); // Connect to server without any verification const TimeMSec timeout = cfgOptionUInt64(cfgOptIoTimeout); IoClient *const tlsClient = tlsClientNewP( sckClientNew(host, cfgOptionUInt(cfgOptTlsServerPort), timeout, timeout), host, timeout, timeout, false); IoSession *const tlsSession = ioClientOpen(tlsClient); // Send ping ProtocolClient *const protocolClient = protocolClientNew( strNewFmt(PROTOCOL_SERVICE_REMOTE " socket protocol on '%s'", strZ(host)), PROTOCOL_SERVICE_REMOTE_STR, ioSessionIoReadP(tlsSession), ioSessionIoWrite(tlsSession)); protocolClientNoExit(protocolClient); protocolClientNoOp(protocolClient); protocolClientFree(protocolClient); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/server/ping.h0000664000175100017510000000123315265101745020222 0ustar00runnerrunner/*********************************************************************************************************************************** Server Ping Command ***********************************************************************************************************************************/ #ifndef COMMAND_SERVER_PING_H #define COMMAND_SERVER_PING_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN void cmdServerPing(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/server/server.c0000664000175100017510000002023215265101745020566 0ustar00runnerrunner/*********************************************************************************************************************************** Server Command ***********************************************************************************************************************************/ #include #include #ifdef HAVE_LIBSYSTEMD #include #endif #include "command/exit.h" #include "command/remote/remote.h" #include "command/server/server.h" #include "common/debug.h" #include "common/fork.h" #include "common/io/socket/server.h" #include "common/io/tls/server.h" #include "config/config.h" #include "config/load.h" #include "protocol/helper.h" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct ServerLocal { MemContext *memContext; // Mem context for server unsigned int argListSize; // Argument list size const char **argList; // Argument list List *processList; // List of child processes bool sigHup; // SIGHUP was caught bool sigTerm; // SIGTERM was caught IoServer *socketServer; // Socket server IoServer *tlsServer; // TLS server } serverLocal; /*********************************************************************************************************************************** Initialization can be redone when options change ***********************************************************************************************************************************/ static void cmdServerInit(void) { // Initialize mem context if (serverLocal.memContext == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(Server, .childQty = MEM_CONTEXT_QTY_MAX) { serverLocal.memContext = MEM_CONTEXT_NEW(); serverLocal.processList = lstNewP(sizeof(pid_t)); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); } MEM_CONTEXT_BEGIN(serverLocal.memContext) { // Free old servers ioServerFree(serverLocal.socketServer); ioServerFree(serverLocal.tlsServer); // Create new servers serverLocal.socketServer = sckServerNew( cfgOptionStr(cfgOptTlsServerAddress), cfgOptionUInt(cfgOptTlsServerPort), cfgOptionUInt64(cfgOptProtocolTimeout)); serverLocal.tlsServer = tlsServerNew( cfgOptionStr(cfgOptTlsServerAddress), cfgOptionStr(cfgOptTlsServerCaFile), cfgOptionStr(cfgOptTlsServerKeyFile), cfgOptionStr(cfgOptTlsServerCertFile), cfgOptionUInt64(cfgOptProtocolTimeout)); } MEM_CONTEXT_END(); } /*********************************************************************************************************************************** Handlers to set flags on signals ***********************************************************************************************************************************/ static void cmdServerSigHup(const int signalType) { (void)signalType; serverLocal.sigHup = true; } static void cmdServerSigTerm(const int signalType) { (void)signalType; serverLocal.sigTerm = true; } /*********************************************************************************************************************************** Handler to reap child processes ***********************************************************************************************************************************/ static void cmdServerSigChild(const int signalType, siginfo_t *const signalInfo, void *const context) { (void)signalType; (void)context; // Find the process and remove it for (unsigned int processIdx = 0; processIdx < lstSize(serverLocal.processList); processIdx++) { if (*(int *)lstGet(serverLocal.processList, processIdx) == signalInfo->si_pid) { ASSERT(signalInfo->si_code == CLD_EXITED); lstRemoveIdx(serverLocal.processList, processIdx); } } } /**********************************************************************************************************************************/ FN_EXTERN void cmdServer(const unsigned int argListSize, const char *argList[]) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, argListSize); FUNCTION_LOG_PARAM(CHARPY, argList); FUNCTION_LOG_END(); // Initialize server cmdServerInit(); // Set arguments used for reload serverLocal.argListSize = argListSize; serverLocal.argList = argList; MEM_CONTEXT_TEMP_BEGIN() { // Set signal handlers sigaction(SIGHUP, &(struct sigaction){.sa_handler = cmdServerSigHup}, NULL); sigaction(SIGTERM, &(struct sigaction){.sa_handler = cmdServerSigTerm}, NULL); sigaction( SIGCHLD, &(struct sigaction){.sa_sigaction = cmdServerSigChild, .sa_flags = SA_NOCLDSTOP | SA_NOCLDWAIT | SA_SIGINFO}, NULL); // Notify systemd that we are ready to accept connections #ifdef HAVE_LIBSYSTEMD sd_notify(0, "READY=1"); #endif // Accept connections indefinitely. The only way to exit this loop is for the process to receive a signal. do { // Accept a new connection IoSession *const socketSession = ioServerAccept(serverLocal.socketServer, NULL); if (socketSession != NULL) { // Fork off the child process pid_t pid = forkSafe(); if (pid == 0) { // Reset SIGCHLD to default sigaction(SIGCHLD, &(struct sigaction){.sa_handler = SIG_DFL}, NULL); // Set standard signal handlers exitInit(); // Close the server socket so we don't hold the port open if the parent exits first ioServerFree(serverLocal.socketServer); // Disable logging and close log file logClose(); // Start standard remote processing if a server is returned ProtocolServer *server = protocolServer(serverLocal.tlsServer, socketSession); if (server != NULL) cmdRemote(server); break; } // Add process to list else lstAdd(serverLocal.processList, &pid); // Free the socket since the child is now using it ioSessionFree(socketSession); } // Reload configuration if (serverLocal.sigHup) { LOG_DETAIL("configuration reload begin"); // Reload configuration cfgLoad(serverLocal.argListSize, serverLocal.argList); // Reinitialize server cmdServerInit(); LOG_DETAIL("configuration reload end"); // Reset flag serverLocal.sigHup = false; } } while (!serverLocal.sigTerm); } MEM_CONTEXT_TEMP_END(); // Notify systemd that we are shutting down #ifdef HAVE_LIBSYSTEMD sd_notify(0, "STOPPING=1"); #endif // Terminate any remaining children on SIGTERM. Disable the callback so it does not fire in the middle of the loop. if (serverLocal.sigTerm) { sigaction(SIGCHLD, &(struct sigaction){.sa_flags = SA_NOCLDSTOP | SA_NOCLDWAIT}, NULL); for (unsigned int processIdx = 0; processIdx < lstSize(serverLocal.processList); processIdx++) { pid_t pid = *(int *)lstGet(serverLocal.processList, processIdx); LOG_WARN_FMT("terminate child process %d", pid); kill(pid, SIGTERM); } } FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/server/server.h0000664000175100017510000000133015265101745020571 0ustar00runnerrunner/*********************************************************************************************************************************** Server Command ***********************************************************************************************************************************/ #ifndef COMMAND_SERVER_SERVER_H #define COMMAND_SERVER_SERVER_H #include /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN void cmdServer(unsigned int argListSize, const char *argList[]); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/stanza/0000775000175100017510000000000015265101745017107 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/stanza/common.c0000664000175100017510000000603715265101745020551 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Commands Handler ***********************************************************************************************************************************/ #include #include "command/check/common.h" #include "command/stanza/common.h" #include "common/debug.h" #include "common/format/format.h" #include "common/log.h" #include "config/config.h" #include "db/helper.h" #include "info/infoPg.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN CipherSpec * cipherSpecGen(const CipherType cipherType, const unsigned int format) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, cipherType); FUNCTION_TEST_PARAM(UINT, format); FUNCTION_TEST_END(); CipherSpec *result; if (cipherType == cipherTypeNone) result = cipherSpecNewNone(); else { MEM_CONTEXT_TEMP_BEGIN() { uint8_t buffer[48]; // 48 is the amount of entropy needed to get a 64 base key cryptoRandomBytes(buffer, sizeof(buffer)); // The pass is the base64 text rather than the bytes it encodes, since the text is what gets stored and what the key // derives from. A new pass derives with the digest its format calls for and that digest is stored with it, so a // reader derives the same key. result = cipherSpecNewP( cipherType, BUFSTR(strNewEncode(encodingBase64, BUF(buffer, sizeof(buffer)))), .digest = repoFormatDigest(format)); cipherSpecMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN(CIPHER_SPEC, result); } /**********************************************************************************************************************************/ FN_EXTERN PgControl pgValidate(void) { FUNCTION_TEST_VOID(); PgControl result = {0}; MEM_CONTEXT_TEMP_BEGIN() { if (cfgOptionBool(cfgOptOnline)) { // Check the primary connections (and standby, if any) and return the primary database object. const DbGetResult dbObject = dbGet(false, true, CFGOPTVAL_BACKUP_STANDBY_N); // Get the pgControl information from the pg*-path deemed to be the primary result = dbPgControl(dbObject.primary); // Check the user configured path and version against the database checkDbConfig(dbPgControl(dbObject.primary).version, dbObject.primaryIdx, dbObject.primary, false); } // If the database is not online, assume that pg1 is the primary else result = pgControlFromFile(storagePg(), cfgOptionStrNull(cfgOptPgVersionForce)); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_TYPE(PgControl, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/stanza/common.h0000664000175100017510000000173415265101745020555 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Commands Handler ***********************************************************************************************************************************/ #ifndef COMMAND_STANZA_COMMON_H #define COMMAND_STANZA_COMMON_H #include "info/infoPg.h" #include "postgres/interface.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Generate a sub pass for a file at this format, which sets the digest since that is what will derive it when it is read back FN_EXTERN CipherSpec *cipherSpecGen(CipherType cipherType, unsigned int format); // Validate and return database information FN_EXTERN PgControl pgValidate(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/stanza/create.c0000664000175100017510000002135215265101745020521 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Create Command ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/check/common.h" #include "command/control/common.h" #include "command/stanza/common.h" #include "command/stanza/create.h" #include "common/debug.h" #include "common/format/format.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "info/infoPg.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdStanzaCreate(void) { FUNCTION_LOG_VOID(logLevelDebug); // Verify that a stop was not issued before proceeding lockStopTest(); MEM_CONTEXT_TEMP_BEGIN() { if (cfgOptionBool(cfgOptForce)) LOG_WARN("option --" CFGOPT_FORCE " is no longer supported"); // Get the version and system information - validating it if the database is online const PgControl pgControl = pgValidate(); // For each repository configured, create the stanza for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { LOG_INFO_FMT( CFGCMD_STANZA_CREATE " for stanza '%s' on %s", strZ(cfgOptionDisplay(cfgOptStanza)), cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); const Storage *const storageRepoReadStanza = storageRepoIdx(repoIdx); const Storage *const storageRepoWriteStanza = storageRepoIdxWrite(repoIdx); InfoArchive *infoArchive = NULL; InfoBackup *infoBackup = NULL; const bool archiveInfoFileExists = storageExistsP(storageRepoReadStanza, INFO_ARCHIVE_PATH_FILE_STR); const bool archiveInfoFileCopyExists = storageExistsP(storageRepoReadStanza, INFO_ARCHIVE_PATH_FILE_COPY_STR); const bool backupInfoFileExists = storageExistsP(storageRepoReadStanza, INFO_BACKUP_PATH_FILE_STR); const bool backupInfoFileCopyExists = storageExistsP(storageRepoReadStanza, INFO_BACKUP_PATH_FILE_COPY_STR); // If neither archive info nor backup info files exist and nothing else exists in the stanza directory // then create the stanza if (!archiveInfoFileExists && !archiveInfoFileCopyExists && !backupInfoFileExists && !backupInfoFileCopyExists) { const bool archiveNotEmpty = strLstSize( storageListP(storageRepoReadStanza, STORAGE_REPO_ARCHIVE_STR)) > 0 ? true : false; const bool backupNotEmpty = strLstSize( storageListP(storageRepoReadStanza, STORAGE_REPO_BACKUP_STR)) > 0 ? true : false; // If something else exists in the backup or archive directories for this stanza, then error if (archiveNotEmpty || backupNotEmpty) { THROW_FMT( PathNotEmptyError, "%s%s%snot empty", (backupNotEmpty ? "backup directory " : ""), (backupNotEmpty && archiveNotEmpty ? "and/or " : ""), (archiveNotEmpty ? "archive directory " : "")); } // Format for the new stanza const unsigned int format = cfgOptionIdxUInt(cfgOptRepoFormat, repoIdx); // If the repo is encrypted, generate a cipher passphrase for encrypting archive files. Format >= 6 adds it by // rotating to the first key id and earlier formats store it as the default key with no id. const CipherSpec *const cipherSpecArchive = cipherSpecGen(cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), format); CipherSpecMap *const cipherSpecMapArchive = cipherSpecMapNew(); if (cipherSpecType(cipherSpecArchive) != cipherTypeNone && format < REPOSITORY_FORMAT_6) cipherSpecMapAdd(cipherSpecMapArchive, CIPHER_SPEC_MAP_ID_DEFAULT_STR, cipherSpecArchive); // Create and save archive info infoArchive = infoArchiveNew(pgControl.version, pgControl.systemId, format, cipherSpecMapArchive); if (cipherSpecType(cipherSpecArchive) != cipherTypeNone && format >= REPOSITORY_FORMAT_6) infoArchiveCipherRotate(infoArchive, cipherSpecArchive, time(NULL)); infoArchiveSaveFile(infoArchive, storageRepoWriteStanza, INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); // If the repo is encrypted, generate a cipher passphrase for encrypting subsequent manifests const CipherSpec *const cipherSpecManifest = cipherSpecGen( cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), format); // Create and save backup info infoBackup = infoBackupNew( pgControl.version, pgControl.systemId, pgControl.catalogVersion, format, cipherSpecManifest); infoBackupSaveFile(infoBackup, storageRepoWriteStanza, INFO_BACKUP_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); } // Else if at least one archive and one backup info file exists, then ensure both are valid else if ((archiveInfoFileExists || archiveInfoFileCopyExists) && (backupInfoFileExists || backupInfoFileCopyExists)) { // Error if there is a mismatch between the archive and backup info files or the database version/system Id matches // current database checkStanzaInfoPg( repoIdx, storageRepoReadStanza, pgControl.version, pgControl.systemId, cfgCipherSpecMainIdx(repoIdx)); // The files are valid - upgrade const String *sourceFile = NULL; const String *destinationFile = NULL; // If the existing files are valid, then, if a file is missing, copy the existing one to the missing one to ensure // there is both a .info and .info.copy if (!archiveInfoFileExists || !archiveInfoFileCopyExists) { sourceFile = archiveInfoFileExists ? INFO_ARCHIVE_PATH_FILE_STR : INFO_ARCHIVE_PATH_FILE_COPY_STR; destinationFile = !archiveInfoFileExists ? INFO_ARCHIVE_PATH_FILE_STR : INFO_ARCHIVE_PATH_FILE_COPY_STR; // Using get and put instead of copy in case the storage is remote storagePutP( storageNewWriteP(storageRepoWriteStanza, destinationFile), storageGetP(storageNewReadP(storageRepoReadStanza, sourceFile))); } if (!backupInfoFileExists || !backupInfoFileCopyExists) { sourceFile = backupInfoFileExists ? INFO_BACKUP_PATH_FILE_STR : INFO_BACKUP_PATH_FILE_COPY_STR; destinationFile = !backupInfoFileExists ? INFO_BACKUP_PATH_FILE_STR : INFO_BACKUP_PATH_FILE_COPY_STR; // Using get and put instead of copy in case the storage is remote storagePutP( storageNewWriteP(storageRepoWriteStanza, destinationFile), storageGetP(storageNewReadP(storageRepoReadStanza, sourceFile))); } // If no files copied, then the stanza was already valid if (sourceFile == NULL) { LOG_INFO_FMT( "stanza '%s' already exists on %s and is valid", strZ(cfgOptionDisplay(cfgOptStanza)), cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); } } // Else if both .info and corresponding .copy file are missing for one but not the other, then error - the user will // have to make a conscious effort to determine if deleting the stanza on this repo is appropriate or other action is else { THROW_FMT( FileMissingError, "%s on %s\n" "HINT: this may be a symptom of repository corruption!", (archiveInfoFileExists || archiveInfoFileCopyExists) ? "archive.info exists but backup.info is missing" : "backup.info exists but archive.info is missing", cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/stanza/create.h0000664000175100017510000000127415265101745020527 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Create Command ***********************************************************************************************************************************/ #ifndef COMMAND_STANZA_CREATE_H #define COMMAND_STANZA_CREATE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process stanza-create FN_EXTERN void cmdStanzaCreate(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/stanza/delete.c0000664000175100017510000001066315265101745020523 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Delete Command ***********************************************************************************************************************************/ #include #include "command/backup/common.h" #include "command/control/common.h" #include "command/stanza/delete.h" #include "common/debug.h" #include "common/memContext.h" #include "config/config.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdStanzaDelete(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { const StringList *const archiveList = storageListP(storageRepo(), STORAGE_REPO_ARCHIVE_STR, .nullOnMissing = true); const StringList *const backupList = storageListP(storageRepo(), STORAGE_REPO_BACKUP_STR, .nullOnMissing = true); // For most drivers, NULL indicates the directory does not exist at all. For those that do not support paths (e.g. S3) an // empty StringList will be returned; in such a case, the directory will attempt to be deleted (this is OK). if (archiveList != NULL || backupList != NULL) { const bool archiveNotEmpty = (archiveList != NULL && !strLstEmpty(archiveList)) ? true : false; const bool backupNotEmpty = (backupList != NULL && !strLstEmpty(backupList)) ? true : false; // If something exists in either directory, then remove if (archiveNotEmpty || backupNotEmpty) { // If the stop file does not exist, then error. This check is required even when --force is issued. if (!storageExistsP(storageLocal(), lockStopFileName(cfgOptionStr(cfgOptStanza)))) { THROW_FMT( FileMissingError, "stop file does not exist for stanza '%s'\n" "HINT: has the pgbackrest stop command been run on this server for this stanza?", strZ(cfgOptionDisplay(cfgOptStanza))); } // If a force has not been issued and Postgres is running, then error if (!cfgOptionBool(cfgOptForce) && storageExistsP(storagePg(), STRDEF(PG_FILE_POSTMTRPID))) { THROW_FMT( PgRunningError, PG_FILE_POSTMTRPID " exists - looks like " PG_NAME " is running. " "To delete stanza '%s' on %s, shut down " PG_NAME " for stanza '%s' and try again, or use --force.", strZ(cfgOptionDisplay(cfgOptStanza)), cfgOptionGroupName(cfgOptGrpRepo, cfgOptionGroupIdxDefault(cfgOptGrpRepo)), strZ(cfgOptionDisplay(cfgOptStanza))); } // Delete the archive info files if (archiveNotEmpty) { storageRemoveP(storageRepoWrite(), INFO_ARCHIVE_PATH_FILE_STR); storageRemoveP(storageRepoWrite(), INFO_ARCHIVE_PATH_FILE_COPY_STR); } // Delete the backup info files if (backupNotEmpty) { storageRemoveP(storageRepoWrite(), INFO_BACKUP_PATH_FILE_STR); storageRemoveP(storageRepoWrite(), INFO_BACKUP_PATH_FILE_COPY_STR); } } // Recursively remove the entire stanza repo if exists. S3 will attempt to remove even if not. if (archiveList != NULL) storagePathRemoveP(storageRepoWrite(), STORAGE_REPO_ARCHIVE_STR, .recurse = true); if (backupList != NULL) storagePathRemoveP(storageRepoWrite(), STORAGE_REPO_BACKUP_STR, .recurse = true); // Remove the stop file - this will not error if the stop file does not exist. If the stanza directories existed but // nothing was in them, then no pgbackrest commands can be in progress without the info files so a stop is technically // not necessary storageRemoveP(storageLocalWrite(), lockStopFileName(cfgOptionStr(cfgOptStanza))); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/stanza/delete.h0000664000175100017510000000127415265101745020526 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Delete Command ***********************************************************************************************************************************/ #ifndef COMMAND_STANZA_DELETE_H #define COMMAND_STANZA_DELETE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process stanza-delete FN_EXTERN void cmdStanzaDelete(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/stanza/upgrade.c0000664000175100017510000001720315265101745020705 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Upgrade Command ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/check/common.h" #include "command/control/common.h" #include "command/stanza/common.h" #include "command/stanza/upgrade.h" #include "common/debug.h" #include "common/format/format.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "info/infoPg.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdStanzaUpgrade(void) { FUNCTION_LOG_VOID(logLevelDebug); // Verify that a stop was not issued before proceeding lockStopTest(); MEM_CONTEXT_TEMP_BEGIN() { // Get the version and system information - validating it if the database is online PgControl pgControl = pgValidate(); // For each repository configured, upgrade the stanza for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { LOG_INFO_FMT( CFGCMD_STANZA_UPGRADE " for stanza '%s' on %s", strZ(cfgOptionDisplay(cfgOptStanza)), cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); const Storage *const storageRepoReadStanza = storageRepoIdx(repoIdx); const Storage *const storageRepoWriteStanza = storageRepoIdxWrite(repoIdx); bool infoArchiveUpgrade = false; bool infoBackupUpgrade = false; // Load the info files (errors if missing) InfoArchive *const infoArchive = infoArchiveLoadFile( storageRepoReadStanza, INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); InfoPgData archiveInfo = infoPgData(infoArchivePg(infoArchive), infoPgDataCurrentId(infoArchivePg(infoArchive))); InfoBackup *const infoBackup = infoBackupLoadFile( storageRepoReadStanza, INFO_BACKUP_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); InfoPgData backupInfo = infoPgData(infoBackupPg(infoBackup), infoPgDataCurrentId(infoBackupPg(infoBackup))); // Determine the format to write. An upgrade interrupted between the two info file saves leaves them at different // formats and the higher of the two is the only target that does not downgrade a file, so it is written to both even // when no format was requested. A format that is not already in the repository must still be requested, since the // option default would otherwise downgrade a repository that has already been upgraded. const unsigned int formatArchive = infoArchiveFormat(infoArchive); const unsigned int formatBackup = infoBackupFormat(infoBackup); unsigned int format = formatArchive > formatBackup ? formatArchive : formatBackup; if (cfgOptionIdxSource(cfgOptRepoFormat, repoIdx) != cfgSourceDefault) { const unsigned int formatRequest = cfgOptionIdxUInt(cfgOptRepoFormat, repoIdx); // Error when the format would be downgraded. Backups and archives written at a newer format would no longer be // gated by the info files, so an older version could read the info files and then fail on newer files. if (formatRequest < format) { THROW_FMT( FormatError, "unable to downgrade repository format from %u to %u\n" "HINT: backups and archives already written at format %u would not be readable by a version that only" " supports format %u.", format, formatRequest, format, formatRequest); } format = formatRequest; } // Since the file save of archive.info and backup.info are not atomic, then check and update each separately. // Update archive if (pgControl.version != archiveInfo.version || pgControl.systemId != archiveInfo.systemId) { infoArchivePgSet(infoArchive, pgControl.version, pgControl.systemId); infoArchiveUpgrade = true; } // Update backup if (pgControl.version != backupInfo.version || pgControl.systemId != backupInfo.systemId) { infoBackupPgSet(infoBackup, pgControl.version, pgControl.systemId, pgControl.catalogVersion); infoBackupUpgrade = true; } // Update the format on both info files together so they never disagree if (format != formatArchive || format != formatBackup) { // Report a repository that was found at two formats. It is repaired here but a prior upgrade did not finish, which // the user has not been told about since the run it happened on did not get far enough to report it. if (formatArchive != formatBackup) LOG_WARN("repository format mismatch from an interrupted " CFGCMD_STANZA_UPGRADE " will be repaired"); // Log the format the repository is migrating from, which is the lower of the two when an interrupted upgrade left // them at different formats. This cannot be undone and a version that does not support the new format will no // longer be able to read the stanza, so report it rather than migrating silently. LOG_INFO_FMT( "upgrade repository format from %u to %u", formatArchive < formatBackup ? formatArchive : formatBackup, format); infoArchiveFormatSet(infoArchive, format); infoBackupFormatSet(infoBackup, format); // A stanza migrated to format 6 keeps the key it already has as id 0 and adds a key for new WAL using rotation if (formatArchive < REPOSITORY_FORMAT_6) { const CipherType cipherType = cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx); if (cipherType != cipherTypeNone) infoArchiveCipherRotate(infoArchive, cipherSpecGen(cipherType, format), time(NULL)); } infoArchiveUpgrade = true; infoBackupUpgrade = true; } // Throw an error if the info files do not match before saving (even if only one needed to be updated) checkStanzaInfo(repoIdx, infoArchivePg(infoArchive), infoBackupPg(infoBackup)); // Save archive info if (infoArchiveUpgrade) infoArchiveSaveFile(infoArchive, storageRepoWriteStanza, INFO_ARCHIVE_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); // Save backup info if (infoBackupUpgrade) infoBackupSaveFile(infoBackup, storageRepoWriteStanza, INFO_BACKUP_PATH_FILE_STR, cfgCipherSpecMainIdx(repoIdx)); if (!(infoArchiveUpgrade || infoBackupUpgrade)) { LOG_INFO_FMT( "stanza '%s' on %s is already up to date", strZ(cfgOptionDisplay(cfgOptStanza)), cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/stanza/upgrade.h0000664000175100017510000000130115265101745020702 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Upgrade Command ***********************************************************************************************************************************/ #ifndef COMMAND_STANZA_UPGRADE_H #define COMMAND_STANZA_UPGRADE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process stanza-upgrade FN_EXTERN void cmdStanzaUpgrade(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/verify/0000775000175100017510000000000015265101745017113 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/verify/file.c0000664000175100017510000001000615265101745020173 0ustar00runnerrunner/*********************************************************************************************************************************** Verify File ***********************************************************************************************************************************/ #include #include "command/verify/file.h" #include "common/crypto/cipherBlock.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/format/cipherBlockFormat.h" #include "common/io/filter/group.h" #include "common/io/filter/sink.h" #include "common/io/filter/size.h" #include "common/io/io.h" #include "common/log.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN VerifyResult verifyFile( const String *const filePathName, const uint64_t offset, const Variant *const limit, const CompressType compressType, const Buffer *const fileChecksum, const uint64_t fileSize, const CipherSpecMap *const cipherSpecMap, const bool formatHeader) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, filePathName); // Fully qualified file name FUNCTION_LOG_PARAM(UINT64, offset); // Offset to read in file FUNCTION_LOG_PARAM(VARIANT, limit); // Limit to read from file FUNCTION_LOG_PARAM(ENUM, compressType); // Compression type FUNCTION_LOG_PARAM(BUFFER, fileChecksum); // Checksum for the file FUNCTION_LOG_PARAM(UINT64, fileSize); // Size of file FUNCTION_LOG_PARAM(CIPHER_SPEC_MAP, cipherSpecMap); // Cipher keys to access the repo file if encrypted FUNCTION_LOG_PARAM(BOOL, formatHeader); // May the file begin with a format header? FUNCTION_LOG_END(); ASSERT(filePathName != NULL); ASSERT(fileChecksum != NULL); ASSERT(limit == NULL || varType(limit) == varTypeUInt64); // Is the file valid? VerifyResult result = verifyOk; MEM_CONTEXT_TEMP_BEGIN() { // Prepare the file for reading IoRead *const read = storageReadIo( storageNewReadP(storageRepo(), filePathName, .ignoreMissing = true, .offset = offset, .limit = limit)); IoFilterGroup *const filterGroup = ioReadFilterGroup(read); // Add decryption filter if (cipherSpecMapSize(cipherSpecMap) != 0) { // WAL contains a format header with the key id. Backup files contain no header so their key is under the default id. if (formatHeader) cipherBlockFormatFilterGroupReadAddMap(filterGroup, cipherSpecMap); else { cipherBlockFilterGroupAdd( filterGroup, cipherModeDecrypt, cipherSpecMapGet(cipherSpecMap, CIPHER_SPEC_MAP_ID_DEFAULT_STR)); } } // Add decompression filter if (compressType != compressTypeNone) ioFilterGroupAdd(filterGroup, decompressFilterP(compressType)); // Add sha1 filter ioFilterGroupAdd(filterGroup, cryptoHashNew(hashTypeSha1)); // Add size filter ioFilterGroupAdd(filterGroup, ioSizeNew()); // Add IoSink so the file data is not transmitted from the remote ioFilterGroupAdd(filterGroup, ioSinkNew()); // If the file exists check the checksum/size if (ioReadDrain(read)) { // Validate checksum if (!bufEq(fileChecksum, pckReadBinP(ioFilterGroupResultP(filterGroup, CRYPTO_HASH_FILTER_TYPE)))) { result = verifyChecksumMismatch; } // If the size can be checked, do so else if (fileSize != pckReadU64P(ioFilterGroupResultP(ioReadFilterGroup(read), SIZE_FILTER_TYPE))) result = verifySizeInvalid; } else result = verifyFileMissing; } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/verify/file.h0000664000175100017510000000272115265101745020205 0ustar00runnerrunner/*********************************************************************************************************************************** Verify File ***********************************************************************************************************************************/ #ifndef COMMAND_VERIFY_FILE_H #define COMMAND_VERIFY_FILE_H #include "common/compress/helper.h" #include "common/format/cipherSpecMap.h" /*********************************************************************************************************************************** File result ***********************************************************************************************************************************/ typedef enum { verifyOk, // Default result - file OK verifyFileMissing, verifyChecksumMismatch, verifySizeInvalid, verifyOtherError, } VerifyResult; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Verify a file in the pgBackRest repository FN_EXTERN VerifyResult verifyFile( const String *filePathName, uint64_t offset, const Variant *limit, CompressType compressType, const Buffer *fileChecksum, uint64_t fileSize, const CipherSpecMap *cipherSpecMap, bool formatHeader); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/verify/protocol.c0000664000175100017510000000362015265101745021121 0ustar00runnerrunner/*********************************************************************************************************************************** Verify Protocol Handler ***********************************************************************************************************************************/ #include #include "command/verify/file.h" #include "command/verify/protocol.h" #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * verifyFileProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Verify file const String *const filePathName = pckReadStrP(param); uint64_t offset = 0; const Variant *limit = NULL; if (pckReadBoolP(param)) { offset = pckReadU64P(param); limit = varNewUInt64(pckReadU64P(param)); } const CompressType compressType = (CompressType)pckReadU32P(param); const Buffer *const fileChecksum = pckReadBinP(param); const uint64_t fileSize = pckReadU64P(param); const CipherSpecMap *const cipherSpecMap = cipherSpecMapNewPack(param); const bool formatHeader = pckReadBoolP(param); // Return result pckWriteU32P( protocolServerResultData(result), verifyFile(filePathName, offset, limit, compressType, fileChecksum, fileSize, cipherSpecMap, formatHeader)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/verify/protocol.h0000664000175100017510000000266215265101745021133 0ustar00runnerrunner/*********************************************************************************************************************************** Verify Protocol Handler ***********************************************************************************************************************************/ #ifndef COMMAND_VERIFY_PROTOCOL_H #define COMMAND_VERIFY_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process protocol requests FN_EXTERN ProtocolServerResult *verifyFileProtocol(PackRead *param); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_VERIFY_FILE STRID5("vf-f", 0x36cd60) #define PROTOCOL_SERVER_HANDLER_VERIFY_LIST \ {.command = PROTOCOL_COMMAND_VERIFY_FILE, .process = verifyFileProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/verify/verify.c0000664000175100017510000026420615265101745020575 0ustar00runnerrunner/*********************************************************************************************************************************** Verify Command Verify contents of the repository. ***********************************************************************************************************************************/ #include #include #include #include #include "command/archive/common.h" #include "command/check/common.h" #include "command/verify/file.h" #include "command/verify/protocol.h" #include "command/verify/verify.h" #include "common/compress/helper.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/format/cipherBlockFormat.h" #include "common/io/fdWrite.h" #include "common/io/io.h" #include "common/log.h" #include "common/regExp.h" #include "config/config.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "info/manifest/manifest.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "protocol/parallel.h" #include "storage/helper.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define VERIFY_STATUS_OK "ok" #define VERIFY_STATUS_ERROR "error" /*********************************************************************************************************************************** Data Types and Structures ***********************************************************************************************************************************/ #define FUNCTION_LOG_VERIFY_ARCHIVE_RESULT_TYPE \ VerifyArchiveResult #define FUNCTION_LOG_VERIFY_ARCHIVE_RESULT_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "VerifyArchiveResult", buffer, bufferSize) #define FUNCTION_LOG_VERIFY_BACKUP_RESULT_TYPE \ VerifyBackupResult #define FUNCTION_LOG_VERIFY_BACKUP_RESULT_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "VerifyBackupResult", buffer, bufferSize) // Structure for verifying repository info files typedef struct VerifyInfoFile { InfoBackup *backup; // Backup.info file contents InfoArchive *archive; // Archive.info file contents Manifest *manifest; // Manifest file contents const String *checksum; // File checksum int errorCode; // Error code else 0 for no error } VerifyInfoFile; // Job data results structures for archive and backup typedef struct VerifyArchiveResult { String *archiveId; // Archive Id (e.g. 9.6-1, 10-2) unsigned int totalWalFile; // Total number of WAL files listed in directory on first read unsigned int totalValidWal; // Total number of WAL that were verified and valid PgWal pgWalInfo; // PG version, WAL size, system id List *walRangeList; // List of WAL file ranges - new item is when WAL is missing } VerifyArchiveResult; // WAL range includes the start/stop of sequential WAL and start/stop includes the timeline (e.g. 000000020000000100000005) typedef struct VerifyWalRange { String *stop; // Last WAL segment in this sequential range String *start; // First WAL segment in this sequential range List *invalidFileList; // After all jobs complete, list of VerifyInvalidFile } VerifyWalRange; // Invalid file information (not missing but files failing verification) - for archive and backup typedef struct VerifyInvalidFile { String *fileName; // Name of the file (includes path within the stanza) VerifyResult reason; // Reason file is invalid (e.g. incorrect checksum) } VerifyInvalidFile; // Status result of a backup typedef enum { backupValid, // Default: All files in backup label repo passed verification backupInvalid, // One of more files in backup label repo failed verification backupMissingManifest, // Backup manifest missing (backup may have expired) backupInProgress, // Backup appeared to be in progress (so was skipped) } VerifyBackupResultStatus; typedef struct VerifyBackupResult { String *backupLabel; // Label assigned to the backup VerifyBackupResultStatus status; // Final status of the backup bool fileVerifyComplete; // Have all the files of the backup completed verification? unsigned int totalFileManifest; // Total number of backup files in the manifest unsigned int totalFileVerify; // Total number of backup files being verified unsigned int totalFileValid; // Total number of backup files that were verified and valid String *backupPrior; // Prior backup that this backup depends on, if any unsigned int pgId; // PG id will be used to find WAL for the backup in the repo unsigned int pgVersion; // PG version will be used with PG id to find WAL in the repo String *archiveStart; // First WAL segment in the backup String *archiveStop; // Last WAL segment in the backup List *invalidFileList; // List of invalid files found in the backup } VerifyBackupResult; // Job data structure for processing and results collection typedef struct VerifyJobData { MemContext *memContext; // Context for memory allocations in this struct StringList *archiveIdList; // List of archive ids to verify StringList *walPathList; // WAL path list for a single archive id StringList *walFileList; // WAL file list for a single WAL path StringList *backupList; // List of backups to verify Manifest *manifest; // Manifest contents with list of files to verify unsigned int manifestFileIdx; // Index of the file within the manifest file list to process String *currentBackup; // In progress backup, if any const InfoPg *pgHistory; // Database history list bool backupProcessing; // Are we processing WAL or are we processing backups const CipherSpec *cipherSpecManifest; // Cipher spec for reading backup manifests const CipherSpecMap *cipherSpecArchive; // Cipher keys for reading WAL files unsigned int jobErrorTotal; // Total errors that occurred during the job execution List *archiveIdResultList; // Archive results List *backupResultList; // Backup results bool enableArchiveFilter; // Only check archives in the specified range const String *archiveStart; // Start of the WAL range to be verified const String *archiveStop; // End of the WAL range to be verified } VerifyJobData; /*********************************************************************************************************************************** Helper function to add a file to an invalid file list ***********************************************************************************************************************************/ static void verifyInvalidFileAdd(List *const invalidFileList, const VerifyResult reason, const String *const fileName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, invalidFileList); // Invalid file list to add the filename to FUNCTION_TEST_PARAM(ENUM, reason); // Reason for invalid file FUNCTION_TEST_PARAM(STRING, fileName); // Name of invalid file FUNCTION_TEST_END(); ASSERT(invalidFileList != NULL); ASSERT(fileName != NULL); MEM_CONTEXT_BEGIN(lstMemContext(invalidFileList)) { const VerifyInvalidFile invalidFile = { .fileName = strDup(fileName), .reason = reason, }; lstAdd(invalidFileList, &invalidFile); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Get status of info files in the repository ***********************************************************************************************************************************/ static VerifyInfoFile verifyInfoFile(const String *const pathFileName, const bool keepFile, const CipherSpec *const cipherSpec) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, pathFileName); // Fully qualified path/file name FUNCTION_LOG_PARAM(BOOL, keepFile); // Should the file be kept in memory? FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); // Cipher spec to open file if encrypted FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(pathFileName != NULL); VerifyInfoFile result = {.errorCode = 0}; MEM_CONTEXT_TEMP_BEGIN() { TRY_BEGIN() { StorageRead *const infoStorageRead = storageNewReadP(storageRepo(), pathFileName); IoRead *const infoRead = storageReadIo(infoStorageRead); ioFilterGroupAdd(ioReadFilterGroup(infoRead), cryptoHashNew(hashTypeSha1)); const bool isBackup = strBeginsWith(pathFileName, INFO_BACKUP_PATH_FILE_STR); const bool isArchive = strBeginsWith(pathFileName, INFO_ARCHIVE_PATH_FILE_STR); // Add decryption after the hash so the checksum is over the file as stored. An info file is read through the filter // that reads its header, a manifest with the spec as it was given since it has no header of its own. if (isBackup || isArchive) cipherBlockFormatFilterGroupReadAdd(ioReadFilterGroup(infoRead), cipherSpec); else cipherBlockFilterGroupAdd(ioReadFilterGroup(infoRead), cipherModeDecrypt, cipherSpec); // If directed to keep the loaded file in memory, then move the file into the result, else drain the io and close it if (keepFile) { if (isBackup) result.backup = infoBackupMove(infoBackupNewLoad(infoRead, cipherSpec), memContextPrior()); else if (isArchive) result.archive = infoArchiveMove(infoArchiveNewLoad(infoRead, cipherSpec), memContextPrior()); else result.manifest = manifestMove(manifestNewLoad(infoRead, cipherSpec), memContextPrior()); } // Else nothing needs the file, so the drain runs the filters else ioReadDrain(infoRead); const Buffer *const filterResult = pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(infoRead), CRYPTO_HASH_FILTER_TYPE)); MEM_CONTEXT_PRIOR_BEGIN() { result.checksum = strNewEncode(encodingHex, filterResult); } MEM_CONTEXT_PRIOR_END(); } CATCH_ANY() { result.errorCode = errorCode(); String *const errorMsg = strCatZ(strNew(), errorMessage()); if (result.errorCode == errorTypeCode(&ChecksumError)) strCat(errorMsg, strNewFmt(" %s", strZ(pathFileName))); LOG_DETAIL(strZ(errorMsg)); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /*********************************************************************************************************************************** Get the archive.info file ***********************************************************************************************************************************/ static InfoArchive * verifyArchiveInfoFile(void) { FUNCTION_LOG_VOID(logLevelDebug); InfoArchive *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get the main info file const VerifyInfoFile verifyArchiveInfo = verifyInfoFile(INFO_ARCHIVE_PATH_FILE_STR, true, cfgCipherSpecMain()); // If the main file did not error, then report on the copy's status and check checksums if (verifyArchiveInfo.errorCode == 0) { result = verifyArchiveInfo.archive; infoArchiveMove(result, memContextPrior()); // Attempt to load the copy and report on it's status but don't keep it in memory const VerifyInfoFile verifyArchiveInfoCopy = verifyInfoFile( INFO_ARCHIVE_PATH_FILE_COPY_STR, false, cfgCipherSpecMain()); // If the copy loaded successfully, then check the checksums if (verifyArchiveInfoCopy.errorCode == 0) { // If the info and info.copy checksums don't match each other than one (or both) of the files could be corrupt so // log a warning but must trust main if (!strEq(verifyArchiveInfo.checksum, verifyArchiveInfoCopy.checksum)) LOG_DETAIL("archive.info.copy does not match archive.info"); } } else { // Attempt to load the copy const VerifyInfoFile verifyArchiveInfoCopy = verifyInfoFile(INFO_ARCHIVE_PATH_FILE_COPY_STR, true, cfgCipherSpecMain()); // If loaded successfully, then return the copy as usable if (verifyArchiveInfoCopy.errorCode == 0) { result = verifyArchiveInfoCopy.archive; infoArchiveMove(result, memContextPrior()); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INFO_ARCHIVE, result); } /*********************************************************************************************************************************** Get the backup.info file ***********************************************************************************************************************************/ static InfoBackup * verifyBackupInfoFile(void) { FUNCTION_LOG_VOID(logLevelDebug); InfoBackup *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get the main info file const VerifyInfoFile verifyBackupInfo = verifyInfoFile(INFO_BACKUP_PATH_FILE_STR, true, cfgCipherSpecMain()); // If the main file did not error, then report on the copy's status and check checksums if (verifyBackupInfo.errorCode == 0) { result = verifyBackupInfo.backup; infoBackupMove(result, memContextPrior()); // Attempt to load the copy and report on it's status but don't keep it in memory const VerifyInfoFile verifyBackupInfoCopy = verifyInfoFile(INFO_BACKUP_PATH_FILE_COPY_STR, false, cfgCipherSpecMain()); // If the copy loaded successfully, then check the checksums if (verifyBackupInfoCopy.errorCode == 0) { // If the info and info.copy checksums don't match each other than one (or both) of the files could be corrupt so // log a warning but must trust main if (!strEq(verifyBackupInfo.checksum, verifyBackupInfoCopy.checksum)) LOG_DETAIL("backup.info.copy does not match backup.info"); } } else { // Attempt to load the copy const VerifyInfoFile verifyBackupInfoCopy = verifyInfoFile(INFO_BACKUP_PATH_FILE_COPY_STR, true, cfgCipherSpecMain()); // If loaded successfully, then return the copy as usable if (verifyBackupInfoCopy.errorCode == 0) { result = verifyBackupInfoCopy.backup; infoBackupMove(result, memContextPrior()); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INFO_BACKUP, result); } /*********************************************************************************************************************************** Get the manifest file ***********************************************************************************************************************************/ static Manifest * verifyManifestFile( VerifyBackupResult *const backupResult, const CipherSpec *const cipherSpecManifest, bool currentBackup, const InfoPg *const pgHistory, unsigned int *const jobErrorTotal) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM_P(VERIFY_BACKUP_RESULT, backupResult); // The result set for the backup being processed FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpecManifest); // Cipher spec to access the manifest file FUNCTION_LOG_PARAM(BOOL, currentBackup); // Is this possibly a backup currently in progress? FUNCTION_LOG_PARAM(INFO_PG, pgHistory); // Database history FUNCTION_LOG_PARAM_P(UINT, jobErrorTotal); // Pointer to the overall job error total FUNCTION_LOG_END(); Manifest *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { const String *const fileName = strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupResult->backupLabel)); // Get the main manifest file const VerifyInfoFile verifyManifestInfo = verifyInfoFile(fileName, true, cipherSpecManifest); // If the main file did not error, then report on the copy's status and check checksums if (verifyManifestInfo.errorCode == 0) { result = verifyManifestInfo.manifest; // The current in-progress backup is only notional until the main file is checked because the backup may have // completed by the time the main manifest is checked here. So having a main manifest file means this backup is not // (or is no longer) the currentBackup. currentBackup = false; // Attempt to load the copy and report on it's status but don't keep it in memory const VerifyInfoFile verifyManifestInfoCopy = verifyInfoFile( strNewFmt("%s%s", strZ(fileName), INFO_COPY_EXT), false, cipherSpecManifest); // If the copy loaded successfully, then check the checksums if (verifyManifestInfoCopy.errorCode == 0) { // If the manifest and manifest.copy checksums don't match each other than one (or both) of the files could be // corrupt so log a warning but trust main if (!strEq(verifyManifestInfo.checksum, verifyManifestInfoCopy.checksum)) LOG_DETAIL_FMT("backup '%s' manifest.copy does not match manifest", strZ(backupResult->backupLabel)); } } else { // If this might be an in-progress backup and the main manifest is simply missing, it is assumed the backup is an // actual in-progress backup and verification is skipped, otherwise, if the main is not simply missing, or this is not // an in-progress backup then attempt to load the copy. if (!(currentBackup && verifyManifestInfo.errorCode == errorTypeCode(&FileMissingError))) { currentBackup = false; const VerifyInfoFile verifyManifestInfoCopy = verifyInfoFile( strNewFmt("%s%s", strZ(fileName), INFO_COPY_EXT), true, cipherSpecManifest); // If loaded successfully, then return the copy as usable if (verifyManifestInfoCopy.errorCode == 0) { LOG_DETAIL_FMT("%s/backup.manifest is missing or unusable, using copy", strZ(backupResult->backupLabel)); result = verifyManifestInfoCopy.manifest; } else if (verifyManifestInfo.errorCode == errorTypeCode(&FileMissingError) && verifyManifestInfoCopy.errorCode == errorTypeCode(&FileMissingError)) { backupResult->status = backupMissingManifest; LOG_DETAIL_FMT("manifest missing for '%s' - backup may have expired", strZ(backupResult->backupLabel)); } } else { backupResult->status = backupInProgress; LOG_INFO_FMT("backup '%s' appears to be in progress, skipping", strZ(backupResult->backupLabel)); } } // If found a usable manifest then check that the database it was based on is in the history if (result != NULL) { bool found = false; const ManifestData *const manData = manifestData(result); // Confirm the PG database information from the manifest is in the history list for (unsigned int infoPgIdx = 0; infoPgIdx < infoPgDataTotal(pgHistory); infoPgIdx++) { const InfoPgData pgHistoryData = infoPgData(pgHistory, infoPgIdx); if (pgHistoryData.id == manData->pgId && pgHistoryData.systemId == manData->pgSystemId && pgHistoryData.version == manData->pgVersion) { found = true; break; } } // If the PG data is not found in the backup.info history, then error and reset the result if (!found) { LOG_INFO_FMT( "'%s' may not be recoverable - PG data (id %u, version %s, system-id %" PRIu64 ") is not in the backup.info" " history, skipping", strZ(backupResult->backupLabel), manData->pgId, strZ(pgVersionToStr(manData->pgVersion)), manData->pgSystemId); manifestFree(result); result = NULL; } else manifestMove(result, memContextPrior()); } // If the result is NULL and the backup status has not yet been set, then the backup is unusable (invalid) if (result == NULL && backupResult->status == backupValid) { backupResult->status = backupInvalid; (*jobErrorTotal)++; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(MANIFEST, result); } /*********************************************************************************************************************************** Check the history in the info files ***********************************************************************************************************************************/ static void verifyPgHistory(const InfoPg *const archiveInfoPg, const InfoPg *const backupInfoPg) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO_PG, archiveInfoPg); // Postgres information from the archive.info file FUNCTION_TEST_PARAM(INFO_PG, backupInfoPg); // Postgres information from the backup.info file FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { // Check archive.info and backup.info current PG data matches. If there is a mismatch, verify cannot continue since // the database is not considered accessible during the verify command so no way to tell which would be valid. checkStanzaInfo(cfgOptionGroupIdxDefault(cfgOptGrpRepo), archiveInfoPg, backupInfoPg); const unsigned int archiveInfoHistoryTotal = infoPgDataTotal(archiveInfoPg); const unsigned int backupInfoHistoryTotal = infoPgDataTotal(backupInfoPg); String *const errMsg = strNewZ("archive and backup history lists do not match"); if (archiveInfoHistoryTotal != backupInfoHistoryTotal) THROW(FormatError, strZ(errMsg)); // Confirm the lists are the same for (unsigned int infoPgIdx = 0; infoPgIdx < archiveInfoHistoryTotal; infoPgIdx++) { const InfoPgData archiveInfoPgHistory = infoPgData(archiveInfoPg, infoPgIdx); const InfoPgData backupInfoPgHistory = infoPgData(backupInfoPg, infoPgIdx); if (archiveInfoPgHistory.id != backupInfoPgHistory.id || archiveInfoPgHistory.systemId != backupInfoPgHistory.systemId || archiveInfoPgHistory.version != backupInfoPgHistory.version) { THROW(FormatError, strZ(errMsg)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Populate the WAL ranges from the provided, sorted, WAL files list for a given archiveId ***********************************************************************************************************************************/ static void verifyCreateArchiveIdRange( const VerifyArchiveResult *const archiveIdResult, StringList *const walFileList, unsigned int *const jobErrorTotal) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VERIFY_ARCHIVE_RESULT, archiveIdResult); // The result set for the archive Id being processed FUNCTION_TEST_PARAM(STRING_LIST, walFileList); // Sorted (ascending) list of WAL files in a timeline FUNCTION_TEST_PARAM_P(UINT, jobErrorTotal); // Pointer to the overall job error total FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(archiveIdResult != NULL); ASSERT(walFileList != NULL); unsigned int walFileIdx = 0; // Initialize the WAL range VerifyWalRange *walRange = NULL; // If there is a WAL range for this archiveID, get the last one. If there is no timeline change then continue updating the last // WAL range. if (!lstEmpty(archiveIdResult->walRangeList) && strEq( strSubN(((VerifyWalRange *)lstGetLast(archiveIdResult->walRangeList))->stop, 0, 8), strSubN(strSubN(strLstGet(walFileList, walFileIdx), 0, WAL_SEGMENT_NAME_SIZE), 0, 8))) { walRange = lstGetLast(archiveIdResult->walRangeList); } do { const String *const walSegment = strSubN(strLstGet(walFileList, walFileIdx), 0, WAL_SEGMENT_NAME_SIZE); // The lists are sorted so look ahead to see if this is a duplicate of the next one in the list if (walFileIdx + 1 < strLstSize(walFileList)) { if (strEq(walSegment, strSubN(strLstGet(walFileList, walFileIdx + 1), 0, WAL_SEGMENT_NAME_SIZE))) { LOG_INFO_FMT("duplicate WAL '%s' for '%s' exists, skipping", strZ(walSegment), strZ(archiveIdResult->archiveId)); (*jobErrorTotal)++; bool foundDup = true; // Remove all duplicates of this WAL, including this WAL, from the list while (walFileIdx < strLstSize(walFileList) && foundDup) { if (strEq(walSegment, strSubN(strLstGet(walFileList, walFileIdx), 0, WAL_SEGMENT_NAME_SIZE))) strLstRemoveIdx(walFileList, walFileIdx); else foundDup = false; } continue; } } // Initialize the range if it has not yet been initialized and continue to next if (walRange == NULL || !strEq( walSegmentNext(walRange->stop, (size_t)archiveIdResult->pgWalInfo.size), walSegment)) { // Add the initialized wal range to the range list MEM_CONTEXT_BEGIN(lstMemContext(archiveIdResult->walRangeList)) { const VerifyWalRange walRangeNew = { .start = strDup(walSegment), .stop = strDup(walSegment), .invalidFileList = lstNewP(sizeof(VerifyInvalidFile), .comparator = lstComparatorStr), }; lstAdd(archiveIdResult->walRangeList, &walRangeNew); } MEM_CONTEXT_END(); // Set the current wal range being processed to what was just added walRange = lstGetLast(archiveIdResult->walRangeList); } // If the next WAL is the appropriate distance away, then there is no gap else { MEM_CONTEXT_BEGIN(lstMemContext(archiveIdResult->walRangeList)) { strFree(walRange->stop); walRange->stop = strDup(walSegment); } MEM_CONTEXT_END(); } walFileIdx++; } while (walFileIdx < strLstSize(walFileList)); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Check if backup is block incremental and populate backup list from its references and set the WAL range to verify based on WAL required to make the backup consistent. ***********************************************************************************************************************************/ static void verifyBackupSet(VerifyJobData *const jobData, const String *const backupLabel) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, jobData); // Pointer to the job data FUNCTION_TEST_PARAM(STRING, backupLabel); // Label of backup to check FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); MEM_CONTEXT_TEMP_BEGIN() { TRY_BEGIN() { const Manifest *const manifest = manifestLoadFile( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)), jobData->cipherSpecManifest); // Check files for block incremental bool hasBlockIncr = false; for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFile file = manifestFile(manifest, fileIdx); if (file.blockIncrMapSize != 0) { hasBlockIncr = true; break; } } // Block incremental backups can depend on any referenced backups through block maps which means we have to verify all // referenced backups as well. ??? Make this more efficient by verifying only required blocks. if (hasBlockIncr) { const StringList *const referenceList = manifestReferenceList(manifest); MEM_CONTEXT_BEGIN(jobData->memContext) { for (unsigned int referenceIdx = 0; referenceIdx < strLstSize(referenceList); referenceIdx++) strLstAddIfMissing(jobData->backupList, strLstGet(referenceList, referenceIdx)); strLstSort(jobData->backupList, sortOrderAsc); } MEM_CONTEXT_END(); } // Save WAL range required to make the backup consistent MEM_CONTEXT_BEGIN(jobData->memContext) { const ManifestData *const manData = manifestData(manifest); jobData->archiveStart = strDup(manData->archiveStart); jobData->archiveStop = strDup(manData->archiveStop); } MEM_CONTEXT_END(); } CATCH_ANY() { jobData->archiveStart = NULL; jobData->archiveStop = NULL; } TRY_END(); jobData->enableArchiveFilter = true; } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Return verify jobs for the archive ***********************************************************************************************************************************/ static ProtocolParallelJob * verifyArchive(VerifyJobData *const jobData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, jobData); FUNCTION_TEST_END(); ProtocolParallelJob *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Process archive files, if any while (!strLstEmpty(jobData->archiveIdList)) { // Add archiveId to the result list if the list is empty or the last processed is not equal to the current archiveId if (lstEmpty(jobData->archiveIdResultList) || !strEq( ((VerifyArchiveResult *)lstGetLast(jobData->archiveIdResultList))->archiveId, strLstGet(jobData->archiveIdList, 0))) { const String *const archiveId = strLstGet(jobData->archiveIdList, 0); MEM_CONTEXT_BEGIN(lstMemContext(jobData->archiveIdResultList)) { const VerifyArchiveResult archiveIdResult = { .archiveId = strDup(archiveId), .walRangeList = lstNewP(sizeof(VerifyWalRange), .comparator = lstComparatorStr), }; lstAdd(jobData->archiveIdResultList, &archiveIdResult); } MEM_CONTEXT_END(); // Free the old WAL path list strLstFree(jobData->walPathList); // Get the WAL paths for the archive Id const String *const archiveIdPath = strNewFmt(STORAGE_REPO_ARCHIVE "/%s", strZ(archiveId)); MEM_CONTEXT_BEGIN(jobData->memContext) { jobData->walPathList = strLstSort( storageListP(storageRepo(), archiveIdPath, .expression = WAL_SEGMENT_DIR_REGEXP_STR), sortOrderAsc); } MEM_CONTEXT_END(); } // If there are WAL paths then get the file lists if (!strLstEmpty(jobData->walPathList)) { // Get the archive id info for the current (last) archive id being processed VerifyArchiveResult *const archiveResult = lstGetLast(jobData->archiveIdResultList); do { const String *const walPath = strLstGet(jobData->walPathList, 0); // Get the WAL files for the first item in the WAL paths list and initialize WAL info and ranges if (strLstEmpty(jobData->walFileList)) { // Free the old WAL file list strLstFree(jobData->walFileList); // Get WAL file list const String *const walFilePath = strNewFmt( STORAGE_REPO_ARCHIVE "/%s/%s", strZ(archiveResult->archiveId), strZ(walPath)); MEM_CONTEXT_BEGIN(jobData->memContext) { jobData->walFileList = strLstSort( storageListP(storageRepo(), walFilePath, .expression = WAL_SEGMENT_FILE_REGEXP_STR), sortOrderAsc); } MEM_CONTEXT_END(); // Filter WAL files if needed if (jobData->enableArchiveFilter) { // If backup manifest is broken, skip all archives to report the error immediately if (!jobData->archiveStart) jobData->walFileList = strLstNew(); // Skip WAL files that come before range while (strLstSize(jobData->walFileList) > 0) { const String *const item = strLstGet(jobData->walFileList, 0); if (strCmp(strSubN(item, 0, WAL_SEGMENT_NAME_SIZE), jobData->archiveStart) < 0) jobData->walFileList = strLstRemoveIdx(jobData->walFileList, 0); else break; } // Skip WAL files that come after range while (strLstSize(jobData->walFileList) > 0) { unsigned int lastIdx = strLstSize(jobData->walFileList) - 1; const String *const item = strLstGet(jobData->walFileList, lastIdx); if (strCmp(strSubN(item, 0, WAL_SEGMENT_NAME_SIZE), jobData->archiveStop) > 0) jobData->walFileList = strLstRemoveIdx(jobData->walFileList, lastIdx); else break; } } if (!strLstEmpty(jobData->walFileList)) { if (archiveResult->pgWalInfo.size == 0) { // Initialize the WAL segment size from the first WAL const String *const walFile = strNewFmt( STORAGE_REPO_ARCHIVE "/%s/%s/%s", strZ(archiveResult->archiveId), strZ(walPath), strZ(strLstGet(jobData->walFileList, 0))); StorageRead *const walRead = storageNewReadP(storageRepo(), walFile); IoFilterGroup *const walFilterGroup = ioReadFilterGroup(storageReadIo(walRead)); // Add decryption filter when required cipherBlockFormatFilterGroupReadAddMap(walFilterGroup, jobData->cipherSpecArchive); // If the file is compressed, add a decompression filter if (compressTypeFromName(walFile) != compressTypeNone) ioFilterGroupAdd(walFilterGroup, decompressFilterP(compressTypeFromName(walFile))); const PgWal walInfo = pgWalFromBuffer( storageGetP(walRead, .exactSize = PG_WAL_HEADER_SIZE), cfgOptionStrNull(cfgOptPgVersionForce)); archiveResult->pgWalInfo.size = walInfo.size; archiveResult->pgWalInfo.version = walInfo.version; } // Add total number of WAL files in the directory to the total WAL - this number will include // duplicates, if any, that will be filtered out and not checked but will be reported as errors in the // log archiveResult->totalWalFile += strLstSize(jobData->walFileList); verifyCreateArchiveIdRange(archiveResult, jobData->walFileList, &jobData->jobErrorTotal); } } // If there are WAL files, then verify them if (!strLstEmpty(jobData->walFileList)) { // Get the fully qualified file name and checksum const String *const fileName = strLstGet(jobData->walFileList, 0); const String *const filePathName = strNewFmt( STORAGE_REPO_ARCHIVE "/%s/%s/%s", strZ(archiveResult->archiveId), strZ(walPath), strZ(fileName)); const Buffer *const checksum = bufNewDecode( encodingHex, strSubN(fileName, WAL_SEGMENT_NAME_SIZE + 1, HASH_TYPE_SHA1_SIZE_HEX)); // Set up the job PackWrite *const param = protocolPackNew(); pckWriteStrP(param, filePathName); pckWriteBoolP(param, false); pckWriteU32P(param, compressTypeFromName(filePathName)); pckWriteBinP(param, checksum); pckWriteU64P(param, archiveResult->pgWalInfo.size); cipherSpecMapPack(param, jobData->cipherSpecArchive); pckWriteBoolP(param, true); // Assign job to result, prepending the archiveId to the key for consistency with backup processing const String *const jobKey = strNewFmt("%s/%s", strZ(archiveResult->archiveId), strZ(filePathName)); MEM_CONTEXT_PRIOR_BEGIN() { result = protocolParallelJobNew(VARSTR(jobKey), PROTOCOL_COMMAND_VERIFY_FILE, param); } MEM_CONTEXT_PRIOR_END(); // Remove the file to process from the list strLstRemoveIdx(jobData->walFileList, 0); // If this is the last file to process for this timeline, then remove the path if (strLstEmpty(jobData->walFileList)) strLstRemoveIdx(jobData->walPathList, 0); } else { // No valid WAL to process (may be only duplicates or nothing in WAL path) - remove WAL path from the list LOG_DETAIL_FMT( "path '%s/%s' does not contain any valid WAL to be processed", strZ(archiveResult->archiveId), strZ(walPath)); strLstRemoveIdx(jobData->walPathList, 0); } // If a job was found to be processed then break out to process it if (result != NULL) break; } while (!strLstEmpty(jobData->walPathList)); // If this is the last timeline to process for this archiveId, then remove the archiveId if (strLstEmpty(jobData->walPathList)) strLstRemoveIdx(jobData->archiveIdList, 0); // If a file was sent to be processed then break so we can process it if (result != NULL) break; } else { // Log that no WAL paths exist in the archive Id dir - remove the archive Id from the list (nothing to process) LOG_DETAIL_FMT("archive path '%s' is empty", strZ(strLstGet(jobData->archiveIdList, 0))); strLstRemoveIdx(jobData->archiveIdList, 0); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } /*********************************************************************************************************************************** Verify the job data backups ***********************************************************************************************************************************/ static ProtocolParallelJob * verifyBackup(VerifyJobData *const jobData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, jobData); FUNCTION_TEST_END(); ProtocolParallelJob *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Process backup files, if any while (!strLstEmpty(jobData->backupList)) { // If result list is empty or the last processed is not equal to the backup being processed, then initialize the backup // data and results if (lstEmpty(jobData->backupResultList) || !strEq( ((VerifyBackupResult *)lstGetLast(jobData->backupResultList))->backupLabel, strLstGet(jobData->backupList, 0))) { MEM_CONTEXT_BEGIN(lstMemContext(jobData->backupResultList)) { const VerifyBackupResult backupResultNew = { .backupLabel = strDup(strLstGet(jobData->backupList, 0)), .invalidFileList = lstNewP(sizeof(VerifyInvalidFile), .comparator = lstComparatorStr), }; // Add the backup to the result list lstAdd(jobData->backupResultList, &backupResultNew); } MEM_CONTEXT_END(); // Get the result just added so it can be updated directly VerifyBackupResult *const backupResult = lstGetLast(jobData->backupResultList); // If currentBackup is set (meaning the newest backup label on disk was not in the db:current section when the // backup.info file was read) and this is the same label, then set inProgressBackup to true, else false. // inProgressBackup may be changed in verifyManifestFile if a main backup.manifest exists since that would indicate // the backup completed during the verify process. const bool inProgressBackup = strEq(jobData->currentBackup, backupResult->backupLabel); // Get a usable backup manifest file Manifest *const manifest = verifyManifestFile( backupResult, jobData->cipherSpecManifest, inProgressBackup, jobData->pgHistory, &jobData->jobErrorTotal); // If a usable backup.manifest file is not found if (manifest == NULL) { // Remove this backup from the processing list strLstRemoveIdx(jobData->backupList, 0); // No files to process so continue to the next backup in the list continue; } // Initialize the backup results and manifest for processing else { // Move the manifest to the jobData for processing jobData->manifest = manifestMove(manifest, jobData->memContext); jobData->manifestFileIdx = 0; const ManifestData *const manData = manifestData(jobData->manifest); MEM_CONTEXT_BEGIN(lstMemContext(jobData->backupResultList)) { backupResult->totalFileManifest = manifestFileTotal(jobData->manifest); backupResult->backupPrior = strDup(manData->backupLabelPrior); backupResult->pgId = manData->pgId; backupResult->pgVersion = manData->pgVersion; backupResult->archiveStart = strDup(manData->archiveStart); backupResult->archiveStop = strDup(manData->archiveStop); } MEM_CONTEXT_END(); } } VerifyBackupResult *const backupResult = lstGetLast(jobData->backupResultList); // Process any files in the manifest if (jobData->manifestFileIdx < manifestFileTotal(jobData->manifest)) { do { const ManifestFile fileData = manifestFile(jobData->manifest, jobData->manifestFileIdx); // Track the files verified in order to determine when the processing of the backup is complete backupResult->totalFileVerify++; // Check the file if it is not zero-length or not bundled if (fileData.size != 0 || !manifestData(jobData->manifest)->bundle) { // Check if the file is referenced in a prior backup const String *fileBackupLabel = NULL; if (fileData.reference != NULL) { // If the prior backup is not in the result list, then that backup was never processed (likely due to // the --set option) so verify the file const unsigned int backupPriorIdx = lstFindIdx(jobData->backupResultList, &fileData.reference); if (backupPriorIdx == LIST_NOT_FOUND) { fileBackupLabel = fileData.reference; } // Else the backup this file references has a result so check the processing state for the referenced // backup else { const VerifyBackupResult *const backupResultPrior = lstGet( jobData->backupResultList, backupPriorIdx); // If the verify-state of the backup is not complete then verify the file if (!backupResultPrior->fileVerifyComplete) { fileBackupLabel = fileData.reference; } // Else skip verification else { const String *const priorFile = strNewFmt( "%s/%s%s", strZ(fileData.reference), strZ(fileData.name), strZ(compressExtStr((manifestData(jobData->manifest))->backupOptionCompressType))); const unsigned int backupPriorInvalidIdx = lstFindIdx( backupResultPrior->invalidFileList, &priorFile); // If the file is in the invalid file list of the prior backup where it is referenced then add // the file as invalid to this backup result and set the backup result status; since already // logged an error on this file, don't log again if (backupPriorInvalidIdx != LIST_NOT_FOUND) { VerifyInvalidFile *invalidFile = lstGet( backupResultPrior->invalidFileList, backupPriorInvalidIdx); verifyInvalidFileAdd( backupResult->invalidFileList, invalidFile->reason, invalidFile->fileName); backupResult->status = backupInvalid; } // Else file in prior backup was valid so increment total valid files for this backup else { backupResult->totalFileValid++; } } } } // Else file is not referenced in a prior backup else fileBackupLabel = backupResult->backupLabel; // If backup label is not null then send it off for processing if (fileBackupLabel != NULL) { // Set up the job PackWrite *const param = protocolPackNew(); const String *const filePathName = backupFileRepoPathP( fileBackupLabel, .manifestName = fileData.name, .bundleId = fileData.bundleId, .compressType = manifestData(jobData->manifest)->backupOptionCompressType, .blockIncr = fileData.blockIncrMapSize != 0); pckWriteStrP(param, filePathName); if (fileData.bundleId != 0) { pckWriteBoolP(param, true); pckWriteU64P(param, fileData.bundleOffset); pckWriteU64P(param, fileData.sizeRepo); } else pckWriteBoolP(param, false); // Use the repo checksum when present if (fileData.checksumRepoSha1 != NULL) { pckWriteU32P(param, compressTypeNone); pckWriteBinP(param, BUF(fileData.checksumRepoSha1, HASH_TYPE_SHA1_SIZE)); pckWriteU64P(param, fileData.sizeRepo); cipherSpecMapPack(param, cipherSpecMapNew()); pckWriteBoolP(param, false); } // Else use the file checksum, which may require additional filters, e.g. decompression else { pckWriteU32P(param, manifestData(jobData->manifest)->backupOptionCompressType); pckWriteBinP(param, BUF(fileData.checksumSha1, HASH_TYPE_SHA1_SIZE)); pckWriteU64P(param, fileData.size); // Backup files contain no key id so their key goes under the default id CipherSpecMap *const cipherSpecMap = cipherSpecMapNew(); cipherSpecMapAdd( cipherSpecMap, CIPHER_SPEC_MAP_ID_DEFAULT_STR, manifestCipherSpec(jobData->manifest)); cipherSpecMapPack(param, cipherSpecMap); pckWriteBoolP(param, false); } // Assign job to result (prepend backup label being processed to the key since some files are in a prior // backup) const String *const jobKey = strNewFmt("%s/%s", strZ(backupResult->backupLabel), strZ(filePathName)); MEM_CONTEXT_PRIOR_BEGIN() { result = protocolParallelJobNew(VARSTR(jobKey), PROTOCOL_COMMAND_VERIFY_FILE, param); } MEM_CONTEXT_PRIOR_END(); } } // Else mark the zero-length file as valid else backupResult->totalFileValid++; // Increment the index to point to the next file jobData->manifestFileIdx++; // If this was the last file to process for this backup, then free the manifest and remove this backup from the // processing list if (jobData->manifestFileIdx == backupResult->totalFileManifest) { manifestFree(jobData->manifest); jobData->manifest = NULL; strLstRemoveIdx(jobData->backupList, 0); } // If a job was found to be processed then break out to process it if (result != NULL) break; } while (jobData->manifestFileIdx < backupResult->totalFileManifest); } else { // Nothing to process so report an error, free the manifest, set the status, and remove the backup from processing // list LOG_INFO_FMT("backup '%s' manifest does not contain any target files to verify", strZ(backupResult->backupLabel)); jobData->jobErrorTotal++; manifestFree(jobData->manifest); jobData->manifest = NULL; backupResult->status = backupInvalid; strLstRemoveIdx(jobData->backupList, 0); } // If a job was found to be processed then break out to process it if (result != NULL) break; } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } /*********************************************************************************************************************************** Process the job data ***********************************************************************************************************************************/ static ProtocolParallelJob * verifyJobCallback(void *const data, const unsigned int clientIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); // Pointer to the job data (void)clientIdx; // Client index (not used for this process) FUNCTION_TEST_END(); ASSERT(data != NULL); // Initialize the result ProtocolParallelJob *result = NULL; VerifyJobData *const jobData = data; if (!jobData->backupProcessing) { result = verifyArchive(jobData); // Set the backupProcessing flag if the archive processing is finished so backup processing can begin immediately after jobData->backupProcessing = strLstEmpty(jobData->archiveIdList); } if (jobData->backupProcessing) { // Only begin backup verification if the last archive result was processed if (result == NULL) result = verifyBackup(jobData); } FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } /*********************************************************************************************************************************** Helper function for returning a string corresponding to the result code ***********************************************************************************************************************************/ static const char * verifyErrorMsg(const VerifyResult verifyResult) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, verifyResult); // Result code from the verifyFile() function FUNCTION_TEST_END(); ASSERT(verifyResult != verifyOk); switch (verifyResult) { case verifyFileMissing: FUNCTION_TEST_RETURN_CONST(STRINGZ, "file missing"); case verifyChecksumMismatch: FUNCTION_TEST_RETURN_CONST(STRINGZ, "invalid checksum"); case verifySizeInvalid: FUNCTION_TEST_RETURN_CONST(STRINGZ, "invalid size"); default: ASSERT(verifyResult == verifyOtherError); FUNCTION_TEST_RETURN_CONST(STRINGZ, "invalid result"); } } /*********************************************************************************************************************************** Helper function to output a log message based on job result that is not verifyOk and return an error count ***********************************************************************************************************************************/ static unsigned int verifyLogInvalidResult( const String *const fileType, const VerifyResult verifyResult, const unsigned int processId, const String *const filePathName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, fileType); // Indicates archive or backup file FUNCTION_TEST_PARAM(ENUM, verifyResult); // Result code from the verifyFile() function FUNCTION_TEST_PARAM(UINT, processId); // Process Id reporting the result FUNCTION_TEST_PARAM(STRING, filePathName); // File for which results are being reported FUNCTION_TEST_END(); ASSERT(fileType != NULL); ASSERT(filePathName != NULL); // Log a warning because the WAL may have gone missing if expire came through and removed it // legitimately so it is not necessarily an error so the jobErrorTotal should not be incremented if (strEq(fileType, STORAGE_REPO_ARCHIVE_STR) && verifyResult == verifyFileMissing) { LOG_WARN_PID_FMT(processId, "%s '%s'", verifyErrorMsg(verifyResult), strZ(filePathName)); FUNCTION_TEST_RETURN(UINT, 0); } LOG_INFO_PID_FMT(processId, "%s '%s'", verifyErrorMsg(verifyResult), strZ(filePathName)); FUNCTION_TEST_RETURN(UINT, 1); } /*********************************************************************************************************************************** Helper function to set the currently processing backup label, if any, and check that the archiveIds are in the db history ***********************************************************************************************************************************/ static String * verifySetBackupCheckArchive( const StringList *const backupList, const InfoBackup *const backupInfo, const StringList *const archiveIdList, const InfoPg *const pgHistory, unsigned int *const jobErrorTotal) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, backupList); // List of backup labels in the backup directory FUNCTION_TEST_PARAM(INFO_BACKUP, backupInfo); // Contents of the backup.info file FUNCTION_TEST_PARAM(STRING_LIST, archiveIdList); // List of archiveIds in the archive directory FUNCTION_TEST_PARAM(INFO_PG, pgHistory); // Pointer to InfoPg of archive.info for accessing PG history FUNCTION_TEST_PARAM_P(UINT, jobErrorTotal); // Pointer to overall job error total FUNCTION_TEST_END(); String *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // If there are backups, set the last backup as current if it is not in backup.info - if it is, then it is complete, else // it will be checked later if (!strLstEmpty(backupList)) { // Get the last backup as current if it is not in backup.info current list const String *const backupLabel = strLstGet(backupList, strLstSize(backupList) - 1); if (!infoBackupLabelExists(backupInfo, backupLabel)) { // Duplicate the string into the prior context MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(backupLabel); } MEM_CONTEXT_PRIOR_END(); } } // If there are archive directories on disk, make sure they are in the database history list if (!strLstEmpty(archiveIdList)) { StringList *const archiveIdHistoryList = strLstNew(); for (unsigned int histIdx = 0; histIdx < infoPgDataTotal(pgHistory); histIdx++) strLstAdd(archiveIdHistoryList, infoPgArchiveId(pgHistory, histIdx)); // Sort the history list strLstSort(strLstComparatorSet(archiveIdHistoryList, archiveIdComparator), sortOrderAsc); String *const missingFromHistory = strNew(); // Check if the archiveId on disk exists in the archive.info history list and report it if not for (unsigned int archiveIdx = 0; archiveIdx < strLstSize(archiveIdList); archiveIdx++) { const String *const archiveId = strLstGet(archiveIdList, archiveIdx); if (!strLstExists(archiveIdHistoryList, archiveId)) strCat(missingFromHistory, (strEmpty(missingFromHistory) ? archiveId : strNewFmt(", %s", strZ(archiveId)))); } if (!strEmpty(missingFromHistory)) { LOG_INFO_FMT("archiveIds '%s' are not in the archive.info history list", strZ(missingFromHistory)); (*jobErrorTotal)++; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Add the file to the invalid file list for the range in which it exists ***********************************************************************************************************************************/ static void verifyAddInvalidWalFile( const List *const walRangeList, const VerifyResult fileResult, const String *const fileName, const String *const walSegment) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, walRangeList); // List of WAL ranges for an archive Id FUNCTION_TEST_PARAM(UINT, fileResult); // Result of verifyFile() FUNCTION_TEST_PARAM(STRING, fileName); // File name (without the REPO prefix) FUNCTION_TEST_PARAM(STRING, walSegment); // WAL segment, i.e. 000000010000000000000005 FUNCTION_TEST_END(); ASSERT(walRangeList != NULL); ASSERT(fileName != NULL); ASSERT(walSegment != NULL); MEM_CONTEXT_TEMP_BEGIN() { for (unsigned int walIdx = 0; walIdx < lstSize(walRangeList); walIdx++) { const VerifyWalRange *const walRange = lstGet(walRangeList, walIdx); // If the WAL segment is less/equal to the stop file then it falls in this range since ranges are sorted by stop file in // ascending order, therefore first one found is the range if (strCmp(walRange->stop, walSegment) >= 0) { // Add the file to the range where it was found and exit the loop verifyInvalidFileAdd(walRange->invalidFileList, fileResult, fileName); break; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Create file errors string ***********************************************************************************************************************************/ static String * verifyCreateFileErrorsStr( const unsigned int errMissing, const unsigned int errChecksum, const unsigned int errSize, const unsigned int errOther, const bool verboseText) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, errMissing); // Number of files missing FUNCTION_TEST_PARAM(UINT, errChecksum); // Number of files with checksum errors FUNCTION_TEST_PARAM(UINT, errSize); // Number of files with invalid size FUNCTION_TEST_PARAM(UINT, errOther); // Number of files with other errors FUNCTION_TEST_PARAM(BOOL, verboseText); // Is verbose output requested FUNCTION_TEST_END(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { // List all if verbose text, otherwise only list if type has errors strCatFmt( result, "\n %s%s%s%s", verboseText || errMissing ? zNewFmt("missing: %u, ", errMissing) : "", verboseText || errChecksum ? zNewFmt("checksum invalid: %u, ", errChecksum) : "", verboseText || errSize ? zNewFmt("size invalid: %u, ", errSize) : "", verboseText || errOther ? zNewFmt("other: %u", errOther) : ""); // Clean up trailing comma when necessary if (strEndsWithZ(result, ", ")) strTruncIdx(result, (int)strSize(result) - 2); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Render the results of the verify command ***********************************************************************************************************************************/ static String * verifyRender(const List *const archiveIdResultList, const List *const backupResultList, const bool verboseText) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, archiveIdResultList); // Result list for all archive Ids in the repo FUNCTION_TEST_PARAM(LIST, backupResultList); // Result list for all backups in the repo FUNCTION_TEST_PARAM(BOOL, verboseText); // Is verbose output requested? FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(archiveIdResultList != NULL); ASSERT(backupResultList != NULL); String *const result = strNew(); // Render archive results if (verboseText && lstEmpty(archiveIdResultList)) strCatZ(result, "\n archiveId: none found"); else { for (unsigned int archiveIdx = 0; archiveIdx < lstSize(archiveIdResultList); archiveIdx++) { const VerifyArchiveResult *const archiveIdResult = lstGet(archiveIdResultList, archiveIdx); if (verboseText || archiveIdResult->totalWalFile - archiveIdResult->totalValidWal != 0) { strCatFmt( result, "\n archiveId: %s, total WAL checked: %u, total valid WAL: %u", strZ(archiveIdResult->archiveId), archiveIdResult->totalWalFile, archiveIdResult->totalValidWal); } if (archiveIdResult->totalWalFile > 0) { unsigned int errMissing = 0; unsigned int errChecksum = 0; unsigned int errSize = 0; unsigned int errOther = 0; for (unsigned int walIdx = 0; walIdx < lstSize(archiveIdResult->walRangeList); walIdx++) { const VerifyWalRange *const walRange = lstGet(archiveIdResult->walRangeList, walIdx); LOG_DETAIL_FMT( "archiveId: %s, wal start: %s, wal stop: %s", strZ(archiveIdResult->archiveId), strZ(walRange->start), strZ(walRange->stop)); unsigned int invalidIdx = 0; while (invalidIdx < lstSize(walRange->invalidFileList)) { VerifyInvalidFile *invalidFile = lstGet(walRange->invalidFileList, invalidIdx); if (invalidFile->reason == verifyFileMissing) errMissing++; else if (invalidFile->reason == verifyChecksumMismatch) errChecksum++; else if (invalidFile->reason == verifySizeInvalid) errSize++; else errOther++; invalidIdx++; } } // Create/append file errors string if (verboseText || errMissing + errChecksum + errSize + errOther > 0) strCat(result, verifyCreateFileErrorsStr(errMissing, errChecksum, errSize, errOther, verboseText)); } } } // Render backup results if (verboseText && lstEmpty(backupResultList)) strCatZ(result, "\n backup: none found"); else { for (unsigned int backupIdx = 0; backupIdx < lstSize(backupResultList); backupIdx++) { const VerifyBackupResult *const backupResult = lstGet(backupResultList, backupIdx); const char *status; switch (backupResult->status) { case backupInvalid: status = "invalid"; break; case backupMissingManifest: status = "manifest missing"; break; case backupInProgress: status = "in-progress"; break; default: { ASSERT(backupResult->status == backupValid); status = "valid"; break; } } if (verboseText || (strcmp(status, "valid") != 0 && strcmp(status, "in-progress") != 0)) { strCatFmt( result, "\n backup: %s, status: %s, total files checked: %u, total valid files: %u", strZ(backupResult->backupLabel), status, backupResult->totalFileVerify, backupResult->totalFileValid); } if (backupResult->totalFileVerify > 0) { unsigned int errMissing = 0; unsigned int errChecksum = 0; unsigned int errSize = 0; unsigned int errOther = 0; for (unsigned int invalidIdx = 0; invalidIdx < lstSize(backupResult->invalidFileList); invalidIdx++) { const VerifyInvalidFile *const invalidFile = lstGet(backupResult->invalidFileList, invalidIdx); if (invalidFile->reason == verifyFileMissing) errMissing++; else if (invalidFile->reason == verifyChecksumMismatch) errChecksum++; else if (invalidFile->reason == verifySizeInvalid) errSize++; else errOther++; } // Create/append file errors string if (verboseText || errMissing + errChecksum + errSize + errOther > 0) strCat(result, verifyCreateFileErrorsStr(errMissing, errChecksum, errSize, errOther, verboseText)); } } } FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Process the verify command ***********************************************************************************************************************************/ static String * verifyProcess(const bool verboseText) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BOOL, verboseText); // Is verbose output requested? FUNCTION_LOG_END(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { unsigned int errorTotal = 0; String *resultStr = strNew(); // Get the repo storage in case it is remote and encryption settings need to be pulled down const Storage *const storage = storageRepo(); // Get a usable backup info file const InfoBackup *const backupInfo = verifyBackupInfoFile(); // If a usable backup.info file is not found, then report an error in the log if (backupInfo == NULL) { strCatZ(resultStr, "\n No usable backup.info file"); errorTotal++; } // Get a usable archive info file const InfoArchive *const archiveInfo = verifyArchiveInfoFile(); // If a usable archive.info file is not found, then report an error in the log if (archiveInfo == NULL) { strCatZ(resultStr, "\n No usable archive.info file"); errorTotal++; } // If both a usable archive info and backup info file were found, then proceed with verification if (archiveInfo != NULL && backupInfo != NULL) { TRY_BEGIN() { // Verify that the archive.info and backup.info current database info and history lists are the same verifyPgHistory(infoArchivePg(archiveInfo), infoBackupPg(backupInfo)); } CATCH_ANY() { strCatFmt(resultStr, "\n%s", errorMessage()); errorTotal++; } TRY_END(); } // If valid info files, then begin process of checking backups and archives in the repo if (errorTotal == 0) { // Initialize the job data VerifyJobData jobData = { .memContext = memContextCurrent(), .walPathList = NULL, .walFileList = strLstNew(), .pgHistory = infoArchivePg(archiveInfo), .cipherSpecManifest = infoBackupCipherSpec(backupInfo), .cipherSpecArchive = infoArchiveCipherSpecMap(archiveInfo), .archiveIdResultList = lstNewP(sizeof(VerifyArchiveResult), .comparator = archiveIdComparator), .backupResultList = lstNewP(sizeof(VerifyBackupResult), .comparator = lstComparatorStr), }; // Use backup label if specified via --set const String *const backupLabel = cfgOptionStrNull(cfgOptSet); const String *backupRegExpStr = backupRegExpP(.full = true, .differential = true, .incremental = true); bool backupLabelInvalid = false; if (backupLabel != NULL) { if (!regExpMatchOne(backupRegExpStr, backupLabel)) { strCatFmt(resultStr, "\n '%s' is not a valid backup label format", strZ(backupLabel)); backupLabelInvalid = true; errorTotal++; } else backupRegExpStr = strNewFmt("^%s$", strZ(backupLabel)); } // Get a list of backups in the repo sorted ascending if (!backupLabelInvalid) { jobData.backupList = strLstSort( storageListP(storage, STORAGE_REPO_BACKUP_STR, .expression = backupRegExpStr), sortOrderAsc); // Warn if backups in the repository are not described in backup.info. Do not add them for processing since it is // possible for backups to exist in the repository but not in backup.info if an error occurs while backups are being // expired. for (unsigned int repoIdx = 0; repoIdx < strLstSize(jobData.backupList); repoIdx++) { const String *const label = strLstGet(jobData.backupList, repoIdx); if (!infoBackupLabelExists(backupInfo, label)) LOG_WARN_FMT("backup '%s' found in the repository but not in " INFO_BACKUP_FILE, strZ(label)); } } if (!backupLabelInvalid && backupLabel != NULL && strLstEmpty(jobData.backupList)) { strCatFmt(resultStr, "\n backup set %s is not valid", strZ(backupLabel)); backupLabelInvalid = true; errorTotal++; } // Get a list of archive ids in the repo (e.g. 9.4-1, 10-2, etc) sorted ascending by the db-id (number after the dash) if (!backupLabelInvalid) { jobData.archiveIdList = strLstSort( strLstComparatorSet( storageListP(storage, STORAGE_REPO_ARCHIVE_STR, .expression = STRDEF(REGEX_ARCHIVE_DIR_DB_VERSION)), archiveIdComparator), sortOrderAsc); } // Check for block map dependencies if --set option is specified if (!backupLabelInvalid && backupLabel != NULL) verifyBackupSet(&jobData, backupLabel); // Check if backup.info contains backups not found in the repository and add them for processing if (backupLabel == NULL) { const StringList *const infoBackupLabelList = infoBackupDataLabelList(backupInfo, NULL); for (unsigned int infoIdx = 0; infoIdx < strLstSize(infoBackupLabelList); infoIdx++) { const String *const infoLabel = strLstGet(infoBackupLabelList, infoIdx); if (strLstFindIdxP(jobData.backupList, infoLabel) == LIST_NOT_FOUND) { LOG_WARN_FMT("backup '%s' found in " INFO_BACKUP_FILE " but not in the repository", strZ(infoLabel)); strLstAdd(jobData.backupList, infoLabel); } } jobData.backupList = strLstSort(jobData.backupList, sortOrderAsc); } // Only begin processing if there are some archives or backups in the repo if (!backupLabelInvalid && (!strLstEmpty(jobData.archiveIdList) || !strLstEmpty(jobData.backupList))) { // Warn if there are no archives or there are no backups in the repo so that the callback need not try to // distinguish between having processed all of the list or if the list was missing in the first place if (strLstEmpty(jobData.archiveIdList) || strLstEmpty(jobData.backupList)) LOG_DETAIL_FMT("no %s exist in the repo", strLstEmpty(jobData.archiveIdList) ? "archives" : "backups"); // If there are no archives to process, then set the processing flag to skip to processing the backups if (strLstEmpty(jobData.archiveIdList)) jobData.backupProcessing = true; // Set current backup if there is one and verify the archive history on disk is in the database history jobData.currentBackup = verifySetBackupCheckArchive( jobData.backupList, backupInfo, jobData.archiveIdList, jobData.pgHistory, &jobData.jobErrorTotal); // Create the parallel executor ProtocolParallel *const parallelExec = protocolParallelNew( cfgOptionUInt64(cfgOptProtocolTimeout) / 2, verifyJobCallback, &jobData); for (unsigned int processIdx = 1; processIdx <= cfgOptionUInt(cfgOptProcessMax); processIdx++) protocolParallelClientAdd(parallelExec, protocolLocalGet(protocolStorageTypeRepo, 0, processIdx)); // Process jobs MEM_CONTEXT_TEMP_RESET_BEGIN() { do { unsigned int completed = protocolParallelProcess(parallelExec); // Process completed jobs for (unsigned int jobIdx = 0; jobIdx < completed; jobIdx++) { // Get the job and job key ProtocolParallelJob *const job = protocolParallelResult(parallelExec); const unsigned int processId = protocolParallelJobProcessId(job); StringList *const filePathLst = strLstNewSplit(varStr(protocolParallelJobKey(job)), FSLASH_STR); // Remove the result and file type identifier and recreate the path file name const String *const resultId = strLstGet(filePathLst, 0); strLstRemoveIdx(filePathLst, 0); const String *const fileType = strLstGet(filePathLst, 0); strLstRemoveIdx(filePathLst, 0); const String *const filePathName = strLstJoin(filePathLst, "/"); // Initialize the result sets VerifyArchiveResult *archiveIdResult = NULL; VerifyBackupResult *backupResult = NULL; // Get archiveId result data if (strEq(fileType, STORAGE_REPO_ARCHIVE_STR)) { // Find the archiveId in the list - assert if not found since this should never happen const unsigned int index = lstFindIdx(jobData.archiveIdResultList, &resultId); ASSERT(index != LIST_NOT_FOUND); archiveIdResult = lstGet(jobData.archiveIdResultList, index); } // Else get the backup result data else { const unsigned int index = lstFindIdx(jobData.backupResultList, &resultId); ASSERT(index != LIST_NOT_FOUND); backupResult = lstGet(jobData.backupResultList, index); } // The job was successful if (protocolParallelJobErrorCode(job) == 0) { const VerifyResult verifyResult = (VerifyResult)pckReadU32P(protocolParallelJobResult(job)); // Update the result set for the type of file being processed if (strEq(fileType, STORAGE_REPO_ARCHIVE_STR)) { if (verifyResult == verifyOk) archiveIdResult->totalValidWal++; else { jobData.jobErrorTotal += verifyLogInvalidResult( fileType, verifyResult, processId, filePathName); // Add invalid file to the WAL range verifyAddInvalidWalFile( archiveIdResult->walRangeList, verifyResult, filePathName, strSubN( strLstGet(filePathLst, strLstSize(filePathLst) - 1), 0, WAL_SEGMENT_NAME_SIZE)); } } else { if (verifyResult == verifyOk) backupResult->totalFileValid++; else { jobData.jobErrorTotal += verifyLogInvalidResult( fileType, verifyResult, processId, filePathName); backupResult->status = backupInvalid; verifyInvalidFileAdd(backupResult->invalidFileList, verifyResult, filePathName); } } } // Else the job errored else { // Log a protocol error and increment the jobErrorTotal LOG_INFO_PID_FMT( processId, "%s %s: [%d] %s", verifyErrorMsg(verifyOtherError), strZ(filePathName), protocolParallelJobErrorCode(job), strZ(protocolParallelJobErrorMessage(job))); jobData.jobErrorTotal++; // Add invalid file with "OtherError" reason to invalid file list if (strEq(fileType, STORAGE_REPO_ARCHIVE_STR)) { // Add invalid file to the WAL range verifyAddInvalidWalFile( archiveIdResult->walRangeList, verifyOtherError, filePathName, strSubN(strLstGet(filePathLst, strLstSize(filePathLst) - 1), 0, WAL_SEGMENT_NAME_SIZE)); } else { backupResult->status = backupInvalid; verifyInvalidFileAdd(backupResult->invalidFileList, verifyOtherError, filePathName); } } // Set backup verification complete for a backup if all files have run through verification if (strEq(fileType, STORAGE_REPO_BACKUP_STR) && backupResult->totalFileVerify == backupResult->totalFileManifest) { backupResult->fileVerifyComplete = true; } // Free the job protocolParallelJobFree(job); } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } while (!protocolParallelDone(parallelExec)); } MEM_CONTEXT_TEMP_END(); // ??? Need to do the final reconciliation - checking backup required WAL against, valid WAL // Report results resultStr = verifyRender(jobData.archiveIdResultList, jobData.backupResultList, verboseText); } else if (!backupLabelInvalid) strCatZ(resultStr, "\n no archives or backups exist in the repo"); errorTotal += jobData.jobErrorTotal; } // If verbose output or errors then output results if (verboseText || errorTotal > 0) { strCatFmt( result, "stanza: %s\nstatus: %s%s", strZ(cfgOptionStr(cfgOptStanza)), errorTotal > 0 ? VERIFY_STATUS_ERROR : VERIFY_STATUS_OK, strZ(resultStr)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void cmdVerify(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { const String *const result = verifyProcess(cfgOptionBool(cfgOptVerbose)); // Output results if any if (!strEmpty(result)) { // Log results LOG_INFO_FMT("%s", strZ(result)); // Output to console when requested if (cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_VERIFY_OUTPUT_TEXT) { ioFdWriteOneStr(STDOUT_FILENO, result); ioFdWriteOneStr(STDOUT_FILENO, LF_STR); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/command/verify/verify.h0000664000175100017510000000127715265101745020577 0ustar00runnerrunner/*********************************************************************************************************************************** Verify Command ***********************************************************************************************************************************/ #ifndef COMMAND_VERIFY_VERIFY_H #define COMMAND_VERIFY_VERIFY_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Verify the contents of the repository FN_EXTERN void cmdVerify(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/0000775000175100017510000000000015265101745015461 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/assert.h0000664000175100017510000001276015265101745017141 0ustar00runnerrunner/*********************************************************************************************************************************** Assert Routines ***********************************************************************************************************************************/ #ifndef COMMON_ASSERT_H #define COMMON_ASSERT_H #include "common/error/error.h" #include "common/macro.h" /*********************************************************************************************************************************** Asserts are used in test code to ensure that certain conditions are true. They are omitted from production builds. ***********************************************************************************************************************************/ #ifdef DEBUG #define ASSERT(condition) \ do \ { \ if (UNLIKELY(!(condition))) \ THROW_FMT(AssertError, "assertion '%s' failed", #condition); \ } \ while (0) // Skip inline asserts when coverage testing because they will not have branch coverage. Generally speaking inline assertions // should be of the "this != NULL" variety which is also caught effectively by Valgrind. #ifndef DEBUG_COVERAGE #define ASSERT_INLINE(condition) \ do \ { \ if (UNLIKELY(!(condition))) \ THROW_FMT(AssertError, "assertion '%s' failed", #condition); \ } \ while (0) #else #define ASSERT_INLINE(condition) #endif // Used when execution reaches an invalid location rather than an invalid condition #define ASSERT_MSG(message) \ THROW_FMT(AssertError, message); // Declare variables that will be used by later assertions with the goal of making them easier to read and maintain #define ASSERT_DECLARE(declaration) \ declaration // Add a parameter to a function that is only used by assertions #define ASSERT_PARAM(param) \ , param #else #define ASSERT(condition) #define ASSERT_INLINE(condition) #define ASSERT_MSG(message) #define ASSERT_DECLARE(declaration) #define ASSERT_PARAM(param) #endif /*********************************************************************************************************************************** Checks are used in production builds to test very important conditions. Be sure to limit use to the most critical cases. ***********************************************************************************************************************************/ #define CHECK(type, condition, message) \ do \ { \ if (UNLIKELY(!(condition))) \ THROW(type, message); \ } \ while (0) #define CHECK_FMT(type, condition, message, ...) \ do \ { \ if (UNLIKELY(!(condition))) \ THROW_FMT(type, message, __VA_ARGS__); \ } \ while (0) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/0000775000175100017510000000000015265101745017314 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/bz2/0000775000175100017510000000000015265101745020011 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/bz2/common.c0000664000175100017510000000403615265101745021450 0ustar00runnerrunner/*********************************************************************************************************************************** BZ2 Common ***********************************************************************************************************************************/ #include #include #include "common/compress/bz2/common.h" #include "common/debug.h" #include "common/memContext.h" /**********************************************************************************************************************************/ FN_EXTERN int bz2Error(const int error) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, error); FUNCTION_TEST_END(); if (error < 0) { const char *errorMsg; const ErrorType *errorType = &AssertError; switch (error) { case BZ_SEQUENCE_ERROR: errorMsg = "sequence error"; break; case BZ_PARAM_ERROR: errorMsg = "parameter error"; break; case BZ_MEM_ERROR: errorMsg = "memory error"; errorType = &MemoryError; break; case BZ_DATA_ERROR: errorMsg = "data error"; errorType = &FormatError; break; case BZ_DATA_ERROR_MAGIC: errorMsg = "data error magic"; errorType = &FormatError; break; case BZ_IO_ERROR: errorMsg = "io error"; break; case BZ_UNEXPECTED_EOF: errorMsg = "unexpected eof"; break; case BZ_OUTBUFF_FULL: errorMsg = "outbuff full"; break; case BZ_CONFIG_ERROR: errorMsg = "config error"; break; default: errorMsg = "unknown error"; break; } THROWP_FMT(errorType, "bz2 error: [%d] %s", error, errorMsg); } FUNCTION_TEST_RETURN(INT, error); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/bz2/common.h0000664000175100017510000000217115265101745021453 0ustar00runnerrunner/*********************************************************************************************************************************** BZ2 Common Developed using the documentation in https://www.sourceware.org/bzip2/manual/manual.html#libprog ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_BZ2_COMMON_H #define COMMON_COMPRESS_BZ2_COMMON_H #include /*********************************************************************************************************************************** BZ2 extension ***********************************************************************************************************************************/ #define BZ2_EXT "bz2" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN int bz2Error(int error); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/bz2/compress.c0000664000175100017510000001632115265101745022013 0ustar00runnerrunner/*********************************************************************************************************************************** BZ2 Compress ***********************************************************************************************************************************/ #include #include #include #include "common/compress/bz2/common.h" #include "common/compress/bz2/compress.h" #include "common/compress/common.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/macro.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Bz2Compress { bz_stream stream; // Compression stream bool inputSame; // Is the same input required on the next process call? bool flushing; // Is input complete and flushing in progress? bool done; // Is compression done? } Bz2Compress; /*********************************************************************************************************************************** Render as string for logging ***********************************************************************************************************************************/ static void bz2CompressToLog(const Bz2Compress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, done: %s, flushing: %s, avail_in: %u}", cvtBoolToConstZ(this->inputSame), cvtBoolToConstZ(this->done), cvtBoolToConstZ(this->flushing), this->stream.avail_in); } #define FUNCTION_LOG_BZ2_COMPRESS_TYPE \ Bz2Compress * #define FUNCTION_LOG_BZ2_COMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, bz2CompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free compression stream ***********************************************************************************************************************************/ static void bz2CompressFreeResource(THIS_VOID) { THIS(Bz2Compress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BZ2_COMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); BZ2_bzCompressEnd(&this->stream); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Compress data ***********************************************************************************************************************************/ static void bz2CompressProcess(THIS_VOID, const Buffer *const uncompressed, Buffer *const compressed) { THIS(Bz2Compress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BZ2_COMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, uncompressed); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->done); ASSERT(compressed != NULL); ASSERT(!this->flushing || uncompressed == NULL); ASSERT(this->flushing || (!this->inputSame || this->stream.avail_in != 0)); // If input is NULL then start flushing if (uncompressed == NULL) { this->stream.avail_in = 0; this->flushing = true; } // Else still have input data else { // Is new input allowed? if (!this->inputSame) { this->stream.avail_in = (unsigned int)bufUsed(uncompressed); // bzip2 does not accept const input buffers #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wcast-qual" this->stream.next_in = (char *)UNCONSTIFY(uint8_t *, bufPtrConst(uncompressed)); #pragma GCC diagnostic pop } } // Initialize compressed output buffer this->stream.avail_out = (unsigned int)bufRemains(compressed); this->stream.next_out = (char *)bufPtr(compressed) + bufUsed(compressed); // Perform compression, check for error const int result = bz2Error(BZ2_bzCompress(&this->stream, this->flushing ? BZ_FINISH : BZ_RUN)); // Set buffer used space bufUsedSet(compressed, bufSize(compressed) - (size_t)this->stream.avail_out); // Is compression done? if (this->flushing && result == BZ_STREAM_END) this->done = true; // Can more input be provided on the next call? this->inputSame = this->flushing ? !this->done : this->stream.avail_in != 0; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is compress done? ***********************************************************************************************************************************/ static bool bz2CompressDone(const THIS_VOID) { THIS(const Bz2Compress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BZ2_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool bz2CompressInputSame(const THIS_VOID) { THIS(const Bz2Compress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BZ2_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * bz2CompressNew(const int level, const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INT, level); (void)raw; // Raw unsupported FUNCTION_LOG_END(); ASSERT(level >= BZ2_COMPRESS_LEVEL_MIN && level <= BZ2_COMPRESS_LEVEL_MAX); OBJ_NEW_BEGIN(Bz2Compress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (Bz2Compress) { .stream = {.bzalloc = NULL}, }; // Initialize context bz2Error(BZ2_bzCompressInit(&this->stream, level, 0, 0)); // Set callback to ensure bz2 stream is freed memContextCallbackSet(objMemContext(this), bz2CompressFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( BZ2_COMPRESS_FILTER_TYPE, this, compressParamList(level, raw), .done = bz2CompressDone, .inOut = bz2CompressProcess, .inputSame = bz2CompressInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/bz2/compress.h0000664000175100017510000000305715265101745022022 0ustar00runnerrunner/*********************************************************************************************************************************** BZ2 Compress Compress IO to the bz2 format. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_BZ2_COMPRESS_H #define COMMON_COMPRESS_BZ2_COMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define BZ2_COMPRESS_FILTER_TYPE STRID5("bz2-cmp", 0x41a3df3420) /*********************************************************************************************************************************** Level constants ***********************************************************************************************************************************/ #define BZ2_COMPRESS_LEVEL_MIN 1 #define BZ2_COMPRESS_LEVEL_MAX 9 /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *bz2CompressNew(int level, bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/bz2/decompress.c0000664000175100017510000001550715265101745022331 0ustar00runnerrunner/*********************************************************************************************************************************** BZ2 Decompress ***********************************************************************************************************************************/ #include #include #include #include "common/compress/bz2/common.h" #include "common/compress/bz2/decompress.h" #include "common/compress/common.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/macro.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Bz2Decompress { bz_stream stream; // Decompression stream state int result; // Result of last operation bool inputSame; // Is the same input required on the next process call? bool done; // Is decompression done? } Bz2Decompress; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void bz2DecompressToLog(const Bz2Decompress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, done: %s, avail_in: %u}", cvtBoolToConstZ(this->inputSame), cvtBoolToConstZ(this->done), this->stream.avail_in); } #define FUNCTION_LOG_BZ2_DECOMPRESS_TYPE \ Bz2Decompress * #define FUNCTION_LOG_BZ2_DECOMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, bz2DecompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free inflate stream ***********************************************************************************************************************************/ static void bz2DecompressFreeResource(THIS_VOID) { THIS(Bz2Decompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BZ2_DECOMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); BZ2_bzDecompressEnd(&this->stream); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Decompress data ***********************************************************************************************************************************/ static void bz2DecompressProcess(THIS_VOID, const Buffer *const compressed, Buffer *const uncompressed) { THIS(Bz2Decompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BZ2_DECOMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_PARAM(BUFFER, uncompressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(uncompressed != NULL); // There should never be a flush because in a valid compressed stream the end of data can be determined and done will be set. // If a flush is received it means the compressed stream terminated early, e.g. a zero-length or truncated file. if (compressed == NULL) THROW(FormatError, "unexpected eof in compressed data"); if (!this->inputSame) { this->stream.avail_in = (unsigned int)bufUsed(compressed); // bzip2 does not accept const input buffers #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wcast-qual" this->stream.next_in = (char *)UNCONSTIFY(uint8_t *, bufPtrConst(compressed)); #pragma GCC diagnostic pop } this->stream.avail_out = (unsigned int)bufRemains(uncompressed); this->stream.next_out = (char *)bufPtr(uncompressed) + bufUsed(uncompressed); this->result = bz2Error(BZ2_bzDecompress(&this->stream)); // Set buffer used space bufUsedSet(uncompressed, bufSize(uncompressed) - (size_t)this->stream.avail_out); // Is decompression done? this->done = this->result == BZ_STREAM_END; // Is the same input expected on the next call? this->inputSame = this->done ? false : this->stream.avail_in != 0; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is decompress done? ***********************************************************************************************************************************/ static bool bz2DecompressDone(const THIS_VOID) { THIS(const Bz2Decompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BZ2_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool bz2DecompressInputSame(const THIS_VOID) { THIS(const Bz2Decompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BZ2_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * bz2DecompressNew(const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); (void)raw; // Raw unsupported FUNCTION_LOG_END(); OBJ_NEW_BEGIN(Bz2Decompress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (Bz2Decompress) { .stream = {.bzalloc = NULL}, }; // Create bz2 stream bz2Error(this->result = BZ2_bzDecompressInit(&this->stream, 0, 0)); // Set free callback to ensure bz2 context is freed memContextCallbackSet(objMemContext(this), bz2DecompressFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( BZ2_DECOMPRESS_FILTER_TYPE, this, decompressParamList(raw), .done = bz2DecompressDone, .inOut = bz2DecompressProcess, .inputSame = bz2DecompressInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/bz2/decompress.h0000664000175100017510000000221415265101745022325 0ustar00runnerrunner/*********************************************************************************************************************************** BZ2 Decompress Decompress IO from the bz2 format. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_BZ2_DECOMPRESS_H #define COMMON_COMPRESS_BZ2_DECOMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define BZ2_DECOMPRESS_FILTER_TYPE STRID5("bz2-dcmp", 0x83464df3420) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *bz2DecompressNew(bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/common.c0000664000175100017510000000322615265101745020753 0ustar00runnerrunner/*********************************************************************************************************************************** Compression Common ***********************************************************************************************************************************/ #include #include "common/compress/common.h" #include "common/debug.h" /**********************************************************************************************************************************/ FN_EXTERN Pack * compressParamList(const int level, const bool raw) { FUNCTION_TEST_BEGIN(); FUNCTION_LOG_PARAM(INT, level); FUNCTION_TEST_PARAM(BOOL, raw); FUNCTION_TEST_END(); Pack *result; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteI32P(packWrite, level); pckWriteBoolP(packWrite, raw); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN Pack * decompressParamList(const bool raw) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, raw); FUNCTION_TEST_END(); Pack *result; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteBoolP(packWrite, raw); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PACK, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/common.h0000664000175100017510000000147715265101745020766 0ustar00runnerrunner/*********************************************************************************************************************************** Compression Common ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_COMMON_H #define COMMON_COMPRESS_COMMON_H #include "common/type/pack.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Build compress param list FN_EXTERN Pack *compressParamList(int level, bool raw); // Build decompress param list FN_EXTERN Pack *decompressParamList(bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/gz/0000775000175100017510000000000015265101745017734 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/gz/common.c0000664000175100017510000000412515265101745021372 0ustar00runnerrunner/*********************************************************************************************************************************** Gz Common ***********************************************************************************************************************************/ #include #include #include "common/compress/gz/common.h" #include "common/debug.h" #include "common/memContext.h" /**********************************************************************************************************************************/ FN_EXTERN int gzError(const int error) { if (error != Z_OK && error != Z_STREAM_END) { const char *errorMsg; const ErrorType *errorType = &AssertError; switch (error) { // Not exactly an error, but since we are not using custom dictionaries it shouldn't be possible to get this result case Z_NEED_DICT: errorMsg = "need dictionary"; break; // We should not get this error -- included for completeness case Z_ERRNO: errorMsg = "file error"; break; case Z_STREAM_ERROR: errorMsg = "stream error"; errorType = &FormatError; break; case Z_DATA_ERROR: errorMsg = "data error"; errorType = &FormatError; break; case Z_MEM_ERROR: errorMsg = "insufficient memory"; errorType = &MemoryError; break; // This error indicates an error in the code -- there should always be space in the buffer case Z_BUF_ERROR: errorMsg = "no space in buffer"; break; case Z_VERSION_ERROR: errorMsg = "incompatible version"; errorType = &FormatError; break; default: errorMsg = "unknown error"; break; } THROWP_FMT(errorType, "zlib threw error: [%d] %s", error, errorMsg); } return error; } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/gz/common.h0000664000175100017510000000266415265101745021405 0ustar00runnerrunner/*********************************************************************************************************************************** Gz Common ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_GZ_COMMON_H #define COMMON_COMPRESS_GZ_COMMON_H /*********************************************************************************************************************************** Gz extension ***********************************************************************************************************************************/ #define GZ_EXT "gz" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define WINDOW_BITS 15 #define WANT_GZ 16 /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process gz errors FN_EXTERN int gzError(int error); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/gz/compress.c0000664000175100017510000001671415265101745021744 0ustar00runnerrunner/*********************************************************************************************************************************** Gz Compress Based on the documentation at https://github.com/madler/zlib/blob/master/zlib.h ***********************************************************************************************************************************/ #include #include #include #include "common/compress/common.h" #include "common/compress/gz/common.h" #include "common/compress/gz/compress.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/macro.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct GzCompress { z_stream stream; // Compression stream state bool inputSame; // Is the same input required on the next process call? bool flushing; // Is input complete and flushing in progress? bool done; // Is compression done? } GzCompress; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void gzCompressToLog(const GzCompress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, done: %s, flushing: %s, availIn: %u}", cvtBoolToConstZ(this->inputSame), cvtBoolToConstZ(this->done), cvtBoolToConstZ(this->flushing), this->stream.avail_in); } #define FUNCTION_LOG_GZ_COMPRESS_TYPE \ GzCompress * #define FUNCTION_LOG_GZ_COMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, gzCompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Compression constants ***********************************************************************************************************************************/ #define MEM_LEVEL 9 /*********************************************************************************************************************************** Free deflate stream ***********************************************************************************************************************************/ static void gzCompressFreeResource(THIS_VOID) { THIS(GzCompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(GZ_COMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); deflateEnd(&this->stream); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Compress data ***********************************************************************************************************************************/ static void gzCompressProcess(THIS_VOID, const Buffer *const uncompressed, Buffer *const compressed) { THIS(GzCompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(GZ_COMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, uncompressed); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->done); ASSERT(compressed != NULL); ASSERT(!this->flushing || uncompressed == NULL); ASSERT(this->flushing || (!this->inputSame || this->stream.avail_in != 0)); // Flushing if (uncompressed == NULL) { this->stream.avail_in = 0; this->flushing = true; } // More input else { // Is new input allowed? if (!this->inputSame) { this->stream.avail_in = (unsigned int)bufUsed(uncompressed); // Not all versions of zlib (and none by default) will accept const input buffers this->stream.next_in = bufPtrConst(uncompressed); } } // Initialize compressed output buffer this->stream.avail_out = (unsigned int)bufRemains(compressed); this->stream.next_out = bufPtr(compressed) + bufUsed(compressed); // Perform compression const int result = gzError(deflate(&this->stream, this->flushing ? Z_FINISH : Z_NO_FLUSH)); // Set buffer used space bufUsedSet(compressed, bufSize(compressed) - (size_t)this->stream.avail_out); // Is compression done? if (this->flushing && result == Z_STREAM_END) this->done = true; // Can more input be provided on the next call? this->inputSame = this->flushing ? !this->done : this->stream.avail_in != 0; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is compress done? ***********************************************************************************************************************************/ static bool gzCompressDone(const THIS_VOID) { THIS(const GzCompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(GZ_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool gzCompressInputSame(const THIS_VOID) { THIS(const GzCompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(GZ_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * gzCompressNew(const int level, const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INT, level); FUNCTION_LOG_PARAM(BOOL, raw); FUNCTION_LOG_END(); ASSERT(level >= GZ_COMPRESS_LEVEL_MIN && level <= GZ_COMPRESS_LEVEL_MAX); OBJ_NEW_BEGIN(GzCompress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (GzCompress) { .stream = {.zalloc = NULL}, }; // Create gz stream gzError(deflateInit2(&this->stream, level, Z_DEFLATED, (raw ? 0 : WANT_GZ) | WINDOW_BITS, MEM_LEVEL, Z_DEFAULT_STRATEGY)); // Set free callback to ensure gz context is freed memContextCallbackSet(objMemContext(this), gzCompressFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( GZ_COMPRESS_FILTER_TYPE, this, compressParamList(level, raw), .done = gzCompressDone, .inOut = gzCompressProcess, .inputSame = gzCompressInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/gz/compress.h0000664000175100017510000000305415265101745021742 0ustar00runnerrunner/*********************************************************************************************************************************** Gz Compress Compress IO using the gz format. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_GZ_COMPRESS_H #define COMMON_COMPRESS_GZ_COMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define GZ_COMPRESS_FILTER_TYPE STRID5("gz-cmp", 0x20d1ef470) /*********************************************************************************************************************************** Level constants ***********************************************************************************************************************************/ #define GZ_COMPRESS_LEVEL_MIN -1 #define GZ_COMPRESS_LEVEL_MAX 9 /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *gzCompressNew(int level, bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/gz/decompress.c0000664000175100017510000001535415265101745022254 0ustar00runnerrunner/*********************************************************************************************************************************** Gz Decompress Based on the documentation at https://github.com/madler/zlib/blob/master/zlib.h ***********************************************************************************************************************************/ #include #include #include #include "common/compress/common.h" #include "common/compress/gz/common.h" #include "common/compress/gz/decompress.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/macro.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct GzDecompress { z_stream stream; // Decompression stream state int result; // Result of last operation bool inputSame; // Is the same input required on the next process call? bool done; // Is decompression done? } GzDecompress; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void gzDecompressToLog(const GzDecompress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, done: %s, availIn: %u}", cvtBoolToConstZ(this->inputSame), cvtBoolToConstZ(this->done), this->stream.avail_in); } #define FUNCTION_LOG_GZ_DECOMPRESS_TYPE \ GzDecompress * #define FUNCTION_LOG_GZ_DECOMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, gzDecompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free inflate stream ***********************************************************************************************************************************/ static void gzDecompressFreeResource(THIS_VOID) { THIS(GzDecompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(GZ_DECOMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); inflateEnd(&this->stream); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Decompress data ***********************************************************************************************************************************/ static void gzDecompressProcess(THIS_VOID, const Buffer *const compressed, Buffer *const uncompressed) { THIS(GzDecompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(GZ_DECOMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_PARAM(BUFFER, uncompressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(uncompressed != NULL); // There should never be a flush because in a valid compressed stream the end of data can be determined and done will be set. // If a flush is received it means the compressed stream terminated early, e.g. a zero-length or truncated file. if (compressed == NULL) THROW(FormatError, "unexpected eof in compressed data"); if (!this->inputSame) { this->stream.avail_in = (unsigned int)bufUsed(compressed); // Not all versions of zlib (and none by default) will accept const input buffers this->stream.next_in = bufPtrConst(compressed); } this->stream.avail_out = (unsigned int)bufRemains(uncompressed); this->stream.next_out = bufPtr(uncompressed) + bufUsed(uncompressed); this->result = gzError(inflate(&this->stream, Z_NO_FLUSH)); // Set buffer used space bufUsedSet(uncompressed, bufSize(uncompressed) - (size_t)this->stream.avail_out); // Is decompression done? this->done = this->result == Z_STREAM_END; // Is the same input expected on the next call? this->inputSame = this->done ? false : this->stream.avail_in != 0; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is decompress done? ***********************************************************************************************************************************/ static bool gzDecompressDone(const THIS_VOID) { THIS(const GzDecompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(GZ_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool gzDecompressInputSame(const THIS_VOID) { THIS(const GzDecompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(GZ_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * gzDecompressNew(const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BOOL, raw); FUNCTION_LOG_END(); OBJ_NEW_BEGIN(GzDecompress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (GzDecompress) { .stream = {.zalloc = NULL}, }; // Create gz stream gzError(this->result = inflateInit2(&this->stream, (raw ? 0 : WANT_GZ) | WINDOW_BITS)); // Set free callback to ensure gz context is freed memContextCallbackSet(objMemContext(this), gzDecompressFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( GZ_DECOMPRESS_FILTER_TYPE, this, decompressParamList(raw), .done = gzDecompressDone, .inOut = gzDecompressProcess, .inputSame = gzDecompressInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/gz/decompress.h0000664000175100017510000000220415265101745022247 0ustar00runnerrunner/*********************************************************************************************************************************** Gz Decompress Decompress IO from the gz format. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_GZ_DECOMPRESS_H #define COMMON_COMPRESS_GZ_DECOMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define GZ_DECOMPRESS_FILTER_TYPE STRID5("gz-dcmp", 0x41a326f470) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *gzDecompressNew(bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/helper.c0000664000175100017510000002505615265101745020747 0ustar00runnerrunner/*********************************************************************************************************************************** Compression Helper ***********************************************************************************************************************************/ #include #include #include "common/compress/bz2/common.h" #include "common/compress/bz2/compress.h" #include "common/compress/bz2/decompress.h" #include "common/compress/gz/common.h" #include "common/compress/gz/compress.h" #include "common/compress/gz/decompress.h" #include "common/compress/helper.h" #include "common/compress/lz4/common.h" #include "common/compress/lz4/compress.h" #include "common/compress/lz4/decompress.h" #include "common/compress/zst/common.h" #include "common/compress/zst/compress.h" #include "common/compress/zst/decompress.h" #include "common/debug.h" #include "common/log.h" #include "version.h" /*********************************************************************************************************************************** Compression type constants ***********************************************************************************************************************************/ #define COMPRESS_TYPE_NONE "none" // Constants for currently unsupported compression types #define XZ_EXT "xz" /*********************************************************************************************************************************** Configuration for supported and future compression types ***********************************************************************************************************************************/ static const struct CompressHelperLocal { StringId typeId; // Compress type id const String *const type; // Compress type -- must be extension without period prefixed const String *const ext; // File extension with period prefixed StringId compressType; // Type of the compression filter IoFilter *(*compressNew)(int, bool); // Function to create new compression filter StringId decompressType; // Type of the decompression filter IoFilter *(*decompressNew)(bool); // Function to create new decompression filter } compressHelperLocal[] = { [compressTypeNone] = { .typeId = STRID5("none", 0x2b9ee0), .type = STRDEF(COMPRESS_TYPE_NONE), .ext = STRDEF(""), }, [compressTypeBz2] = { .typeId = STRID5("bz2", 0x73420), .type = STRDEF(BZ2_EXT), .ext = STRDEF("." BZ2_EXT), .compressType = BZ2_COMPRESS_FILTER_TYPE, .compressNew = bz2CompressNew, .decompressType = BZ2_DECOMPRESS_FILTER_TYPE, .decompressNew = bz2DecompressNew, }, [compressTypeGz] = { .typeId = STRID5("gz", 0x3470), .type = STRDEF(GZ_EXT), .ext = STRDEF("." GZ_EXT), .compressType = GZ_COMPRESS_FILTER_TYPE, .compressNew = gzCompressNew, .decompressType = GZ_DECOMPRESS_FILTER_TYPE, .decompressNew = gzDecompressNew, }, [compressTypeLz4] = { .typeId = STRID6("lz4", 0x2068c1), .type = STRDEF(LZ4_EXT), .ext = STRDEF("." LZ4_EXT), .compressType = LZ4_COMPRESS_FILTER_TYPE, .compressNew = lz4CompressNew, .decompressType = LZ4_DECOMPRESS_FILTER_TYPE, .decompressNew = lz4DecompressNew, }, [compressTypeZst] = { .typeId = STRID5("zst", 0x527a0), .type = STRDEF(ZST_EXT), .ext = STRDEF("." ZST_EXT), #ifdef HAVE_LIBZST .compressType = ZST_COMPRESS_FILTER_TYPE, .compressNew = zstCompressNew, .decompressType = ZST_DECOMPRESS_FILTER_TYPE, .decompressNew = zstDecompressNew, #endif }, [compressTypeXz] = { .typeId = STRID5("xz", 0x3580), .type = STRDEF(XZ_EXT), .ext = STRDEF("." XZ_EXT), }, }; /**********************************************************************************************************************************/ FN_EXTERN CompressType compressTypeEnum(const StringId type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_END(); ASSERT(type != 0); CompressType result = compressTypeNone; for (; result < LENGTH_OF(compressHelperLocal); result++) { if (type == compressHelperLocal[result].typeId) break; } if (result == LENGTH_OF(compressHelperLocal)) THROW_FMT(AssertError, "invalid compression type '%s'", zNewStrId(type)); FUNCTION_TEST_RETURN(ENUM, result); } /**********************************************************************************************************************************/ static void compressTypePresent(const CompressType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_END(); ASSERT(type < LENGTH_OF(compressHelperLocal)); if (type != compressTypeNone && compressHelperLocal[type].compressNew == NULL) THROW_FMT(OptionInvalidValueError, PROJECT_NAME " not built with %s support", strZ(compressHelperLocal[type].type)); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN const String * compressTypeStr(const CompressType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_END(); ASSERT(type < LENGTH_OF(compressHelperLocal)); FUNCTION_TEST_RETURN_CONST(STRING, compressHelperLocal[type].type); } /**********************************************************************************************************************************/ FN_EXTERN CompressType compressTypeFromName(const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); CompressType result = compressTypeNone + 1; for (; result < LENGTH_OF(compressHelperLocal); result++) { if (strEndsWith(name, compressHelperLocal[result].ext)) break; } if (result == LENGTH_OF(compressHelperLocal)) result = compressTypeNone; FUNCTION_TEST_RETURN(ENUM, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * compressFilter(const CompressType type, const int level, const CompressFilterParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(INT, level); FUNCTION_TEST_PARAM(BOOL, param.raw); FUNCTION_TEST_END(); ASSERT(type < LENGTH_OF(compressHelperLocal)); ASSERT(type != compressTypeNone); compressTypePresent(type); FUNCTION_TEST_RETURN(IO_FILTER, compressHelperLocal[type].compressNew(level, param.raw)); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * compressFilterPack(const StringId filterType, const Pack *const filterParam) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, filterType); FUNCTION_LOG_PARAM(VARIANT_LIST, filterParam); FUNCTION_LOG_END(); ASSERT(filterType != 0); IoFilter *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { for (CompressType compressIdx = compressTypeNone + 1; compressIdx < LENGTH_OF(compressHelperLocal); compressIdx++) { const struct CompressHelperLocal *compress = &compressHelperLocal[compressIdx]; if (filterType == compress->compressType) { ASSERT(filterParam != NULL); PackRead *const paramRead = pckReadNew(filterParam); const int level = pckReadI32P(paramRead); const bool raw = pckReadBoolP(paramRead); result = ioFilterMove(compress->compressNew(level, raw), memContextPrior()); break; } else if (filterType == compress->decompressType) { result = ioFilterMove(compress->decompressNew(pckReadBoolP(pckReadNew(filterParam))), memContextPrior()); break; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(IO_FILTER, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * decompressFilter(const CompressType type, const DecompressFilterParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(BOOL, param.raw); FUNCTION_TEST_END(); ASSERT(type < LENGTH_OF(compressHelperLocal)); ASSERT(type != compressTypeNone); compressTypePresent(type); FUNCTION_TEST_RETURN(IO_FILTER, compressHelperLocal[type].decompressNew(param.raw)); } /**********************************************************************************************************************************/ FN_EXTERN const String * compressExtStr(const CompressType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_END(); ASSERT(type < LENGTH_OF(compressHelperLocal)); FUNCTION_TEST_RETURN_CONST(STRING, compressHelperLocal[type].ext); } /**********************************************************************************************************************************/ FN_EXTERN void compressExtCat(String *const file, const CompressType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, file); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_END(); ASSERT(file != NULL); strCat(file, compressExtStr(type)); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * compressExtStrip(const String *const file, const CompressType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, file); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_END(); ASSERT(file != NULL); if (!strEndsWith(file, compressExtStr(type))) THROW_FMT(FormatError, "'%s' must have '%s' extension", strZ(file), strZ(compressExtStr(type))); FUNCTION_TEST_RETURN(STRING, strSubN(file, 0, strSize(file) - strSize(compressExtStr(type)))); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/helper.h0000664000175100017510000001215715265101745020752 0ustar00runnerrunner/*********************************************************************************************************************************** Compression Helper Abstract compression types so that the calling code does not need to worry about the individual characteristics of the compression type. All compress calls should use this module rather than the individual filters, even if a specific compression type is required. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_HELPER_H #define COMMON_COMPRESS_HELPER_H #include /*********************************************************************************************************************************** Available compression types ***********************************************************************************************************************************/ typedef enum { compressTypeNone, // No compression compressTypeBz2, // bzip2 compressTypeGz, // gzip compressTypeLz4, // lz4 compressTypeZst, // zstandard // These types have not been implemented but are included here so older versions can identify compression types added by future // versions. In that sense this list is speculative, but these seem to be all the types that are likely to be added in the // foreseeable future. compressTypeXz, // xz/lzma } CompressType; #include #include /*********************************************************************************************************************************** Compression types as a regexp. In the future this regexp will be generated automatically at build time but we want to wait until the build code is migrated to C to do that. For now just define it statically since we don't expect it to change very often. If a supported type is not in this list then it should cause an integration test to fail. ***********************************************************************************************************************************/ #define COMPRESS_TYPE_REGEXP "(\\.gz|\\.lz4|\\.zst|\\.xz|\\.bz2)" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Get enum from a compression type string FN_EXTERN CompressType compressTypeEnum(StringId type); // Get string representation of a compression type. This is the extension without the period. FN_EXTERN const String *compressTypeStr(CompressType type); // Get compression type from a (file) name by checking the extension. If the extension is not a supported compression type then // compressType none is returned, even if the file is compressed with some unknown type. FN_EXTERN CompressType compressTypeFromName(const String *name); // Compression filter for the specified type. Error when compress type is none or invalid. typedef struct CompressFilterParam { VAR_PARAM_HEADER; bool raw; // Omit headers, checksum, etc. when possible } CompressFilterParam; #define compressFilterP(type, level, ...) \ compressFilter(type, level, (CompressFilterParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN IoFilter *compressFilter(CompressType type, int level, CompressFilterParam param); // Compression/decompression filter based on string type and a parameter list. This is useful when a filter must be created on a // remote system since the filter type and parameters can be passed through a protocol. FN_EXTERN IoFilter *compressFilterPack(StringId filterType, const Pack *filterParam); // Decompression filter for the specified type. Error when compress type is none or invalid. typedef struct DecompressFilterParam { VAR_PARAM_HEADER; bool raw; // Omit headers, checksum, etc. when possible } DecompressFilterParam; #define decompressFilterP(type, ...) \ decompressFilter(type, (DecompressFilterParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN IoFilter *decompressFilter(CompressType type, DecompressFilterParam param); // Get extension for the current compression type FN_EXTERN const String *compressExtStr(CompressType type); // Add extension for current compression type to the file FN_EXTERN void compressExtCat(String *file, CompressType type); // Remove the specified compression extension. Error when the extension is not correct. FN_EXTERN String *compressExtStrip(const String *file, CompressType type); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/lz4/0000775000175100017510000000000015265101745020025 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/lz4/common.c0000664000175100017510000000154415265101745021465 0ustar00runnerrunner/*********************************************************************************************************************************** LZ4 Common ***********************************************************************************************************************************/ #include #include #include "common/compress/lz4/common.h" #include "common/debug.h" /**********************************************************************************************************************************/ FN_EXTERN LZ4F_errorCode_t lz4Error(const LZ4F_errorCode_t error) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, error); FUNCTION_TEST_END(); if (LZ4F_isError(error)) THROW_FMT(FormatError, "lz4 error: [%zd] %s", (ssize_t)error, LZ4F_getErrorName(error)); FUNCTION_TEST_RETURN_TYPE(LZ4F_errorCode_t, error); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/lz4/common.h0000664000175100017510000000203615265101745021467 0ustar00runnerrunner/*********************************************************************************************************************************** LZ4 Common ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_LZ4_COMMON_H #define COMMON_COMPRESS_LZ4_COMMON_H /*********************************************************************************************************************************** LZ4 extension ***********************************************************************************************************************************/ #define LZ4_EXT "lz4" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process lz4 errors FN_EXTERN size_t lz4Error(size_t error); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/lz4/compress.c0000664000175100017510000002563515265101745022037 0ustar00runnerrunner/*********************************************************************************************************************************** LZ4 Compress Developed against version r131 using the documentation in https://github.com/lz4/lz4/blob/r131/lib/lz4frame.h. ***********************************************************************************************************************************/ #include #include #include #include #include "common/compress/common.h" #include "common/compress/lz4/common.h" #include "common/compress/lz4/compress.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Older versions of lz4 do not define the max header size. This seems to be the max for any version. ***********************************************************************************************************************************/ #ifndef LZ4F_HEADER_SIZE_MAX #define LZ4F_HEADER_SIZE_MAX 19 #endif /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Lz4Compress { LZ4F_compressionContext_t context; // LZ4 compression context LZ4F_preferences_t prefs; // Preferences -- just compress level set Buffer *buffer; // For when the output buffer can't accept all compressed data bool first; // Is this the first call to process? bool inputSame; // Is the same input required on the next process call? bool flushing; // Is input complete and flushing in progress? } Lz4Compress; /*********************************************************************************************************************************** Render as string for logging ***********************************************************************************************************************************/ static void lz4CompressToLog(const Lz4Compress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{level: %d, first: %s, inputSame: %s, flushing: %s}", this->prefs.compressionLevel, cvtBoolToConstZ(this->first), cvtBoolToConstZ(this->inputSame), cvtBoolToConstZ(this->flushing)); } #define FUNCTION_LOG_LZ4_COMPRESS_TYPE \ Lz4Compress * #define FUNCTION_LOG_LZ4_COMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, lz4CompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free compression context ***********************************************************************************************************************************/ static void lz4CompressFreeResource(THIS_VOID) { THIS(Lz4Compress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(LZ4_COMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); LZ4F_freeCompressionContext(this->context); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Compress data ***********************************************************************************************************************************/ // Helper to return a buffer where output will be written. If there is enough space in the provided output buffer then use it, // otherwise allocate an internal buffer to hold the compressed data. Once we start using the internal buffer we'll need to continue // using it until it is completely flushed. static Buffer * lz4CompressBuffer(Lz4Compress *const this, const size_t required, Buffer *const output) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LZ4_COMPRESS, this); FUNCTION_TEST_PARAM(SIZE, required); FUNCTION_TEST_PARAM(BUFFER, output); FUNCTION_TEST_END(); Buffer *result = output; // Is an internal buffer required to hold the compressed data? if (!bufEmpty(this->buffer) || required >= bufRemains(output)) { // Reallocate buffer if it is not large enough if (required >= bufRemains(this->buffer)) bufResize(this->buffer, bufUsed(this->buffer) + required); result = this->buffer; } FUNCTION_TEST_RETURN(BUFFER, result); } // Helper to flush output data to compressed buffer static void lz4CompressFlush(Lz4Compress *const this, Buffer *const output, Buffer *const compressed) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LZ4_COMPRESS, this); FUNCTION_TEST_PARAM(BUFFER, output); FUNCTION_TEST_PARAM(BUFFER, compressed); FUNCTION_TEST_END(); // If the compressed buffer can hold all the output if (bufRemains(compressed) >= bufUsed(output)) { bufCat(compressed, output); bufUsedZero(output); this->inputSame = false; } // Else flush as much as possible and set inputSame to flush more once the compressed buffer has been emptied else { size_t catSize = bufRemains(compressed); bufCatSub(compressed, output, 0, catSize); memmove(bufPtr(output), bufPtr(output) + catSize, bufUsed(output) - catSize); bufUsedSet(output, bufUsed(output) - catSize); this->inputSame = true; } FUNCTION_TEST_RETURN_VOID(); } static void lz4CompressProcess(THIS_VOID, const Buffer *const uncompressed, Buffer *const compressed) { THIS(Lz4Compress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(LZ4_COMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, uncompressed); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!(this->flushing && !this->inputSame)); ASSERT(this->context != NULL); ASSERT(compressed != NULL); ASSERT(!this->flushing || uncompressed == NULL); // Flush overflow output to the compressed buffer if (this->inputSame) { lz4CompressFlush(this, this->buffer, compressed); } else { Buffer *output = NULL; // If first call to process then begin compression if (this->first) { output = lz4CompressBuffer(this, LZ4F_HEADER_SIZE_MAX, compressed); bufUsedInc( output, lz4Error(LZ4F_compressBegin(this->context, bufRemainsPtr(output), bufRemains(output), &this->prefs))); this->first = false; } // Normal processing call if (uncompressed != NULL) { output = lz4CompressBuffer(this, lz4Error(LZ4F_compressBound(bufUsed(uncompressed), &this->prefs)), compressed); bufUsedInc( output, lz4Error( LZ4F_compressUpdate( this->context, bufRemainsPtr(output), bufRemains(output), bufPtrConst(uncompressed), bufUsed(uncompressed), NULL))); } // Else flush remaining output else { // Pass 1 as src size to help allocate enough space for the final flush. This is required for some versions that don't // allocate enough memory unless autoFlush is enabled. Other versions fail if autoFlush is only enabled before the final // flush. This will hopefully work across all versions even if it does allocate a larger buffer than needed. output = lz4CompressBuffer(this, lz4Error(LZ4F_compressBound(1, &this->prefs)), compressed); bufUsedInc(output, lz4Error(LZ4F_compressEnd(this->context, bufRemainsPtr(output), bufRemains(output), NULL))); this->flushing = true; } // If the output buffer was allocated locally it will need to be flushed to the compressed buffer if (output != compressed) lz4CompressFlush(this, output, compressed); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is compress done? ***********************************************************************************************************************************/ static bool lz4CompressDone(const THIS_VOID) { THIS(const Lz4Compress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LZ4_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->flushing && !this->inputSame); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool lz4CompressInputSame(const THIS_VOID) { THIS(const Lz4Compress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LZ4_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * lz4CompressNew(const int level, const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INT, level); FUNCTION_LOG_PARAM(BOOL, raw); FUNCTION_LOG_END(); ASSERT(level >= LZ4_COMPRESS_LEVEL_MIN && level <= LZ4_COMPRESS_LEVEL_MAX); OBJ_NEW_BEGIN(Lz4Compress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (Lz4Compress) { .prefs = { .compressionLevel = level, .frameInfo = {.contentChecksumFlag = raw ? LZ4F_noContentChecksum : LZ4F_contentChecksumEnabled}, }, .first = true, .buffer = bufNew(0), }; // Create lz4 context lz4Error(LZ4F_createCompressionContext(&this->context, LZ4F_VERSION)); // Set callback to ensure lz4 context is freed memContextCallbackSet(objMemContext(this), lz4CompressFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( LZ4_COMPRESS_FILTER_TYPE, this, compressParamList(level, raw), .done = lz4CompressDone, .inOut = lz4CompressProcess, .inputSame = lz4CompressInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/lz4/compress.h0000664000175100017510000000306315265101745022033 0ustar00runnerrunner/*********************************************************************************************************************************** LZ4 Compress Compress IO to the lz4 format. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_LZ4_COMPRESS_H #define COMMON_COMPRESS_LZ4_COMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define LZ4_COMPRESS_FILTER_TYPE STRID6("lz4-cmp", 0x103436e068c1) /*********************************************************************************************************************************** Level constants ***********************************************************************************************************************************/ #define LZ4_COMPRESS_LEVEL_MIN -5 #define LZ4_COMPRESS_LEVEL_MAX 12 /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *lz4CompressNew(int level, bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/lz4/decompress.c0000664000175100017510000001566315265101745022350 0ustar00runnerrunner/*********************************************************************************************************************************** LZ4 Decompress ***********************************************************************************************************************************/ #include #include #include #include "common/compress/common.h" #include "common/compress/lz4/common.h" #include "common/compress/lz4/decompress.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Lz4Decompress { LZ4F_decompressionContext_t context; // LZ4 decompression context bool inputSame; // Is the same input required on the next process call? size_t inputOffset; // Current offset from the start of the buffer bool frameDone; // Has the current frame completed? bool done; // Is decompression done? } Lz4Decompress; /*********************************************************************************************************************************** Render as string for logging ***********************************************************************************************************************************/ static void lz4DecompressToLog(const Lz4Decompress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, inputOffset: %zu, frameDone %s, done: %s}", cvtBoolToConstZ(this->inputSame), this->inputOffset, cvtBoolToConstZ(this->frameDone), cvtBoolToConstZ(this->done)); } #define FUNCTION_LOG_LZ4_DECOMPRESS_TYPE \ Lz4Decompress * #define FUNCTION_LOG_LZ4_DECOMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, lz4DecompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free decompression context ***********************************************************************************************************************************/ static void lz4DecompressFreeResource(THIS_VOID) { THIS(Lz4Decompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(LZ4_DECOMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); LZ4F_freeDecompressionContext(this->context); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Decompress data ***********************************************************************************************************************************/ static void lz4DecompressProcess(THIS_VOID, const Buffer *const compressed, Buffer *const decompressed) { THIS(Lz4Decompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(LZ4_DECOMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_PARAM(BUFFER, decompressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->context != NULL); ASSERT(decompressed != NULL); // When there is no more input then decompression is done if (compressed == NULL) { // If the current frame being decompressed was not completed then error if (!this->frameDone) THROW(FormatError, "unexpected eof in compressed data"); this->done = true; } else { // Decompress as much data as possible size_t srcSize = bufUsed(compressed) - this->inputOffset; size_t dstSize = bufRemains(decompressed); this->frameDone = lz4Error( LZ4F_decompress( this->context, bufRemainsPtr(decompressed), &dstSize, bufPtrConst(compressed) + this->inputOffset, &srcSize, NULL)) == 0; bufUsedInc(decompressed, dstSize); // If the compressed data was not fully processed then update the offset and set inputSame if (srcSize < bufUsed(compressed) - this->inputOffset) { this->inputOffset += srcSize; this->inputSame = true; } // Else all compressed data was processed else { this->inputOffset = 0; this->inputSame = false; } } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is decompress done? ***********************************************************************************************************************************/ static bool lz4DecompressDone(const THIS_VOID) { THIS(const Lz4Decompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LZ4_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool lz4DecompressInputSame(const THIS_VOID) { THIS(const Lz4Decompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LZ4_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * lz4DecompressNew(const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); (void)raw; // Not required for decompress FUNCTION_LOG_END(); OBJ_NEW_BEGIN(Lz4Decompress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (Lz4Decompress){0}; // Create lz4 context lz4Error(LZ4F_createDecompressionContext(&this->context, LZ4F_VERSION)); // Set callback to ensure lz4 context is freed memContextCallbackSet(objMemContext(this), lz4DecompressFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( LZ4_DECOMPRESS_FILTER_TYPE, this, decompressParamList(raw), .done = lz4DecompressDone, .inOut = lz4DecompressProcess, .inputSame = lz4DecompressInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/lz4/decompress.h0000664000175100017510000000221515265101745022342 0ustar00runnerrunner/*********************************************************************************************************************************** LZ4 Decompress Decompress IO from the lz4 format. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_LZ4_DECOMPRESS_H #define COMMON_COMPRESS_LZ4_DECOMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define LZ4_DECOMPRESS_FILTER_TYPE STRID6("lz4-dcmp", 0x40d0c46e068c1) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *lz4DecompressNew(bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/zst/0000775000175100017510000000000015265101745020134 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/zst/common.c0000664000175100017510000000200615265101745021566 0ustar00runnerrunner/*********************************************************************************************************************************** ZST Common ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBZST #include // Check the version -- this is done in configure but it makes sense to be sure #if ZSTD_VERSION_MAJOR < 1 #error "ZSTD_VERSION_MAJOR must be >= 1" #endif #include "common/compress/zst/common.h" #include "common/debug.h" /**********************************************************************************************************************************/ FN_EXTERN size_t zstError(const size_t error) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, error); FUNCTION_TEST_END(); if (ZSTD_isError(error)) THROW_FMT(FormatError, "zst error: [%zd] %s", (ssize_t)error, ZSTD_getErrorName(error)); FUNCTION_TEST_RETURN(SIZE, error); } #endif // HAVE_LIBZST ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/zst/common.h0000664000175100017510000000224715265101745021602 0ustar00runnerrunner/*********************************************************************************************************************************** ZST Common Developed using the documentation in https://github.com/facebook/zstd/blob/v1.0.0/lib/zstd.h. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_ZST_COMMON_H #define COMMON_COMPRESS_ZST_COMMON_H #include /*********************************************************************************************************************************** ZST extension ***********************************************************************************************************************************/ #define ZST_EXT "zst" #ifdef HAVE_LIBZST /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN size_t zstError(size_t error); #endif // HAVE_LIBZST #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/zst/compress.c0000664000175100017510000001636215265101745022143 0ustar00runnerrunner/*********************************************************************************************************************************** ZST Compress ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBZST #include #include "common/compress/common.h" #include "common/compress/zst/common.h" #include "common/compress/zst/compress.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ZstCompress { ZSTD_CStream *context; // Compression context int level; // Compression level bool inputSame; // Is the same input required on the next process call? size_t inputOffset; // Current offset in input buffer bool flushing; // Is input complete and flushing in progress? } ZstCompress; /*********************************************************************************************************************************** Render as string for logging ***********************************************************************************************************************************/ static void zstCompressToLog(const ZstCompress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{level: %d, inputSame: %s, inputOffset: %zu, flushing: %s}", this->level, cvtBoolToConstZ(this->inputSame), this->inputOffset, cvtBoolToConstZ(this->flushing)); } #define FUNCTION_LOG_ZST_COMPRESS_TYPE \ ZstCompress * #define FUNCTION_LOG_ZST_COMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, zstCompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free compression context ***********************************************************************************************************************************/ static void zstCompressFreeResource(THIS_VOID) { THIS(ZstCompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ZST_COMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ZSTD_freeCStream(this->context); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Compress data ***********************************************************************************************************************************/ static void zstCompressProcess(THIS_VOID, const Buffer *const uncompressed, Buffer *const compressed) { THIS(ZstCompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ZST_COMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, uncompressed); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!(this->flushing && !this->inputSame)); ASSERT(this->context != NULL); ASSERT(compressed != NULL); ASSERT(!this->flushing || uncompressed == NULL); // Initialize output buffer ZSTD_outBuffer out = {.dst = bufRemainsPtr(compressed), .size = bufRemains(compressed)}; // If input is NULL then start flushing if (uncompressed == NULL) { this->flushing = true; this->inputSame = zstError(ZSTD_endStream(this->context, &out)) != 0; } // Else still have input data else { // Initialize input buffer ZSTD_inBuffer in = { .src = bufPtrConst(uncompressed) + this->inputOffset, .size = bufUsed(uncompressed) - this->inputOffset, }; // Perform compression zstError(ZSTD_compressStream(this->context, &out, &in)); // If the input buffer was not entirely consumed then set inputSame and store the offset where processing will restart if (in.pos < in.size) { // Output buffer should be completely full ASSERT(out.pos == out.size); this->inputSame = true; this->inputOffset += in.pos; } // Else ready for more input else { this->inputSame = false; this->inputOffset = 0; } } bufUsedInc(compressed, out.pos); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is compress done? ***********************************************************************************************************************************/ static bool zstCompressDone(const THIS_VOID) { THIS(const ZstCompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ZST_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->flushing && !this->inputSame); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool zstCompressInputSame(const THIS_VOID) { THIS(const ZstCompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ZST_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * zstCompressNew(const int level, const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INT, level); (void)raw; // Raw unsupported FUNCTION_LOG_END(); ASSERT(level >= ZST_COMPRESS_LEVEL_MIN && level <= ZST_COMPRESS_LEVEL_MAX); OBJ_NEW_BEGIN(ZstCompress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (ZstCompress) { .context = ZSTD_createCStream(), .level = level, }; // Set callback to ensure zst context is freed memContextCallbackSet(objMemContext(this), zstCompressFreeResource, this); // Initialize context zstError(ZSTD_initCStream(this->context, this->level)); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( ZST_COMPRESS_FILTER_TYPE, this, compressParamList(level, raw), .done = zstCompressDone, .inOut = zstCompressProcess, .inputSame = zstCompressInputSame)); } #endif // HAVE_LIBZST ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/zst/compress.h0000664000175100017510000000313415265101745022141 0ustar00runnerrunner/*********************************************************************************************************************************** ZST Compress Compress IO to the zst format. ***********************************************************************************************************************************/ #ifdef HAVE_LIBZST #ifndef COMMON_COMPRESS_ZST_COMPRESS_H #define COMMON_COMPRESS_ZST_COMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define ZST_COMPRESS_FILTER_TYPE STRID5("zst-cmp", 0x41a3dd27a0) /*********************************************************************************************************************************** Level constants ***********************************************************************************************************************************/ #define ZST_COMPRESS_LEVEL_MIN -7 #define ZST_COMPRESS_LEVEL_MAX 22 /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *zstCompressNew(int level, bool raw); #endif #endif // HAVE_LIBZST ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/zst/decompress.c0000664000175100017510000001611215265101745022445 0ustar00runnerrunner/*********************************************************************************************************************************** ZST Decompress ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBZST #include #include "common/compress/common.h" #include "common/compress/zst/common.h" #include "common/compress/zst/decompress.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ZstDecompress { ZSTD_DStream *context; // Decompression context bool inputSame; // Is the same input required on the next process call? size_t inputOffset; // Current offset in input buffer bool frameDone; // Has the current frame completed? bool done; // Is decompression done? } ZstDecompress; /*********************************************************************************************************************************** Render as string for logging ***********************************************************************************************************************************/ static void zstDecompressToLog(const ZstDecompress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, inputOffset: %zu, frameDone %s, done: %s}", cvtBoolToConstZ(this->inputSame), this->inputOffset, cvtBoolToConstZ(this->frameDone), cvtBoolToConstZ(this->done)); } #define FUNCTION_LOG_ZST_DECOMPRESS_TYPE \ ZstDecompress * #define FUNCTION_LOG_ZST_DECOMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, zstDecompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free decompression context ***********************************************************************************************************************************/ static void zstDecompressFreeResource(THIS_VOID) { THIS(ZstDecompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ZST_DECOMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ZSTD_freeDStream(this->context); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Decompress data ***********************************************************************************************************************************/ static void zstDecompressProcess(THIS_VOID, const Buffer *const compressed, Buffer *const decompressed) { THIS(ZstDecompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ZST_DECOMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_PARAM(BUFFER, decompressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->context != NULL); ASSERT(decompressed != NULL); // When there is no more input then decompression is done if (compressed == NULL) { // If the current frame being decompressed was not completed then error if (!this->frameDone) THROW(FormatError, "unexpected eof in compressed data"); this->done = true; } else { // Initialize input/output buffer ZSTD_inBuffer in = {.src = bufPtrConst(compressed) + this->inputOffset, .size = bufUsed(compressed) - this->inputOffset}; ZSTD_outBuffer out = {.dst = bufRemainsPtr(decompressed), .size = bufRemains(decompressed)}; // Perform decompression. Track frame done so we can detect unexpected EOF. this->frameDone = zstError(ZSTD_decompressStream(this->context, &out, &in)) == 0; bufUsedInc(decompressed, out.pos); // If the input buffer was not entirely consumed then set inputSame and store the offset where processing will restart if (in.pos < in.size) { // Output buffer should be completely full ASSERT(out.pos == out.size); this->inputSame = true; this->inputOffset += in.pos; } // Else ready for more input else { this->inputOffset = 0; this->inputSame = false; } } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is decompress done? ***********************************************************************************************************************************/ static bool zstDecompressDone(const THIS_VOID) { THIS(const ZstDecompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ZST_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool zstDecompressInputSame(const THIS_VOID) { THIS(const ZstDecompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ZST_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * zstDecompressNew(const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); (void)raw; // Raw unsupported FUNCTION_LOG_END(); OBJ_NEW_BEGIN(ZstDecompress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (ZstDecompress) { .context = ZSTD_createDStream(), }; // Set callback to ensure zst context is freed memContextCallbackSet(objMemContext(this), zstDecompressFreeResource, this); // Initialize context zstError(ZSTD_initDStream(this->context)); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( ZST_DECOMPRESS_FILTER_TYPE, this, decompressParamList(raw), .done = zstDecompressDone, .inOut = zstDecompressProcess, .inputSame = zstDecompressInputSame)); } #endif // HAVE_LIBZST ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/compress/zst/decompress.h0000664000175100017510000000226615265101745022457 0ustar00runnerrunner/*********************************************************************************************************************************** ZST Decompress Decompress IO from the zst format. ***********************************************************************************************************************************/ #ifdef HAVE_LIBZST #ifndef COMMON_COMPRESS_ZST_DECOMPRESS_H #define COMMON_COMPRESS_ZST_DECOMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define ZST_DECOMPRESS_FILTER_TYPE STRID5("zst-dcmp", 0x83464dd27a0) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *zstDecompressNew(bool raw); #endif #endif // HAVE_LIBZST ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/0000775000175100017510000000000015265101745017001 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/cipherBlock.c0000664000175100017510000004504315265101745021400 0ustar00runnerrunner/*********************************************************************************************************************************** Block Cipher ***********************************************************************************************************************************/ #include #include #include #include #include "common/crypto/cipherBlock.h" #include "common/crypto/common.h" #include "common/crypto/common.intern.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/convert.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct CipherBlock { CipherMode mode; // Mode encrypt/decrypt bool raw; // Omit header magic to save space bool saltDone; // Has the salt been read/generated? bool processDone; // Has any data been processed? const Buffer *pass; // Passphrase used to generate encryption key size_t headerSize; // Size of header read during decrypt uint8_t header[CIPHER_BLOCK_MAGIC_SIZE + PKCS5_SALT_LEN]; // Buffer to hold partial header during decrypt const EVP_CIPHER *cipher; // Cipher object const EVP_MD *digest; // Message digest object EVP_CIPHER_CTX *cipherContext; // Encrypt/decrypt context Buffer *buffer; // Internal buffer in case destination buffer isn't large enough bool inputSame; // Is the same input required on next process call? bool done; // Is processing done? } CipherBlock; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void cipherBlockToLog(const CipherBlock *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{inputSame: %s, done: %s}", cvtBoolToConstZ(this->inputSame), cvtBoolToConstZ(this->done)); } #define FUNCTION_LOG_CIPHER_BLOCK_TYPE \ CipherBlock * #define FUNCTION_LOG_CIPHER_BLOCK_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, cipherBlockToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free cipher context ***********************************************************************************************************************************/ static void cipherBlockFreeResource(THIS_VOID) { THIS(CipherBlock); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CIPHER_BLOCK, this); FUNCTION_LOG_END(); ASSERT(this != NULL); EVP_CIPHER_CTX_free(this->cipherContext); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Determine how large the destination buffer should be ***********************************************************************************************************************************/ static size_t cipherBlockProcessSize(const CipherBlock *const this, const size_t sourceSize) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CIPHER_BLOCK, this); FUNCTION_LOG_PARAM(SIZE, sourceSize); FUNCTION_LOG_END(); ASSERT(this != NULL); // Destination size is source size plus one extra block size_t destinationSize = sourceSize + EVP_MAX_BLOCK_LENGTH; // On encrypt the header size must be included before the first block if (this->mode == cipherModeEncrypt && !this->saltDone) destinationSize += CIPHER_BLOCK_MAGIC_SIZE + PKCS5_SALT_LEN; FUNCTION_LOG_RETURN(SIZE, destinationSize); } /*********************************************************************************************************************************** Encrypt/decrypt data ***********************************************************************************************************************************/ static size_t cipherBlockProcessBlock(CipherBlock *const this, const uint8_t *source, size_t sourceSize, uint8_t *destination) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CIPHER_BLOCK, this); FUNCTION_LOG_PARAM_P(BYTEDATA, source); FUNCTION_LOG_PARAM(SIZE, sourceSize); FUNCTION_LOG_PARAM_P(BYTEDATA, destination); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(source != NULL || sourceSize == 0); ASSERT(destination != NULL); // Actual destination size size_t destinationSize = 0; // If the salt has not been generated/read yet if (!this->saltDone) { const uint8_t *salt = NULL; // On encrypt the salt is generated if (this->mode == cipherModeEncrypt) { // Add magic to the destination buffer so openssl knows the file is salted if (!this->raw) { memcpy(destination, CIPHER_BLOCK_MAGIC, CIPHER_BLOCK_MAGIC_SIZE); destination += CIPHER_BLOCK_MAGIC_SIZE; destinationSize += CIPHER_BLOCK_MAGIC_SIZE; } // Add salt to the destination buffer cryptoRandomBytes(destination, PKCS5_SALT_LEN); salt = destination; destination += PKCS5_SALT_LEN; destinationSize += PKCS5_SALT_LEN; } // On decrypt the salt is read from the header else if (sourceSize > 0) { // Check if the entire header has been read const size_t headerExpected = this->raw ? PKCS5_SALT_LEN : CIPHER_BLOCK_MAGIC_SIZE + PKCS5_SALT_LEN; if (this->headerSize + sourceSize >= headerExpected) { // Copy header (or remains of header) from source into the header buffer memcpy(this->header + this->headerSize, source, headerExpected - this->headerSize); salt = this->header + (this->raw ? 0 : CIPHER_BLOCK_MAGIC_SIZE); // Advance source and source size by the number of bytes read source += headerExpected - this->headerSize; sourceSize -= headerExpected - this->headerSize; // The first bytes of the file to decrypt should be equal to the magic. If not then this is not an encrypted // file, or at least not in a format we recognize. if (!this->raw && memcmp(this->header, CIPHER_BLOCK_MAGIC, CIPHER_BLOCK_MAGIC_SIZE) != 0) THROW(CryptoError, "cipher header invalid"); } // Else copy what was provided into the header buffer and return 0 else { memcpy(this->header + this->headerSize, source, sourceSize); this->headerSize += sourceSize; // Indicate that there is nothing left to process sourceSize = 0; } } // If salt generation/read is done if (salt) { // Generate key and initialization vector uint8_t key[EVP_MAX_KEY_LENGTH]; uint8_t initVector[EVP_MAX_IV_LENGTH]; cryptoError( EVP_BytesToKey( this->cipher, this->digest, salt, bufPtrConst(this->pass), (int)bufSize(this->pass), 1, key, initVector) == 0, "unable to generate key and initialization vector"); // Create context to track cipher cryptoError(!(this->cipherContext = EVP_CIPHER_CTX_new()), "unable to create context"); // Set free callback to ensure cipher context is freed memContextCallbackSet(objMemContext(this), cipherBlockFreeResource, this); // Initialize cipher cryptoError( !EVP_CipherInit_ex(this->cipherContext, this->cipher, NULL, key, initVector, this->mode == cipherModeEncrypt), "unable to initialize cipher"); this->saltDone = true; } } // Recheck that source size > 0 as the bytes may have been consumed reading the header if (sourceSize > 0) { // Process the data int destinationUpdateSize = 0; cryptoError( !EVP_CipherUpdate(this->cipherContext, destination, &destinationUpdateSize, source, (int)sourceSize), "unable to process cipher"); destinationSize += (size_t)destinationUpdateSize; // Note that data has been processed so flush is valid this->processDone = true; } // Return actual destination size FUNCTION_LOG_RETURN(SIZE, destinationSize); } /*********************************************************************************************************************************** Flush the remaining data ***********************************************************************************************************************************/ static size_t cipherBlockFlush(CipherBlock *const this, Buffer *const destination) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CIPHER_BLOCK, this); FUNCTION_LOG_PARAM(BUFFER, destination); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(destination != NULL); // Actual destination size int destinationSize = 0; // If no header was processed then error if (!this->saltDone) THROW(CryptoError, "cipher header missing"); // Only flush remaining data if some data was processed if (!EVP_CipherFinal(this->cipherContext, bufRemainsPtr(destination), &destinationSize)) THROW(CryptoError, "unable to flush"); // Return actual destination size FUNCTION_LOG_RETURN(SIZE, (size_t)destinationSize); } /*********************************************************************************************************************************** Process function used by C filter ***********************************************************************************************************************************/ static void cipherBlockProcess(THIS_VOID, const Buffer *const source, Buffer *const destination) { THIS(CipherBlock); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CIPHER_BLOCK, this); FUNCTION_LOG_PARAM(BUFFER, source); FUNCTION_LOG_PARAM(BUFFER, destination); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(destination != NULL); ASSERT(bufRemains(destination) > 0); // Copy already buffered bytes if (this->buffer != NULL && !bufEmpty(this->buffer)) { if (bufRemains(destination) >= bufUsed(this->buffer)) { bufCat(destination, this->buffer); bufUsedZero(this->buffer); this->inputSame = false; } else { const size_t catSize = bufRemains(destination); bufCatSub(destination, this->buffer, 0, catSize); memmove(bufPtr(this->buffer), bufPtr(this->buffer) + catSize, bufUsed(this->buffer) - catSize); bufUsedSet(this->buffer, bufUsed(this->buffer) - catSize); this->inputSame = true; } } else { ASSERT(this->buffer == NULL || bufEmpty(this->buffer)); // Determine how much space is required in the output buffer Buffer *outputActual = destination; const size_t destinationSize = cipherBlockProcessSize(this, source == NULL ? 0 : bufUsed(source)); if (destinationSize > bufRemains(destination)) { // Allocate the buffer if needed MEM_CONTEXT_OBJ_BEGIN(this) { if (this->buffer == NULL) { this->buffer = bufNew(destinationSize); } // Resize buffer if needed else bufResize(this->buffer, destinationSize); } MEM_CONTEXT_OBJ_END(); outputActual = this->buffer; } // Encrypt/decrypt bytes size_t destinationSizeActual; if (source == NULL) { // If salt was not generated it means that process() was never called with any data. It's OK to encrypt a zero byte file // but we need to call process to generate the header. if (!this->saltDone) { destinationSizeActual = cipherBlockProcessBlock(this, NULL, 0, bufRemainsPtr(outputActual)); bufUsedInc(outputActual, destinationSizeActual); } destinationSizeActual = cipherBlockFlush(this, outputActual); this->done = true; } else { destinationSizeActual = cipherBlockProcessBlock( this, bufPtrConst(source), bufUsed(source), bufRemainsPtr(outputActual)); } bufUsedInc(outputActual, destinationSizeActual); // Copy from buffer to destination if needed if (this->buffer != NULL && !bufEmpty(this->buffer)) cipherBlockProcess(this, source, destination); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is cipher done? ***********************************************************************************************************************************/ static bool cipherBlockDone(const THIS_VOID) { THIS(const CipherBlock); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CIPHER_BLOCK, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done && !this->inputSame); } /*********************************************************************************************************************************** Should the same input be provided again? ***********************************************************************************************************************************/ static bool cipherBlockInputSame(const THIS_VOID) { THIS(const CipherBlock); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CIPHER_BLOCK, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * cipherBlockNew(const CipherMode mode, const CipherSpec *const cipherSpec, const CipherBlockNewParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, mode); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_PARAM(ENUM, param.header); FUNCTION_LOG_END(); ASSERT(cipherSpec != NULL); ASSERT(cipherSpecType(cipherSpec) != cipherTypeNone); ASSERT(cipherSpecDigest(cipherSpec) != 0); ASSERT(cipherSpecPass(cipherSpec) != NULL && !bufEmpty(cipherSpecPass(cipherSpec))); // Init crypto subsystem cryptoInit(); // Lookup cipher by name. This means the ciphers passed in must exactly match a name expected by OpenSSL. This is a good thing // since the name required by the openssl command-line tool will match what is used by pgBackRest. char *const cipherTypeZ = zNewStrId(cipherSpecType(cipherSpec)); const EVP_CIPHER *cipher = EVP_get_cipherbyname(cipherTypeZ); if (!cipher) THROW_FMT(AssertError, "unable to load cipher '%s'", cipherTypeZ); zFree(cipherTypeZ); // Lookup digest, which the spec supplies since the pass and the digest are chosen together const EVP_MD *const digest = cryptoDigest(cipherSpecDigest(cipherSpec)); OBJ_NEW_BEGIN(CipherBlock, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (CipherBlock) { .mode = mode, .raw = param.header == cipherBlockHeaderNone, .cipher = cipher, .digest = digest, .pass = bufDup(cipherSpecPass(cipherSpec)), }; } OBJ_NEW_END(); // Create param list Pack *paramList; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteU64P(packWrite, mode); cipherSpecPack(packWrite, cipherSpec); pckWriteU32P(packWrite, param.header); pckWriteEndP(packWrite); paramList = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( CIPHER_BLOCK_FILTER_TYPE, this, paramList, .done = cipherBlockDone, .inOut = cipherBlockProcess, .inputSame = cipherBlockInputSame)); } FN_EXTERN IoFilter * cipherBlockNewPack(const Pack *const paramList) { IoFilter *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackRead *const paramListPack = pckReadNew(paramList); const CipherMode cipherMode = (CipherMode)pckReadU64P(paramListPack); const CipherSpec *const cipherSpec = cipherSpecNewPack(paramListPack); const CipherBlockHeader header = (CipherBlockHeader)pckReadU32P(paramListPack); result = ioFilterMove(cipherBlockNewP(cipherMode, cipherSpec, .header = header), memContextPrior()); } MEM_CONTEXT_TEMP_END(); return result; } /**********************************************************************************************************************************/ FN_EXTERN IoFilterGroup * cipherBlockFilterGroupAdd(IoFilterGroup *const filterGroup, const CipherMode mode, const CipherSpec *const cipherSpec) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_LOG_PARAM(STRING_ID, mode); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_END(); ASSERT(filterGroup != NULL); ASSERT(cipherSpec != NULL); if (cipherSpecType(cipherSpec) != cipherTypeNone) ioFilterGroupAdd(filterGroup, cipherBlockNewP(mode, cipherSpec)); FUNCTION_LOG_RETURN(IO_FILTER_GROUP, filterGroup); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/cipherBlock.h0000664000175100017510000000650715265101745021407 0ustar00runnerrunner/*********************************************************************************************************************************** Block Cipher Header ***********************************************************************************************************************************/ #ifndef COMMON_CRYPTO_CIPHERBLOCK_H #define COMMON_CRYPTO_CIPHERBLOCK_H #include "common/crypto/spec.h" #include "common/io/filter/group.h" /*********************************************************************************************************************************** Magic constant for salted encrypt, written before the salt unless the header is none. Anything written in place of the magic must be the same size so that what follows still begins with the salt. ***********************************************************************************************************************************/ #define CIPHER_BLOCK_MAGIC "Salted__" #define CIPHER_BLOCK_MAGIC_SIZE (sizeof(CIPHER_BLOCK_MAGIC) - 1) /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define CIPHER_BLOCK_FILTER_TYPE STRID5("cipher-blk", 0x16c16e45441230) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ // What precedes the content of a file. The magic is what openssl writes and is the default. None saves the eight bytes the magic // takes for a file nothing but this version reads, e.g. a file bundled into a backup, and for a file that something else has // written in place of the magic, e.g. the format header. typedef enum { cipherBlockHeaderMagic = 0, // Salted magic openssl writes cipherBlockHeaderNone, // Nothing, to save space } CipherBlockHeader; typedef struct CipherBlockNewParam { VAR_PARAM_HEADER; CipherBlockHeader header; // What precedes the content, the magic by default } CipherBlockNewParam; #define cipherBlockNewP(mode, cipherSpec, ...) \ cipherBlockNew(mode, cipherSpec, (CipherBlockNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN IoFilter *cipherBlockNew(CipherMode mode, const CipherSpec *cipherSpec, CipherBlockNewParam param); FN_EXTERN IoFilter *cipherBlockNewPack(const Pack *paramList); /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Add a block cipher to an io object. Nothing is added when the repository is not encrypted. FN_EXTERN IoFilterGroup *cipherBlockFilterGroupAdd(IoFilterGroup *filterGroup, CipherMode mode, const CipherSpec *cipherSpec); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/common.c0000664000175100017510000000712715265101745020444 0ustar00runnerrunner/*********************************************************************************************************************************** Crypto Common ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "common/crypto/common.h" #include "common/crypto/common.intern.h" #include "common/debug.h" #include "common/log.h" /*********************************************************************************************************************************** Flag to indicate if OpenSSL has already been initialized ***********************************************************************************************************************************/ static bool cryptoInitDone = false; /**********************************************************************************************************************************/ FN_EXTERN void cryptoError(const bool error, const char *const description) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, error); FUNCTION_TEST_PARAM(STRINGZ, description); FUNCTION_TEST_END(); if (error) cryptoErrorCode(ERR_get_error(), description); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void cryptoErrorCode(const unsigned long code, const char *const description) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, code); FUNCTION_TEST_PARAM(STRINGZ, description); FUNCTION_TEST_END(); const char *const errorMessage = ERR_reason_error_string(code); THROW_FMT(CryptoError, "%s: [%lu] %s", description, code, errorMessage == NULL ? "no details available" : errorMessage); FUNCTION_TEST_NO_RETURN(); } /**********************************************************************************************************************************/ FN_EXTERN void cryptoInit(void) { FUNCTION_LOG_VOID(logLevelTrace); if (!cryptoInitDone) { // Load crypto strings and algorithms OPENSSL_init_crypto( OPENSSL_INIT_LOAD_CRYPTO_STRINGS | OPENSSL_INIT_ADD_ALL_CIPHERS | OPENSSL_INIT_ADD_ALL_DIGESTS, NULL); // Initialization OPENSSL_init_ssl(OPENSSL_INIT_LOAD_CONFIG, NULL); // Mark crypto as initialized cryptoInitDone = true; } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cryptoRandomBytes(uint8_t *const buffer, const size_t size) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(BYTEDATA, buffer); FUNCTION_LOG_PARAM(SIZE, size); FUNCTION_LOG_END(); ASSERT(buffer != NULL); ASSERT(size > 0); // A failure must not go unnoticed since the caller would be left with bytes that are not random cryptoError(RAND_bytes(buffer, (int)size) != 1, "unable to generate random bytes"); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN const EVP_MD * cryptoDigest(const HashType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_END(); char typeZ[STRID_MAX + 1]; strIdToZ(type, typeZ); const EVP_MD *const result = EVP_get_digestbyname(typeZ); if (result == NULL) THROW_FMT(AssertError, "unable to load digest '%s'", typeZ); FUNCTION_TEST_RETURN_TYPE_CONST_P(EVP_MD, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/common.h0000664000175100017510000000434015265101745020443 0ustar00runnerrunner/*********************************************************************************************************************************** Crypto Common ***********************************************************************************************************************************/ #ifndef COMMON_CRYPTO_COMMON_H #define COMMON_CRYPTO_COMMON_H #include /*********************************************************************************************************************************** Cipher modes ***********************************************************************************************************************************/ typedef enum { cipherModeEncrypt = STRID5("encrypt", 0x521990dc50), cipherModeDecrypt = STRID5("decrypt", 0x521990ca40), } CipherMode; /*********************************************************************************************************************************** Cipher types ***********************************************************************************************************************************/ typedef enum { cipherTypeNone = STRID5("none", 0x2b9ee0), cipherTypeAes256Cbc = STRID5("aes-256-cbc", 0xc43dfbbcdcca10), } CipherType; /*********************************************************************************************************************************** Hash types ***********************************************************************************************************************************/ typedef enum { hashTypeMd5 = STRID5("md5", 0x748d0), hashTypeSha1 = STRID6("sha1", 0x7412131), hashTypeSha256 = STRID5("sha256", 0x3dde05130), } HashType; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize crypto FN_EXTERN void cryptoInit(void); // Throw crypto errors FN_EXTERN void cryptoError(bool error, const char *description); FN_EXTERN noreturn void cryptoErrorCode(unsigned long code, const char *description); // Generate random bytes FN_EXTERN void cryptoRandomBytes(uint8_t *buffer, size_t size); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/common.intern.h0000664000175100017510000000206315265101745021741 0ustar00runnerrunner/*********************************************************************************************************************************** Crypto Common Internal Openssl types are used here, so this is included only by the crypto modules that work with them rather than by everything that needs a hash type. ***********************************************************************************************************************************/ #ifndef COMMON_CRYPTO_COMMON_INTERN_H #define COMMON_CRYPTO_COMMON_INTERN_H #include #include "common/crypto/common.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Digest openssl knows by the name of the hash type. Error when there is no such digest, which is a coding error since a hash type // is not read from user input. FN_EXTERN const EVP_MD *cryptoDigest(HashType type); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/hash.c0000664000175100017510000002570715265101745020103 0ustar00runnerrunner/*********************************************************************************************************************************** Cryptographic Hash ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/crypto/common.h" #include "common/crypto/common.intern.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Hashes for zero-length files (i.e., seed value) ***********************************************************************************************************************************/ BUFFER_EXTERN( HASH_TYPE_SHA1_ZERO_BUF, 0xda, 0x39, 0xa3, 0xee, 0x5e, 0x6b, 0x4b, 0x0d, 0x32, 0x55, 0xbf, 0xef, 0x95, 0x60, 0x18, 0x90, 0xaf, 0xd8, 0x07, 0x09); BUFFER_EXTERN( HASH_TYPE_SHA256_ZERO_BUF, 0xe3, 0xb0, 0xc4, 0x42, 0x98, 0xfc, 0x1c, 0x14, 0x9a, 0xfb, 0xf4, 0xc8, 0x99, 0x6f, 0xb9, 0x24, 0x27, 0xae, 0x41, 0xe4, 0x64, 0x9b, 0x93, 0x4c, 0xa4, 0x95, 0x99, 0x1b, 0x78, 0x52, 0xb8, 0x55); /*********************************************************************************************************************************** Include local MD5 code ***********************************************************************************************************************************/ #include "common/crypto/md5.vendor.c.inc" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct CryptoHash { const EVP_MD *hashType; // Hash type (sha1, md5, etc.) EVP_MD_CTX *hashContext; // Message hash context MD5_CTX md5Context; // MD5 context (used to bypass FIPS restrictions) Buffer *hash; // Hash in binary form } CryptoHash; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_CRYPTO_HASH_TYPE \ CryptoHash * #define FUNCTION_LOG_CRYPTO_HASH_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "CryptoHash", buffer, bufferSize) /*********************************************************************************************************************************** Free hash context ***********************************************************************************************************************************/ static void cryptoHashFreeResource(THIS_VOID) { THIS(CryptoHash); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CRYPTO_HASH, this); FUNCTION_LOG_END(); ASSERT(this != NULL); EVP_MD_CTX_free(this->hashContext); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Add message data to the hash from a Buffer ***********************************************************************************************************************************/ static void cryptoHashProcess(THIS_VOID, const Buffer *const message) { THIS(CryptoHash); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CRYPTO_HASH, this); FUNCTION_LOG_PARAM(BUFFER, message); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->hash == NULL); ASSERT(message != NULL); // Standard OpenSSL implementation if (this->hashContext != NULL) { cryptoError(!EVP_DigestUpdate(this->hashContext, bufPtrConst(message), bufUsed(message)), "unable to process message hash"); } // Else local MD5 implementation else MD5_Update(&this->md5Context, bufPtrConst(message), bufUsed(message)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get binary representation of the hash ***********************************************************************************************************************************/ static const Buffer * cryptoHash(CryptoHash *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CRYPTO_HASH, this); FUNCTION_LOG_END(); ASSERT(this != NULL); if (this->hash == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { // Standard OpenSSL implementation if (this->hashContext != NULL) { this->hash = bufNew((size_t)EVP_MD_size(this->hashType)); cryptoError(!EVP_DigestFinal_ex(this->hashContext, bufPtr(this->hash), NULL), "unable to finalize message hash"); } // Else local MD5 implementation else { this->hash = bufNew(HASH_TYPE_M5_SIZE); MD5_Final(bufPtr(this->hash), &this->md5Context); } bufUsedSet(this->hash, bufSize(this->hash)); } MEM_CONTEXT_OBJ_END(); } FUNCTION_LOG_RETURN(BUFFER, this->hash); } /*********************************************************************************************************************************** Get string representation of the hash as a filter result ***********************************************************************************************************************************/ static Pack * cryptoHashResult(THIS_VOID) { THIS(CryptoHash); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CRYPTO_HASH, this); FUNCTION_LOG_END(); ASSERT(this != NULL); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteBinP(packWrite, cryptoHash(this)); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * cryptoHashNew(const HashType type) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_END(); ASSERT(type != 0); // Init crypto subsystem cryptoInit(); OBJ_NEW_BEGIN(CryptoHash, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (CryptoHash){0}; // Use local MD5 implementation since FIPS-enabled systems do not allow MD5. This is a bit misguided since there are valid // cases for using MD5 which do not involve, for example, password hashes. Since popular object stores, e.g. S3, require // MD5 for verifying payload integrity we are simply forced to provide MD5 functionality. if (type == hashTypeMd5) { MD5_Init(&this->md5Context); } // Else use the standard OpenSSL implementation else { this->hashType = cryptoDigest(type); // Create context cryptoError((this->hashContext = EVP_MD_CTX_new()) == NULL, "unable to create hash context"); // Set free callback to ensure hash context is freed memContextCallbackSet(objMemContext(this), cryptoHashFreeResource, this); // Initialize context cryptoError(!EVP_DigestInit_ex(this->hashContext, this->hashType, NULL), "unable to initialize hash context"); } } OBJ_NEW_END(); // Create param list Pack *paramList; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteStrIdP(packWrite, type); pckWriteEndP(packWrite); paramList = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP(CRYPTO_HASH_FILTER_TYPE, this, paramList, .in = cryptoHashProcess, .result = cryptoHashResult)); } FN_EXTERN IoFilter * cryptoHashNewPack(const Pack *const paramList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK, paramList); FUNCTION_TEST_END(); IoFilter *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { result = ioFilterMove(cryptoHashNew(pckReadStrIdP(pckReadNew(paramList))), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(IO_FILTER, result); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * cryptoHashOne(const HashType type, const Buffer *const message) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(BUFFER, message); FUNCTION_LOG_END(); ASSERT(type != 0); ASSERT(message != NULL); Buffer *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { IoFilter *const hash = cryptoHashNew(type); if (!bufEmpty(message)) ioFilterProcessIn(hash, message); const Buffer *const buffer = cryptoHash((CryptoHash *)ioFilterDriver(hash)); MEM_CONTEXT_PRIOR_BEGIN() { result = bufDup(buffer); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BUFFER, result); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * cryptoHmacOne(const HashType type, const Buffer *const key, const Buffer *const message) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(BUFFER, key); FUNCTION_LOG_PARAM(BUFFER, message); FUNCTION_LOG_END(); ASSERT(type != 0); ASSERT(key != NULL); ASSERT(message != NULL); // Init crypto subsystem cryptoInit(); const EVP_MD *const hashType = cryptoDigest(type); // Allocate a buffer to hold the hmac Buffer *const result = bufNew((size_t)EVP_MD_size(hashType)); bufUsedSet(result, bufSize(result)); // Calculate the HMAC cryptoError( HMAC(hashType, bufPtrConst(key), (int)bufUsed(key), bufPtrConst(message), bufUsed(message), bufPtr(result), NULL) == NULL, "unable to calculate hmac"); FUNCTION_LOG_RETURN(BUFFER, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/hash.h0000664000175100017510000000652515265101745020105 0ustar00runnerrunner/*********************************************************************************************************************************** Cryptographic Hash Generate a hash (sha1, md5, etc.) from a string, Buffer, or using an IoFilter. ***********************************************************************************************************************************/ #ifndef COMMON_CRYPTO_HASH_H #define COMMON_CRYPTO_HASH_H #include "common/crypto/common.h" #include "common/io/filter/filter.h" #include "common/type/string.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define CRYPTO_HASH_FILTER_TYPE STRID5("hash", 0x44c280) /*********************************************************************************************************************************** Hashes for zero-length files (i.e., starting hash) ***********************************************************************************************************************************/ #define HASH_TYPE_MD5_ZERO "d41d8cd98f00b204e9800998ecf8427e" #define HASH_TYPE_SHA1_ZERO "da39a3ee5e6b4b0d3255bfef95601890afd80709" BUFFER_DECLARE(HASH_TYPE_SHA1_ZERO_BUF); #define HASH_TYPE_SHA256_ZERO \ "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" BUFFER_DECLARE(HASH_TYPE_SHA256_ZERO_BUF); /*********************************************************************************************************************************** Hash type sizes ***********************************************************************************************************************************/ #define HASH_TYPE_M5_SIZE 16 #define HASH_TYPE_MD5_SIZE_HEX (HASH_TYPE_M5_SIZE * 2) #define HASH_TYPE_SHA1_SIZE 20 #define HASH_TYPE_SHA1_SIZE_HEX (HASH_TYPE_SHA1_SIZE * 2) #define HASH_TYPE_SHA256_SIZE 32 #define HASH_TYPE_SHA256_SIZE_HEX (HASH_TYPE_SHA256_SIZE * 2) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *cryptoHashNew(HashType type); FN_EXTERN IoFilter *cryptoHashNewPack(const Pack *paramList); /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Get hash for one buffer FN_EXTERN Buffer *cryptoHashOne(HashType type, const Buffer *message); // Get hmac for one message/key FN_EXTERN Buffer *cryptoHmacOne(HashType type, const Buffer *key, const Buffer *message); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/md5.vendor.c.inc0000664000175100017510000002131515265101745021700 0ustar00runnerrunner/* * This is an OpenSSL-compatible implementation of the RSA Data Security, Inc. * MD5 Message-Digest Algorithm (RFC 1321). * * Homepage: * http://openwall.info/wiki/people/solar/software/public-domain-source-code/md5 * * Author: * Alexander Peslyak, better known as Solar Designer * * This software was written by Alexander Peslyak in 2001. No copyright is * claimed, and the software is hereby placed in the public domain. * In case this attempt to disclaim copyright and place the software in the * public domain is deemed null and void, then the software is * Copyright (c) 2001 Alexander Peslyak and it is hereby released to the * general public under the following terms: * * Redistribution and use in source and binary forms, with or without * modification, are permitted. * * There's ABSOLUTELY NO WARRANTY, express or implied. * * (This is a heavily cut-down "BSD license".) * * This differs from Colin Plumb's older public domain implementation in that * no exactly 32-bit integer data type is required (any 32-bit or wider * unsigned integer data type will do), there's no compile-time endianness * configuration, and the function prototypes match OpenSSL's. No code from * Colin Plumb's implementation has been reused; this comment merely compares * the properties of the two independent implementations. * * The primary goals of this implementation are portability and ease of use. * It is meant to be fast, but not as fast as possible. Some known * optimizations are not included to reduce source code size and avoid * compile-time configuration. */ #include #include /* Any 32-bit or wider unsigned integer data type will do */ typedef unsigned int MD5_u32plus; typedef struct { MD5_u32plus lo, hi; MD5_u32plus a, b, c, d; uint8_t buffer[64]; MD5_u32plus block[16]; } MD5_CTX; /* * The basic MD5 functions. * * F and G are optimized compared to their RFC 1321 definitions for * architectures that lack an AND-NOT instruction, just like in Colin Plumb's * implementation. */ #define F(x, y, z) ((z) ^ ((x) & ((y) ^ (z)))) #define G(x, y, z) ((y) ^ ((z) & ((x) ^ (y)))) #define H(x, y, z) (((x) ^ (y)) ^ (z)) #define H2(x, y, z) ((x) ^ ((y) ^ (z))) #define I(x, y, z) ((y) ^ ((x) | ~(z))) /* * The MD5 transformation for all four rounds. */ #define STEP(f, a, b, c, d, x, t, s) \ (a) += f((b), (c), (d)) + (x) + (t); \ (a) = (((a) << (s)) | (((a) & 0xffffffff) >> (32 - (s)))); \ (a) += (b); /* * SET reads 4 input bytes in little-endian byte order and stores them in a * properly aligned word in host byte order. * * The check for little-endian architectures that tolerate unaligned memory * accesses is just an optimization. Nothing will break if it fails to detect * a suitable architecture. * * Unfortunately, this optimization may be a C strict aliasing rules violation * if the caller's data buffer has effective type that cannot be aliased by * MD5_u32plus. In practice, this problem may occur if these MD5 routines are * inlined into a calling function, or with future and dangerously advanced * link-time optimizations. For the time being, keeping these MD5 routines in * their own translation unit avoids the problem. */ #if defined(__i386__) || defined(__x86_64__) || defined(__vax__) #define SET(n) \ (*(const MD5_u32plus *)&ptr[(n) * 4]) #define GET(n) \ SET(n) #else #define SET(n) \ (ctx->block[(n)] = \ (MD5_u32plus)ptr[(n) * 4] | \ ((MD5_u32plus)ptr[(n) * 4 + 1] << 8) | \ ((MD5_u32plus)ptr[(n) * 4 + 2] << 16) | \ ((MD5_u32plus)ptr[(n) * 4 + 3] << 24)) #define GET(n) \ (ctx->block[(n)]) #endif /* * This processes one or more 64-byte data blocks, but does NOT update the bit * counters. There are no alignment requirements. */ static const void *body(MD5_CTX *ctx, const void *data, unsigned long size) { const uint8_t *ptr; MD5_u32plus a, b, c, d; MD5_u32plus saved_a, saved_b, saved_c, saved_d; ptr = (const uint8_t *)data; a = ctx->a; b = ctx->b; c = ctx->c; d = ctx->d; do { saved_a = a; saved_b = b; saved_c = c; saved_d = d; /* Round 1 */ STEP(F, a, b, c, d, SET(0), 0xd76aa478, 7) STEP(F, d, a, b, c, SET(1), 0xe8c7b756, 12) STEP(F, c, d, a, b, SET(2), 0x242070db, 17) STEP(F, b, c, d, a, SET(3), 0xc1bdceee, 22) STEP(F, a, b, c, d, SET(4), 0xf57c0faf, 7) STEP(F, d, a, b, c, SET(5), 0x4787c62a, 12) STEP(F, c, d, a, b, SET(6), 0xa8304613, 17) STEP(F, b, c, d, a, SET(7), 0xfd469501, 22) STEP(F, a, b, c, d, SET(8), 0x698098d8, 7) STEP(F, d, a, b, c, SET(9), 0x8b44f7af, 12) STEP(F, c, d, a, b, SET(10), 0xffff5bb1, 17) STEP(F, b, c, d, a, SET(11), 0x895cd7be, 22) STEP(F, a, b, c, d, SET(12), 0x6b901122, 7) STEP(F, d, a, b, c, SET(13), 0xfd987193, 12) STEP(F, c, d, a, b, SET(14), 0xa679438e, 17) STEP(F, b, c, d, a, SET(15), 0x49b40821, 22) /* Round 2 */ STEP(G, a, b, c, d, GET(1), 0xf61e2562, 5) STEP(G, d, a, b, c, GET(6), 0xc040b340, 9) STEP(G, c, d, a, b, GET(11), 0x265e5a51, 14) STEP(G, b, c, d, a, GET(0), 0xe9b6c7aa, 20) STEP(G, a, b, c, d, GET(5), 0xd62f105d, 5) STEP(G, d, a, b, c, GET(10), 0x02441453, 9) STEP(G, c, d, a, b, GET(15), 0xd8a1e681, 14) STEP(G, b, c, d, a, GET(4), 0xe7d3fbc8, 20) STEP(G, a, b, c, d, GET(9), 0x21e1cde6, 5) STEP(G, d, a, b, c, GET(14), 0xc33707d6, 9) STEP(G, c, d, a, b, GET(3), 0xf4d50d87, 14) STEP(G, b, c, d, a, GET(8), 0x455a14ed, 20) STEP(G, a, b, c, d, GET(13), 0xa9e3e905, 5) STEP(G, d, a, b, c, GET(2), 0xfcefa3f8, 9) STEP(G, c, d, a, b, GET(7), 0x676f02d9, 14) STEP(G, b, c, d, a, GET(12), 0x8d2a4c8a, 20) /* Round 3 */ STEP(H, a, b, c, d, GET(5), 0xfffa3942, 4) STEP(H2, d, a, b, c, GET(8), 0x8771f681, 11) STEP(H, c, d, a, b, GET(11), 0x6d9d6122, 16) STEP(H2, b, c, d, a, GET(14), 0xfde5380c, 23) STEP(H, a, b, c, d, GET(1), 0xa4beea44, 4) STEP(H2, d, a, b, c, GET(4), 0x4bdecfa9, 11) STEP(H, c, d, a, b, GET(7), 0xf6bb4b60, 16) STEP(H2, b, c, d, a, GET(10), 0xbebfbc70, 23) STEP(H, a, b, c, d, GET(13), 0x289b7ec6, 4) STEP(H2, d, a, b, c, GET(0), 0xeaa127fa, 11) STEP(H, c, d, a, b, GET(3), 0xd4ef3085, 16) STEP(H2, b, c, d, a, GET(6), 0x04881d05, 23) STEP(H, a, b, c, d, GET(9), 0xd9d4d039, 4) STEP(H2, d, a, b, c, GET(12), 0xe6db99e5, 11) STEP(H, c, d, a, b, GET(15), 0x1fa27cf8, 16) STEP(H2, b, c, d, a, GET(2), 0xc4ac5665, 23) /* Round 4 */ STEP(I, a, b, c, d, GET(0), 0xf4292244, 6) STEP(I, d, a, b, c, GET(7), 0x432aff97, 10) STEP(I, c, d, a, b, GET(14), 0xab9423a7, 15) STEP(I, b, c, d, a, GET(5), 0xfc93a039, 21) STEP(I, a, b, c, d, GET(12), 0x655b59c3, 6) STEP(I, d, a, b, c, GET(3), 0x8f0ccc92, 10) STEP(I, c, d, a, b, GET(10), 0xffeff47d, 15) STEP(I, b, c, d, a, GET(1), 0x85845dd1, 21) STEP(I, a, b, c, d, GET(8), 0x6fa87e4f, 6) STEP(I, d, a, b, c, GET(15), 0xfe2ce6e0, 10) STEP(I, c, d, a, b, GET(6), 0xa3014314, 15) STEP(I, b, c, d, a, GET(13), 0x4e0811a1, 21) STEP(I, a, b, c, d, GET(4), 0xf7537e82, 6) STEP(I, d, a, b, c, GET(11), 0xbd3af235, 10) STEP(I, c, d, a, b, GET(2), 0x2ad7d2bb, 15) STEP(I, b, c, d, a, GET(9), 0xeb86d391, 21) a += saved_a; b += saved_b; c += saved_c; d += saved_d; ptr += 64; } while (size -= 64); ctx->a = a; ctx->b = b; ctx->c = c; ctx->d = d; return ptr; } static void MD5_Init(MD5_CTX *ctx) { *ctx = (MD5_CTX) { .a = 0x67452301, .b = 0xefcdab89, .c = 0x98badcfe, .d = 0x10325476, }; } static void MD5_Update(MD5_CTX *ctx, const void *data, unsigned long size) { MD5_u32plus saved_lo; unsigned long used, available; saved_lo = ctx->lo; if ((ctx->lo = (saved_lo + size) & 0x1fffffff) < saved_lo) ctx->hi++; ctx->hi += (MD5_u32plus)(size >> 29); used = saved_lo & 0x3f; if (used) { available = 64 - used; if (size < available) { memcpy(&ctx->buffer[used], data, size); return; } memcpy(&ctx->buffer[used], data, available); data = (const uint8_t *)data + available; size -= available; body(ctx, ctx->buffer, 64); } if (size >= 64) { data = body(ctx, data, size & ~(unsigned long)0x3f); size &= 0x3f; } memcpy(ctx->buffer, data, size); } #define OUT(dst, src) \ (dst)[0] = (uint8_t)(src); \ (dst)[1] = (uint8_t)((src) >> 8); \ (dst)[2] = (uint8_t)((src) >> 16); \ (dst)[3] = (uint8_t)((src) >> 24); static void MD5_Final(uint8_t *result, MD5_CTX *ctx) { unsigned long used, available; used = ctx->lo & 0x3f; ctx->buffer[used++] = 0x80; available = 64 - used; if (available < 8) { memset(&ctx->buffer[used], 0, available); body(ctx, ctx->buffer, 64); used = 0; available = 64; } memset(&ctx->buffer[used], 0, available - 8); ctx->lo <<= 3; OUT(&ctx->buffer[56], ctx->lo) OUT(&ctx->buffer[60], ctx->hi) body(ctx, ctx->buffer, 64); OUT(&result[0], ctx->a) OUT(&result[4], ctx->b) OUT(&result[8], ctx->c) OUT(&result[12], ctx->d) memset(ctx, 0, sizeof(*ctx)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/spec.c0000664000175100017510000001111315265101745020074 0ustar00runnerrunner/*********************************************************************************************************************************** Cipher Spec ***********************************************************************************************************************************/ #include #include "common/crypto/spec.h" #include "common/debug.h" #include "common/log.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct CipherSpec { CipherSpecPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN CipherSpec * cipherSpecNew(const CipherType type, const Buffer *const pass, const CipherSpecNewParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_PARAM(BUFFER, pass); FUNCTION_TEST_PARAM(STRING_ID, param.digest); FUNCTION_TEST_END(); ASSERT((type == cipherTypeNone) == (pass == NULL)); ASSERT(pass == NULL || !bufEmpty(pass)); OBJ_NEW_BEGIN(CipherSpec, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (CipherSpec){.pub = {.type = type}}; if (this->pub.type != cipherTypeNone) { this->pub.digest = param.digest; this->pub.pass = bufDup(pass); } } OBJ_NEW_END(); FUNCTION_TEST_RETURN(CIPHER_SPEC, this); } /**********************************************************************************************************************************/ FN_EXTERN CipherSpec * cipherSpecNewPack(PackRead *const packRead) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, packRead); FUNCTION_TEST_END(); ASSERT(packRead != NULL); OBJ_NEW_BEGIN(CipherSpec, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (CipherSpec){.pub = {.type = (CipherType)pckReadStrIdP(packRead)}}; // Nothing else was written when there is no cipher. The pass is read here rather than copied in since this context is the // one it belongs in. if (this->pub.type != cipherTypeNone) { this->pub.digest = (HashType)pckReadStrIdP(packRead); this->pub.pass = pckReadBinP(packRead); } } OBJ_NEW_END(); FUNCTION_TEST_RETURN(CIPHER_SPEC, this); } /**********************************************************************************************************************************/ FN_EXTERN CipherSpec * cipherSpecDup(const CipherSpec *const this, const CipherSpecDupParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CIPHER_SPEC, this); FUNCTION_TEST_PARAM(STRING_ID, param.digestDefault); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN( CIPHER_SPEC, cipherSpecNewP( cipherSpecType(this), cipherSpecPass(this), .digest = cipherSpecDigest(this) == 0 ? param.digestDefault : cipherSpecDigest(this))); } /**********************************************************************************************************************************/ FN_EXTERN void cipherSpecPack(PackWrite *const packWrite, const CipherSpec *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, packWrite); FUNCTION_TEST_PARAM(CIPHER_SPEC, this); FUNCTION_TEST_END(); ASSERT(packWrite != NULL); ASSERT(this != NULL); pckWriteStrIdP(packWrite, cipherSpecType(this)); // Nothing else is needed when there is no cipher if (cipherSpecType(this) != cipherTypeNone) { pckWriteStrIdP(packWrite, cipherSpecDigest(this)); pckWriteBinP(packWrite, cipherSpecPass(this)); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cipherSpecToLog(const CipherSpec *const this, StringStatic *const debugLog) { char typeZ[STRID_MAX + 1]; strIdToZ(cipherSpecType(this), typeZ); strStcFmt(debugLog, "{type: %s", typeZ); // There is no digest when there is no cipher or when the format decides it. The pass is never logged. if (cipherSpecDigest(this) != 0) { char digestZ[STRID_MAX + 1]; strIdToZ(cipherSpecDigest(this), digestZ); strStcFmt(debugLog, ", digest: %s", digestZ); } strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/spec.h0000664000175100017510000001355115265101745020111 0ustar00runnerrunner/*********************************************************************************************************************************** Cipher Spec Everything needed to encrypt or decrypt, kept together so that adding to it does not mean changing every function and protocol message that contains it. The pass holds the bytes used to derive the key. The digest travels with the pass because the two are chosen together and deriving with the wrong digest gives a wrong key instead of an error. The pass is a buffer rather than a string because it may be binary or it may be text and nothing here needs to know which. It is copied into the object, so the caller can release whatever it read the pass from. An unset digest means the repository format of the file being read or written specifies it. There is no digest or pass when the type is none, and the pass is never logged. ***********************************************************************************************************************************/ #ifndef COMMON_CRYPTO_SPEC_H #define COMMON_CRYPTO_SPEC_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct CipherSpec CipherSpec; #include "common/crypto/common.h" #include "common/type/buffer.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ // Create from a pass, which is the key bytes or the passphrase text typedef struct CipherSpecNewParam { VAR_PARAM_HEADER; HashType digest; // Digest to derive the key with, unset when specified by format } CipherSpecNewParam; #define cipherSpecNewP(type, pass, ...) \ cipherSpecNew(type, pass, (CipherSpecNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN CipherSpec *cipherSpecNew(CipherType type, const Buffer *pass, CipherSpecNewParam param); // Create for data that is not encrypted FN_INLINE_ALWAYS CipherSpec * cipherSpecNewNone(void) { return cipherSpecNewP(cipherTypeNone, NULL); } // Create from a pack written by cipherSpecPack() FN_EXTERN CipherSpec *cipherSpecNewPack(PackRead *packRead); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct CipherSpecPub { CipherType type; // Cipher type, none when not encrypted HashType digest; // Digest the pass derives with, unset when specified by format const Buffer *pass; // Passphrase text or key bytes } CipherSpecPub; // Cipher type FN_INLINE_ALWAYS CipherType cipherSpecType(const CipherSpec *const this) { return THIS_PUB(CipherSpec)->type; } // Digest the pass derives the key with, unset when specified by the format FN_INLINE_ALWAYS HashType cipherSpecDigest(const CipherSpec *const this) { return THIS_PUB(CipherSpec)->digest; } // Passphrase text or key bytes FN_INLINE_ALWAYS const Buffer * cipherSpecPass(const CipherSpec *const this) { return THIS_PUB(CipherSpec)->pass; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Duplicate typedef struct CipherSpecDupParam { VAR_PARAM_HEADER; HashType digestDefault; // Digest to use when the spec has none } CipherSpecDupParam; #define cipherSpecDupP(this, ...) \ cipherSpecDup(this, (CipherSpecDupParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN CipherSpec *cipherSpecDup(const CipherSpec *this, CipherSpecDupParam param); // Write to a pack so it can be passed over a protocol FN_EXTERN void cipherSpecPack(PackWrite *packWrite, const CipherSpec *this); // Move to a new parent mem context FN_INLINE_ALWAYS CipherSpec * cipherSpecMove(CipherSpec *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void cipherSpecFree(CipherSpec *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void cipherSpecToLog(const CipherSpec *this, StringStatic *debugLog); #define FUNCTION_LOG_CIPHER_SPEC_TYPE \ CipherSpec * #define FUNCTION_LOG_CIPHER_SPEC_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, cipherSpecToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/xxhash.c0000664000175100017510000001317015265101745020452 0ustar00runnerrunner/*********************************************************************************************************************************** xxHash Interface ***********************************************************************************************************************************/ #include #include "common/crypto/xxhash.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Include local xxHash code ***********************************************************************************************************************************/ #define XXH_INLINE_ALL #include "common/crypto/xxhash.vendor.c.inc" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct XxHash { size_t size; // Size of hash to return XXH3_state_t *state; // xxHash state } XxHash; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_XX_HASH_TYPE \ XxHash * #define FUNCTION_LOG_XX_HASH_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "XxHash", buffer, bufferSize) /*********************************************************************************************************************************** Free hash context ***********************************************************************************************************************************/ static void xxHashFreeResource(THIS_VOID) { THIS(XxHash); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(XX_HASH, this); FUNCTION_LOG_END(); ASSERT(this != NULL); XXH3_freeState(this->state); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Add message data to the hash from a Buffer ***********************************************************************************************************************************/ static void xxHashProcess(THIS_VOID, const Buffer *const message) { THIS(XxHash); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(XX_HASH, this); FUNCTION_LOG_PARAM(BUFFER, message); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(message != NULL); XXH3_128bits_update(this->state, bufPtrConst(message), bufUsed(message)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get string representation of the hash as a filter result ***********************************************************************************************************************************/ static Pack * xxHashResult(THIS_VOID) { THIS(XxHash); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(XX_HASH, this); FUNCTION_LOG_END(); ASSERT(this != NULL); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); XXH128_canonical_t canonical; XXH128_canonicalFromHash(&canonical, XXH3_128bits_digest(this->state)); pckWriteBinP(packWrite, BUF(canonical.digest, this->size)); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * xxHashNew(const size_t size) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SIZE, size); FUNCTION_LOG_END(); ASSERT(size >= 1 && size <= XX_HASH_SIZE_MAX); OBJ_NEW_BEGIN(XxHash, .callbackQty = 1) { *this = (XxHash){.size = size}; this->state = XXH3_createState(); XXH3_128bits_reset(this->state); // Set free callback to ensure hash context is freed memContextCallbackSet(objMemContext(this), xxHashFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_FILTER, ioFilterNewP(XX_HASH_FILTER_TYPE, this, NULL, .in = xxHashProcess, .result = xxHashResult)); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * xxHashOne(const size_t size, const Buffer *const message) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SIZE, size); FUNCTION_LOG_PARAM(BUFFER, message); FUNCTION_LOG_END(); ASSERT(size >= 1 && size <= XX_HASH_SIZE_MAX); ASSERT(message != NULL); Buffer *const result = bufNew(size); XXH128_canonical_t canonical; XXH128_canonicalFromHash(&canonical, XXH3_128bits(bufPtrConst(message), bufUsed(message))); memcpy(bufPtr(result), canonical.digest, size); bufUsedSet(result, size); FUNCTION_LOG_RETURN(BUFFER, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/xxhash.h0000664000175100017510000000456615265101745020470 0ustar00runnerrunner/*********************************************************************************************************************************** xxHash Interface Generate a 128-bit hash and return the specified number of bytes from the canonical representation. The 128-bit variant is only slightly slower than the 64-bit variant so this makes getting up to 16 bytes of hash value fairly simple. We consider this safe because xxHash claims to have excellent dispersion, zstd uses the lower 32-bits of a 64-bit hash for content checksums, xxHash has been tested with SMHasher, and the "128-bit variant has better mixing and strength than the 64-bit variant, even without counting the significantly larger output size". ***********************************************************************************************************************************/ #ifndef COMMON_CRYPTO_XXHASH_H #define COMMON_CRYPTO_XXHASH_H #include "common/io/filter/filter.h" #include "common/type/buffer.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define XX_HASH_FILTER_TYPE STRID5("xxhash", 0x1130a3180) /*********************************************************************************************************************************** Max size of hash ***********************************************************************************************************************************/ #define XX_HASH_SIZE_MAX 16 /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *xxHashNew(size_t size); /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Get hash for one buffer FN_EXTERN Buffer *xxHashOne(size_t size, const Buffer *message); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/crypto/xxhash.vendor.c.inc0000664000175100017510000075625015265101745022533 0ustar00runnerrunner/* * xxHash - Extremely Fast Hash algorithm * Header File * Copyright (C) 2012-2021 Yann Collet * * BSD 2-Clause License (https://www.opensource.org/licenses/bsd-license.php) * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are * met: * * * Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above * copyright notice, this list of conditions and the following disclaimer * in the documentation and/or other materials provided with the * distribution. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * You can contact the author at: * - xxHash homepage: https://www.xxhash.com * - xxHash source repository: https://github.com/Cyan4973/xxHash */ /*! * @mainpage xxHash * * xxHash is an extremely fast non-cryptographic hash algorithm, working at RAM speed * limits. * * It is proposed in four flavors, in three families: * 1. @ref XXH32_family * - Classic 32-bit hash function. Simple, compact, and runs on almost all * 32-bit and 64-bit systems. * 2. @ref XXH64_family * - Classic 64-bit adaptation of XXH32. Just as simple, and runs well on most * 64-bit systems (but _not_ 32-bit systems). * 3. @ref XXH3_family * - Modern 64-bit and 128-bit hash function family which features improved * strength and performance across the board, especially on smaller data. * It benefits greatly from SIMD and 64-bit without requiring it. * * Benchmarks * --- * The reference system uses an Intel i7-9700K CPU, and runs Ubuntu x64 20.04. * The open source benchmark program is compiled with clang v10.0 using -O3 flag. * * | Hash Name | ISA ext | Width | Large Data Speed | Small Data Velocity | * | -------------------- | ------- | ----: | ---------------: | ------------------: | * | XXH3_64bits() | @b AVX2 | 64 | 59.4 GB/s | 133.1 | * | MeowHash | AES-NI | 128 | 58.2 GB/s | 52.5 | * | XXH3_128bits() | @b AVX2 | 128 | 57.9 GB/s | 118.1 | * | CLHash | PCLMUL | 64 | 37.1 GB/s | 58.1 | * | XXH3_64bits() | @b SSE2 | 64 | 31.5 GB/s | 133.1 | * | XXH3_128bits() | @b SSE2 | 128 | 29.6 GB/s | 118.1 | * | RAM sequential read | | N/A | 28.0 GB/s | N/A | * | ahash | AES-NI | 64 | 22.5 GB/s | 107.2 | * | City64 | | 64 | 22.0 GB/s | 76.6 | * | T1ha2 | | 64 | 22.0 GB/s | 99.0 | * | City128 | | 128 | 21.7 GB/s | 57.7 | * | FarmHash | AES-NI | 64 | 21.3 GB/s | 71.9 | * | XXH64() | | 64 | 19.4 GB/s | 71.0 | * | SpookyHash | | 64 | 19.3 GB/s | 53.2 | * | Mum | | 64 | 18.0 GB/s | 67.0 | * | CRC32C | SSE4.2 | 32 | 13.0 GB/s | 57.9 | * | XXH32() | | 32 | 9.7 GB/s | 71.9 | * | City32 | | 32 | 9.1 GB/s | 66.0 | * | Blake3* | @b AVX2 | 256 | 4.4 GB/s | 8.1 | * | Murmur3 | | 32 | 3.9 GB/s | 56.1 | * | SipHash* | | 64 | 3.0 GB/s | 43.2 | * | Blake3* | @b SSE2 | 256 | 2.4 GB/s | 8.1 | * | HighwayHash | | 64 | 1.4 GB/s | 6.0 | * | FNV64 | | 64 | 1.2 GB/s | 62.7 | * | Blake2* | | 256 | 1.1 GB/s | 5.1 | * | SHA1* | | 160 | 0.8 GB/s | 5.6 | * | MD5* | | 128 | 0.6 GB/s | 7.8 | * @note * - Hashes which require a specific ISA extension are noted. SSE2 is also noted, * even though it is mandatory on x64. * - Hashes with an asterisk are cryptographic. Note that MD5 is non-cryptographic * by modern standards. * - Small data velocity is a rough average of algorithm's efficiency for small * data. For more accurate information, see the wiki. * - More benchmarks and strength tests are found on the wiki: * https://github.com/Cyan4973/xxHash/wiki * * Usage * ------ * All xxHash variants use a similar API. Changing the algorithm is a trivial * substitution. * * @pre * For functions which take an input and length parameter, the following * requirements are assumed: * - The range from [`input`, `input + length`) is valid, readable memory. * - The only exception is if the `length` is `0`, `input` may be `NULL`. * - For C++, the objects must have the *TriviallyCopyable* property, as the * functions access bytes directly as if it was an array of `unsigned char`. * * @anchor single_shot_example * **Single Shot** * * These functions are stateless functions which hash a contiguous block of memory, * immediately returning the result. They are the easiest and usually the fastest * option. * * XXH32(), XXH64(), XXH3_64bits(), XXH3_128bits() * * @code{.c} * #include * #include "xxhash.h" * * // Example for a function which hashes a null terminated string with XXH32(). * XXH32_hash_t hash_string(const char* string, XXH32_hash_t seed) * { * // NULL pointers are only valid if the length is zero * size_t length = (string == NULL) ? 0 : strlen(string); * return XXH32(string, length, seed); * } * @endcode * * @anchor streaming_example * **Streaming** * * These groups of functions allow incremental hashing of unknown size, even * more than what would fit in a size_t. * * XXH32_reset(), XXH64_reset(), XXH3_64bits_reset(), XXH3_128bits_reset() * * @code{.c} * #include * #include * #include "xxhash.h" * // Example for a function which hashes a FILE incrementally with XXH3_64bits(). * XXH64_hash_t hashFile(FILE* f) * { * // Allocate a state struct. Do not just use malloc() or new. * XXH3_state_t* state = XXH3_createState(); * assert(state != NULL && "Out of memory!"); * // Reset the state to start a new hashing session. * XXH3_64bits_reset(state); * char buffer[4096]; * size_t count; * // Read the file in chunks * while ((count = fread(buffer, 1, sizeof(buffer), f)) != 0) { * // Run update() as many times as necessary to process the data * XXH3_64bits_update(state, buffer, count); * } * // Retrieve the finalized hash. This will not change the state. * XXH64_hash_t result = XXH3_64bits_digest(state); * // Free the state. Do not use free(). * XXH3_freeState(state); * return result; * } * @endcode * * @file xxhash.h * xxHash prototypes and implementation */ #if defined (__cplusplus) extern "C" { #endif /* **************************** * INLINE mode ******************************/ /*! * @defgroup public Public API * Contains details on the public xxHash functions. * @{ */ #ifdef XXH_DOXYGEN /*! * @brief Gives access to internal state declaration, required for static allocation. * * Incompatible with dynamic linking, due to risks of ABI changes. * * Usage: * @code{.c} * #define XXH_STATIC_LINKING_ONLY * #include "xxhash.h" * @endcode */ # define XXH_STATIC_LINKING_ONLY /* Do not undef XXH_STATIC_LINKING_ONLY for Doxygen */ /*! * @brief Gives access to internal definitions. * * Usage: * @code{.c} * #define XXH_STATIC_LINKING_ONLY * #define XXH_IMPLEMENTATION * #include "xxhash.h" * @endcode */ # define XXH_IMPLEMENTATION /* Do not undef XXH_IMPLEMENTATION for Doxygen */ /*! * @brief Exposes the implementation and marks all functions as `inline`. * * Use these build macros to inline xxhash into the target unit. * Inlining improves performance on small inputs, especially when the length is * expressed as a compile-time constant: * * https://fastcompression.blogspot.com/2018/03/xxhash-for-small-keys-impressive-power.html * * It also keeps xxHash symbols private to the unit, so they are not exported. * * Usage: * @code{.c} * #define XXH_INLINE_ALL * #include "xxhash.h" * @endcode * Do not compile and link xxhash.o as a separate object, as it is not useful. */ # define XXH_INLINE_ALL # undef XXH_INLINE_ALL /*! * @brief Exposes the implementation without marking functions as inline. */ # define XXH_PRIVATE_API # undef XXH_PRIVATE_API /*! * @brief Emulate a namespace by transparently prefixing all symbols. * * If you want to include _and expose_ xxHash functions from within your own * library, but also want to avoid symbol collisions with other libraries which * may also include xxHash, you can use @ref XXH_NAMESPACE to automatically prefix * any public symbol from xxhash library with the value of @ref XXH_NAMESPACE * (therefore, avoid empty or numeric values). * * Note that no change is required within the calling program as long as it * includes `xxhash.h`: Regular symbol names will be automatically translated * by this header. */ # define XXH_NAMESPACE /* YOUR NAME HERE */ # undef XXH_NAMESPACE #endif #if (defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)) \ && !defined(XXH_INLINE_ALL_31684351384) /* this section should be traversed only once */ # define XXH_INLINE_ALL_31684351384 /* give access to the advanced API, required to compile implementations */ # undef XXH_STATIC_LINKING_ONLY /* avoid macro redef */ # define XXH_STATIC_LINKING_ONLY /* make all functions private */ # undef XXH_PUBLIC_API # if defined(__GNUC__) # define XXH_PUBLIC_API static __inline __attribute__((unused)) # elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) # define XXH_PUBLIC_API static inline # elif defined(_MSC_VER) # define XXH_PUBLIC_API static __inline # else /* note: this version may generate warnings for unused static functions */ # define XXH_PUBLIC_API static # endif /* * This part deals with the special case where a unit wants to inline xxHash, * but "xxhash.h" has previously been included without XXH_INLINE_ALL, * such as part of some previously included *.h header file. * Without further action, the new include would just be ignored, * and functions would effectively _not_ be inlined (silent failure). * The following macros solve this situation by prefixing all inlined names, * avoiding naming collision with previous inclusions. */ /* Before that, we unconditionally #undef all symbols, * in case they were already defined with XXH_NAMESPACE. * They will then be redefined for XXH_INLINE_ALL */ # undef XXH_versionNumber /* XXH32 */ # undef XXH32 # undef XXH32_createState # undef XXH32_freeState # undef XXH32_reset # undef XXH32_update # undef XXH32_digest # undef XXH32_copyState # undef XXH32_canonicalFromHash # undef XXH32_hashFromCanonical /* XXH64 */ # undef XXH64 # undef XXH64_createState # undef XXH64_freeState # undef XXH64_reset # undef XXH64_update # undef XXH64_digest # undef XXH64_copyState # undef XXH64_canonicalFromHash # undef XXH64_hashFromCanonical /* XXH3_64bits */ # undef XXH3_64bits # undef XXH3_64bits_withSecret # undef XXH3_64bits_withSeed # undef XXH3_64bits_withSecretandSeed # undef XXH3_createState # undef XXH3_freeState # undef XXH3_copyState # undef XXH3_64bits_reset # undef XXH3_64bits_reset_withSeed # undef XXH3_64bits_reset_withSecret # undef XXH3_64bits_update # undef XXH3_64bits_digest # undef XXH3_generateSecret /* XXH3_128bits */ # undef XXH128 # undef XXH3_128bits # undef XXH3_128bits_withSeed # undef XXH3_128bits_withSecret # undef XXH3_128bits_reset # undef XXH3_128bits_reset_withSeed # undef XXH3_128bits_reset_withSecret # undef XXH3_128bits_reset_withSecretandSeed # undef XXH3_128bits_update # undef XXH3_128bits_digest # undef XXH128_isEqual # undef XXH128_cmp # undef XXH128_canonicalFromHash # undef XXH128_hashFromCanonical /* Finally, free the namespace itself */ # undef XXH_NAMESPACE /* employ the namespace for XXH_INLINE_ALL */ # define XXH_NAMESPACE XXH_INLINE_ /* * Some identifiers (enums, type names) are not symbols, * but they must nonetheless be renamed to avoid redeclaration. * Alternative solution: do not redeclare them. * However, this requires some #ifdefs, and has a more dispersed impact. * Meanwhile, renaming can be achieved in a single place. */ # define XXH_IPREF(Id) XXH_NAMESPACE ## Id # define XXH_OK XXH_IPREF(XXH_OK) # define XXH_ERROR XXH_IPREF(XXH_ERROR) # define XXH_errorcode XXH_IPREF(XXH_errorcode) # define XXH32_canonical_t XXH_IPREF(XXH32_canonical_t) # define XXH64_canonical_t XXH_IPREF(XXH64_canonical_t) # define XXH128_canonical_t XXH_IPREF(XXH128_canonical_t) # define XXH32_state_s XXH_IPREF(XXH32_state_s) # define XXH32_state_t XXH_IPREF(XXH32_state_t) # define XXH64_state_s XXH_IPREF(XXH64_state_s) # define XXH64_state_t XXH_IPREF(XXH64_state_t) # define XXH3_state_s XXH_IPREF(XXH3_state_s) # define XXH3_state_t XXH_IPREF(XXH3_state_t) # define XXH128_hash_t XXH_IPREF(XXH128_hash_t) /* Ensure the header is parsed again, even if it was previously included */ # undef XXHASH_H_5627135585666179 # undef XXHASH_H_STATIC_13879238742 #endif /* XXH_INLINE_ALL || XXH_PRIVATE_API */ /* **************************************************************** * Stable API *****************************************************************/ #ifndef XXHASH_H_5627135585666179 #define XXHASH_H_5627135585666179 1 /*! @brief Marks a global symbol. */ #if !defined(XXH_INLINE_ALL) && !defined(XXH_PRIVATE_API) # if defined(WIN32) && defined(_MSC_VER) && (defined(XXH_IMPORT) || defined(XXH_EXPORT)) # ifdef XXH_EXPORT # define XXH_PUBLIC_API __declspec(dllexport) # elif XXH_IMPORT # define XXH_PUBLIC_API __declspec(dllimport) # endif # else # define XXH_PUBLIC_API /* do nothing */ # endif #endif #ifdef XXH_NAMESPACE # define XXH_CAT(A,B) A##B # define XXH_NAME2(A,B) XXH_CAT(A,B) # define XXH_versionNumber XXH_NAME2(XXH_NAMESPACE, XXH_versionNumber) /* XXH32 */ # define XXH32 XXH_NAME2(XXH_NAMESPACE, XXH32) # define XXH32_createState XXH_NAME2(XXH_NAMESPACE, XXH32_createState) # define XXH32_freeState XXH_NAME2(XXH_NAMESPACE, XXH32_freeState) # define XXH32_reset XXH_NAME2(XXH_NAMESPACE, XXH32_reset) # define XXH32_update XXH_NAME2(XXH_NAMESPACE, XXH32_update) # define XXH32_digest XXH_NAME2(XXH_NAMESPACE, XXH32_digest) # define XXH32_copyState XXH_NAME2(XXH_NAMESPACE, XXH32_copyState) # define XXH32_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH32_canonicalFromHash) # define XXH32_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH32_hashFromCanonical) /* XXH64 */ # define XXH64 XXH_NAME2(XXH_NAMESPACE, XXH64) # define XXH64_createState XXH_NAME2(XXH_NAMESPACE, XXH64_createState) # define XXH64_freeState XXH_NAME2(XXH_NAMESPACE, XXH64_freeState) # define XXH64_reset XXH_NAME2(XXH_NAMESPACE, XXH64_reset) # define XXH64_update XXH_NAME2(XXH_NAMESPACE, XXH64_update) # define XXH64_digest XXH_NAME2(XXH_NAMESPACE, XXH64_digest) # define XXH64_copyState XXH_NAME2(XXH_NAMESPACE, XXH64_copyState) # define XXH64_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH64_canonicalFromHash) # define XXH64_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH64_hashFromCanonical) /* XXH3_64bits */ # define XXH3_64bits XXH_NAME2(XXH_NAMESPACE, XXH3_64bits) # define XXH3_64bits_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSecret) # define XXH3_64bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSeed) # define XXH3_64bits_withSecretandSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSecretandSeed) # define XXH3_createState XXH_NAME2(XXH_NAMESPACE, XXH3_createState) # define XXH3_freeState XXH_NAME2(XXH_NAMESPACE, XXH3_freeState) # define XXH3_copyState XXH_NAME2(XXH_NAMESPACE, XXH3_copyState) # define XXH3_64bits_reset XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset) # define XXH3_64bits_reset_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset_withSeed) # define XXH3_64bits_reset_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset_withSecret) # define XXH3_64bits_reset_withSecretandSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset_withSecretandSeed) # define XXH3_64bits_update XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_update) # define XXH3_64bits_digest XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_digest) # define XXH3_generateSecret XXH_NAME2(XXH_NAMESPACE, XXH3_generateSecret) # define XXH3_generateSecret_fromSeed XXH_NAME2(XXH_NAMESPACE, XXH3_generateSecret_fromSeed) /* XXH3_128bits */ # define XXH128 XXH_NAME2(XXH_NAMESPACE, XXH128) # define XXH3_128bits XXH_NAME2(XXH_NAMESPACE, XXH3_128bits) # define XXH3_128bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSeed) # define XXH3_128bits_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSecret) # define XXH3_128bits_withSecretandSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSecretandSeed) # define XXH3_128bits_reset XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset) # define XXH3_128bits_reset_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset_withSeed) # define XXH3_128bits_reset_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset_withSecret) # define XXH3_128bits_reset_withSecretandSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset_withSecretandSeed) # define XXH3_128bits_update XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_update) # define XXH3_128bits_digest XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_digest) # define XXH128_isEqual XXH_NAME2(XXH_NAMESPACE, XXH128_isEqual) # define XXH128_cmp XXH_NAME2(XXH_NAMESPACE, XXH128_cmp) # define XXH128_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH128_canonicalFromHash) # define XXH128_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH128_hashFromCanonical) #endif /* ************************************* * Compiler specifics ***************************************/ /* specific declaration modes for Windows */ #if !defined(XXH_INLINE_ALL) && !defined(XXH_PRIVATE_API) # if defined(WIN32) && defined(_MSC_VER) && (defined(XXH_IMPORT) || defined(XXH_EXPORT)) # ifdef XXH_EXPORT # define XXH_PUBLIC_API __declspec(dllexport) # elif XXH_IMPORT # define XXH_PUBLIC_API __declspec(dllimport) # endif # else # define XXH_PUBLIC_API /* do nothing */ # endif #endif #if defined (__GNUC__) # define XXH_CONSTF __attribute__((const)) # define XXH_PUREF __attribute__((pure)) # define XXH_MALLOCF __attribute__((malloc)) #else # define XXH_CONSTF /* disable */ # define XXH_PUREF # define XXH_MALLOCF #endif /* ************************************* * Version ***************************************/ #define XXH_VERSION_MAJOR 0 #define XXH_VERSION_MINOR 8 #define XXH_VERSION_RELEASE 2 /*! @brief Version number, encoded as two digits each */ #define XXH_VERSION_NUMBER (XXH_VERSION_MAJOR *100*100 + XXH_VERSION_MINOR *100 + XXH_VERSION_RELEASE) /*! * @brief Obtains the xxHash version. * * This is mostly useful when xxHash is compiled as a shared library, * since the returned value comes from the library, as opposed to header file. * * @return @ref XXH_VERSION_NUMBER of the invoked library. */ XXH_PUBLIC_API XXH_CONSTF unsigned XXH_versionNumber (void); /* **************************** * Common basic types ******************************/ #include /* size_t */ /*! * @brief Exit code for the streaming API. */ typedef enum { XXH_OK = 0, /*!< OK */ XXH_ERROR /*!< Error */ } XXH_errorcode; /*-********************************************************************** * 32-bit hash ************************************************************************/ #if defined(XXH_DOXYGEN) /* Don't show include */ /*! * @brief An unsigned 32-bit integer. * * Not necessarily defined to `uint32_t` but functionally equivalent. */ typedef uint32_t XXH32_hash_t; #elif !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) # include typedef uint32_t XXH32_hash_t; #else # include # if UINT_MAX == 0xFFFFFFFFUL typedef unsigned int XXH32_hash_t; # elif ULONG_MAX == 0xFFFFFFFFUL typedef unsigned long XXH32_hash_t; # else # error "unsupported platform: need a 32-bit type" # endif #endif /*! * @} * * @defgroup XXH32_family XXH32 family * @ingroup public * Contains functions used in the classic 32-bit xxHash algorithm. * * @note * XXH32 is useful for older platforms, with no or poor 64-bit performance. * Note that the @ref XXH3_family provides competitive speed for both 32-bit * and 64-bit systems, and offers true 64/128 bit hash results. * * @see @ref XXH64_family, @ref XXH3_family : Other xxHash families * @see @ref XXH32_impl for implementation details * @{ */ /*! * @brief Calculates the 32-bit hash of @p input using xxHash32. * * Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark): 5.4 GB/s * * See @ref single_shot_example "Single Shot Example" for an example. * * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * @param seed The 32-bit seed to alter the hash's output predictably. * * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return The calculated 32-bit hash value. * * @see * XXH64(), XXH3_64bits_withSeed(), XXH3_128bits_withSeed(), XXH128(): * Direct equivalents for the other variants of xxHash. * @see * XXH32_createState(), XXH32_update(), XXH32_digest(): Streaming version. */ XXH_PUBLIC_API XXH_PUREF XXH32_hash_t XXH32 (const void* input, size_t length, XXH32_hash_t seed); #ifndef XXH_NO_STREAM /*! * Streaming functions generate the xxHash value from an incremental input. * This method is slower than single-call functions, due to state management. * For small inputs, prefer `XXH32()` and `XXH64()`, which are better optimized. * * An XXH state must first be allocated using `XXH*_createState()`. * * Start a new hash by initializing the state with a seed using `XXH*_reset()`. * * Then, feed the hash state by calling `XXH*_update()` as many times as necessary. * * The function returns an error code, with 0 meaning OK, and any other value * meaning there is an error. * * Finally, a hash value can be produced anytime, by using `XXH*_digest()`. * This function returns the nn-bits hash as an int or long long. * * It's still possible to continue inserting input into the hash state after a * digest, and generate new hash values later on by invoking `XXH*_digest()`. * * When done, release the state using `XXH*_freeState()`. * * @see streaming_example at the top of @ref xxhash.h for an example. */ /*! * @typedef struct XXH32_state_s XXH32_state_t * @brief The opaque state struct for the XXH32 streaming API. * * @see XXH32_state_s for details. */ typedef struct XXH32_state_s XXH32_state_t; /*! * @brief Allocates an @ref XXH32_state_t. * * Must be freed with XXH32_freeState(). * @return An allocated XXH32_state_t on success, `NULL` on failure. */ XXH_PUBLIC_API XXH_MALLOCF XXH32_state_t* XXH32_createState(void); /*! * @brief Frees an @ref XXH32_state_t. * * Must be allocated with XXH32_createState(). * @param statePtr A pointer to an @ref XXH32_state_t allocated with @ref XXH32_createState(). * @return XXH_OK. */ XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr); /*! * @brief Copies one @ref XXH32_state_t to another. * * @param dst_state The state to copy to. * @param src_state The state to copy from. * @pre * @p dst_state and @p src_state must not be `NULL` and must not overlap. */ XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dst_state, const XXH32_state_t* src_state); /*! * @brief Resets an @ref XXH32_state_t to begin a new hash. * * This function resets and seeds a state. Call it before @ref XXH32_update(). * * @param statePtr The state struct to reset. * @param seed The 32-bit seed to alter the hash result predictably. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH32_reset (XXH32_state_t* statePtr, XXH32_hash_t seed); /*! * @brief Consumes a block of @p input to an @ref XXH32_state_t. * * Call this to incrementally consume blocks of data. * * @param statePtr The state struct to update. * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * * @pre * @p statePtr must not be `NULL`. * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length); /*! * @brief Returns the calculated hash value from an @ref XXH32_state_t. * * @note * Calling XXH32_digest() will not affect @p statePtr, so you can update, * digest, and update again. * * @param statePtr The state struct to calculate the hash from. * * @pre * @p statePtr must not be `NULL`. * * @return The calculated xxHash32 value from that state. */ XXH_PUBLIC_API XXH_PUREF XXH32_hash_t XXH32_digest (const XXH32_state_t* statePtr); #endif /* !XXH_NO_STREAM */ /******* Canonical representation *******/ /* * The default return values from XXH functions are unsigned 32 and 64 bit * integers. * This the simplest and fastest format for further post-processing. * * However, this leaves open the question of what is the order on the byte level, * since little and big endian conventions will store the same number differently. * * The canonical representation settles this issue by mandating big-endian * convention, the same convention as human-readable numbers (large digits first). * * When writing hash values to storage, sending them over a network, or printing * them, it's highly recommended to use the canonical representation to ensure * portability across a wider range of systems, present and future. * * The following functions allow transformation of hash values to and from * canonical format. */ /*! * @brief Canonical (big endian) representation of @ref XXH32_hash_t. */ typedef struct { unsigned char digest[4]; /*!< Hash bytes, big endian */ } XXH32_canonical_t; /*! * @brief Converts an @ref XXH32_hash_t to a big endian @ref XXH32_canonical_t. * * @param dst The @ref XXH32_canonical_t pointer to be stored to. * @param hash The @ref XXH32_hash_t to be converted. * * @pre * @p dst must not be `NULL`. */ XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash); /*! * @brief Converts an @ref XXH32_canonical_t to a native @ref XXH32_hash_t. * * @param src The @ref XXH32_canonical_t to convert. * * @pre * @p src must not be `NULL`. * * @return The converted hash. */ XXH_PUBLIC_API XXH_PUREF XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src); /*! @cond Doxygen ignores this part */ #ifdef __has_attribute # define XXH_HAS_ATTRIBUTE(x) __has_attribute(x) #else # define XXH_HAS_ATTRIBUTE(x) 0 #endif /*! @endcond */ /*! @cond Doxygen ignores this part */ /* * C23 __STDC_VERSION__ number hasn't been specified yet. For now * leave as `201711L` (C17 + 1). * TODO: Update to correct value when its been specified. */ #define XXH_C23_VN 201711L /*! @endcond */ /*! @cond Doxygen ignores this part */ /* C-language Attributes are added in C23. */ #if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= XXH_C23_VN) && defined(__has_c_attribute) # define XXH_HAS_C_ATTRIBUTE(x) __has_c_attribute(x) #else # define XXH_HAS_C_ATTRIBUTE(x) 0 #endif /*! @endcond */ /*! @cond Doxygen ignores this part */ #if defined(__cplusplus) && defined(__has_cpp_attribute) # define XXH_HAS_CPP_ATTRIBUTE(x) __has_cpp_attribute(x) #else # define XXH_HAS_CPP_ATTRIBUTE(x) 0 #endif /*! @endcond */ /*! @cond Doxygen ignores this part */ /* * Define XXH_FALLTHROUGH macro for annotating switch case with the 'fallthrough' attribute * introduced in CPP17 and C23. * CPP17 : https://en.cppreference.com/w/cpp/language/attributes/fallthrough * C23 : https://en.cppreference.com/w/c/language/attributes/fallthrough */ #if XXH_HAS_C_ATTRIBUTE(fallthrough) || XXH_HAS_CPP_ATTRIBUTE(fallthrough) # define XXH_FALLTHROUGH [[fallthrough]] #elif XXH_HAS_ATTRIBUTE(__fallthrough__) # define XXH_FALLTHROUGH __attribute__ ((__fallthrough__)) #else # define XXH_FALLTHROUGH /* fallthrough */ #endif /*! @endcond */ /*! @cond Doxygen ignores this part */ /* * Define XXH_NOESCAPE for annotated pointers in public API. * https://clang.llvm.org/docs/AttributeReference.html#noescape * As of writing this, only supported by clang. */ #if XXH_HAS_ATTRIBUTE(noescape) # define XXH_NOESCAPE __attribute__((noescape)) #else # define XXH_NOESCAPE #endif /*! @endcond */ /*! * @} * @ingroup public * @{ */ #ifndef XXH_NO_LONG_LONG /*-********************************************************************** * 64-bit hash ************************************************************************/ #if defined(XXH_DOXYGEN) /* don't include */ /*! * @brief An unsigned 64-bit integer. * * Not necessarily defined to `uint64_t` but functionally equivalent. */ typedef uint64_t XXH64_hash_t; #elif !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) # include typedef uint64_t XXH64_hash_t; #else # include # if defined(__LP64__) && ULONG_MAX == 0xFFFFFFFFFFFFFFFFULL /* LP64 ABI says uint64_t is unsigned long */ typedef unsigned long XXH64_hash_t; # else /* the following type must have a width of 64-bit */ typedef unsigned long long XXH64_hash_t; # endif #endif /*! * @} * * @defgroup XXH64_family XXH64 family * @ingroup public * @{ * Contains functions used in the classic 64-bit xxHash algorithm. * * @note * XXH3 provides competitive speed for both 32-bit and 64-bit systems, * and offers true 64/128 bit hash results. * It provides better speed for systems with vector processing capabilities. */ /*! * @brief Calculates the 64-bit hash of @p input using xxHash64. * * This function usually runs faster on 64-bit systems, but slower on 32-bit * systems (see benchmark). * * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * @param seed The 64-bit seed to alter the hash's output predictably. * * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return The calculated 64-bit hash. * * @see * XXH32(), XXH3_64bits_withSeed(), XXH3_128bits_withSeed(), XXH128(): * Direct equivalents for the other variants of xxHash. * @see * XXH64_createState(), XXH64_update(), XXH64_digest(): Streaming version. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH64(XXH_NOESCAPE const void* input, size_t length, XXH64_hash_t seed); /******* Streaming *******/ #ifndef XXH_NO_STREAM /*! * @brief The opaque state struct for the XXH64 streaming API. * * @see XXH64_state_s for details. */ typedef struct XXH64_state_s XXH64_state_t; /* incomplete type */ /*! * @brief Allocates an @ref XXH64_state_t. * * Must be freed with XXH64_freeState(). * @return An allocated XXH64_state_t on success, `NULL` on failure. */ XXH_PUBLIC_API XXH_MALLOCF XXH64_state_t* XXH64_createState(void); /*! * @brief Frees an @ref XXH64_state_t. * * Must be allocated with XXH64_createState(). * @param statePtr A pointer to an @ref XXH64_state_t allocated with @ref XXH64_createState(). * @return XXH_OK. */ XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr); /*! * @brief Copies one @ref XXH64_state_t to another. * * @param dst_state The state to copy to. * @param src_state The state to copy from. * @pre * @p dst_state and @p src_state must not be `NULL` and must not overlap. */ XXH_PUBLIC_API void XXH64_copyState(XXH_NOESCAPE XXH64_state_t* dst_state, const XXH64_state_t* src_state); /*! * @brief Resets an @ref XXH64_state_t to begin a new hash. * * This function resets and seeds a state. Call it before @ref XXH64_update(). * * @param statePtr The state struct to reset. * @param seed The 64-bit seed to alter the hash result predictably. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH64_reset (XXH_NOESCAPE XXH64_state_t* statePtr, XXH64_hash_t seed); /*! * @brief Consumes a block of @p input to an @ref XXH64_state_t. * * Call this to incrementally consume blocks of data. * * @param statePtr The state struct to update. * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * * @pre * @p statePtr must not be `NULL`. * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH_NOESCAPE XXH64_state_t* statePtr, XXH_NOESCAPE const void* input, size_t length); /*! * @brief Returns the calculated hash value from an @ref XXH64_state_t. * * @note * Calling XXH64_digest() will not affect @p statePtr, so you can update, * digest, and update again. * * @param statePtr The state struct to calculate the hash from. * * @pre * @p statePtr must not be `NULL`. * * @return The calculated xxHash64 value from that state. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH64_digest (XXH_NOESCAPE const XXH64_state_t* statePtr); #endif /* !XXH_NO_STREAM */ /******* Canonical representation *******/ /*! * @brief Canonical (big endian) representation of @ref XXH64_hash_t. */ typedef struct { unsigned char digest[sizeof(XXH64_hash_t)]; } XXH64_canonical_t; /*! * @brief Converts an @ref XXH64_hash_t to a big endian @ref XXH64_canonical_t. * * @param dst The @ref XXH64_canonical_t pointer to be stored to. * @param hash The @ref XXH64_hash_t to be converted. * * @pre * @p dst must not be `NULL`. */ XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH_NOESCAPE XXH64_canonical_t* dst, XXH64_hash_t hash); /*! * @brief Converts an @ref XXH64_canonical_t to a native @ref XXH64_hash_t. * * @param src The @ref XXH64_canonical_t to convert. * * @pre * @p src must not be `NULL`. * * @return The converted hash. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH64_hashFromCanonical(XXH_NOESCAPE const XXH64_canonical_t* src); #ifndef XXH_NO_XXH3 /*! * @} * ************************************************************************ * @defgroup XXH3_family XXH3 family * @ingroup public * @{ * * XXH3 is a more recent hash algorithm featuring: * - Improved speed for both small and large inputs * - True 64-bit and 128-bit outputs * - SIMD acceleration * - Improved 32-bit viability * * Speed analysis methodology is explained here: * * https://fastcompression.blogspot.com/2019/03/presenting-xxh3.html * * Compared to XXH64, expect XXH3 to run approximately * ~2x faster on large inputs and >3x faster on small ones, * exact differences vary depending on platform. * * XXH3's speed benefits greatly from SIMD and 64-bit arithmetic, * but does not require it. * Most 32-bit and 64-bit targets that can run XXH32 smoothly can run XXH3 * at competitive speeds, even without vector support. Further details are * explained in the implementation. * * XXH3 has a fast scalar implementation, but it also includes accelerated SIMD * implementations for many common platforms: * - AVX512 * - AVX2 * - SSE2 * - ARM NEON * - WebAssembly SIMD128 * - POWER8 VSX * - s390x ZVector * This can be controlled via the @ref XXH_VECTOR macro, but it automatically * selects the best version according to predefined macros. For the x86 family, an * automatic runtime dispatcher is included separately in @ref xxh_x86dispatch.c. * * XXH3 implementation is portable: * it has a generic C90 formulation that can be compiled on any platform, * all implementations generate exactly the same hash value on all platforms. * Starting from v0.8.0, it's also labelled "stable", meaning that * any future version will also generate the same hash value. * * XXH3 offers 2 variants, _64bits and _128bits. * * When only 64 bits are needed, prefer invoking the _64bits variant, as it * reduces the amount of mixing, resulting in faster speed on small inputs. * It's also generally simpler to manipulate a scalar return type than a struct. * * The API supports one-shot hashing, streaming mode, and custom secrets. */ /*-********************************************************************** * XXH3 64-bit variant ************************************************************************/ /*! * @brief 64-bit unseeded variant of XXH3. * * This is equivalent to @ref XXH3_64bits_withSeed() with a seed of 0, however * it may have slightly better performance due to constant propagation of the * defaults. * * @see * XXH32(), XXH64(), XXH3_128bits(): equivalent for the other xxHash algorithms * @see * XXH3_64bits_withSeed(), XXH3_64bits_withSecret(): other seeding variants * @see * XXH3_64bits_reset(), XXH3_64bits_update(), XXH3_64bits_digest(): Streaming version. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH3_64bits(XXH_NOESCAPE const void* input, size_t length); /*! * @brief 64-bit seeded variant of XXH3 * * This variant generates a custom secret on the fly based on default secret * altered using the `seed` value. * * While this operation is decently fast, note that it's not completely free. * * @note * seed == 0 produces the same results as @ref XXH3_64bits(). * * @param input The data to hash * @param length The length * @param seed The 64-bit seed to alter the state. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH3_64bits_withSeed(XXH_NOESCAPE const void* input, size_t length, XXH64_hash_t seed); /*! * The bare minimum size for a custom secret. * * @see * XXH3_64bits_withSecret(), XXH3_64bits_reset_withSecret(), * XXH3_128bits_withSecret(), XXH3_128bits_reset_withSecret(). */ #define XXH3_SECRET_SIZE_MIN 136 /*! * @brief 64-bit variant of XXH3 with a custom "secret". * * It's possible to provide any blob of bytes as a "secret" to generate the hash. * This makes it more difficult for an external actor to prepare an intentional collision. * The main condition is that secretSize *must* be large enough (>= XXH3_SECRET_SIZE_MIN). * However, the quality of the secret impacts the dispersion of the hash algorithm. * Therefore, the secret _must_ look like a bunch of random bytes. * Avoid "trivial" or structured data such as repeated sequences or a text document. * Whenever in doubt about the "randomness" of the blob of bytes, * consider employing "XXH3_generateSecret()" instead (see below). * It will generate a proper high entropy secret derived from the blob of bytes. * Another advantage of using XXH3_generateSecret() is that * it guarantees that all bits within the initial blob of bytes * will impact every bit of the output. * This is not necessarily the case when using the blob of bytes directly * because, when hashing _small_ inputs, only a portion of the secret is employed. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH3_64bits_withSecret(XXH_NOESCAPE const void* data, size_t len, XXH_NOESCAPE const void* secret, size_t secretSize); /******* Streaming *******/ #ifndef XXH_NO_STREAM /* * Streaming requires state maintenance. * This operation costs memory and CPU. * As a consequence, streaming is slower than one-shot hashing. * For better performance, prefer one-shot functions whenever applicable. */ /*! * @brief The state struct for the XXH3 streaming API. * * @see XXH3_state_s for details. */ typedef struct XXH3_state_s XXH3_state_t; XXH_PUBLIC_API XXH_MALLOCF XXH3_state_t* XXH3_createState(void); XXH_PUBLIC_API XXH_errorcode XXH3_freeState(XXH3_state_t* statePtr); /*! * @brief Copies one @ref XXH3_state_t to another. * * @param dst_state The state to copy to. * @param src_state The state to copy from. * @pre * @p dst_state and @p src_state must not be `NULL` and must not overlap. */ XXH_PUBLIC_API void XXH3_copyState(XXH_NOESCAPE XXH3_state_t* dst_state, XXH_NOESCAPE const XXH3_state_t* src_state); /*! * @brief Resets an @ref XXH3_state_t to begin a new hash. * * This function resets `statePtr` and generate a secret with default parameters. Call it before @ref XXH3_64bits_update(). * Digest will be equivalent to `XXH3_64bits()`. * * @param statePtr The state struct to reset. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. * */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset(XXH_NOESCAPE XXH3_state_t* statePtr); /*! * @brief Resets an @ref XXH3_state_t with 64-bit seed to begin a new hash. * * This function resets `statePtr` and generate a secret from `seed`. Call it before @ref XXH3_64bits_update(). * Digest will be equivalent to `XXH3_64bits_withSeed()`. * * @param statePtr The state struct to reset. * @param seed The 64-bit seed to alter the state. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. * */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH64_hash_t seed); /*! * XXH3_64bits_reset_withSecret(): * `secret` is referenced, it _must outlive_ the hash streaming session. * Similar to one-shot API, `secretSize` must be >= `XXH3_SECRET_SIZE_MIN`, * and the quality of produced hash values depends on secret's entropy * (secret's content should look like a bunch of random bytes). * When in doubt about the randomness of a candidate `secret`, * consider employing `XXH3_generateSecret()` instead (see below). */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecret(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize); /*! * @brief Consumes a block of @p input to an @ref XXH3_state_t. * * Call this to incrementally consume blocks of data. * * @param statePtr The state struct to update. * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * * @pre * @p statePtr must not be `NULL`. * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_update (XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* input, size_t length); /*! * @brief Returns the calculated XXH3 64-bit hash value from an @ref XXH3_state_t. * * @note * Calling XXH3_64bits_digest() will not affect @p statePtr, so you can update, * digest, and update again. * * @param statePtr The state struct to calculate the hash from. * * @pre * @p statePtr must not be `NULL`. * * @return The calculated XXH3 64-bit hash value from that state. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH3_64bits_digest (XXH_NOESCAPE const XXH3_state_t* statePtr); #endif /* !XXH_NO_STREAM */ /* note : canonical representation of XXH3 is the same as XXH64 * since they both produce XXH64_hash_t values */ /*-********************************************************************** * XXH3 128-bit variant ************************************************************************/ /*! * @brief The return value from 128-bit hashes. * * Stored in little endian order, although the fields themselves are in native * endianness. */ typedef struct { XXH64_hash_t low64; /*!< `value & 0xFFFFFFFFFFFFFFFF` */ XXH64_hash_t high64; /*!< `value >> 64` */ } XXH128_hash_t; /*! * @brief Unseeded 128-bit variant of XXH3 * * The 128-bit variant of XXH3 has more strength, but it has a bit of overhead * for shorter inputs. * * This is equivalent to @ref XXH3_128bits_withSeed() with a seed of 0, however * it may have slightly better performance due to constant propagation of the * defaults. * * @see * XXH32(), XXH64(), XXH3_64bits(): equivalent for the other xxHash algorithms * @see * XXH3_128bits_withSeed(), XXH3_128bits_withSecret(): other seeding variants * @see * XXH3_128bits_reset(), XXH3_128bits_update(), XXH3_128bits_digest(): Streaming version. */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH3_128bits(XXH_NOESCAPE const void* data, size_t len); /*! @brief Seeded 128-bit variant of XXH3. @see XXH3_64bits_withSeed(). */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH3_128bits_withSeed(XXH_NOESCAPE const void* data, size_t len, XXH64_hash_t seed); /*! @brief Custom secret 128-bit variant of XXH3. @see XXH3_64bits_withSecret(). */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH3_128bits_withSecret(XXH_NOESCAPE const void* data, size_t len, XXH_NOESCAPE const void* secret, size_t secretSize); /******* Streaming *******/ #ifndef XXH_NO_STREAM /* * Streaming requires state maintenance. * This operation costs memory and CPU. * As a consequence, streaming is slower than one-shot hashing. * For better performance, prefer one-shot functions whenever applicable. * * XXH3_128bits uses the same XXH3_state_t as XXH3_64bits(). * Use already declared XXH3_createState() and XXH3_freeState(). * * All reset and streaming functions have same meaning as their 64-bit counterpart. */ /*! * @brief Resets an @ref XXH3_state_t to begin a new hash. * * This function resets `statePtr` and generate a secret with default parameters. Call it before @ref XXH3_128bits_update(). * Digest will be equivalent to `XXH3_128bits()`. * * @param statePtr The state struct to reset. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. * */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset(XXH_NOESCAPE XXH3_state_t* statePtr); /*! * @brief Resets an @ref XXH3_state_t with 64-bit seed to begin a new hash. * * This function resets `statePtr` and generate a secret from `seed`. Call it before @ref XXH3_128bits_update(). * Digest will be equivalent to `XXH3_128bits_withSeed()`. * * @param statePtr The state struct to reset. * @param seed The 64-bit seed to alter the state. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. * */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH64_hash_t seed); /*! @brief Custom secret 128-bit variant of XXH3. @see XXH_64bits_reset_withSecret(). */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecret(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize); /*! * @brief Consumes a block of @p input to an @ref XXH3_state_t. * * Call this to incrementally consume blocks of data. * * @param statePtr The state struct to update. * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * * @pre * @p statePtr must not be `NULL`. * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_update (XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* input, size_t length); /*! * @brief Returns the calculated XXH3 128-bit hash value from an @ref XXH3_state_t. * * @note * Calling XXH3_128bits_digest() will not affect @p statePtr, so you can update, * digest, and update again. * * @param statePtr The state struct to calculate the hash from. * * @pre * @p statePtr must not be `NULL`. * * @return The calculated XXH3 128-bit hash value from that state. */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH3_128bits_digest (XXH_NOESCAPE const XXH3_state_t* statePtr); #endif /* !XXH_NO_STREAM */ /* Following helper functions make it possible to compare XXH128_hast_t values. * Since XXH128_hash_t is a structure, this capability is not offered by the language. * Note: For better performance, these functions can be inlined using XXH_INLINE_ALL */ /*! * XXH128_isEqual(): * Return: 1 if `h1` and `h2` are equal, 0 if they are not. */ XXH_PUBLIC_API XXH_PUREF int XXH128_isEqual(XXH128_hash_t h1, XXH128_hash_t h2); /*! * @brief Compares two @ref XXH128_hash_t * This comparator is compatible with stdlib's `qsort()`/`bsearch()`. * * @return: >0 if *h128_1 > *h128_2 * =0 if *h128_1 == *h128_2 * <0 if *h128_1 < *h128_2 */ XXH_PUBLIC_API XXH_PUREF int XXH128_cmp(XXH_NOESCAPE const void* h128_1, XXH_NOESCAPE const void* h128_2); /******* Canonical representation *******/ typedef struct { unsigned char digest[sizeof(XXH128_hash_t)]; } XXH128_canonical_t; /*! * @brief Converts an @ref XXH128_hash_t to a big endian @ref XXH128_canonical_t. * * @param dst The @ref XXH128_canonical_t pointer to be stored to. * @param hash The @ref XXH128_hash_t to be converted. * * @pre * @p dst must not be `NULL`. */ XXH_PUBLIC_API void XXH128_canonicalFromHash(XXH_NOESCAPE XXH128_canonical_t* dst, XXH128_hash_t hash); /*! * @brief Converts an @ref XXH128_canonical_t to a native @ref XXH128_hash_t. * * @param src The @ref XXH128_canonical_t to convert. * * @pre * @p src must not be `NULL`. * * @return The converted hash. */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH128_hashFromCanonical(XXH_NOESCAPE const XXH128_canonical_t* src); #endif /* !XXH_NO_XXH3 */ #endif /* XXH_NO_LONG_LONG */ /*! * @} */ #endif /* XXHASH_H_5627135585666179 */ #if defined(XXH_STATIC_LINKING_ONLY) && !defined(XXHASH_H_STATIC_13879238742) #define XXHASH_H_STATIC_13879238742 /* **************************************************************************** * This section contains declarations which are not guaranteed to remain stable. * They may change in future versions, becoming incompatible with a different * version of the library. * These declarations should only be used with static linking. * Never use them in association with dynamic linking! ***************************************************************************** */ /* * These definitions are only present to allow static allocation * of XXH states, on stack or in a struct, for example. * Never **ever** access their members directly. */ /*! * @internal * @brief Structure for XXH32 streaming API. * * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. Otherwise it is * an opaque type. This allows fields to safely be changed. * * Typedef'd to @ref XXH32_state_t. * Do not access the members of this struct directly. * @see XXH64_state_s, XXH3_state_s */ struct XXH32_state_s { XXH32_hash_t total_len_32; /*!< Total length hashed, modulo 2^32 */ XXH32_hash_t large_len; /*!< Whether the hash is >= 16 (handles @ref total_len_32 overflow) */ XXH32_hash_t v[4]; /*!< Accumulator lanes */ XXH32_hash_t mem32[4]; /*!< Internal buffer for partial reads. Treated as unsigned char[16]. */ XXH32_hash_t memsize; /*!< Amount of data in @ref mem32 */ XXH32_hash_t reserved; /*!< Reserved field. Do not read nor write to it. */ }; /* typedef'd to XXH32_state_t */ #ifndef XXH_NO_LONG_LONG /* defined when there is no 64-bit support */ /*! * @internal * @brief Structure for XXH64 streaming API. * * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. Otherwise it is * an opaque type. This allows fields to safely be changed. * * Typedef'd to @ref XXH64_state_t. * Do not access the members of this struct directly. * @see XXH32_state_s, XXH3_state_s */ struct XXH64_state_s { XXH64_hash_t total_len; /*!< Total length hashed. This is always 64-bit. */ XXH64_hash_t v[4]; /*!< Accumulator lanes */ XXH64_hash_t mem64[4]; /*!< Internal buffer for partial reads. Treated as unsigned char[32]. */ XXH32_hash_t memsize; /*!< Amount of data in @ref mem64 */ XXH32_hash_t reserved32; /*!< Reserved field, needed for padding anyways*/ XXH64_hash_t reserved64; /*!< Reserved field. Do not read or write to it. */ }; /* typedef'd to XXH64_state_t */ #ifndef XXH_NO_XXH3 #if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L) /* >= C11 */ # include # define XXH_ALIGN(n) alignas(n) #elif defined(__cplusplus) && (__cplusplus >= 201103L) /* >= C++11 */ /* In C++ alignas() is a keyword */ # define XXH_ALIGN(n) alignas(n) #elif defined(__GNUC__) # define XXH_ALIGN(n) __attribute__ ((aligned(n))) #elif defined(_MSC_VER) # define XXH_ALIGN(n) __declspec(align(n)) #else # define XXH_ALIGN(n) /* disabled */ #endif /* Old GCC versions only accept the attribute after the type in structures. */ #if !(defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L)) /* C11+ */ \ && ! (defined(__cplusplus) && (__cplusplus >= 201103L)) /* >= C++11 */ \ && defined(__GNUC__) # define XXH_ALIGN_MEMBER(align, type) type XXH_ALIGN(align) #else # define XXH_ALIGN_MEMBER(align, type) XXH_ALIGN(align) type #endif /*! * @brief The size of the internal XXH3 buffer. * * This is the optimal update size for incremental hashing. * * @see XXH3_64b_update(), XXH3_128b_update(). */ #define XXH3_INTERNALBUFFER_SIZE 256 /*! * @internal * @brief Default size of the secret buffer (and @ref XXH3_kSecret). * * This is the size used in @ref XXH3_kSecret and the seeded functions. * * Not to be confused with @ref XXH3_SECRET_SIZE_MIN. */ #define XXH3_SECRET_DEFAULT_SIZE 192 /*! * @internal * @brief Structure for XXH3 streaming API. * * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. * Otherwise it is an opaque type. * Never use this definition in combination with dynamic library. * This allows fields to safely be changed in the future. * * @note ** This structure has a strict alignment requirement of 64 bytes!! ** * Do not allocate this with `malloc()` or `new`, * it will not be sufficiently aligned. * Use @ref XXH3_createState() and @ref XXH3_freeState(), or stack allocation. * * Typedef'd to @ref XXH3_state_t. * Do never access the members of this struct directly. * * @see XXH3_INITSTATE() for stack initialization. * @see XXH3_createState(), XXH3_freeState(). * @see XXH32_state_s, XXH64_state_s */ struct XXH3_state_s { XXH_ALIGN_MEMBER(64, XXH64_hash_t acc[8]); /*!< The 8 accumulators. See @ref XXH32_state_s::v and @ref XXH64_state_s::v */ XXH_ALIGN_MEMBER(64, unsigned char customSecret[XXH3_SECRET_DEFAULT_SIZE]); /*!< Used to store a custom secret generated from a seed. */ XXH_ALIGN_MEMBER(64, unsigned char buffer[XXH3_INTERNALBUFFER_SIZE]); /*!< The internal buffer. @see XXH32_state_s::mem32 */ XXH32_hash_t bufferedSize; /*!< The amount of memory in @ref buffer, @see XXH32_state_s::memsize */ XXH32_hash_t useSeed; /*!< Reserved field. Needed for padding on 64-bit. */ size_t nbStripesSoFar; /*!< Number or stripes processed. */ XXH64_hash_t totalLen; /*!< Total length hashed. 64-bit even on 32-bit targets. */ size_t nbStripesPerBlock; /*!< Number of stripes per block. */ size_t secretLimit; /*!< Size of @ref customSecret or @ref extSecret */ XXH64_hash_t seed; /*!< Seed for _withSeed variants. Must be zero otherwise, @see XXH3_INITSTATE() */ XXH64_hash_t reserved64; /*!< Reserved field. */ const unsigned char* extSecret; /*!< Reference to an external secret for the _withSecret variants, NULL * for other variants. */ /* note: there may be some padding at the end due to alignment on 64 bytes */ }; /* typedef'd to XXH3_state_t */ #undef XXH_ALIGN_MEMBER /*! * @brief Initializes a stack-allocated `XXH3_state_s`. * * When the @ref XXH3_state_t structure is merely emplaced on stack, * it should be initialized with XXH3_INITSTATE() or a memset() * in case its first reset uses XXH3_NNbits_reset_withSeed(). * This init can be omitted if the first reset uses default or _withSecret mode. * This operation isn't necessary when the state is created with XXH3_createState(). * Note that this doesn't prepare the state for a streaming operation, * it's still necessary to use XXH3_NNbits_reset*() afterwards. */ #define XXH3_INITSTATE(XXH3_state_ptr) \ do { \ XXH3_state_t* tmp_xxh3_state_ptr = (XXH3_state_ptr); \ tmp_xxh3_state_ptr->seed = 0; \ tmp_xxh3_state_ptr->extSecret = NULL; \ } while(0) /*! * simple alias to pre-selected XXH3_128bits variant */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH128(XXH_NOESCAPE const void* data, size_t len, XXH64_hash_t seed); /* === Experimental API === */ /* Symbols defined below must be considered tied to a specific library version. */ /*! * XXH3_generateSecret(): * * Derive a high-entropy secret from any user-defined content, named customSeed. * The generated secret can be used in combination with `*_withSecret()` functions. * The `_withSecret()` variants are useful to provide a higher level of protection * than 64-bit seed, as it becomes much more difficult for an external actor to * guess how to impact the calculation logic. * * The function accepts as input a custom seed of any length and any content, * and derives from it a high-entropy secret of length @p secretSize into an * already allocated buffer @p secretBuffer. * * The generated secret can then be used with any `*_withSecret()` variant. * The functions @ref XXH3_128bits_withSecret(), @ref XXH3_64bits_withSecret(), * @ref XXH3_128bits_reset_withSecret() and @ref XXH3_64bits_reset_withSecret() * are part of this list. They all accept a `secret` parameter * which must be large enough for implementation reasons (>= @ref XXH3_SECRET_SIZE_MIN) * _and_ feature very high entropy (consist of random-looking bytes). * These conditions can be a high bar to meet, so @ref XXH3_generateSecret() can * be employed to ensure proper quality. * * @p customSeed can be anything. It can have any size, even small ones, * and its content can be anything, even "poor entropy" sources such as a bunch * of zeroes. The resulting `secret` will nonetheless provide all required qualities. * * @pre * - @p secretSize must be >= @ref XXH3_SECRET_SIZE_MIN * - When @p customSeedSize > 0, supplying NULL as customSeed is undefined behavior. * * Example code: * @code{.c} * #include * #include * #include * #define XXH_STATIC_LINKING_ONLY // expose unstable API * #include "xxhash.h" * // Hashes argv[2] using the entropy from argv[1]. * int main(int argc, char* argv[]) * { * char secret[XXH3_SECRET_SIZE_MIN]; * if (argv != 3) { return 1; } * XXH3_generateSecret(secret, sizeof(secret), argv[1], strlen(argv[1])); * XXH64_hash_t h = XXH3_64bits_withSecret( * argv[2], strlen(argv[2]), * secret, sizeof(secret) * ); * printf("%016llx\n", (unsigned long long) h); * } * @endcode */ XXH_PUBLIC_API XXH_errorcode XXH3_generateSecret(XXH_NOESCAPE void* secretBuffer, size_t secretSize, XXH_NOESCAPE const void* customSeed, size_t customSeedSize); /*! * @brief Generate the same secret as the _withSeed() variants. * * The generated secret can be used in combination with *`*_withSecret()` and `_withSecretandSeed()` variants. * * Example C++ `std::string` hash class: * @code{.cpp} * #include * #define XXH_STATIC_LINKING_ONLY // expose unstable API * #include "xxhash.h" * // Slow, seeds each time * class HashSlow { * XXH64_hash_t seed; * public: * HashSlow(XXH64_hash_t s) : seed{s} {} * size_t operator()(const std::string& x) const { * return size_t{XXH3_64bits_withSeed(x.c_str(), x.length(), seed)}; * } * }; * // Fast, caches the seeded secret for future uses. * class HashFast { * unsigned char secret[XXH3_SECRET_SIZE_MIN]; * public: * HashFast(XXH64_hash_t s) { * XXH3_generateSecret_fromSeed(secret, seed); * } * size_t operator()(const std::string& x) const { * return size_t{ * XXH3_64bits_withSecret(x.c_str(), x.length(), secret, sizeof(secret)) * }; * } * }; * @endcode * @param secretBuffer A writable buffer of @ref XXH3_SECRET_SIZE_MIN bytes * @param seed The seed to seed the state. */ XXH_PUBLIC_API void XXH3_generateSecret_fromSeed(XXH_NOESCAPE void* secretBuffer, XXH64_hash_t seed); /*! * These variants generate hash values using either * @p seed for "short" keys (< XXH3_MIDSIZE_MAX = 240 bytes) * or @p secret for "large" keys (>= XXH3_MIDSIZE_MAX). * * This generally benefits speed, compared to `_withSeed()` or `_withSecret()`. * `_withSeed()` has to generate the secret on the fly for "large" keys. * It's fast, but can be perceptible for "not so large" keys (< 1 KB). * `_withSecret()` has to generate the masks on the fly for "small" keys, * which requires more instructions than _withSeed() variants. * Therefore, _withSecretandSeed variant combines the best of both worlds. * * When @p secret has been generated by XXH3_generateSecret_fromSeed(), * this variant produces *exactly* the same results as `_withSeed()` variant, * hence offering only a pure speed benefit on "large" input, * by skipping the need to regenerate the secret for every large input. * * Another usage scenario is to hash the secret to a 64-bit hash value, * for example with XXH3_64bits(), which then becomes the seed, * and then employ both the seed and the secret in _withSecretandSeed(). * On top of speed, an added benefit is that each bit in the secret * has a 50% chance to swap each bit in the output, via its impact to the seed. * * This is not guaranteed when using the secret directly in "small data" scenarios, * because only portions of the secret are employed for small data. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH3_64bits_withSecretandSeed(XXH_NOESCAPE const void* data, size_t len, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed); /*! @copydoc XXH3_64bits_withSecretandSeed() */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH3_128bits_withSecretandSeed(XXH_NOESCAPE const void* input, size_t length, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed64); #ifndef XXH_NO_STREAM /*! @copydoc XXH3_64bits_withSecretandSeed() */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecretandSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed64); /*! @copydoc XXH3_64bits_withSecretandSeed() */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecretandSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed64); #endif /* !XXH_NO_STREAM */ #endif /* !XXH_NO_XXH3 */ #endif /* XXH_NO_LONG_LONG */ #if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API) # define XXH_IMPLEMENTATION #endif #endif /* defined(XXH_STATIC_LINKING_ONLY) && !defined(XXHASH_H_STATIC_13879238742) */ /* ======================================================================== */ /* ======================================================================== */ /* ======================================================================== */ /*-********************************************************************** * xxHash implementation *-********************************************************************** * xxHash's implementation used to be hosted inside xxhash.c. * * However, inlining requires implementation to be visible to the compiler, * hence be included alongside the header. * Previously, implementation was hosted inside xxhash.c, * which was then #included when inlining was activated. * This construction created issues with a few build and install systems, * as it required xxhash.c to be stored in /include directory. * * xxHash implementation is now directly integrated within xxhash.h. * As a consequence, xxhash.c is no longer needed in /include. * * xxhash.c is still available and is still useful. * In a "normal" setup, when xxhash is not inlined, * xxhash.h only exposes the prototypes and public symbols, * while xxhash.c can be built into an object file xxhash.o * which can then be linked into the final binary. ************************************************************************/ #if ( defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API) \ || defined(XXH_IMPLEMENTATION) ) && !defined(XXH_IMPLEM_13a8737387) # define XXH_IMPLEM_13a8737387 /* ************************************* * Tuning parameters ***************************************/ /*! * @defgroup tuning Tuning parameters * @{ * * Various macros to control xxHash's behavior. */ #ifdef XXH_DOXYGEN /*! * @brief Define this to disable 64-bit code. * * Useful if only using the @ref XXH32_family and you have a strict C90 compiler. */ # define XXH_NO_LONG_LONG # undef XXH_NO_LONG_LONG /* don't actually */ /*! * @brief Controls how unaligned memory is accessed. * * By default, access to unaligned memory is controlled by `memcpy()`, which is * safe and portable. * * Unfortunately, on some target/compiler combinations, the generated assembly * is sub-optimal. * * The below switch allow selection of a different access method * in the search for improved performance. * * @par Possible options: * * - `XXH_FORCE_MEMORY_ACCESS=0` (default): `memcpy` * @par * Use `memcpy()`. Safe and portable. Note that most modern compilers will * eliminate the function call and treat it as an unaligned access. * * - `XXH_FORCE_MEMORY_ACCESS=1`: `__attribute__((aligned(1)))` * @par * Depends on compiler extensions and is therefore not portable. * This method is safe _if_ your compiler supports it, * and *generally* as fast or faster than `memcpy`. * * - `XXH_FORCE_MEMORY_ACCESS=2`: Direct cast * @par * Casts directly and dereferences. This method doesn't depend on the * compiler, but it violates the C standard as it directly dereferences an * unaligned pointer. It can generate buggy code on targets which do not * support unaligned memory accesses, but in some circumstances, it's the * only known way to get the most performance. * * - `XXH_FORCE_MEMORY_ACCESS=3`: Byteshift * @par * Also portable. This can generate the best code on old compilers which don't * inline small `memcpy()` calls, and it might also be faster on big-endian * systems which lack a native byteswap instruction. However, some compilers * will emit literal byteshifts even if the target supports unaligned access. * * * @warning * Methods 1 and 2 rely on implementation-defined behavior. Use these with * care, as what works on one compiler/platform/optimization level may cause * another to read garbage data or even crash. * * See https://fastcompression.blogspot.com/2015/08/accessing-unaligned-memory.html for details. * * Prefer these methods in priority order (0 > 3 > 1 > 2) */ # define XXH_FORCE_MEMORY_ACCESS 0 /*! * @def XXH_SIZE_OPT * @brief Controls how much xxHash optimizes for size. * * xxHash, when compiled, tends to result in a rather large binary size. This * is mostly due to heavy usage to forced inlining and constant folding of the * @ref XXH3_family to increase performance. * * However, some developers prefer size over speed. This option can * significantly reduce the size of the generated code. When using the `-Os` * or `-Oz` options on GCC or Clang, this is defined to 1 by default, * otherwise it is defined to 0. * * Most of these size optimizations can be controlled manually. * * This is a number from 0-2. * - `XXH_SIZE_OPT` == 0: Default. xxHash makes no size optimizations. Speed * comes first. * - `XXH_SIZE_OPT` == 1: Default for `-Os` and `-Oz`. xxHash is more * conservative and disables hacks that increase code size. It implies the * options @ref XXH_NO_INLINE_HINTS == 1, @ref XXH_FORCE_ALIGN_CHECK == 0, * and @ref XXH3_NEON_LANES == 8 if they are not already defined. * - `XXH_SIZE_OPT` == 2: xxHash tries to make itself as small as possible. * Performance may cry. For example, the single shot functions just use the * streaming API. */ # define XXH_SIZE_OPT 0 /*! * @def XXH_FORCE_ALIGN_CHECK * @brief If defined to non-zero, adds a special path for aligned inputs (XXH32() * and XXH64() only). * * This is an important performance trick for architectures without decent * unaligned memory access performance. * * It checks for input alignment, and when conditions are met, uses a "fast * path" employing direct 32-bit/64-bit reads, resulting in _dramatically * faster_ read speed. * * The check costs one initial branch per hash, which is generally negligible, * but not zero. * * Moreover, it's not useful to generate an additional code path if memory * access uses the same instruction for both aligned and unaligned * addresses (e.g. x86 and aarch64). * * In these cases, the alignment check can be removed by setting this macro to 0. * Then the code will always use unaligned memory access. * Align check is automatically disabled on x86, x64, ARM64, and some ARM chips * which are platforms known to offer good unaligned memory accesses performance. * * It is also disabled by default when @ref XXH_SIZE_OPT >= 1. * * This option does not affect XXH3 (only XXH32 and XXH64). */ # define XXH_FORCE_ALIGN_CHECK 0 /*! * @def XXH_NO_INLINE_HINTS * @brief When non-zero, sets all functions to `static`. * * By default, xxHash tries to force the compiler to inline almost all internal * functions. * * This can usually improve performance due to reduced jumping and improved * constant folding, but significantly increases the size of the binary which * might not be favorable. * * Additionally, sometimes the forced inlining can be detrimental to performance, * depending on the architecture. * * XXH_NO_INLINE_HINTS marks all internal functions as static, giving the * compiler full control on whether to inline or not. * * When not optimizing (-O0), using `-fno-inline` with GCC or Clang, or if * @ref XXH_SIZE_OPT >= 1, this will automatically be defined. */ # define XXH_NO_INLINE_HINTS 0 /*! * @def XXH3_INLINE_SECRET * @brief Determines whether to inline the XXH3 withSecret code. * * When the secret size is known, the compiler can improve the performance * of XXH3_64bits_withSecret() and XXH3_128bits_withSecret(). * * However, if the secret size is not known, it doesn't have any benefit. This * happens when xxHash is compiled into a global symbol. Therefore, if * @ref XXH_INLINE_ALL is *not* defined, this will be defined to 0. * * Additionally, this defaults to 0 on GCC 12+, which has an issue with function pointers * that are *sometimes* force inline on -Og, and it is impossible to automatically * detect this optimization level. */ # define XXH3_INLINE_SECRET 0 /*! * @def XXH32_ENDJMP * @brief Whether to use a jump for `XXH32_finalize`. * * For performance, `XXH32_finalize` uses multiple branches in the finalizer. * This is generally preferable for performance, * but depending on exact architecture, a jmp may be preferable. * * This setting is only possibly making a difference for very small inputs. */ # define XXH32_ENDJMP 0 /*! * @internal * @brief Redefines old internal names. * * For compatibility with code that uses xxHash's internals before the names * were changed to improve namespacing. There is no other reason to use this. */ # define XXH_OLD_NAMES # undef XXH_OLD_NAMES /* don't actually use, it is ugly. */ /*! * @def XXH_NO_STREAM * @brief Disables the streaming API. * * When xxHash is not inlined and the streaming functions are not used, disabling * the streaming functions can improve code size significantly, especially with * the @ref XXH3_family which tends to make constant folded copies of itself. */ # define XXH_NO_STREAM # undef XXH_NO_STREAM /* don't actually */ #endif /* XXH_DOXYGEN */ /*! * @} */ #ifndef XXH_FORCE_MEMORY_ACCESS /* can be defined externally, on command line for example */ /* prefer __packed__ structures (method 1) for GCC * < ARMv7 with unaligned access (e.g. Raspbian armhf) still uses byte shifting, so we use memcpy * which for some reason does unaligned loads. */ # if defined(__GNUC__) && !(defined(__ARM_ARCH) && __ARM_ARCH < 7 && defined(__ARM_FEATURE_UNALIGNED)) # define XXH_FORCE_MEMORY_ACCESS 1 # endif #endif #ifndef XXH_SIZE_OPT /* default to 1 for -Os or -Oz */ # if (defined(__GNUC__) || defined(__clang__)) && defined(__OPTIMIZE_SIZE__) # define XXH_SIZE_OPT 1 # else # define XXH_SIZE_OPT 0 # endif #endif #ifndef XXH_FORCE_ALIGN_CHECK /* can be defined externally */ /* don't check on sizeopt, x86, aarch64, or arm when unaligned access is available */ # if XXH_SIZE_OPT >= 1 || \ defined(__i386) || defined(__x86_64__) || defined(__aarch64__) || defined(__ARM_FEATURE_UNALIGNED) \ || defined(_M_IX86) || defined(_M_X64) || defined(_M_ARM64) || defined(_M_ARM) /* visual */ # define XXH_FORCE_ALIGN_CHECK 0 # else # define XXH_FORCE_ALIGN_CHECK 1 # endif #endif #ifndef XXH_NO_INLINE_HINTS # if XXH_SIZE_OPT >= 1 || defined(__NO_INLINE__) /* -O0, -fno-inline */ # define XXH_NO_INLINE_HINTS 1 # else # define XXH_NO_INLINE_HINTS 0 # endif #endif #ifndef XXH3_INLINE_SECRET # if (defined(__GNUC__) && !defined(__clang__) && __GNUC__ >= 12) \ || !defined(XXH_INLINE_ALL) # define XXH3_INLINE_SECRET 0 # else # define XXH3_INLINE_SECRET 1 # endif #endif #ifndef XXH32_ENDJMP /* generally preferable for performance */ # define XXH32_ENDJMP 0 #endif /*! * @defgroup impl Implementation * @{ */ /* ************************************* * Includes & Memory related functions ***************************************/ #if defined(XXH_NO_STREAM) /* nothing */ #elif defined(XXH_NO_STDLIB) /* When requesting to disable any mention of stdlib, * the library loses the ability to invoked malloc / free. * In practice, it means that functions like `XXH*_createState()` * will always fail, and return NULL. * This flag is useful in situations where * xxhash.h is integrated into some kernel, embedded or limited environment * without access to dynamic allocation. */ static XXH_CONSTF void* XXH_malloc(size_t s) { (void)s; return NULL; } static void XXH_free(void* p) { (void)p; } #else /* * Modify the local functions below should you wish to use * different memory routines for malloc() and free() */ #include /*! * @internal * @brief Modify this function to use a different routine than malloc(). */ static XXH_MALLOCF void* XXH_malloc(size_t s) { return malloc(s); } /*! * @internal * @brief Modify this function to use a different routine than free(). */ static void XXH_free(void* p) { free(p); } #endif /* XXH_NO_STDLIB */ #include /*! * @internal * @brief Modify this function to use a different routine than memcpy(). */ static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); } #include /* ULLONG_MAX */ /* ************************************* * Compiler Specific Options ***************************************/ #ifdef _MSC_VER /* Visual Studio warning fix */ # pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ #endif #if XXH_NO_INLINE_HINTS /* disable inlining hints */ # if defined(__GNUC__) || defined(__clang__) # define XXH_FORCE_INLINE static __attribute__((unused)) # else # define XXH_FORCE_INLINE static # endif # define XXH_NO_INLINE static /* enable inlining hints */ #elif defined(__GNUC__) || defined(__clang__) # define XXH_FORCE_INLINE static __inline__ __attribute__((always_inline, unused)) # define XXH_NO_INLINE static __attribute__((noinline)) #elif defined(_MSC_VER) /* Visual Studio */ # define XXH_FORCE_INLINE static __forceinline # define XXH_NO_INLINE static __declspec(noinline) #elif defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L)) /* C99 */ # define XXH_FORCE_INLINE static inline # define XXH_NO_INLINE static #else # define XXH_FORCE_INLINE static # define XXH_NO_INLINE static #endif #if XXH3_INLINE_SECRET # define XXH3_WITH_SECRET_INLINE XXH_FORCE_INLINE #else # define XXH3_WITH_SECRET_INLINE XXH_NO_INLINE #endif /* ************************************* * Debug ***************************************/ /*! * @ingroup tuning * @def XXH_DEBUGLEVEL * @brief Sets the debugging level. * * XXH_DEBUGLEVEL is expected to be defined externally, typically via the * compiler's command line options. The value must be a number. */ #ifndef XXH_DEBUGLEVEL # ifdef DEBUGLEVEL /* backwards compat */ # define XXH_DEBUGLEVEL DEBUGLEVEL # else # define XXH_DEBUGLEVEL 0 # endif #endif #if (XXH_DEBUGLEVEL>=1) # include /* note: can still be disabled with NDEBUG */ # define XXH_ASSERT(c) assert(c) #else # if defined(__INTEL_COMPILER) # define XXH_ASSERT(c) XXH_ASSUME((unsigned char) (c)) # else # define XXH_ASSERT(c) XXH_ASSUME(c) # endif #endif /* note: use after variable declarations */ #ifndef XXH_STATIC_ASSERT # if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L) /* C11 */ # define XXH_STATIC_ASSERT_WITH_MESSAGE(c,m) do { _Static_assert((c),m); } while(0) # elif defined(__cplusplus) && (__cplusplus >= 201103L) /* C++11 */ # define XXH_STATIC_ASSERT_WITH_MESSAGE(c,m) do { static_assert((c),m); } while(0) # else # define XXH_STATIC_ASSERT_WITH_MESSAGE(c,m) do { struct xxh_sa { char x[(c) ? 1 : -1]; }; } while(0) # endif # define XXH_STATIC_ASSERT(c) XXH_STATIC_ASSERT_WITH_MESSAGE((c),#c) #endif /*! * @internal * @def XXH_COMPILER_GUARD(var) * @brief Used to prevent unwanted optimizations for @p var. * * It uses an empty GCC inline assembly statement with a register constraint * which forces @p var into a general purpose register (eg eax, ebx, ecx * on x86) and marks it as modified. * * This is used in a few places to avoid unwanted autovectorization (e.g. * XXH32_round()). All vectorization we want is explicit via intrinsics, * and _usually_ isn't wanted elsewhere. * * We also use it to prevent unwanted constant folding for AArch64 in * XXH3_initCustomSecret_scalar(). */ #if defined(__GNUC__) || defined(__clang__) # define XXH_COMPILER_GUARD(var) __asm__("" : "+r" (var)) #else # define XXH_COMPILER_GUARD(var) ((void)0) #endif /* Specifically for NEON vectors which use the "w" constraint, on * Clang. */ #if defined(__clang__) && defined(__ARM_ARCH) && !defined(__wasm__) # define XXH_COMPILER_GUARD_CLANG_NEON(var) __asm__("" : "+w" (var)) #else # define XXH_COMPILER_GUARD_CLANG_NEON(var) ((void)0) #endif /* ************************************* * Basic Types ***************************************/ #if !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) # include typedef uint8_t xxh_u8; #else typedef unsigned char xxh_u8; #endif typedef XXH32_hash_t xxh_u32; #ifdef XXH_OLD_NAMES # warning "XXH_OLD_NAMES is planned to be removed starting v0.9. If the program depends on it, consider moving away from it by employing newer type names directly" # define BYTE xxh_u8 # define U8 xxh_u8 # define U32 xxh_u32 #endif /* *** Memory access *** */ /*! * @internal * @fn xxh_u32 XXH_read32(const void* ptr) * @brief Reads an unaligned 32-bit integer from @p ptr in native endianness. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * * @param ptr The pointer to read from. * @return The 32-bit native endian integer from the bytes at @p ptr. */ /*! * @internal * @fn xxh_u32 XXH_readLE32(const void* ptr) * @brief Reads an unaligned 32-bit little endian integer from @p ptr. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * * @param ptr The pointer to read from. * @return The 32-bit little endian integer from the bytes at @p ptr. */ /*! * @internal * @fn xxh_u32 XXH_readBE32(const void* ptr) * @brief Reads an unaligned 32-bit big endian integer from @p ptr. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * * @param ptr The pointer to read from. * @return The 32-bit big endian integer from the bytes at @p ptr. */ /*! * @internal * @fn xxh_u32 XXH_readLE32_align(const void* ptr, XXH_alignment align) * @brief Like @ref XXH_readLE32(), but has an option for aligned reads. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * Note that when @ref XXH_FORCE_ALIGN_CHECK == 0, the @p align parameter is * always @ref XXH_alignment::XXH_unaligned. * * @param ptr The pointer to read from. * @param align Whether @p ptr is aligned. * @pre * If @p align == @ref XXH_alignment::XXH_aligned, @p ptr must be 4 byte * aligned. * @return The 32-bit little endian integer from the bytes at @p ptr. */ #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) /* * Manual byteshift. Best for old compilers which don't inline memcpy. * We actually directly use XXH_readLE32 and XXH_readBE32. */ #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2)) /* * Force direct memory access. Only works on CPU which support unaligned memory * access in hardware. */ static xxh_u32 XXH_read32(const void* memPtr) { return *(const xxh_u32*) memPtr; } #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1)) /* * __attribute__((aligned(1))) is supported by gcc and clang. Originally the * documentation claimed that it only increased the alignment, but actually it * can decrease it on gcc, clang, and icc: * https://gcc.gnu.org/bugzilla/show_bug.cgi?id=69502, * https://gcc.godbolt.org/z/xYez1j67Y. */ #ifdef XXH_OLD_NAMES typedef union { xxh_u32 u32; } __attribute__((packed)) unalign; #endif static xxh_u32 XXH_read32(const void* ptr) { typedef __attribute__((aligned(1))) xxh_u32 xxh_unalign32; return *((const xxh_unalign32*)ptr); } #else /* * Portable and safe solution. Generally efficient. * see: https://fastcompression.blogspot.com/2015/08/accessing-unaligned-memory.html */ static xxh_u32 XXH_read32(const void* memPtr) { xxh_u32 val; XXH_memcpy(&val, memPtr, sizeof(val)); return val; } #endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */ /* *** Endianness *** */ /*! * @ingroup tuning * @def XXH_CPU_LITTLE_ENDIAN * @brief Whether the target is little endian. * * Defined to 1 if the target is little endian, or 0 if it is big endian. * It can be defined externally, for example on the compiler command line. * * If it is not defined, * a runtime check (which is usually constant folded) is used instead. * * @note * This is not necessarily defined to an integer constant. * * @see XXH_isLittleEndian() for the runtime check. */ #ifndef XXH_CPU_LITTLE_ENDIAN /* * Try to detect endianness automatically, to avoid the nonstandard behavior * in `XXH_isLittleEndian()` */ # if defined(_WIN32) /* Windows is always little endian */ \ || defined(__LITTLE_ENDIAN__) \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) # define XXH_CPU_LITTLE_ENDIAN 1 # elif defined(__BIG_ENDIAN__) \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) # define XXH_CPU_LITTLE_ENDIAN 0 # else /*! * @internal * @brief Runtime check for @ref XXH_CPU_LITTLE_ENDIAN. * * Most compilers will constant fold this. */ static int XXH_isLittleEndian(void) { /* * Portable and well-defined behavior. * Don't use static: it is detrimental to performance. */ const union { xxh_u32 u; xxh_u8 c[4]; } one = { 1 }; return one.c[0]; } # define XXH_CPU_LITTLE_ENDIAN XXH_isLittleEndian() # endif #endif /* **************************************** * Compiler-specific Functions and Macros ******************************************/ #define XXH_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) #ifdef __has_builtin # define XXH_HAS_BUILTIN(x) __has_builtin(x) #else # define XXH_HAS_BUILTIN(x) 0 #endif /* * C23 and future versions have standard "unreachable()". * Once it has been implemented reliably we can add it as an * additional case: * * ``` * #if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= XXH_C23_VN) * # include * # ifdef unreachable * # define XXH_UNREACHABLE() unreachable() * # endif * #endif * ``` * * Note C++23 also has std::unreachable() which can be detected * as follows: * ``` * #if defined(__cpp_lib_unreachable) && (__cpp_lib_unreachable >= 202202L) * # include * # define XXH_UNREACHABLE() std::unreachable() * #endif * ``` * NB: `__cpp_lib_unreachable` is defined in the `` header. * We don't use that as including `` in `extern "C"` blocks * doesn't work on GCC12 */ #if XXH_HAS_BUILTIN(__builtin_unreachable) # define XXH_UNREACHABLE() __builtin_unreachable() #elif defined(_MSC_VER) # define XXH_UNREACHABLE() __assume(0) #else # define XXH_UNREACHABLE() #endif #if XXH_HAS_BUILTIN(__builtin_assume) # define XXH_ASSUME(c) __builtin_assume(c) #else # define XXH_ASSUME(c) if (!(c)) { XXH_UNREACHABLE(); } #endif /*! * @internal * @def XXH_rotl32(x,r) * @brief 32-bit rotate left. * * @param x The 32-bit integer to be rotated. * @param r The number of bits to rotate. * @pre * @p r > 0 && @p r < 32 * @note * @p x and @p r may be evaluated multiple times. * @return The rotated result. */ #if !defined(NO_CLANG_BUILTIN) && XXH_HAS_BUILTIN(__builtin_rotateleft32) \ && XXH_HAS_BUILTIN(__builtin_rotateleft64) # define XXH_rotl32 __builtin_rotateleft32 # define XXH_rotl64 __builtin_rotateleft64 /* Note: although _rotl exists for minGW (GCC under windows), performance seems poor */ #elif defined(_MSC_VER) # define XXH_rotl32(x,r) _rotl(x,r) # define XXH_rotl64(x,r) _rotl64(x,r) #else # define XXH_rotl32(x,r) (((x) << (r)) | ((x) >> (32 - (r)))) # define XXH_rotl64(x,r) (((x) << (r)) | ((x) >> (64 - (r)))) #endif /*! * @internal * @fn xxh_u32 XXH_swap32(xxh_u32 x) * @brief A 32-bit byteswap. * * @param x The 32-bit integer to byteswap. * @return @p x, byteswapped. */ #if defined(_MSC_VER) /* Visual Studio */ # define XXH_swap32 _byteswap_ulong #elif XXH_GCC_VERSION >= 403 # define XXH_swap32 __builtin_bswap32 #else static xxh_u32 XXH_swap32 (xxh_u32 x) { return ((x << 24) & 0xff000000 ) | ((x << 8) & 0x00ff0000 ) | ((x >> 8) & 0x0000ff00 ) | ((x >> 24) & 0x000000ff ); } #endif /* *************************** * Memory reads *****************************/ /*! * @internal * @brief Enum to indicate whether a pointer is aligned. */ typedef enum { XXH_aligned, /*!< Aligned */ XXH_unaligned /*!< Possibly unaligned */ } XXH_alignment; /* * XXH_FORCE_MEMORY_ACCESS==3 is an endian-independent byteshift load. * * This is ideal for older compilers which don't inline memcpy. */ #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) XXH_FORCE_INLINE xxh_u32 XXH_readLE32(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[0] | ((xxh_u32)bytePtr[1] << 8) | ((xxh_u32)bytePtr[2] << 16) | ((xxh_u32)bytePtr[3] << 24); } XXH_FORCE_INLINE xxh_u32 XXH_readBE32(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[3] | ((xxh_u32)bytePtr[2] << 8) | ((xxh_u32)bytePtr[1] << 16) | ((xxh_u32)bytePtr[0] << 24); } #else XXH_FORCE_INLINE xxh_u32 XXH_readLE32(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr)); } static xxh_u32 XXH_readBE32(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_swap32(XXH_read32(ptr)) : XXH_read32(ptr); } #endif XXH_FORCE_INLINE xxh_u32 XXH_readLE32_align(const void* ptr, XXH_alignment align) { if (align==XXH_unaligned) { return XXH_readLE32(ptr); } else { return XXH_CPU_LITTLE_ENDIAN ? *(const xxh_u32*)ptr : XXH_swap32(*(const xxh_u32*)ptr); } } /* ************************************* * Misc ***************************************/ /*! @ingroup public */ XXH_PUBLIC_API unsigned XXH_versionNumber (void) { return XXH_VERSION_NUMBER; } /* ******************************************************************* * 32-bit hash functions *********************************************************************/ /*! * @} * @defgroup XXH32_impl XXH32 implementation * @ingroup impl * * Details on the XXH32 implementation. * @{ */ /* #define instead of static const, to be used as initializers */ #define XXH_PRIME32_1 0x9E3779B1U /*!< 0b10011110001101110111100110110001 */ #define XXH_PRIME32_2 0x85EBCA77U /*!< 0b10000101111010111100101001110111 */ #define XXH_PRIME32_3 0xC2B2AE3DU /*!< 0b11000010101100101010111000111101 */ #define XXH_PRIME32_4 0x27D4EB2FU /*!< 0b00100111110101001110101100101111 */ #define XXH_PRIME32_5 0x165667B1U /*!< 0b00010110010101100110011110110001 */ #ifdef XXH_OLD_NAMES # define PRIME32_1 XXH_PRIME32_1 # define PRIME32_2 XXH_PRIME32_2 # define PRIME32_3 XXH_PRIME32_3 # define PRIME32_4 XXH_PRIME32_4 # define PRIME32_5 XXH_PRIME32_5 #endif /*! * @internal * @brief Normal stripe processing routine. * * This shuffles the bits so that any bit from @p input impacts several bits in * @p acc. * * @param acc The accumulator lane. * @param input The stripe of input to mix. * @return The mixed accumulator lane. */ static xxh_u32 XXH32_round(xxh_u32 acc, xxh_u32 input) { acc += input * XXH_PRIME32_2; acc = XXH_rotl32(acc, 13); acc *= XXH_PRIME32_1; #if (defined(__SSE4_1__) || defined(__aarch64__) || defined(__wasm_simd128__)) && !defined(XXH_ENABLE_AUTOVECTORIZE) /* * UGLY HACK: * A compiler fence is the only thing that prevents GCC and Clang from * autovectorizing the XXH32 loop (pragmas and attributes don't work for some * reason) without globally disabling SSE4.1. * * The reason we want to avoid vectorization is because despite working on * 4 integers at a time, there are multiple factors slowing XXH32 down on * SSE4: * - There's a ridiculous amount of lag from pmulld (10 cycles of latency on * newer chips!) making it slightly slower to multiply four integers at * once compared to four integers independently. Even when pmulld was * fastest, Sandy/Ivy Bridge, it is still not worth it to go into SSE * just to multiply unless doing a long operation. * * - Four instructions are required to rotate, * movqda tmp, v // not required with VEX encoding * pslld tmp, 13 // tmp <<= 13 * psrld v, 19 // x >>= 19 * por v, tmp // x |= tmp * compared to one for scalar: * roll v, 13 // reliably fast across the board * shldl v, v, 13 // Sandy Bridge and later prefer this for some reason * * - Instruction level parallelism is actually more beneficial here because * the SIMD actually serializes this operation: While v1 is rotating, v2 * can load data, while v3 can multiply. SSE forces them to operate * together. * * This is also enabled on AArch64, as Clang is *very aggressive* in vectorizing * the loop. NEON is only faster on the A53, and with the newer cores, it is less * than half the speed. * * Additionally, this is used on WASM SIMD128 because it JITs to the same * SIMD instructions and has the same issue. */ XXH_COMPILER_GUARD(acc); #endif return acc; } /*! * @internal * @brief Mixes all bits to finalize the hash. * * The final mix ensures that all input bits have a chance to impact any bit in * the output digest, resulting in an unbiased distribution. * * @param hash The hash to avalanche. * @return The avalanched hash. */ static xxh_u32 XXH32_avalanche(xxh_u32 hash) { hash ^= hash >> 15; hash *= XXH_PRIME32_2; hash ^= hash >> 13; hash *= XXH_PRIME32_3; hash ^= hash >> 16; return hash; } #define XXH_get32bits(p) XXH_readLE32_align(p, align) /*! * @internal * @brief Processes the last 0-15 bytes of @p ptr. * * There may be up to 15 bytes remaining to consume from the input. * This final stage will digest them to ensure that all input bytes are present * in the final mix. * * @param hash The hash to finalize. * @param ptr The pointer to the remaining input. * @param len The remaining length, modulo 16. * @param align Whether @p ptr is aligned. * @return The finalized hash. * @see XXH64_finalize(). */ static XXH_PUREF xxh_u32 XXH32_finalize(xxh_u32 hash, const xxh_u8* ptr, size_t len, XXH_alignment align) { #define XXH_PROCESS1 do { \ hash += (*ptr++) * XXH_PRIME32_5; \ hash = XXH_rotl32(hash, 11) * XXH_PRIME32_1; \ } while (0) #define XXH_PROCESS4 do { \ hash += XXH_get32bits(ptr) * XXH_PRIME32_3; \ ptr += 4; \ hash = XXH_rotl32(hash, 17) * XXH_PRIME32_4; \ } while (0) if (ptr==NULL) XXH_ASSERT(len == 0); /* Compact rerolled version; generally faster */ if (!XXH32_ENDJMP) { len &= 15; while (len >= 4) { XXH_PROCESS4; len -= 4; } while (len > 0) { XXH_PROCESS1; --len; } return XXH32_avalanche(hash); } else { switch(len&15) /* or switch(bEnd - p) */ { case 12: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 8: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 4: XXH_PROCESS4; return XXH32_avalanche(hash); case 13: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 9: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 5: XXH_PROCESS4; XXH_PROCESS1; return XXH32_avalanche(hash); case 14: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 10: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 6: XXH_PROCESS4; XXH_PROCESS1; XXH_PROCESS1; return XXH32_avalanche(hash); case 15: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 11: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 7: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 3: XXH_PROCESS1; XXH_FALLTHROUGH; /* fallthrough */ case 2: XXH_PROCESS1; XXH_FALLTHROUGH; /* fallthrough */ case 1: XXH_PROCESS1; XXH_FALLTHROUGH; /* fallthrough */ case 0: return XXH32_avalanche(hash); } XXH_ASSERT(0); return hash; /* reaching this point is deemed impossible */ } } #ifdef XXH_OLD_NAMES # define PROCESS1 XXH_PROCESS1 # define PROCESS4 XXH_PROCESS4 #else # undef XXH_PROCESS1 # undef XXH_PROCESS4 #endif /*! * @internal * @brief The implementation for @ref XXH32(). * * @param input , len , seed Directly passed from @ref XXH32(). * @param align Whether @p input is aligned. * @return The calculated hash. */ XXH_FORCE_INLINE XXH_PUREF xxh_u32 XXH32_endian_align(const xxh_u8* input, size_t len, xxh_u32 seed, XXH_alignment align) { xxh_u32 h32; if (input==NULL) XXH_ASSERT(len == 0); if (len>=16) { const xxh_u8* const bEnd = input + len; const xxh_u8* const limit = bEnd - 15; xxh_u32 v1 = seed + XXH_PRIME32_1 + XXH_PRIME32_2; xxh_u32 v2 = seed + XXH_PRIME32_2; xxh_u32 v3 = seed + 0; xxh_u32 v4 = seed - XXH_PRIME32_1; do { v1 = XXH32_round(v1, XXH_get32bits(input)); input += 4; v2 = XXH32_round(v2, XXH_get32bits(input)); input += 4; v3 = XXH32_round(v3, XXH_get32bits(input)); input += 4; v4 = XXH32_round(v4, XXH_get32bits(input)); input += 4; } while (input < limit); h32 = XXH_rotl32(v1, 1) + XXH_rotl32(v2, 7) + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18); } else { h32 = seed + XXH_PRIME32_5; } h32 += (xxh_u32)len; return XXH32_finalize(h32, input, len&15, align); } /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t len, XXH32_hash_t seed) { #if !defined(XXH_NO_STREAM) && XXH_SIZE_OPT >= 2 /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ XXH32_state_t state; XXH32_reset(&state, seed); XXH32_update(&state, (const xxh_u8*)input, len); return XXH32_digest(&state); #else if (XXH_FORCE_ALIGN_CHECK) { if ((((size_t)input) & 3) == 0) { /* Input is 4-bytes aligned, leverage the speed benefit */ return XXH32_endian_align((const xxh_u8*)input, len, seed, XXH_aligned); } } return XXH32_endian_align((const xxh_u8*)input, len, seed, XXH_unaligned); #endif } /******* Hash streaming *******/ #ifndef XXH_NO_STREAM /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void) { return (XXH32_state_t*)XXH_malloc(sizeof(XXH32_state_t)); } /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr) { XXH_free(statePtr); return XXH_OK; } /*! @ingroup XXH32_family */ XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dstState, const XXH32_state_t* srcState) { XXH_memcpy(dstState, srcState, sizeof(*dstState)); } /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH_errorcode XXH32_reset(XXH32_state_t* statePtr, XXH32_hash_t seed) { XXH_ASSERT(statePtr != NULL); memset(statePtr, 0, sizeof(*statePtr)); statePtr->v[0] = seed + XXH_PRIME32_1 + XXH_PRIME32_2; statePtr->v[1] = seed + XXH_PRIME32_2; statePtr->v[2] = seed + 0; statePtr->v[3] = seed - XXH_PRIME32_1; return XXH_OK; } /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH_errorcode XXH32_update(XXH32_state_t* state, const void* input, size_t len) { if (input==NULL) { XXH_ASSERT(len == 0); return XXH_OK; } { const xxh_u8* p = (const xxh_u8*)input; const xxh_u8* const bEnd = p + len; state->total_len_32 += (XXH32_hash_t)len; state->large_len |= (XXH32_hash_t)((len>=16) | (state->total_len_32>=16)); if (state->memsize + len < 16) { /* fill in tmp buffer */ XXH_memcpy((xxh_u8*)(state->mem32) + state->memsize, input, len); state->memsize += (XXH32_hash_t)len; return XXH_OK; } if (state->memsize) { /* some data left from previous update */ XXH_memcpy((xxh_u8*)(state->mem32) + state->memsize, input, 16-state->memsize); { const xxh_u32* p32 = state->mem32; state->v[0] = XXH32_round(state->v[0], XXH_readLE32(p32)); p32++; state->v[1] = XXH32_round(state->v[1], XXH_readLE32(p32)); p32++; state->v[2] = XXH32_round(state->v[2], XXH_readLE32(p32)); p32++; state->v[3] = XXH32_round(state->v[3], XXH_readLE32(p32)); } p += 16-state->memsize; state->memsize = 0; } if (p <= bEnd-16) { const xxh_u8* const limit = bEnd - 16; do { state->v[0] = XXH32_round(state->v[0], XXH_readLE32(p)); p+=4; state->v[1] = XXH32_round(state->v[1], XXH_readLE32(p)); p+=4; state->v[2] = XXH32_round(state->v[2], XXH_readLE32(p)); p+=4; state->v[3] = XXH32_round(state->v[3], XXH_readLE32(p)); p+=4; } while (p<=limit); } if (p < bEnd) { XXH_memcpy(state->mem32, p, (size_t)(bEnd-p)); state->memsize = (unsigned)(bEnd-p); } } return XXH_OK; } /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH32_hash_t XXH32_digest(const XXH32_state_t* state) { xxh_u32 h32; if (state->large_len) { h32 = XXH_rotl32(state->v[0], 1) + XXH_rotl32(state->v[1], 7) + XXH_rotl32(state->v[2], 12) + XXH_rotl32(state->v[3], 18); } else { h32 = state->v[2] /* == seed */ + XXH_PRIME32_5; } h32 += state->total_len_32; return XXH32_finalize(h32, (const xxh_u8*)state->mem32, state->memsize, XXH_aligned); } #endif /* !XXH_NO_STREAM */ /******* Canonical representation *******/ /*! * @ingroup XXH32_family * The default return values from XXH functions are unsigned 32 and 64 bit * integers. * * The canonical representation uses big endian convention, the same convention * as human-readable numbers (large digits first). * * This way, hash values can be written into a file or buffer, remaining * comparable across different systems. * * The following functions allow transformation of hash values to and from their * canonical format. */ XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash) { XXH_STATIC_ASSERT(sizeof(XXH32_canonical_t) == sizeof(XXH32_hash_t)); if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap32(hash); XXH_memcpy(dst, &hash, sizeof(*dst)); } /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src) { return XXH_readBE32(src); } #ifndef XXH_NO_LONG_LONG /* ******************************************************************* * 64-bit hash functions *********************************************************************/ /*! * @} * @ingroup impl * @{ */ /******* Memory access *******/ typedef XXH64_hash_t xxh_u64; #ifdef XXH_OLD_NAMES # define U64 xxh_u64 #endif #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) /* * Manual byteshift. Best for old compilers which don't inline memcpy. * We actually directly use XXH_readLE64 and XXH_readBE64. */ #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2)) /* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */ static xxh_u64 XXH_read64(const void* memPtr) { return *(const xxh_u64*) memPtr; } #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1)) /* * __attribute__((aligned(1))) is supported by gcc and clang. Originally the * documentation claimed that it only increased the alignment, but actually it * can decrease it on gcc, clang, and icc: * https://gcc.gnu.org/bugzilla/show_bug.cgi?id=69502, * https://gcc.godbolt.org/z/xYez1j67Y. */ #ifdef XXH_OLD_NAMES typedef union { xxh_u32 u32; xxh_u64 u64; } __attribute__((packed)) unalign64; #endif static xxh_u64 XXH_read64(const void* ptr) { typedef __attribute__((aligned(1))) xxh_u64 xxh_unalign64; return *((const xxh_unalign64*)ptr); } #else /* * Portable and safe solution. Generally efficient. * see: https://fastcompression.blogspot.com/2015/08/accessing-unaligned-memory.html */ static xxh_u64 XXH_read64(const void* memPtr) { xxh_u64 val; XXH_memcpy(&val, memPtr, sizeof(val)); return val; } #endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */ #if defined(_MSC_VER) /* Visual Studio */ # define XXH_swap64 _byteswap_uint64 #elif XXH_GCC_VERSION >= 403 # define XXH_swap64 __builtin_bswap64 #else static xxh_u64 XXH_swap64(xxh_u64 x) { return ((x << 56) & 0xff00000000000000ULL) | ((x << 40) & 0x00ff000000000000ULL) | ((x << 24) & 0x0000ff0000000000ULL) | ((x << 8) & 0x000000ff00000000ULL) | ((x >> 8) & 0x00000000ff000000ULL) | ((x >> 24) & 0x0000000000ff0000ULL) | ((x >> 40) & 0x000000000000ff00ULL) | ((x >> 56) & 0x00000000000000ffULL); } #endif /* XXH_FORCE_MEMORY_ACCESS==3 is an endian-independent byteshift load. */ #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) XXH_FORCE_INLINE xxh_u64 XXH_readLE64(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[0] | ((xxh_u64)bytePtr[1] << 8) | ((xxh_u64)bytePtr[2] << 16) | ((xxh_u64)bytePtr[3] << 24) | ((xxh_u64)bytePtr[4] << 32) | ((xxh_u64)bytePtr[5] << 40) | ((xxh_u64)bytePtr[6] << 48) | ((xxh_u64)bytePtr[7] << 56); } XXH_FORCE_INLINE xxh_u64 XXH_readBE64(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[7] | ((xxh_u64)bytePtr[6] << 8) | ((xxh_u64)bytePtr[5] << 16) | ((xxh_u64)bytePtr[4] << 24) | ((xxh_u64)bytePtr[3] << 32) | ((xxh_u64)bytePtr[2] << 40) | ((xxh_u64)bytePtr[1] << 48) | ((xxh_u64)bytePtr[0] << 56); } #else XXH_FORCE_INLINE xxh_u64 XXH_readLE64(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr)); } static xxh_u64 XXH_readBE64(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_swap64(XXH_read64(ptr)) : XXH_read64(ptr); } #endif XXH_FORCE_INLINE xxh_u64 XXH_readLE64_align(const void* ptr, XXH_alignment align) { if (align==XXH_unaligned) return XXH_readLE64(ptr); else return XXH_CPU_LITTLE_ENDIAN ? *(const xxh_u64*)ptr : XXH_swap64(*(const xxh_u64*)ptr); } /******* xxh64 *******/ /*! * @} * @defgroup XXH64_impl XXH64 implementation * @ingroup impl * * Details on the XXH64 implementation. * @{ */ /* #define rather that static const, to be used as initializers */ #define XXH_PRIME64_1 0x9E3779B185EBCA87ULL /*!< 0b1001111000110111011110011011000110000101111010111100101010000111 */ #define XXH_PRIME64_2 0xC2B2AE3D27D4EB4FULL /*!< 0b1100001010110010101011100011110100100111110101001110101101001111 */ #define XXH_PRIME64_3 0x165667B19E3779F9ULL /*!< 0b0001011001010110011001111011000110011110001101110111100111111001 */ #define XXH_PRIME64_4 0x85EBCA77C2B2AE63ULL /*!< 0b1000010111101011110010100111011111000010101100101010111001100011 */ #define XXH_PRIME64_5 0x27D4EB2F165667C5ULL /*!< 0b0010011111010100111010110010111100010110010101100110011111000101 */ #ifdef XXH_OLD_NAMES # define PRIME64_1 XXH_PRIME64_1 # define PRIME64_2 XXH_PRIME64_2 # define PRIME64_3 XXH_PRIME64_3 # define PRIME64_4 XXH_PRIME64_4 # define PRIME64_5 XXH_PRIME64_5 #endif /*! @copydoc XXH32_round */ static xxh_u64 XXH64_round(xxh_u64 acc, xxh_u64 input) { acc += input * XXH_PRIME64_2; acc = XXH_rotl64(acc, 31); acc *= XXH_PRIME64_1; return acc; } static xxh_u64 XXH64_mergeRound(xxh_u64 acc, xxh_u64 val) { val = XXH64_round(0, val); acc ^= val; acc = acc * XXH_PRIME64_1 + XXH_PRIME64_4; return acc; } /*! @copydoc XXH32_avalanche */ static xxh_u64 XXH64_avalanche(xxh_u64 hash) { hash ^= hash >> 33; hash *= XXH_PRIME64_2; hash ^= hash >> 29; hash *= XXH_PRIME64_3; hash ^= hash >> 32; return hash; } #define XXH_get64bits(p) XXH_readLE64_align(p, align) /*! * @internal * @brief Processes the last 0-31 bytes of @p ptr. * * There may be up to 31 bytes remaining to consume from the input. * This final stage will digest them to ensure that all input bytes are present * in the final mix. * * @param hash The hash to finalize. * @param ptr The pointer to the remaining input. * @param len The remaining length, modulo 32. * @param align Whether @p ptr is aligned. * @return The finalized hash * @see XXH32_finalize(). */ static XXH_PUREF xxh_u64 XXH64_finalize(xxh_u64 hash, const xxh_u8* ptr, size_t len, XXH_alignment align) { if (ptr==NULL) XXH_ASSERT(len == 0); len &= 31; while (len >= 8) { xxh_u64 const k1 = XXH64_round(0, XXH_get64bits(ptr)); ptr += 8; hash ^= k1; hash = XXH_rotl64(hash,27) * XXH_PRIME64_1 + XXH_PRIME64_4; len -= 8; } if (len >= 4) { hash ^= (xxh_u64)(XXH_get32bits(ptr)) * XXH_PRIME64_1; ptr += 4; hash = XXH_rotl64(hash, 23) * XXH_PRIME64_2 + XXH_PRIME64_3; len -= 4; } while (len > 0) { hash ^= (*ptr++) * XXH_PRIME64_5; hash = XXH_rotl64(hash, 11) * XXH_PRIME64_1; --len; } return XXH64_avalanche(hash); } #ifdef XXH_OLD_NAMES # define PROCESS1_64 XXH_PROCESS1_64 # define PROCESS4_64 XXH_PROCESS4_64 # define PROCESS8_64 XXH_PROCESS8_64 #else # undef XXH_PROCESS1_64 # undef XXH_PROCESS4_64 # undef XXH_PROCESS8_64 #endif /*! * @internal * @brief The implementation for @ref XXH64(). * * @param input , len , seed Directly passed from @ref XXH64(). * @param align Whether @p input is aligned. * @return The calculated hash. */ XXH_FORCE_INLINE XXH_PUREF xxh_u64 XXH64_endian_align(const xxh_u8* input, size_t len, xxh_u64 seed, XXH_alignment align) { xxh_u64 h64; if (input==NULL) XXH_ASSERT(len == 0); if (len>=32) { const xxh_u8* const bEnd = input + len; const xxh_u8* const limit = bEnd - 31; xxh_u64 v1 = seed + XXH_PRIME64_1 + XXH_PRIME64_2; xxh_u64 v2 = seed + XXH_PRIME64_2; xxh_u64 v3 = seed + 0; xxh_u64 v4 = seed - XXH_PRIME64_1; do { v1 = XXH64_round(v1, XXH_get64bits(input)); input+=8; v2 = XXH64_round(v2, XXH_get64bits(input)); input+=8; v3 = XXH64_round(v3, XXH_get64bits(input)); input+=8; v4 = XXH64_round(v4, XXH_get64bits(input)); input+=8; } while (input= 2 /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ XXH64_state_t state; XXH64_reset(&state, seed); XXH64_update(&state, (const xxh_u8*)input, len); return XXH64_digest(&state); #else if (XXH_FORCE_ALIGN_CHECK) { if ((((size_t)input) & 7)==0) { /* Input is aligned, let's leverage the speed advantage */ return XXH64_endian_align((const xxh_u8*)input, len, seed, XXH_aligned); } } return XXH64_endian_align((const xxh_u8*)input, len, seed, XXH_unaligned); #endif } /******* Hash Streaming *******/ #ifndef XXH_NO_STREAM /*! @ingroup XXH64_family*/ XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void) { return (XXH64_state_t*)XXH_malloc(sizeof(XXH64_state_t)); } /*! @ingroup XXH64_family */ XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr) { XXH_free(statePtr); return XXH_OK; } /*! @ingroup XXH64_family */ XXH_PUBLIC_API void XXH64_copyState(XXH_NOESCAPE XXH64_state_t* dstState, const XXH64_state_t* srcState) { XXH_memcpy(dstState, srcState, sizeof(*dstState)); } /*! @ingroup XXH64_family */ XXH_PUBLIC_API XXH_errorcode XXH64_reset(XXH_NOESCAPE XXH64_state_t* statePtr, XXH64_hash_t seed) { XXH_ASSERT(statePtr != NULL); memset(statePtr, 0, sizeof(*statePtr)); statePtr->v[0] = seed + XXH_PRIME64_1 + XXH_PRIME64_2; statePtr->v[1] = seed + XXH_PRIME64_2; statePtr->v[2] = seed + 0; statePtr->v[3] = seed - XXH_PRIME64_1; return XXH_OK; } /*! @ingroup XXH64_family */ XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH_NOESCAPE XXH64_state_t* state, XXH_NOESCAPE const void* input, size_t len) { if (input==NULL) { XXH_ASSERT(len == 0); return XXH_OK; } { const xxh_u8* p = (const xxh_u8*)input; const xxh_u8* const bEnd = p + len; state->total_len += len; if (state->memsize + len < 32) { /* fill in tmp buffer */ XXH_memcpy(((xxh_u8*)state->mem64) + state->memsize, input, len); state->memsize += (xxh_u32)len; return XXH_OK; } if (state->memsize) { /* tmp buffer is full */ XXH_memcpy(((xxh_u8*)state->mem64) + state->memsize, input, 32-state->memsize); state->v[0] = XXH64_round(state->v[0], XXH_readLE64(state->mem64+0)); state->v[1] = XXH64_round(state->v[1], XXH_readLE64(state->mem64+1)); state->v[2] = XXH64_round(state->v[2], XXH_readLE64(state->mem64+2)); state->v[3] = XXH64_round(state->v[3], XXH_readLE64(state->mem64+3)); p += 32 - state->memsize; state->memsize = 0; } if (p+32 <= bEnd) { const xxh_u8* const limit = bEnd - 32; do { state->v[0] = XXH64_round(state->v[0], XXH_readLE64(p)); p+=8; state->v[1] = XXH64_round(state->v[1], XXH_readLE64(p)); p+=8; state->v[2] = XXH64_round(state->v[2], XXH_readLE64(p)); p+=8; state->v[3] = XXH64_round(state->v[3], XXH_readLE64(p)); p+=8; } while (p<=limit); } if (p < bEnd) { XXH_memcpy(state->mem64, p, (size_t)(bEnd-p)); state->memsize = (unsigned)(bEnd-p); } } return XXH_OK; } /*! @ingroup XXH64_family */ XXH_PUBLIC_API XXH64_hash_t XXH64_digest(XXH_NOESCAPE const XXH64_state_t* state) { xxh_u64 h64; if (state->total_len >= 32) { h64 = XXH_rotl64(state->v[0], 1) + XXH_rotl64(state->v[1], 7) + XXH_rotl64(state->v[2], 12) + XXH_rotl64(state->v[3], 18); h64 = XXH64_mergeRound(h64, state->v[0]); h64 = XXH64_mergeRound(h64, state->v[1]); h64 = XXH64_mergeRound(h64, state->v[2]); h64 = XXH64_mergeRound(h64, state->v[3]); } else { h64 = state->v[2] /*seed*/ + XXH_PRIME64_5; } h64 += (xxh_u64) state->total_len; return XXH64_finalize(h64, (const xxh_u8*)state->mem64, (size_t)state->total_len, XXH_aligned); } #endif /* !XXH_NO_STREAM */ /******* Canonical representation *******/ /*! @ingroup XXH64_family */ XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH_NOESCAPE XXH64_canonical_t* dst, XXH64_hash_t hash) { XXH_STATIC_ASSERT(sizeof(XXH64_canonical_t) == sizeof(XXH64_hash_t)); if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap64(hash); XXH_memcpy(dst, &hash, sizeof(*dst)); } /*! @ingroup XXH64_family */ XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(XXH_NOESCAPE const XXH64_canonical_t* src) { return XXH_readBE64(src); } #ifndef XXH_NO_XXH3 /* ********************************************************************* * XXH3 * New generation hash designed for speed on small keys and vectorization ************************************************************************ */ /*! * @} * @defgroup XXH3_impl XXH3 implementation * @ingroup impl * @{ */ /* === Compiler specifics === */ #if ((defined(sun) || defined(__sun)) && __cplusplus) /* Solaris includes __STDC_VERSION__ with C++. Tested with GCC 5.5 */ # define XXH_RESTRICT /* disable */ #elif defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* >= C99 */ # define XXH_RESTRICT restrict #elif (defined (__GNUC__) && ((__GNUC__ > 3) || (__GNUC__ == 3 && __GNUC_MINOR__ >= 1))) \ || (defined (__clang__)) \ || (defined (_MSC_VER) && (_MSC_VER >= 1400)) \ || (defined (__INTEL_COMPILER) && (__INTEL_COMPILER >= 1300)) /* * There are a LOT more compilers that recognize __restrict but this * covers the major ones. */ # define XXH_RESTRICT __restrict #else # define XXH_RESTRICT /* disable */ #endif #if (defined(__GNUC__) && (__GNUC__ >= 3)) \ || (defined(__INTEL_COMPILER) && (__INTEL_COMPILER >= 800)) \ || defined(__clang__) # define XXH_likely(x) __builtin_expect(x, 1) # define XXH_unlikely(x) __builtin_expect(x, 0) #else # define XXH_likely(x) (x) # define XXH_unlikely(x) (x) #endif #ifndef XXH_HAS_INCLUDE # ifdef __has_include # define XXH_HAS_INCLUDE(x) __has_include(x) # else # define XXH_HAS_INCLUDE(x) 0 # endif #endif #if defined(__GNUC__) || defined(__clang__) # if defined(__ARM_FEATURE_SVE) # include # endif # if defined(__ARM_NEON__) || defined(__ARM_NEON) \ || (defined(_M_ARM) && _M_ARM >= 7) \ || defined(_M_ARM64) || defined(_M_ARM64EC) \ || (defined(__wasm_simd128__) && XXH_HAS_INCLUDE()) /* WASM SIMD128 via SIMDe */ # define inline __inline__ /* circumvent a clang bug */ # include # undef inline # elif defined(__AVX2__) # include # elif defined(__SSE2__) # include # endif #endif #if defined(_MSC_VER) # include #endif /* * One goal of XXH3 is to make it fast on both 32-bit and 64-bit, while * remaining a true 64-bit/128-bit hash function. * * This is done by prioritizing a subset of 64-bit operations that can be * emulated without too many steps on the average 32-bit machine. * * For example, these two lines seem similar, and run equally fast on 64-bit: * * xxh_u64 x; * x ^= (x >> 47); // good * x ^= (x >> 13); // bad * * However, to a 32-bit machine, there is a major difference. * * x ^= (x >> 47) looks like this: * * x.lo ^= (x.hi >> (47 - 32)); * * while x ^= (x >> 13) looks like this: * * // note: funnel shifts are not usually cheap. * x.lo ^= (x.lo >> 13) | (x.hi << (32 - 13)); * x.hi ^= (x.hi >> 13); * * The first one is significantly faster than the second, simply because the * shift is larger than 32. This means: * - All the bits we need are in the upper 32 bits, so we can ignore the lower * 32 bits in the shift. * - The shift result will always fit in the lower 32 bits, and therefore, * we can ignore the upper 32 bits in the xor. * * Thanks to this optimization, XXH3 only requires these features to be efficient: * * - Usable unaligned access * - A 32-bit or 64-bit ALU * - If 32-bit, a decent ADC instruction * - A 32 or 64-bit multiply with a 64-bit result * - For the 128-bit variant, a decent byteswap helps short inputs. * * The first two are already required by XXH32, and almost all 32-bit and 64-bit * platforms which can run XXH32 can run XXH3 efficiently. * * Thumb-1, the classic 16-bit only subset of ARM's instruction set, is one * notable exception. * * First of all, Thumb-1 lacks support for the UMULL instruction which * performs the important long multiply. This means numerous __aeabi_lmul * calls. * * Second of all, the 8 functional registers are just not enough. * Setup for __aeabi_lmul, byteshift loads, pointers, and all arithmetic need * Lo registers, and this shuffling results in thousands more MOVs than A32. * * A32 and T32 don't have this limitation. They can access all 14 registers, * do a 32->64 multiply with UMULL, and the flexible operand allowing free * shifts is helpful, too. * * Therefore, we do a quick sanity check. * * If compiling Thumb-1 for a target which supports ARM instructions, we will * emit a warning, as it is not a "sane" platform to compile for. * * Usually, if this happens, it is because of an accident and you probably need * to specify -march, as you likely meant to compile for a newer architecture. * * Credit: large sections of the vectorial and asm source code paths * have been contributed by @easyaspi314 */ #if defined(__thumb__) && !defined(__thumb2__) && defined(__ARM_ARCH_ISA_ARM) # warning "XXH3 is highly inefficient without ARM or Thumb-2." #endif /* ========================================== * Vectorization detection * ========================================== */ #ifdef XXH_DOXYGEN /*! * @ingroup tuning * @brief Overrides the vectorization implementation chosen for XXH3. * * Can be defined to 0 to disable SIMD or any of the values mentioned in * @ref XXH_VECTOR_TYPE. * * If this is not defined, it uses predefined macros to determine the best * implementation. */ # define XXH_VECTOR XXH_SCALAR /*! * @ingroup tuning * @brief Possible values for @ref XXH_VECTOR. * * Note that these are actually implemented as macros. * * If this is not defined, it is detected automatically. * internal macro XXH_X86DISPATCH overrides this. */ enum XXH_VECTOR_TYPE /* fake enum */ { XXH_SCALAR = 0, /*!< Portable scalar version */ XXH_SSE2 = 1, /*!< * SSE2 for Pentium 4, Opteron, all x86_64. * * @note SSE2 is also guaranteed on Windows 10, macOS, and * Android x86. */ XXH_AVX2 = 2, /*!< AVX2 for Haswell and Bulldozer */ XXH_AVX512 = 3, /*!< AVX512 for Skylake and Icelake */ XXH_NEON = 4, /*!< * NEON for most ARMv7-A, all AArch64, and WASM SIMD128 * via the SIMDeverywhere polyfill provided with the * Emscripten SDK. */ XXH_VSX = 5, /*!< VSX and ZVector for POWER8/z13 (64-bit) */ XXH_SVE = 6, /*!< SVE for some ARMv8-A and ARMv9-A */ }; /*! * @ingroup tuning * @brief Selects the minimum alignment for XXH3's accumulators. * * When using SIMD, this should match the alignment required for said vector * type, so, for example, 32 for AVX2. * * Default: Auto detected. */ # define XXH_ACC_ALIGN 8 #endif /* Actual definition */ #ifndef XXH_DOXYGEN # define XXH_SCALAR 0 # define XXH_SSE2 1 # define XXH_AVX2 2 # define XXH_AVX512 3 # define XXH_NEON 4 # define XXH_VSX 5 # define XXH_SVE 6 #endif #ifndef XXH_VECTOR /* can be defined on command line */ # if defined(__ARM_FEATURE_SVE) # define XXH_VECTOR XXH_SVE # elif ( \ defined(__ARM_NEON__) || defined(__ARM_NEON) /* gcc */ \ || defined(_M_ARM) || defined(_M_ARM64) || defined(_M_ARM64EC) /* msvc */ \ || (defined(__wasm_simd128__) && XXH_HAS_INCLUDE()) /* wasm simd128 via SIMDe */ \ ) && ( \ defined(_WIN32) || defined(__LITTLE_ENDIAN__) /* little endian only */ \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) \ ) # define XXH_VECTOR XXH_NEON # elif defined(__AVX512F__) # define XXH_VECTOR XXH_AVX512 # elif defined(__AVX2__) # define XXH_VECTOR XXH_AVX2 # elif defined(__SSE2__) || defined(_M_AMD64) || defined(_M_X64) || (defined(_M_IX86_FP) && (_M_IX86_FP == 2)) # define XXH_VECTOR XXH_SSE2 # elif (defined(__PPC64__) && defined(__POWER8_VECTOR__)) \ || (defined(__s390x__) && defined(__VEC__)) \ && defined(__GNUC__) /* TODO: IBM XL */ # define XXH_VECTOR XXH_VSX # else # define XXH_VECTOR XXH_SCALAR # endif #endif /* __ARM_FEATURE_SVE is only supported by GCC & Clang. */ #if (XXH_VECTOR == XXH_SVE) && !defined(__ARM_FEATURE_SVE) # ifdef _MSC_VER # pragma warning(once : 4606) # else # warning "__ARM_FEATURE_SVE isn't supported. Use SCALAR instead." # endif # undef XXH_VECTOR # define XXH_VECTOR XXH_SCALAR #endif /* * Controls the alignment of the accumulator, * for compatibility with aligned vector loads, which are usually faster. */ #ifndef XXH_ACC_ALIGN # if defined(XXH_X86DISPATCH) # define XXH_ACC_ALIGN 64 /* for compatibility with avx512 */ # elif XXH_VECTOR == XXH_SCALAR /* scalar */ # define XXH_ACC_ALIGN 8 # elif XXH_VECTOR == XXH_SSE2 /* sse2 */ # define XXH_ACC_ALIGN 16 # elif XXH_VECTOR == XXH_AVX2 /* avx2 */ # define XXH_ACC_ALIGN 32 # elif XXH_VECTOR == XXH_NEON /* neon */ # define XXH_ACC_ALIGN 16 # elif XXH_VECTOR == XXH_VSX /* vsx */ # define XXH_ACC_ALIGN 16 # elif XXH_VECTOR == XXH_AVX512 /* avx512 */ # define XXH_ACC_ALIGN 64 # elif XXH_VECTOR == XXH_SVE /* sve */ # define XXH_ACC_ALIGN 64 # endif #endif #if defined(XXH_X86DISPATCH) || XXH_VECTOR == XXH_SSE2 \ || XXH_VECTOR == XXH_AVX2 || XXH_VECTOR == XXH_AVX512 # define XXH_SEC_ALIGN XXH_ACC_ALIGN #elif XXH_VECTOR == XXH_SVE # define XXH_SEC_ALIGN XXH_ACC_ALIGN #else # define XXH_SEC_ALIGN 8 #endif #if defined(__GNUC__) || defined(__clang__) # define XXH_ALIASING __attribute__((may_alias)) #else # define XXH_ALIASING /* nothing */ #endif /* * UGLY HACK: * GCC usually generates the best code with -O3 for xxHash. * * However, when targeting AVX2, it is overzealous in its unrolling resulting * in code roughly 3/4 the speed of Clang. * * There are other issues, such as GCC splitting _mm256_loadu_si256 into * _mm_loadu_si128 + _mm256_inserti128_si256. This is an optimization which * only applies to Sandy and Ivy Bridge... which don't even support AVX2. * * That is why when compiling the AVX2 version, it is recommended to use either * -O2 -mavx2 -march=haswell * or * -O2 -mavx2 -mno-avx256-split-unaligned-load * for decent performance, or to use Clang instead. * * Fortunately, we can control the first one with a pragma that forces GCC into * -O2, but the other one we can't control without "failed to inline always * inline function due to target mismatch" warnings. */ #if XXH_VECTOR == XXH_AVX2 /* AVX2 */ \ && defined(__GNUC__) && !defined(__clang__) /* GCC, not Clang */ \ && defined(__OPTIMIZE__) && XXH_SIZE_OPT <= 0 /* respect -O0 and -Os */ # pragma GCC push_options # pragma GCC optimize("-O2") #endif #if XXH_VECTOR == XXH_NEON /* * UGLY HACK: While AArch64 GCC on Linux does not seem to care, on macOS, GCC -O3 * optimizes out the entire hashLong loop because of the aliasing violation. * * However, GCC is also inefficient at load-store optimization with vld1q/vst1q, * so the only option is to mark it as aliasing. */ typedef uint64x2_t xxh_aliasing_uint64x2_t XXH_ALIASING; /*! * @internal * @brief `vld1q_u64` but faster and alignment-safe. * * On AArch64, unaligned access is always safe, but on ARMv7-a, it is only * *conditionally* safe (`vld1` has an alignment bit like `movdq[ua]` in x86). * * GCC for AArch64 sees `vld1q_u8` as an intrinsic instead of a load, so it * prohibits load-store optimizations. Therefore, a direct dereference is used. * * Otherwise, `vld1q_u8` is used with `vreinterpretq_u8_u64` to do a safe * unaligned load. */ #if defined(__aarch64__) && defined(__GNUC__) && !defined(__clang__) XXH_FORCE_INLINE uint64x2_t XXH_vld1q_u64(void const* ptr) /* silence -Wcast-align */ { return *(xxh_aliasing_uint64x2_t const *)ptr; } #else XXH_FORCE_INLINE uint64x2_t XXH_vld1q_u64(void const* ptr) { return vreinterpretq_u64_u8(vld1q_u8((uint8_t const*)ptr)); } #endif /*! * @internal * @brief `vmlal_u32` on low and high halves of a vector. * * This is a workaround for AArch64 GCC < 11 which implemented arm_neon.h with * inline assembly and were therefore incapable of merging the `vget_{low, high}_u32` * with `vmlal_u32`. */ #if defined(__aarch64__) && defined(__GNUC__) && !defined(__clang__) && __GNUC__ < 11 XXH_FORCE_INLINE uint64x2_t XXH_vmlal_low_u32(uint64x2_t acc, uint32x4_t lhs, uint32x4_t rhs) { /* Inline assembly is the only way */ __asm__("umlal %0.2d, %1.2s, %2.2s" : "+w" (acc) : "w" (lhs), "w" (rhs)); return acc; } XXH_FORCE_INLINE uint64x2_t XXH_vmlal_high_u32(uint64x2_t acc, uint32x4_t lhs, uint32x4_t rhs) { /* This intrinsic works as expected */ return vmlal_high_u32(acc, lhs, rhs); } #else /* Portable intrinsic versions */ XXH_FORCE_INLINE uint64x2_t XXH_vmlal_low_u32(uint64x2_t acc, uint32x4_t lhs, uint32x4_t rhs) { return vmlal_u32(acc, vget_low_u32(lhs), vget_low_u32(rhs)); } /*! @copydoc XXH_vmlal_low_u32 * Assume the compiler converts this to vmlal_high_u32 on aarch64 */ XXH_FORCE_INLINE uint64x2_t XXH_vmlal_high_u32(uint64x2_t acc, uint32x4_t lhs, uint32x4_t rhs) { return vmlal_u32(acc, vget_high_u32(lhs), vget_high_u32(rhs)); } #endif /*! * @ingroup tuning * @brief Controls the NEON to scalar ratio for XXH3 * * This can be set to 2, 4, 6, or 8. * * ARM Cortex CPUs are _very_ sensitive to how their pipelines are used. * * For example, the Cortex-A73 can dispatch 3 micro-ops per cycle, but only 2 of those * can be NEON. If you are only using NEON instructions, you are only using 2/3 of the CPU * bandwidth. * * This is even more noticeable on the more advanced cores like the Cortex-A76 which * can dispatch 8 micro-ops per cycle, but still only 2 NEON micro-ops at once. * * Therefore, to make the most out of the pipeline, it is beneficial to run 6 NEON lanes * and 2 scalar lanes, which is chosen by default. * * This does not apply to Apple processors or 32-bit processors, which run better with * full NEON. These will default to 8. Additionally, size-optimized builds run 8 lanes. * * This change benefits CPUs with large micro-op buffers without negatively affecting * most other CPUs: * * | Chipset | Dispatch type | NEON only | 6:2 hybrid | Diff. | * |:----------------------|:--------------------|----------:|-----------:|------:| * | Snapdragon 730 (A76) | 2 NEON/8 micro-ops | 8.8 GB/s | 10.1 GB/s | ~16% | * | Snapdragon 835 (A73) | 2 NEON/3 micro-ops | 5.1 GB/s | 5.3 GB/s | ~5% | * | Marvell PXA1928 (A53) | In-order dual-issue | 1.9 GB/s | 1.9 GB/s | 0% | * | Apple M1 | 4 NEON/8 micro-ops | 37.3 GB/s | 36.1 GB/s | ~-3% | * * It also seems to fix some bad codegen on GCC, making it almost as fast as clang. * * When using WASM SIMD128, if this is 2 or 6, SIMDe will scalarize 2 of the lanes meaning * it effectively becomes worse 4. * * @see XXH3_accumulate_512_neon() */ # ifndef XXH3_NEON_LANES # if (defined(__aarch64__) || defined(__arm64__) || defined(_M_ARM64) || defined(_M_ARM64EC)) \ && !defined(__APPLE__) && XXH_SIZE_OPT <= 0 # define XXH3_NEON_LANES 6 # else # define XXH3_NEON_LANES XXH_ACC_NB # endif # endif #endif /* XXH_VECTOR == XXH_NEON */ /* * VSX and Z Vector helpers. * * This is very messy, and any pull requests to clean this up are welcome. * * There are a lot of problems with supporting VSX and s390x, due to * inconsistent intrinsics, spotty coverage, and multiple endiannesses. */ #if XXH_VECTOR == XXH_VSX /* Annoyingly, these headers _may_ define three macros: `bool`, `vector`, * and `pixel`. This is a problem for obvious reasons. * * These keywords are unnecessary; the spec literally says they are * equivalent to `__bool`, `__vector`, and `__pixel` and may be undef'd * after including the header. * * We use pragma push_macro/pop_macro to keep the namespace clean. */ # pragma push_macro("bool") # pragma push_macro("vector") # pragma push_macro("pixel") /* silence potential macro redefined warnings */ # undef bool # undef vector # undef pixel # if defined(__s390x__) # include # else # include # endif /* Restore the original macro values, if applicable. */ # pragma pop_macro("pixel") # pragma pop_macro("vector") # pragma pop_macro("bool") typedef __vector unsigned long long xxh_u64x2; typedef __vector unsigned char xxh_u8x16; typedef __vector unsigned xxh_u32x4; /* * UGLY HACK: Similar to aarch64 macOS GCC, s390x GCC has the same aliasing issue. */ typedef xxh_u64x2 xxh_aliasing_u64x2 XXH_ALIASING; # ifndef XXH_VSX_BE # if defined(__BIG_ENDIAN__) \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) # define XXH_VSX_BE 1 # elif defined(__VEC_ELEMENT_REG_ORDER__) && __VEC_ELEMENT_REG_ORDER__ == __ORDER_BIG_ENDIAN__ # warning "-maltivec=be is not recommended. Please use native endianness." # define XXH_VSX_BE 1 # else # define XXH_VSX_BE 0 # endif # endif /* !defined(XXH_VSX_BE) */ # if XXH_VSX_BE # if defined(__POWER9_VECTOR__) || (defined(__clang__) && defined(__s390x__)) # define XXH_vec_revb vec_revb # else /*! * A polyfill for POWER9's vec_revb(). */ XXH_FORCE_INLINE xxh_u64x2 XXH_vec_revb(xxh_u64x2 val) { xxh_u8x16 const vByteSwap = { 0x07, 0x06, 0x05, 0x04, 0x03, 0x02, 0x01, 0x00, 0x0F, 0x0E, 0x0D, 0x0C, 0x0B, 0x0A, 0x09, 0x08 }; return vec_perm(val, val, vByteSwap); } # endif # endif /* XXH_VSX_BE */ /*! * Performs an unaligned vector load and byte swaps it on big endian. */ XXH_FORCE_INLINE xxh_u64x2 XXH_vec_loadu(const void *ptr) { xxh_u64x2 ret; XXH_memcpy(&ret, ptr, sizeof(xxh_u64x2)); # if XXH_VSX_BE ret = XXH_vec_revb(ret); # endif return ret; } /* * vec_mulo and vec_mule are very problematic intrinsics on PowerPC * * These intrinsics weren't added until GCC 8, despite existing for a while, * and they are endian dependent. Also, their meaning swap depending on version. * */ # if defined(__s390x__) /* s390x is always big endian, no issue on this platform */ # define XXH_vec_mulo vec_mulo # define XXH_vec_mule vec_mule # elif defined(__clang__) && XXH_HAS_BUILTIN(__builtin_altivec_vmuleuw) && !defined(__ibmxl__) /* Clang has a better way to control this, we can just use the builtin which doesn't swap. */ /* The IBM XL Compiler (which defined __clang__) only implements the vec_* operations */ # define XXH_vec_mulo __builtin_altivec_vmulouw # define XXH_vec_mule __builtin_altivec_vmuleuw # else /* gcc needs inline assembly */ /* Adapted from https://github.com/google/highwayhash/blob/master/highwayhash/hh_vsx.h. */ XXH_FORCE_INLINE xxh_u64x2 XXH_vec_mulo(xxh_u32x4 a, xxh_u32x4 b) { xxh_u64x2 result; __asm__("vmulouw %0, %1, %2" : "=v" (result) : "v" (a), "v" (b)); return result; } XXH_FORCE_INLINE xxh_u64x2 XXH_vec_mule(xxh_u32x4 a, xxh_u32x4 b) { xxh_u64x2 result; __asm__("vmuleuw %0, %1, %2" : "=v" (result) : "v" (a), "v" (b)); return result; } # endif /* XXH_vec_mulo, XXH_vec_mule */ #endif /* XXH_VECTOR == XXH_VSX */ #if XXH_VECTOR == XXH_SVE #define ACCRND(acc, offset) \ do { \ svuint64_t input_vec = svld1_u64(mask, xinput + offset); \ svuint64_t secret_vec = svld1_u64(mask, xsecret + offset); \ svuint64_t mixed = sveor_u64_x(mask, secret_vec, input_vec); \ svuint64_t swapped = svtbl_u64(input_vec, kSwap); \ svuint64_t mixed_lo = svextw_u64_x(mask, mixed); \ svuint64_t mixed_hi = svlsr_n_u64_x(mask, mixed, 32); \ svuint64_t mul = svmad_u64_x(mask, mixed_lo, mixed_hi, swapped); \ acc = svadd_u64_x(mask, acc, mul); \ } while (0) #endif /* XXH_VECTOR == XXH_SVE */ /* prefetch * can be disabled, by declaring XXH_NO_PREFETCH build macro */ #if defined(XXH_NO_PREFETCH) # define XXH_PREFETCH(ptr) (void)(ptr) /* disabled */ #else # if XXH_SIZE_OPT >= 1 # define XXH_PREFETCH(ptr) (void)(ptr) # elif defined(_MSC_VER) && (defined(_M_X64) || defined(_M_IX86)) /* _mm_prefetch() not defined outside of x86/x64 */ # include /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */ # define XXH_PREFETCH(ptr) _mm_prefetch((const char*)(ptr), _MM_HINT_T0) # elif defined(__GNUC__) && ( (__GNUC__ >= 4) || ( (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) ) ) # define XXH_PREFETCH(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 3 /* locality */) # else # define XXH_PREFETCH(ptr) (void)(ptr) /* disabled */ # endif #endif /* XXH_NO_PREFETCH */ /* ========================================== * XXH3 default settings * ========================================== */ #define XXH_SECRET_DEFAULT_SIZE 192 /* minimum XXH3_SECRET_SIZE_MIN */ #if (XXH_SECRET_DEFAULT_SIZE < XXH3_SECRET_SIZE_MIN) # error "default keyset is not large enough" #endif /*! Pseudorandom secret taken directly from FARSH. */ XXH_ALIGN(64) static const xxh_u8 XXH3_kSecret[XXH_SECRET_DEFAULT_SIZE] = { 0xb8, 0xfe, 0x6c, 0x39, 0x23, 0xa4, 0x4b, 0xbe, 0x7c, 0x01, 0x81, 0x2c, 0xf7, 0x21, 0xad, 0x1c, 0xde, 0xd4, 0x6d, 0xe9, 0x83, 0x90, 0x97, 0xdb, 0x72, 0x40, 0xa4, 0xa4, 0xb7, 0xb3, 0x67, 0x1f, 0xcb, 0x79, 0xe6, 0x4e, 0xcc, 0xc0, 0xe5, 0x78, 0x82, 0x5a, 0xd0, 0x7d, 0xcc, 0xff, 0x72, 0x21, 0xb8, 0x08, 0x46, 0x74, 0xf7, 0x43, 0x24, 0x8e, 0xe0, 0x35, 0x90, 0xe6, 0x81, 0x3a, 0x26, 0x4c, 0x3c, 0x28, 0x52, 0xbb, 0x91, 0xc3, 0x00, 0xcb, 0x88, 0xd0, 0x65, 0x8b, 0x1b, 0x53, 0x2e, 0xa3, 0x71, 0x64, 0x48, 0x97, 0xa2, 0x0d, 0xf9, 0x4e, 0x38, 0x19, 0xef, 0x46, 0xa9, 0xde, 0xac, 0xd8, 0xa8, 0xfa, 0x76, 0x3f, 0xe3, 0x9c, 0x34, 0x3f, 0xf9, 0xdc, 0xbb, 0xc7, 0xc7, 0x0b, 0x4f, 0x1d, 0x8a, 0x51, 0xe0, 0x4b, 0xcd, 0xb4, 0x59, 0x31, 0xc8, 0x9f, 0x7e, 0xc9, 0xd9, 0x78, 0x73, 0x64, 0xea, 0xc5, 0xac, 0x83, 0x34, 0xd3, 0xeb, 0xc3, 0xc5, 0x81, 0xa0, 0xff, 0xfa, 0x13, 0x63, 0xeb, 0x17, 0x0d, 0xdd, 0x51, 0xb7, 0xf0, 0xda, 0x49, 0xd3, 0x16, 0x55, 0x26, 0x29, 0xd4, 0x68, 0x9e, 0x2b, 0x16, 0xbe, 0x58, 0x7d, 0x47, 0xa1, 0xfc, 0x8f, 0xf8, 0xb8, 0xd1, 0x7a, 0xd0, 0x31, 0xce, 0x45, 0xcb, 0x3a, 0x8f, 0x95, 0x16, 0x04, 0x28, 0xaf, 0xd7, 0xfb, 0xca, 0xbb, 0x4b, 0x40, 0x7e, }; static const xxh_u64 PRIME_MX1 = 0x165667919E3779F9ULL; /*!< 0b0001011001010110011001111001000110011110001101110111100111111001 */ static const xxh_u64 PRIME_MX2 = 0x9FB21C651E98DF25ULL; /*!< 0b1001111110110010000111000110010100011110100110001101111100100101 */ #ifdef XXH_OLD_NAMES # define kSecret XXH3_kSecret #endif #ifdef XXH_DOXYGEN /*! * @brief Calculates a 32-bit to 64-bit long multiply. * * Implemented as a macro. * * Wraps `__emulu` on MSVC x86 because it tends to call `__allmul` when it doesn't * need to (but it shouldn't need to anyways, it is about 7 instructions to do * a 64x64 multiply...). Since we know that this will _always_ emit `MULL`, we * use that instead of the normal method. * * If you are compiling for platforms like Thumb-1 and don't have a better option, * you may also want to write your own long multiply routine here. * * @param x, y Numbers to be multiplied * @return 64-bit product of the low 32 bits of @p x and @p y. */ XXH_FORCE_INLINE xxh_u64 XXH_mult32to64(xxh_u64 x, xxh_u64 y) { return (x & 0xFFFFFFFF) * (y & 0xFFFFFFFF); } #elif defined(_MSC_VER) && defined(_M_IX86) # define XXH_mult32to64(x, y) __emulu((unsigned)(x), (unsigned)(y)) #else /* * Downcast + upcast is usually better than masking on older compilers like * GCC 4.2 (especially 32-bit ones), all without affecting newer compilers. * * The other method, (x & 0xFFFFFFFF) * (y & 0xFFFFFFFF), will AND both operands * and perform a full 64x64 multiply -- entirely redundant on 32-bit. */ # define XXH_mult32to64(x, y) ((xxh_u64)(xxh_u32)(x) * (xxh_u64)(xxh_u32)(y)) #endif /*! * @brief Calculates a 64->128-bit long multiply. * * Uses `__uint128_t` and `_umul128` if available, otherwise uses a scalar * version. * * @param lhs , rhs The 64-bit integers to be multiplied * @return The 128-bit result represented in an @ref XXH128_hash_t. */ static XXH128_hash_t XXH_mult64to128(xxh_u64 lhs, xxh_u64 rhs) { /* * GCC/Clang __uint128_t method. * * On most 64-bit targets, GCC and Clang define a __uint128_t type. * This is usually the best way as it usually uses a native long 64-bit * multiply, such as MULQ on x86_64 or MUL + UMULH on aarch64. * * Usually. * * Despite being a 32-bit platform, Clang (and emscripten) define this type * despite not having the arithmetic for it. This results in a laggy * compiler builtin call which calculates a full 128-bit multiply. * In that case it is best to use the portable one. * https://github.com/Cyan4973/xxHash/issues/211#issuecomment-515575677 */ #if (defined(__GNUC__) || defined(__clang__)) && !defined(__wasm__) \ && defined(__SIZEOF_INT128__) \ || (defined(_INTEGRAL_MAX_BITS) && _INTEGRAL_MAX_BITS >= 128) __uint128_t const product = (__uint128_t)lhs * (__uint128_t)rhs; XXH128_hash_t r128; r128.low64 = (xxh_u64)(product); r128.high64 = (xxh_u64)(product >> 64); return r128; /* * MSVC for x64's _umul128 method. * * xxh_u64 _umul128(xxh_u64 Multiplier, xxh_u64 Multiplicand, xxh_u64 *HighProduct); * * This compiles to single operand MUL on x64. */ #elif (defined(_M_X64) || defined(_M_IA64)) && !defined(_M_ARM64EC) #ifndef _MSC_VER # pragma intrinsic(_umul128) #endif xxh_u64 product_high; xxh_u64 const product_low = _umul128(lhs, rhs, &product_high); XXH128_hash_t r128; r128.low64 = product_low; r128.high64 = product_high; return r128; /* * MSVC for ARM64's __umulh method. * * This compiles to the same MUL + UMULH as GCC/Clang's __uint128_t method. */ #elif defined(_M_ARM64) || defined(_M_ARM64EC) #ifndef _MSC_VER # pragma intrinsic(__umulh) #endif XXH128_hash_t r128; r128.low64 = lhs * rhs; r128.high64 = __umulh(lhs, rhs); return r128; #else /* * Portable scalar method. Optimized for 32-bit and 64-bit ALUs. * * This is a fast and simple grade school multiply, which is shown below * with base 10 arithmetic instead of base 0x100000000. * * 9 3 // D2 lhs = 93 * x 7 5 // D2 rhs = 75 * ---------- * 1 5 // D2 lo_lo = (93 % 10) * (75 % 10) = 15 * 4 5 | // D2 hi_lo = (93 / 10) * (75 % 10) = 45 * 2 1 | // D2 lo_hi = (93 % 10) * (75 / 10) = 21 * + 6 3 | | // D2 hi_hi = (93 / 10) * (75 / 10) = 63 * --------- * 2 7 | // D2 cross = (15 / 10) + (45 % 10) + 21 = 27 * + 6 7 | | // D2 upper = (27 / 10) + (45 / 10) + 63 = 67 * --------- * 6 9 7 5 // D4 res = (27 * 10) + (15 % 10) + (67 * 100) = 6975 * * The reasons for adding the products like this are: * 1. It avoids manual carry tracking. Just like how * (9 * 9) + 9 + 9 = 99, the same applies with this for UINT64_MAX. * This avoids a lot of complexity. * * 2. It hints for, and on Clang, compiles to, the powerful UMAAL * instruction available in ARM's Digital Signal Processing extension * in 32-bit ARMv6 and later, which is shown below: * * void UMAAL(xxh_u32 *RdLo, xxh_u32 *RdHi, xxh_u32 Rn, xxh_u32 Rm) * { * xxh_u64 product = (xxh_u64)*RdLo * (xxh_u64)*RdHi + Rn + Rm; * *RdLo = (xxh_u32)(product & 0xFFFFFFFF); * *RdHi = (xxh_u32)(product >> 32); * } * * This instruction was designed for efficient long multiplication, and * allows this to be calculated in only 4 instructions at speeds * comparable to some 64-bit ALUs. * * 3. It isn't terrible on other platforms. Usually this will be a couple * of 32-bit ADD/ADCs. */ /* First calculate all of the cross products. */ xxh_u64 const lo_lo = XXH_mult32to64(lhs & 0xFFFFFFFF, rhs & 0xFFFFFFFF); xxh_u64 const hi_lo = XXH_mult32to64(lhs >> 32, rhs & 0xFFFFFFFF); xxh_u64 const lo_hi = XXH_mult32to64(lhs & 0xFFFFFFFF, rhs >> 32); xxh_u64 const hi_hi = XXH_mult32to64(lhs >> 32, rhs >> 32); /* Now add the products together. These will never overflow. */ xxh_u64 const cross = (lo_lo >> 32) + (hi_lo & 0xFFFFFFFF) + lo_hi; xxh_u64 const upper = (hi_lo >> 32) + (cross >> 32) + hi_hi; xxh_u64 const lower = (cross << 32) | (lo_lo & 0xFFFFFFFF); XXH128_hash_t r128; r128.low64 = lower; r128.high64 = upper; return r128; #endif } /*! * @brief Calculates a 64-bit to 128-bit multiply, then XOR folds it. * * The reason for the separate function is to prevent passing too many structs * around by value. This will hopefully inline the multiply, but we don't force it. * * @param lhs , rhs The 64-bit integers to multiply * @return The low 64 bits of the product XOR'd by the high 64 bits. * @see XXH_mult64to128() */ static xxh_u64 XXH3_mul128_fold64(xxh_u64 lhs, xxh_u64 rhs) { XXH128_hash_t product = XXH_mult64to128(lhs, rhs); return product.low64 ^ product.high64; } /*! Seems to produce slightly better code on GCC for some reason. */ XXH_FORCE_INLINE XXH_CONSTF xxh_u64 XXH_xorshift64(xxh_u64 v64, int shift) { XXH_ASSERT(0 <= shift && shift < 64); return v64 ^ (v64 >> shift); } /* * This is a fast avalanche stage, * suitable when input bits are already partially mixed */ static XXH64_hash_t XXH3_avalanche(xxh_u64 h64) { h64 = XXH_xorshift64(h64, 37); h64 *= PRIME_MX1; h64 = XXH_xorshift64(h64, 32); return h64; } /* * This is a stronger avalanche, * inspired by Pelle Evensen's rrmxmx * preferable when input has not been previously mixed */ static XXH64_hash_t XXH3_rrmxmx(xxh_u64 h64, xxh_u64 len) { /* this mix is inspired by Pelle Evensen's rrmxmx */ h64 ^= XXH_rotl64(h64, 49) ^ XXH_rotl64(h64, 24); h64 *= PRIME_MX2; h64 ^= (h64 >> 35) + len ; h64 *= PRIME_MX2; return XXH_xorshift64(h64, 28); } /* ========================================== * Short keys * ========================================== * One of the shortcomings of XXH32 and XXH64 was that their performance was * sub-optimal on short lengths. It used an iterative algorithm which strongly * favored lengths that were a multiple of 4 or 8. * * Instead of iterating over individual inputs, we use a set of single shot * functions which piece together a range of lengths and operate in constant time. * * Additionally, the number of multiplies has been significantly reduced. This * reduces latency, especially when emulating 64-bit multiplies on 32-bit. * * Depending on the platform, this may or may not be faster than XXH32, but it * is almost guaranteed to be faster than XXH64. */ /* * At very short lengths, there isn't enough input to fully hide secrets, or use * the entire secret. * * There is also only a limited amount of mixing we can do before significantly * impacting performance. * * Therefore, we use different sections of the secret and always mix two secret * samples with an XOR. This should have no effect on performance on the * seedless or withSeed variants because everything _should_ be constant folded * by modern compilers. * * The XOR mixing hides individual parts of the secret and increases entropy. * * This adds an extra layer of strength for custom secrets. */ XXH_FORCE_INLINE XXH_PUREF XXH64_hash_t XXH3_len_1to3_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(1 <= len && len <= 3); XXH_ASSERT(secret != NULL); /* * len = 1: combined = { input[0], 0x01, input[0], input[0] } * len = 2: combined = { input[1], 0x02, input[0], input[1] } * len = 3: combined = { input[2], 0x03, input[0], input[1] } */ { xxh_u8 const c1 = input[0]; xxh_u8 const c2 = input[len >> 1]; xxh_u8 const c3 = input[len - 1]; xxh_u32 const combined = ((xxh_u32)c1 << 16) | ((xxh_u32)c2 << 24) | ((xxh_u32)c3 << 0) | ((xxh_u32)len << 8); xxh_u64 const bitflip = (XXH_readLE32(secret) ^ XXH_readLE32(secret+4)) + seed; xxh_u64 const keyed = (xxh_u64)combined ^ bitflip; return XXH64_avalanche(keyed); } } XXH_FORCE_INLINE XXH_PUREF XXH64_hash_t XXH3_len_4to8_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(4 <= len && len <= 8); seed ^= (xxh_u64)XXH_swap32((xxh_u32)seed) << 32; { xxh_u32 const input1 = XXH_readLE32(input); xxh_u32 const input2 = XXH_readLE32(input + len - 4); xxh_u64 const bitflip = (XXH_readLE64(secret+8) ^ XXH_readLE64(secret+16)) - seed; xxh_u64 const input64 = input2 + (((xxh_u64)input1) << 32); xxh_u64 const keyed = input64 ^ bitflip; return XXH3_rrmxmx(keyed, len); } } XXH_FORCE_INLINE XXH_PUREF XXH64_hash_t XXH3_len_9to16_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(9 <= len && len <= 16); { xxh_u64 const bitflip1 = (XXH_readLE64(secret+24) ^ XXH_readLE64(secret+32)) + seed; xxh_u64 const bitflip2 = (XXH_readLE64(secret+40) ^ XXH_readLE64(secret+48)) - seed; xxh_u64 const input_lo = XXH_readLE64(input) ^ bitflip1; xxh_u64 const input_hi = XXH_readLE64(input + len - 8) ^ bitflip2; xxh_u64 const acc = len + XXH_swap64(input_lo) + input_hi + XXH3_mul128_fold64(input_lo, input_hi); return XXH3_avalanche(acc); } } XXH_FORCE_INLINE XXH_PUREF XXH64_hash_t XXH3_len_0to16_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(len <= 16); { if (XXH_likely(len > 8)) return XXH3_len_9to16_64b(input, len, secret, seed); if (XXH_likely(len >= 4)) return XXH3_len_4to8_64b(input, len, secret, seed); if (len) return XXH3_len_1to3_64b(input, len, secret, seed); return XXH64_avalanche(seed ^ (XXH_readLE64(secret+56) ^ XXH_readLE64(secret+64))); } } /* * DISCLAIMER: There are known *seed-dependent* multicollisions here due to * multiplication by zero, affecting hashes of lengths 17 to 240. * * However, they are very unlikely. * * Keep this in mind when using the unseeded XXH3_64bits() variant: As with all * unseeded non-cryptographic hashes, it does not attempt to defend itself * against specially crafted inputs, only random inputs. * * Compared to classic UMAC where a 1 in 2^31 chance of 4 consecutive bytes * cancelling out the secret is taken an arbitrary number of times (addressed * in XXH3_accumulate_512), this collision is very unlikely with random inputs * and/or proper seeding: * * This only has a 1 in 2^63 chance of 8 consecutive bytes cancelling out, in a * function that is only called up to 16 times per hash with up to 240 bytes of * input. * * This is not too bad for a non-cryptographic hash function, especially with * only 64 bit outputs. * * The 128-bit variant (which trades some speed for strength) is NOT affected * by this, although it is always a good idea to use a proper seed if you care * about strength. */ XXH_FORCE_INLINE xxh_u64 XXH3_mix16B(const xxh_u8* XXH_RESTRICT input, const xxh_u8* XXH_RESTRICT secret, xxh_u64 seed64) { #if defined(__GNUC__) && !defined(__clang__) /* GCC, not Clang */ \ && defined(__i386__) && defined(__SSE2__) /* x86 + SSE2 */ \ && !defined(XXH_ENABLE_AUTOVECTORIZE) /* Define to disable like XXH32 hack */ /* * UGLY HACK: * GCC for x86 tends to autovectorize the 128-bit multiply, resulting in * slower code. * * By forcing seed64 into a register, we disrupt the cost model and * cause it to scalarize. See `XXH32_round()` * * FIXME: Clang's output is still _much_ faster -- On an AMD Ryzen 3600, * XXH3_64bits @ len=240 runs at 4.6 GB/s with Clang 9, but 3.3 GB/s on * GCC 9.2, despite both emitting scalar code. * * GCC generates much better scalar code than Clang for the rest of XXH3, * which is why finding a more optimal codepath is an interest. */ XXH_COMPILER_GUARD(seed64); #endif { xxh_u64 const input_lo = XXH_readLE64(input); xxh_u64 const input_hi = XXH_readLE64(input+8); return XXH3_mul128_fold64( input_lo ^ (XXH_readLE64(secret) + seed64), input_hi ^ (XXH_readLE64(secret+8) - seed64) ); } } /* For mid range keys, XXH3 uses a Mum-hash variant. */ XXH_FORCE_INLINE XXH_PUREF XXH64_hash_t XXH3_len_17to128_64b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(16 < len && len <= 128); { xxh_u64 acc = len * XXH_PRIME64_1; #if XXH_SIZE_OPT >= 1 /* Smaller and cleaner, but slightly slower. */ unsigned int i = (unsigned int)(len - 1) / 32; do { acc += XXH3_mix16B(input+16 * i, secret+32*i, seed); acc += XXH3_mix16B(input+len-16*(i+1), secret+32*i+16, seed); } while (i-- != 0); #else if (len > 32) { if (len > 64) { if (len > 96) { acc += XXH3_mix16B(input+48, secret+96, seed); acc += XXH3_mix16B(input+len-64, secret+112, seed); } acc += XXH3_mix16B(input+32, secret+64, seed); acc += XXH3_mix16B(input+len-48, secret+80, seed); } acc += XXH3_mix16B(input+16, secret+32, seed); acc += XXH3_mix16B(input+len-32, secret+48, seed); } acc += XXH3_mix16B(input+0, secret+0, seed); acc += XXH3_mix16B(input+len-16, secret+16, seed); #endif return XXH3_avalanche(acc); } } #define XXH3_MIDSIZE_MAX 240 XXH_NO_INLINE XXH_PUREF XXH64_hash_t XXH3_len_129to240_64b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(128 < len && len <= XXH3_MIDSIZE_MAX); #define XXH3_MIDSIZE_STARTOFFSET 3 #define XXH3_MIDSIZE_LASTOFFSET 17 { xxh_u64 acc = len * XXH_PRIME64_1; xxh_u64 acc_end; unsigned int const nbRounds = (unsigned int)len / 16; unsigned int i; XXH_ASSERT(128 < len && len <= XXH3_MIDSIZE_MAX); for (i=0; i<8; i++) { acc += XXH3_mix16B(input+(16*i), secret+(16*i), seed); } /* last bytes */ acc_end = XXH3_mix16B(input + len - 16, secret + XXH3_SECRET_SIZE_MIN - XXH3_MIDSIZE_LASTOFFSET, seed); XXH_ASSERT(nbRounds >= 8); acc = XXH3_avalanche(acc); #if defined(__clang__) /* Clang */ \ && (defined(__ARM_NEON) || defined(__ARM_NEON__)) /* NEON */ \ && !defined(XXH_ENABLE_AUTOVECTORIZE) /* Define to disable */ /* * UGLY HACK: * Clang for ARMv7-A tries to vectorize this loop, similar to GCC x86. * In everywhere else, it uses scalar code. * * For 64->128-bit multiplies, even if the NEON was 100% optimal, it * would still be slower than UMAAL (see XXH_mult64to128). * * Unfortunately, Clang doesn't handle the long multiplies properly and * converts them to the nonexistent "vmulq_u64" intrinsic, which is then * scalarized into an ugly mess of VMOV.32 instructions. * * This mess is difficult to avoid without turning autovectorization * off completely, but they are usually relatively minor and/or not * worth it to fix. * * This loop is the easiest to fix, as unlike XXH32, this pragma * _actually works_ because it is a loop vectorization instead of an * SLP vectorization. */ #pragma clang loop vectorize(disable) #endif for (i=8 ; i < nbRounds; i++) { /* * Prevents clang for unrolling the acc loop and interleaving with this one. */ XXH_COMPILER_GUARD(acc); acc_end += XXH3_mix16B(input+(16*i), secret+(16*(i-8)) + XXH3_MIDSIZE_STARTOFFSET, seed); } return XXH3_avalanche(acc + acc_end); } } /* ======= Long Keys ======= */ #define XXH_STRIPE_LEN 64 #define XXH_SECRET_CONSUME_RATE 8 /* nb of secret bytes consumed at each accumulation */ #define XXH_ACC_NB (XXH_STRIPE_LEN / sizeof(xxh_u64)) #ifdef XXH_OLD_NAMES # define STRIPE_LEN XXH_STRIPE_LEN # define ACC_NB XXH_ACC_NB #endif #ifndef XXH_PREFETCH_DIST # ifdef __clang__ # define XXH_PREFETCH_DIST 320 # else # if (XXH_VECTOR == XXH_AVX512) # define XXH_PREFETCH_DIST 512 # else # define XXH_PREFETCH_DIST 384 # endif # endif /* __clang__ */ #endif /* XXH_PREFETCH_DIST */ /* * These macros are to generate an XXH3_accumulate() function. * The two arguments select the name suffix and target attribute. * * The name of this symbol is XXH3_accumulate_() and it calls * XXH3_accumulate_512_(). * * It may be useful to hand implement this function if the compiler fails to * optimize the inline function. */ #define XXH3_ACCUMULATE_TEMPLATE(name) \ void \ XXH3_accumulate_##name(xxh_u64* XXH_RESTRICT acc, \ const xxh_u8* XXH_RESTRICT input, \ const xxh_u8* XXH_RESTRICT secret, \ size_t nbStripes) \ { \ size_t n; \ for (n = 0; n < nbStripes; n++ ) { \ const xxh_u8* const in = input + n*XXH_STRIPE_LEN; \ XXH_PREFETCH(in + XXH_PREFETCH_DIST); \ XXH3_accumulate_512_##name( \ acc, \ in, \ secret + n*XXH_SECRET_CONSUME_RATE); \ } \ } XXH_FORCE_INLINE void XXH_writeLE64(void* dst, xxh_u64 v64) { if (!XXH_CPU_LITTLE_ENDIAN) v64 = XXH_swap64(v64); XXH_memcpy(dst, &v64, sizeof(v64)); } /* Several intrinsic functions below are supposed to accept __int64 as argument, * as documented in https://software.intel.com/sites/landingpage/IntrinsicsGuide/ . * However, several environments do not define __int64 type, * requiring a workaround. */ #if !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) typedef int64_t xxh_i64; #else /* the following type must have a width of 64-bit */ typedef long long xxh_i64; #endif /* * XXH3_accumulate_512 is the tightest loop for long inputs, and it is the most optimized. * * It is a hardened version of UMAC, based off of FARSH's implementation. * * This was chosen because it adapts quite well to 32-bit, 64-bit, and SIMD * implementations, and it is ridiculously fast. * * We harden it by mixing the original input to the accumulators as well as the product. * * This means that in the (relatively likely) case of a multiply by zero, the * original input is preserved. * * On 128-bit inputs, we swap 64-bit pairs when we add the input to improve * cross-pollination, as otherwise the upper and lower halves would be * essentially independent. * * This doesn't matter on 64-bit hashes since they all get merged together in * the end, so we skip the extra step. * * Both XXH3_64bits and XXH3_128bits use this subroutine. */ #if (XXH_VECTOR == XXH_AVX512) \ || (defined(XXH_DISPATCH_AVX512) && XXH_DISPATCH_AVX512 != 0) #ifndef XXH_TARGET_AVX512 # define XXH_TARGET_AVX512 /* disable attribute target */ #endif XXH_FORCE_INLINE XXH_TARGET_AVX512 void XXH3_accumulate_512_avx512(void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { __m512i* const xacc = (__m512i *) acc; XXH_ASSERT((((size_t)acc) & 63) == 0); XXH_STATIC_ASSERT(XXH_STRIPE_LEN == sizeof(__m512i)); { /* data_vec = input[0]; */ __m512i const data_vec = _mm512_loadu_si512 (input); /* key_vec = secret[0]; */ __m512i const key_vec = _mm512_loadu_si512 (secret); /* data_key = data_vec ^ key_vec; */ __m512i const data_key = _mm512_xor_si512 (data_vec, key_vec); /* data_key_lo = data_key >> 32; */ __m512i const data_key_lo = _mm512_srli_epi64 (data_key, 32); /* product = (data_key & 0xffffffff) * (data_key_lo & 0xffffffff); */ __m512i const product = _mm512_mul_epu32 (data_key, data_key_lo); /* xacc[0] += swap(data_vec); */ __m512i const data_swap = _mm512_shuffle_epi32(data_vec, (_MM_PERM_ENUM)_MM_SHUFFLE(1, 0, 3, 2)); __m512i const sum = _mm512_add_epi64(*xacc, data_swap); /* xacc[0] += product; */ *xacc = _mm512_add_epi64(product, sum); } } XXH_FORCE_INLINE XXH_TARGET_AVX512 XXH3_ACCUMULATE_TEMPLATE(avx512) /* * XXH3_scrambleAcc: Scrambles the accumulators to improve mixing. * * Multiplication isn't perfect, as explained by Google in HighwayHash: * * // Multiplication mixes/scrambles bytes 0-7 of the 64-bit result to * // varying degrees. In descending order of goodness, bytes * // 3 4 2 5 1 6 0 7 have quality 228 224 164 160 100 96 36 32. * // As expected, the upper and lower bytes are much worse. * * Source: https://github.com/google/highwayhash/blob/0aaf66b/highwayhash/hh_avx2.h#L291 * * Since our algorithm uses a pseudorandom secret to add some variance into the * mix, we don't need to (or want to) mix as often or as much as HighwayHash does. * * This isn't as tight as XXH3_accumulate, but still written in SIMD to avoid * extraction. * * Both XXH3_64bits and XXH3_128bits use this subroutine. */ XXH_FORCE_INLINE XXH_TARGET_AVX512 void XXH3_scrambleAcc_avx512(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 63) == 0); XXH_STATIC_ASSERT(XXH_STRIPE_LEN == sizeof(__m512i)); { __m512i* const xacc = (__m512i*) acc; const __m512i prime32 = _mm512_set1_epi32((int)XXH_PRIME32_1); /* xacc[0] ^= (xacc[0] >> 47) */ __m512i const acc_vec = *xacc; __m512i const shifted = _mm512_srli_epi64 (acc_vec, 47); /* xacc[0] ^= secret; */ __m512i const key_vec = _mm512_loadu_si512 (secret); __m512i const data_key = _mm512_ternarylogic_epi32(key_vec, acc_vec, shifted, 0x96 /* key_vec ^ acc_vec ^ shifted */); /* xacc[0] *= XXH_PRIME32_1; */ __m512i const data_key_hi = _mm512_srli_epi64 (data_key, 32); __m512i const prod_lo = _mm512_mul_epu32 (data_key, prime32); __m512i const prod_hi = _mm512_mul_epu32 (data_key_hi, prime32); *xacc = _mm512_add_epi64(prod_lo, _mm512_slli_epi64(prod_hi, 32)); } } XXH_FORCE_INLINE XXH_TARGET_AVX512 void XXH3_initCustomSecret_avx512(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 63) == 0); XXH_STATIC_ASSERT(XXH_SEC_ALIGN == 64); XXH_ASSERT(((size_t)customSecret & 63) == 0); (void)(&XXH_writeLE64); { int const nbRounds = XXH_SECRET_DEFAULT_SIZE / sizeof(__m512i); __m512i const seed_pos = _mm512_set1_epi64((xxh_i64)seed64); __m512i const seed = _mm512_mask_sub_epi64(seed_pos, 0xAA, _mm512_set1_epi8(0), seed_pos); const __m512i* const src = (const __m512i*) ((const void*) XXH3_kSecret); __m512i* const dest = ( __m512i*) customSecret; int i; XXH_ASSERT(((size_t)src & 63) == 0); /* control alignment */ XXH_ASSERT(((size_t)dest & 63) == 0); for (i=0; i < nbRounds; ++i) { dest[i] = _mm512_add_epi64(_mm512_load_si512(src + i), seed); } } } #endif #if (XXH_VECTOR == XXH_AVX2) \ || (defined(XXH_DISPATCH_AVX2) && XXH_DISPATCH_AVX2 != 0) #ifndef XXH_TARGET_AVX2 # define XXH_TARGET_AVX2 /* disable attribute target */ #endif XXH_FORCE_INLINE XXH_TARGET_AVX2 void XXH3_accumulate_512_avx2( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 31) == 0); { __m256i* const xacc = (__m256i *) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm256_loadu_si256 requires a const __m256i * pointer for some reason. */ const __m256i* const xinput = (const __m256i *) input; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm256_loadu_si256 requires a const __m256i * pointer for some reason. */ const __m256i* const xsecret = (const __m256i *) secret; size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m256i); i++) { /* data_vec = xinput[i]; */ __m256i const data_vec = _mm256_loadu_si256 (xinput+i); /* key_vec = xsecret[i]; */ __m256i const key_vec = _mm256_loadu_si256 (xsecret+i); /* data_key = data_vec ^ key_vec; */ __m256i const data_key = _mm256_xor_si256 (data_vec, key_vec); /* data_key_lo = data_key >> 32; */ __m256i const data_key_lo = _mm256_srli_epi64 (data_key, 32); /* product = (data_key & 0xffffffff) * (data_key_lo & 0xffffffff); */ __m256i const product = _mm256_mul_epu32 (data_key, data_key_lo); /* xacc[i] += swap(data_vec); */ __m256i const data_swap = _mm256_shuffle_epi32(data_vec, _MM_SHUFFLE(1, 0, 3, 2)); __m256i const sum = _mm256_add_epi64(xacc[i], data_swap); /* xacc[i] += product; */ xacc[i] = _mm256_add_epi64(product, sum); } } } XXH_FORCE_INLINE XXH_TARGET_AVX2 XXH3_ACCUMULATE_TEMPLATE(avx2) XXH_FORCE_INLINE XXH_TARGET_AVX2 void XXH3_scrambleAcc_avx2(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 31) == 0); { __m256i* const xacc = (__m256i*) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm256_loadu_si256 requires a const __m256i * pointer for some reason. */ const __m256i* const xsecret = (const __m256i *) secret; const __m256i prime32 = _mm256_set1_epi32((int)XXH_PRIME32_1); size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m256i); i++) { /* xacc[i] ^= (xacc[i] >> 47) */ __m256i const acc_vec = xacc[i]; __m256i const shifted = _mm256_srli_epi64 (acc_vec, 47); __m256i const data_vec = _mm256_xor_si256 (acc_vec, shifted); /* xacc[i] ^= xsecret; */ __m256i const key_vec = _mm256_loadu_si256 (xsecret+i); __m256i const data_key = _mm256_xor_si256 (data_vec, key_vec); /* xacc[i] *= XXH_PRIME32_1; */ __m256i const data_key_hi = _mm256_srli_epi64 (data_key, 32); __m256i const prod_lo = _mm256_mul_epu32 (data_key, prime32); __m256i const prod_hi = _mm256_mul_epu32 (data_key_hi, prime32); xacc[i] = _mm256_add_epi64(prod_lo, _mm256_slli_epi64(prod_hi, 32)); } } } XXH_FORCE_INLINE XXH_TARGET_AVX2 void XXH3_initCustomSecret_avx2(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 31) == 0); XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE / sizeof(__m256i)) == 6); XXH_STATIC_ASSERT(XXH_SEC_ALIGN <= 64); (void)(&XXH_writeLE64); XXH_PREFETCH(customSecret); { __m256i const seed = _mm256_set_epi64x((xxh_i64)(0U - seed64), (xxh_i64)seed64, (xxh_i64)(0U - seed64), (xxh_i64)seed64); const __m256i* const src = (const __m256i*) ((const void*) XXH3_kSecret); __m256i* dest = ( __m256i*) customSecret; # if defined(__GNUC__) || defined(__clang__) /* * On GCC & Clang, marking 'dest' as modified will cause the compiler: * - do not extract the secret from sse registers in the internal loop * - use less common registers, and avoid pushing these reg into stack */ XXH_COMPILER_GUARD(dest); # endif XXH_ASSERT(((size_t)src & 31) == 0); /* control alignment */ XXH_ASSERT(((size_t)dest & 31) == 0); /* GCC -O2 need unroll loop manually */ dest[0] = _mm256_add_epi64(_mm256_load_si256(src+0), seed); dest[1] = _mm256_add_epi64(_mm256_load_si256(src+1), seed); dest[2] = _mm256_add_epi64(_mm256_load_si256(src+2), seed); dest[3] = _mm256_add_epi64(_mm256_load_si256(src+3), seed); dest[4] = _mm256_add_epi64(_mm256_load_si256(src+4), seed); dest[5] = _mm256_add_epi64(_mm256_load_si256(src+5), seed); } } #endif /* x86dispatch always generates SSE2 */ #if (XXH_VECTOR == XXH_SSE2) || defined(XXH_X86DISPATCH) #ifndef XXH_TARGET_SSE2 # define XXH_TARGET_SSE2 /* disable attribute target */ #endif XXH_FORCE_INLINE XXH_TARGET_SSE2 void XXH3_accumulate_512_sse2( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { /* SSE2 is just a half-scale version of the AVX2 version. */ XXH_ASSERT((((size_t)acc) & 15) == 0); { __m128i* const xacc = (__m128i *) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm_loadu_si128 requires a const __m128i * pointer for some reason. */ const __m128i* const xinput = (const __m128i *) input; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm_loadu_si128 requires a const __m128i * pointer for some reason. */ const __m128i* const xsecret = (const __m128i *) secret; size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m128i); i++) { /* data_vec = xinput[i]; */ __m128i const data_vec = _mm_loadu_si128 (xinput+i); /* key_vec = xsecret[i]; */ __m128i const key_vec = _mm_loadu_si128 (xsecret+i); /* data_key = data_vec ^ key_vec; */ __m128i const data_key = _mm_xor_si128 (data_vec, key_vec); /* data_key_lo = data_key >> 32; */ __m128i const data_key_lo = _mm_shuffle_epi32 (data_key, _MM_SHUFFLE(0, 3, 0, 1)); /* product = (data_key & 0xffffffff) * (data_key_lo & 0xffffffff); */ __m128i const product = _mm_mul_epu32 (data_key, data_key_lo); /* xacc[i] += swap(data_vec); */ __m128i const data_swap = _mm_shuffle_epi32(data_vec, _MM_SHUFFLE(1,0,3,2)); __m128i const sum = _mm_add_epi64(xacc[i], data_swap); /* xacc[i] += product; */ xacc[i] = _mm_add_epi64(product, sum); } } } XXH_FORCE_INLINE XXH_TARGET_SSE2 XXH3_ACCUMULATE_TEMPLATE(sse2) XXH_FORCE_INLINE XXH_TARGET_SSE2 void XXH3_scrambleAcc_sse2(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); { __m128i* const xacc = (__m128i*) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm_loadu_si128 requires a const __m128i * pointer for some reason. */ const __m128i* const xsecret = (const __m128i *) secret; const __m128i prime32 = _mm_set1_epi32((int)XXH_PRIME32_1); size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m128i); i++) { /* xacc[i] ^= (xacc[i] >> 47) */ __m128i const acc_vec = xacc[i]; __m128i const shifted = _mm_srli_epi64 (acc_vec, 47); __m128i const data_vec = _mm_xor_si128 (acc_vec, shifted); /* xacc[i] ^= xsecret[i]; */ __m128i const key_vec = _mm_loadu_si128 (xsecret+i); __m128i const data_key = _mm_xor_si128 (data_vec, key_vec); /* xacc[i] *= XXH_PRIME32_1; */ __m128i const data_key_hi = _mm_shuffle_epi32 (data_key, _MM_SHUFFLE(0, 3, 0, 1)); __m128i const prod_lo = _mm_mul_epu32 (data_key, prime32); __m128i const prod_hi = _mm_mul_epu32 (data_key_hi, prime32); xacc[i] = _mm_add_epi64(prod_lo, _mm_slli_epi64(prod_hi, 32)); } } } XXH_FORCE_INLINE XXH_TARGET_SSE2 void XXH3_initCustomSecret_sse2(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 15) == 0); (void)(&XXH_writeLE64); { int const nbRounds = XXH_SECRET_DEFAULT_SIZE / sizeof(__m128i); # if defined(_MSC_VER) && defined(_M_IX86) && _MSC_VER < 1900 /* MSVC 32bit mode does not support _mm_set_epi64x before 2015 */ XXH_ALIGN(16) const xxh_i64 seed64x2[2] = { (xxh_i64)seed64, (xxh_i64)(0U - seed64) }; __m128i const seed = _mm_load_si128((__m128i const*)seed64x2); # else __m128i const seed = _mm_set_epi64x((xxh_i64)(0U - seed64), (xxh_i64)seed64); # endif int i; const void* const src16 = XXH3_kSecret; __m128i* dst16 = (__m128i*) customSecret; # if defined(__GNUC__) || defined(__clang__) /* * On GCC & Clang, marking 'dest' as modified will cause the compiler: * - do not extract the secret from sse registers in the internal loop * - use less common registers, and avoid pushing these reg into stack */ XXH_COMPILER_GUARD(dst16); # endif XXH_ASSERT(((size_t)src16 & 15) == 0); /* control alignment */ XXH_ASSERT(((size_t)dst16 & 15) == 0); for (i=0; i < nbRounds; ++i) { dst16[i] = _mm_add_epi64(_mm_load_si128((const __m128i *)src16+i), seed); } } } #endif #if (XXH_VECTOR == XXH_NEON) /* forward declarations for the scalar routines */ XXH_FORCE_INLINE void XXH3_scalarRound(void* XXH_RESTRICT acc, void const* XXH_RESTRICT input, void const* XXH_RESTRICT secret, size_t lane); XXH_FORCE_INLINE void XXH3_scalarScrambleRound(void* XXH_RESTRICT acc, void const* XXH_RESTRICT secret, size_t lane); /*! * @internal * @brief The bulk processing loop for NEON and WASM SIMD128. * * The NEON code path is actually partially scalar when running on AArch64. This * is to optimize the pipelining and can have up to 15% speedup depending on the * CPU, and it also mitigates some GCC codegen issues. * * @see XXH3_NEON_LANES for configuring this and details about this optimization. * * NEON's 32-bit to 64-bit long multiply takes a half vector of 32-bit * integers instead of the other platforms which mask full 64-bit vectors, * so the setup is more complicated than just shifting right. * * Additionally, there is an optimization for 4 lanes at once noted below. * * Since, as stated, the most optimal amount of lanes for Cortexes is 6, * there needs to be *three* versions of the accumulate operation used * for the remaining 2 lanes. * * WASM's SIMD128 uses SIMDe's arm_neon.h polyfill because the intrinsics overlap * nearly perfectly. */ XXH_FORCE_INLINE void XXH3_accumulate_512_neon( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); XXH_STATIC_ASSERT(XXH3_NEON_LANES > 0 && XXH3_NEON_LANES <= XXH_ACC_NB && XXH3_NEON_LANES % 2 == 0); { /* GCC for darwin arm64 does not like aliasing here */ xxh_aliasing_uint64x2_t* const xacc = (xxh_aliasing_uint64x2_t*) acc; /* We don't use a uint32x4_t pointer because it causes bus errors on ARMv7. */ uint8_t const* xinput = (const uint8_t *) input; uint8_t const* xsecret = (const uint8_t *) secret; size_t i; #ifdef __wasm_simd128__ /* * On WASM SIMD128, Clang emits direct address loads when XXH3_kSecret * is constant propagated, which results in it converting it to this * inside the loop: * * a = v128.load(XXH3_kSecret + 0 + $secret_offset, offset = 0) * b = v128.load(XXH3_kSecret + 16 + $secret_offset, offset = 0) * ... * * This requires a full 32-bit address immediate (and therefore a 6 byte * instruction) as well as an add for each offset. * * Putting an asm guard prevents it from folding (at the cost of losing * the alignment hint), and uses the free offset in `v128.load` instead * of adding secret_offset each time which overall reduces code size by * about a kilobyte and improves performance. */ XXH_COMPILER_GUARD(xsecret); #endif /* Scalar lanes use the normal scalarRound routine */ for (i = XXH3_NEON_LANES; i < XXH_ACC_NB; i++) { XXH3_scalarRound(acc, input, secret, i); } i = 0; /* 4 NEON lanes at a time. */ for (; i+1 < XXH3_NEON_LANES / 2; i+=2) { /* data_vec = xinput[i]; */ uint64x2_t data_vec_1 = XXH_vld1q_u64(xinput + (i * 16)); uint64x2_t data_vec_2 = XXH_vld1q_u64(xinput + ((i+1) * 16)); /* key_vec = xsecret[i]; */ uint64x2_t key_vec_1 = XXH_vld1q_u64(xsecret + (i * 16)); uint64x2_t key_vec_2 = XXH_vld1q_u64(xsecret + ((i+1) * 16)); /* data_swap = swap(data_vec) */ uint64x2_t data_swap_1 = vextq_u64(data_vec_1, data_vec_1, 1); uint64x2_t data_swap_2 = vextq_u64(data_vec_2, data_vec_2, 1); /* data_key = data_vec ^ key_vec; */ uint64x2_t data_key_1 = veorq_u64(data_vec_1, key_vec_1); uint64x2_t data_key_2 = veorq_u64(data_vec_2, key_vec_2); /* * If we reinterpret the 64x2 vectors as 32x4 vectors, we can use a * de-interleave operation for 4 lanes in 1 step with `vuzpq_u32` to * get one vector with the low 32 bits of each lane, and one vector * with the high 32 bits of each lane. * * The intrinsic returns a double vector because the original ARMv7-a * instruction modified both arguments in place. AArch64 and SIMD128 emit * two instructions from this intrinsic. * * [ dk11L | dk11H | dk12L | dk12H ] -> [ dk11L | dk12L | dk21L | dk22L ] * [ dk21L | dk21H | dk22L | dk22H ] -> [ dk11H | dk12H | dk21H | dk22H ] */ uint32x4x2_t unzipped = vuzpq_u32( vreinterpretq_u32_u64(data_key_1), vreinterpretq_u32_u64(data_key_2) ); /* data_key_lo = data_key & 0xFFFFFFFF */ uint32x4_t data_key_lo = unzipped.val[0]; /* data_key_hi = data_key >> 32 */ uint32x4_t data_key_hi = unzipped.val[1]; /* * Then, we can split the vectors horizontally and multiply which, as for most * widening intrinsics, have a variant that works on both high half vectors * for free on AArch64. A similar instruction is available on SIMD128. * * sum = data_swap + (u64x2) data_key_lo * (u64x2) data_key_hi */ uint64x2_t sum_1 = XXH_vmlal_low_u32(data_swap_1, data_key_lo, data_key_hi); uint64x2_t sum_2 = XXH_vmlal_high_u32(data_swap_2, data_key_lo, data_key_hi); /* * Clang reorders * a += b * c; // umlal swap.2d, dkl.2s, dkh.2s * c += a; // add acc.2d, acc.2d, swap.2d * to * c += a; // add acc.2d, acc.2d, swap.2d * c += b * c; // umlal acc.2d, dkl.2s, dkh.2s * * While it would make sense in theory since the addition is faster, * for reasons likely related to umlal being limited to certain NEON * pipelines, this is worse. A compiler guard fixes this. */ XXH_COMPILER_GUARD_CLANG_NEON(sum_1); XXH_COMPILER_GUARD_CLANG_NEON(sum_2); /* xacc[i] = acc_vec + sum; */ xacc[i] = vaddq_u64(xacc[i], sum_1); xacc[i+1] = vaddq_u64(xacc[i+1], sum_2); } /* Operate on the remaining NEON lanes 2 at a time. */ for (; i < XXH3_NEON_LANES / 2; i++) { /* data_vec = xinput[i]; */ uint64x2_t data_vec = XXH_vld1q_u64(xinput + (i * 16)); /* key_vec = xsecret[i]; */ uint64x2_t key_vec = XXH_vld1q_u64(xsecret + (i * 16)); /* acc_vec_2 = swap(data_vec) */ uint64x2_t data_swap = vextq_u64(data_vec, data_vec, 1); /* data_key = data_vec ^ key_vec; */ uint64x2_t data_key = veorq_u64(data_vec, key_vec); /* For two lanes, just use VMOVN and VSHRN. */ /* data_key_lo = data_key & 0xFFFFFFFF; */ uint32x2_t data_key_lo = vmovn_u64(data_key); /* data_key_hi = data_key >> 32; */ uint32x2_t data_key_hi = vshrn_n_u64(data_key, 32); /* sum = data_swap + (u64x2) data_key_lo * (u64x2) data_key_hi; */ uint64x2_t sum = vmlal_u32(data_swap, data_key_lo, data_key_hi); /* Same Clang workaround as before */ XXH_COMPILER_GUARD_CLANG_NEON(sum); /* xacc[i] = acc_vec + sum; */ xacc[i] = vaddq_u64 (xacc[i], sum); } } } XXH_FORCE_INLINE XXH3_ACCUMULATE_TEMPLATE(neon) XXH_FORCE_INLINE void XXH3_scrambleAcc_neon(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); { xxh_aliasing_uint64x2_t* xacc = (xxh_aliasing_uint64x2_t*) acc; uint8_t const* xsecret = (uint8_t const*) secret; size_t i; /* WASM uses operator overloads and doesn't need these. */ #ifndef __wasm_simd128__ /* { prime32_1, prime32_1 } */ uint32x2_t const kPrimeLo = vdup_n_u32(XXH_PRIME32_1); /* { 0, prime32_1, 0, prime32_1 } */ uint32x4_t const kPrimeHi = vreinterpretq_u32_u64(vdupq_n_u64((xxh_u64)XXH_PRIME32_1 << 32)); #endif /* AArch64 uses both scalar and neon at the same time */ for (i = XXH3_NEON_LANES; i < XXH_ACC_NB; i++) { XXH3_scalarScrambleRound(acc, secret, i); } for (i=0; i < XXH3_NEON_LANES / 2; i++) { /* xacc[i] ^= (xacc[i] >> 47); */ uint64x2_t acc_vec = xacc[i]; uint64x2_t shifted = vshrq_n_u64(acc_vec, 47); uint64x2_t data_vec = veorq_u64(acc_vec, shifted); /* xacc[i] ^= xsecret[i]; */ uint64x2_t key_vec = XXH_vld1q_u64(xsecret + (i * 16)); uint64x2_t data_key = veorq_u64(data_vec, key_vec); /* xacc[i] *= XXH_PRIME32_1 */ #ifdef __wasm_simd128__ /* SIMD128 has multiply by u64x2, use it instead of expanding and scalarizing */ xacc[i] = data_key * XXH_PRIME32_1; #else /* * Expanded version with portable NEON intrinsics * * lo(x) * lo(y) + (hi(x) * lo(y) << 32) * * prod_hi = hi(data_key) * lo(prime) << 32 * * Since we only need 32 bits of this multiply a trick can be used, reinterpreting the vector * as a uint32x4_t and multiplying by { 0, prime, 0, prime } to cancel out the unwanted bits * and avoid the shift. */ uint32x4_t prod_hi = vmulq_u32 (vreinterpretq_u32_u64(data_key), kPrimeHi); /* Extract low bits for vmlal_u32 */ uint32x2_t data_key_lo = vmovn_u64(data_key); /* xacc[i] = prod_hi + lo(data_key) * XXH_PRIME32_1; */ xacc[i] = vmlal_u32(vreinterpretq_u64_u32(prod_hi), data_key_lo, kPrimeLo); #endif } } } #endif #if (XXH_VECTOR == XXH_VSX) XXH_FORCE_INLINE void XXH3_accumulate_512_vsx( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { /* presumed aligned */ xxh_aliasing_u64x2* const xacc = (xxh_aliasing_u64x2*) acc; xxh_u8 const* const xinput = (xxh_u8 const*) input; /* no alignment restriction */ xxh_u8 const* const xsecret = (xxh_u8 const*) secret; /* no alignment restriction */ xxh_u64x2 const v32 = { 32, 32 }; size_t i; for (i = 0; i < XXH_STRIPE_LEN / sizeof(xxh_u64x2); i++) { /* data_vec = xinput[i]; */ xxh_u64x2 const data_vec = XXH_vec_loadu(xinput + 16*i); /* key_vec = xsecret[i]; */ xxh_u64x2 const key_vec = XXH_vec_loadu(xsecret + 16*i); xxh_u64x2 const data_key = data_vec ^ key_vec; /* shuffled = (data_key << 32) | (data_key >> 32); */ xxh_u32x4 const shuffled = (xxh_u32x4)vec_rl(data_key, v32); /* product = ((xxh_u64x2)data_key & 0xFFFFFFFF) * ((xxh_u64x2)shuffled & 0xFFFFFFFF); */ xxh_u64x2 const product = XXH_vec_mulo((xxh_u32x4)data_key, shuffled); /* acc_vec = xacc[i]; */ xxh_u64x2 acc_vec = xacc[i]; acc_vec += product; /* swap high and low halves */ #ifdef __s390x__ acc_vec += vec_permi(data_vec, data_vec, 2); #else acc_vec += vec_xxpermdi(data_vec, data_vec, 2); #endif xacc[i] = acc_vec; } } XXH_FORCE_INLINE XXH3_ACCUMULATE_TEMPLATE(vsx) XXH_FORCE_INLINE void XXH3_scrambleAcc_vsx(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); { xxh_aliasing_u64x2* const xacc = (xxh_aliasing_u64x2*) acc; const xxh_u8* const xsecret = (const xxh_u8*) secret; /* constants */ xxh_u64x2 const v32 = { 32, 32 }; xxh_u64x2 const v47 = { 47, 47 }; xxh_u32x4 const prime = { XXH_PRIME32_1, XXH_PRIME32_1, XXH_PRIME32_1, XXH_PRIME32_1 }; size_t i; for (i = 0; i < XXH_STRIPE_LEN / sizeof(xxh_u64x2); i++) { /* xacc[i] ^= (xacc[i] >> 47); */ xxh_u64x2 const acc_vec = xacc[i]; xxh_u64x2 const data_vec = acc_vec ^ (acc_vec >> v47); /* xacc[i] ^= xsecret[i]; */ xxh_u64x2 const key_vec = XXH_vec_loadu(xsecret + 16*i); xxh_u64x2 const data_key = data_vec ^ key_vec; /* xacc[i] *= XXH_PRIME32_1 */ /* prod_lo = ((xxh_u64x2)data_key & 0xFFFFFFFF) * ((xxh_u64x2)prime & 0xFFFFFFFF); */ xxh_u64x2 const prod_even = XXH_vec_mule((xxh_u32x4)data_key, prime); /* prod_hi = ((xxh_u64x2)data_key >> 32) * ((xxh_u64x2)prime >> 32); */ xxh_u64x2 const prod_odd = XXH_vec_mulo((xxh_u32x4)data_key, prime); xacc[i] = prod_odd + (prod_even << v32); } } } #endif #if (XXH_VECTOR == XXH_SVE) XXH_FORCE_INLINE void XXH3_accumulate_512_sve( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { uint64_t *xacc = (uint64_t *)acc; const uint64_t *xinput = (const uint64_t *)(const void *)input; const uint64_t *xsecret = (const uint64_t *)(const void *)secret; svuint64_t kSwap = sveor_n_u64_z(svptrue_b64(), svindex_u64(0, 1), 1); uint64_t element_count = svcntd(); if (element_count >= 8) { svbool_t mask = svptrue_pat_b64(SV_VL8); svuint64_t vacc = svld1_u64(mask, xacc); ACCRND(vacc, 0); svst1_u64(mask, xacc, vacc); } else if (element_count == 2) { /* sve128 */ svbool_t mask = svptrue_pat_b64(SV_VL2); svuint64_t acc0 = svld1_u64(mask, xacc + 0); svuint64_t acc1 = svld1_u64(mask, xacc + 2); svuint64_t acc2 = svld1_u64(mask, xacc + 4); svuint64_t acc3 = svld1_u64(mask, xacc + 6); ACCRND(acc0, 0); ACCRND(acc1, 2); ACCRND(acc2, 4); ACCRND(acc3, 6); svst1_u64(mask, xacc + 0, acc0); svst1_u64(mask, xacc + 2, acc1); svst1_u64(mask, xacc + 4, acc2); svst1_u64(mask, xacc + 6, acc3); } else { svbool_t mask = svptrue_pat_b64(SV_VL4); svuint64_t acc0 = svld1_u64(mask, xacc + 0); svuint64_t acc1 = svld1_u64(mask, xacc + 4); ACCRND(acc0, 0); ACCRND(acc1, 4); svst1_u64(mask, xacc + 0, acc0); svst1_u64(mask, xacc + 4, acc1); } } XXH_FORCE_INLINE void XXH3_accumulate_sve(xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT input, const xxh_u8* XXH_RESTRICT secret, size_t nbStripes) { if (nbStripes != 0) { uint64_t *xacc = (uint64_t *)acc; const uint64_t *xinput = (const uint64_t *)(const void *)input; const uint64_t *xsecret = (const uint64_t *)(const void *)secret; svuint64_t kSwap = sveor_n_u64_z(svptrue_b64(), svindex_u64(0, 1), 1); uint64_t element_count = svcntd(); if (element_count >= 8) { svbool_t mask = svptrue_pat_b64(SV_VL8); svuint64_t vacc = svld1_u64(mask, xacc + 0); do { /* svprfd(svbool_t, void *, enum svfprop); */ svprfd(mask, xinput + 128, SV_PLDL1STRM); ACCRND(vacc, 0); xinput += 8; xsecret += 1; nbStripes--; } while (nbStripes != 0); svst1_u64(mask, xacc + 0, vacc); } else if (element_count == 2) { /* sve128 */ svbool_t mask = svptrue_pat_b64(SV_VL2); svuint64_t acc0 = svld1_u64(mask, xacc + 0); svuint64_t acc1 = svld1_u64(mask, xacc + 2); svuint64_t acc2 = svld1_u64(mask, xacc + 4); svuint64_t acc3 = svld1_u64(mask, xacc + 6); do { svprfd(mask, xinput + 128, SV_PLDL1STRM); ACCRND(acc0, 0); ACCRND(acc1, 2); ACCRND(acc2, 4); ACCRND(acc3, 6); xinput += 8; xsecret += 1; nbStripes--; } while (nbStripes != 0); svst1_u64(mask, xacc + 0, acc0); svst1_u64(mask, xacc + 2, acc1); svst1_u64(mask, xacc + 4, acc2); svst1_u64(mask, xacc + 6, acc3); } else { svbool_t mask = svptrue_pat_b64(SV_VL4); svuint64_t acc0 = svld1_u64(mask, xacc + 0); svuint64_t acc1 = svld1_u64(mask, xacc + 4); do { svprfd(mask, xinput + 128, SV_PLDL1STRM); ACCRND(acc0, 0); ACCRND(acc1, 4); xinput += 8; xsecret += 1; nbStripes--; } while (nbStripes != 0); svst1_u64(mask, xacc + 0, acc0); svst1_u64(mask, xacc + 4, acc1); } } } #endif /* scalar variants - universal */ #if defined(__aarch64__) && (defined(__GNUC__) || defined(__clang__)) /* * In XXH3_scalarRound(), GCC and Clang have a similar codegen issue, where they * emit an excess mask and a full 64-bit multiply-add (MADD X-form). * * While this might not seem like much, as AArch64 is a 64-bit architecture, only * big Cortex designs have a full 64-bit multiplier. * * On the little cores, the smaller 32-bit multiplier is used, and full 64-bit * multiplies expand to 2-3 multiplies in microcode. This has a major penalty * of up to 4 latency cycles and 2 stall cycles in the multiply pipeline. * * Thankfully, AArch64 still provides the 32-bit long multiply-add (UMADDL) which does * not have this penalty and does the mask automatically. */ XXH_FORCE_INLINE xxh_u64 XXH_mult32to64_add64(xxh_u64 lhs, xxh_u64 rhs, xxh_u64 acc) { xxh_u64 ret; /* note: %x = 64-bit register, %w = 32-bit register */ __asm__("umaddl %x0, %w1, %w2, %x3" : "=r" (ret) : "r" (lhs), "r" (rhs), "r" (acc)); return ret; } #else XXH_FORCE_INLINE xxh_u64 XXH_mult32to64_add64(xxh_u64 lhs, xxh_u64 rhs, xxh_u64 acc) { return XXH_mult32to64((xxh_u32)lhs, (xxh_u32)rhs) + acc; } #endif /*! * @internal * @brief Scalar round for @ref XXH3_accumulate_512_scalar(). * * This is extracted to its own function because the NEON path uses a combination * of NEON and scalar. */ XXH_FORCE_INLINE void XXH3_scalarRound(void* XXH_RESTRICT acc, void const* XXH_RESTRICT input, void const* XXH_RESTRICT secret, size_t lane) { xxh_u64* xacc = (xxh_u64*) acc; xxh_u8 const* xinput = (xxh_u8 const*) input; xxh_u8 const* xsecret = (xxh_u8 const*) secret; XXH_ASSERT(lane < XXH_ACC_NB); XXH_ASSERT(((size_t)acc & (XXH_ACC_ALIGN-1)) == 0); { xxh_u64 const data_val = XXH_readLE64(xinput + lane * 8); xxh_u64 const data_key = data_val ^ XXH_readLE64(xsecret + lane * 8); xacc[lane ^ 1] += data_val; /* swap adjacent lanes */ xacc[lane] = XXH_mult32to64_add64(data_key /* & 0xFFFFFFFF */, data_key >> 32, xacc[lane]); } } /*! * @internal * @brief Processes a 64 byte block of data using the scalar path. */ XXH_FORCE_INLINE void XXH3_accumulate_512_scalar(void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { size_t i; /* ARM GCC refuses to unroll this loop, resulting in a 24% slowdown on ARMv6. */ #if defined(__GNUC__) && !defined(__clang__) \ && (defined(__arm__) || defined(__thumb2__)) \ && defined(__ARM_FEATURE_UNALIGNED) /* no unaligned access just wastes bytes */ \ && XXH_SIZE_OPT <= 0 # pragma GCC unroll 8 #endif for (i=0; i < XXH_ACC_NB; i++) { XXH3_scalarRound(acc, input, secret, i); } } XXH_FORCE_INLINE XXH3_ACCUMULATE_TEMPLATE(scalar) /*! * @internal * @brief Scalar scramble step for @ref XXH3_scrambleAcc_scalar(). * * This is extracted to its own function because the NEON path uses a combination * of NEON and scalar. */ XXH_FORCE_INLINE void XXH3_scalarScrambleRound(void* XXH_RESTRICT acc, void const* XXH_RESTRICT secret, size_t lane) { xxh_u64* const xacc = (xxh_u64*) acc; /* presumed aligned */ const xxh_u8* const xsecret = (const xxh_u8*) secret; /* no alignment restriction */ XXH_ASSERT((((size_t)acc) & (XXH_ACC_ALIGN-1)) == 0); XXH_ASSERT(lane < XXH_ACC_NB); { xxh_u64 const key64 = XXH_readLE64(xsecret + lane * 8); xxh_u64 acc64 = xacc[lane]; acc64 = XXH_xorshift64(acc64, 47); acc64 ^= key64; acc64 *= XXH_PRIME32_1; xacc[lane] = acc64; } } /*! * @internal * @brief Scrambles the accumulators after a large chunk has been read */ XXH_FORCE_INLINE void XXH3_scrambleAcc_scalar(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { size_t i; for (i=0; i < XXH_ACC_NB; i++) { XXH3_scalarScrambleRound(acc, secret, i); } } XXH_FORCE_INLINE void XXH3_initCustomSecret_scalar(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { /* * We need a separate pointer for the hack below, * which requires a non-const pointer. * Any decent compiler will optimize this out otherwise. */ const xxh_u8* kSecretPtr = XXH3_kSecret; XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 15) == 0); #if defined(__GNUC__) && defined(__aarch64__) /* * UGLY HACK: * GCC and Clang generate a bunch of MOV/MOVK pairs for aarch64, and they are * placed sequentially, in order, at the top of the unrolled loop. * * While MOVK is great for generating constants (2 cycles for a 64-bit * constant compared to 4 cycles for LDR), it fights for bandwidth with * the arithmetic instructions. * * I L S * MOVK * MOVK * MOVK * MOVK * ADD * SUB STR * STR * By forcing loads from memory (as the asm line causes the compiler to assume * that XXH3_kSecretPtr has been changed), the pipelines are used more * efficiently: * I L S * LDR * ADD LDR * SUB STR * STR * * See XXH3_NEON_LANES for details on the pipsline. * * XXH3_64bits_withSeed, len == 256, Snapdragon 835 * without hack: 2654.4 MB/s * with hack: 3202.9 MB/s */ XXH_COMPILER_GUARD(kSecretPtr); #endif { int const nbRounds = XXH_SECRET_DEFAULT_SIZE / 16; int i; for (i=0; i < nbRounds; i++) { /* * The asm hack causes the compiler to assume that kSecretPtr aliases with * customSecret, and on aarch64, this prevented LDP from merging two * loads together for free. Putting the loads together before the stores * properly generates LDP. */ xxh_u64 lo = XXH_readLE64(kSecretPtr + 16*i) + seed64; xxh_u64 hi = XXH_readLE64(kSecretPtr + 16*i + 8) - seed64; XXH_writeLE64((xxh_u8*)customSecret + 16*i, lo); XXH_writeLE64((xxh_u8*)customSecret + 16*i + 8, hi); } } } typedef void (*XXH3_f_accumulate)(xxh_u64* XXH_RESTRICT, const xxh_u8* XXH_RESTRICT, const xxh_u8* XXH_RESTRICT, size_t); typedef void (*XXH3_f_scrambleAcc)(void* XXH_RESTRICT, const void*); typedef void (*XXH3_f_initCustomSecret)(void* XXH_RESTRICT, xxh_u64); #if (XXH_VECTOR == XXH_AVX512) #define XXH3_accumulate_512 XXH3_accumulate_512_avx512 #define XXH3_accumulate XXH3_accumulate_avx512 #define XXH3_scrambleAcc XXH3_scrambleAcc_avx512 #define XXH3_initCustomSecret XXH3_initCustomSecret_avx512 #elif (XXH_VECTOR == XXH_AVX2) #define XXH3_accumulate_512 XXH3_accumulate_512_avx2 #define XXH3_accumulate XXH3_accumulate_avx2 #define XXH3_scrambleAcc XXH3_scrambleAcc_avx2 #define XXH3_initCustomSecret XXH3_initCustomSecret_avx2 #elif (XXH_VECTOR == XXH_SSE2) #define XXH3_accumulate_512 XXH3_accumulate_512_sse2 #define XXH3_accumulate XXH3_accumulate_sse2 #define XXH3_scrambleAcc XXH3_scrambleAcc_sse2 #define XXH3_initCustomSecret XXH3_initCustomSecret_sse2 #elif (XXH_VECTOR == XXH_NEON) #define XXH3_accumulate_512 XXH3_accumulate_512_neon #define XXH3_accumulate XXH3_accumulate_neon #define XXH3_scrambleAcc XXH3_scrambleAcc_neon #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #elif (XXH_VECTOR == XXH_VSX) #define XXH3_accumulate_512 XXH3_accumulate_512_vsx #define XXH3_accumulate XXH3_accumulate_vsx #define XXH3_scrambleAcc XXH3_scrambleAcc_vsx #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #elif (XXH_VECTOR == XXH_SVE) #define XXH3_accumulate_512 XXH3_accumulate_512_sve #define XXH3_accumulate XXH3_accumulate_sve #define XXH3_scrambleAcc XXH3_scrambleAcc_scalar #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #else /* scalar */ #define XXH3_accumulate_512 XXH3_accumulate_512_scalar #define XXH3_accumulate XXH3_accumulate_scalar #define XXH3_scrambleAcc XXH3_scrambleAcc_scalar #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #endif #if XXH_SIZE_OPT >= 1 /* don't do SIMD for initialization */ # undef XXH3_initCustomSecret # define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #endif XXH_FORCE_INLINE void XXH3_hashLong_internal_loop(xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble) { size_t const nbStripesPerBlock = (secretSize - XXH_STRIPE_LEN) / XXH_SECRET_CONSUME_RATE; size_t const block_len = XXH_STRIPE_LEN * nbStripesPerBlock; size_t const nb_blocks = (len - 1) / block_len; size_t n; XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); for (n = 0; n < nb_blocks; n++) { f_acc(acc, input + n*block_len, secret, nbStripesPerBlock); f_scramble(acc, secret + secretSize - XXH_STRIPE_LEN); } /* last partial block */ XXH_ASSERT(len > XXH_STRIPE_LEN); { size_t const nbStripes = ((len - 1) - (block_len * nb_blocks)) / XXH_STRIPE_LEN; XXH_ASSERT(nbStripes <= (secretSize / XXH_SECRET_CONSUME_RATE)); f_acc(acc, input + nb_blocks*block_len, secret, nbStripes); /* last stripe */ { const xxh_u8* const p = input + len - XXH_STRIPE_LEN; #define XXH_SECRET_LASTACC_START 7 /* not aligned on 8, last secret is different from acc & scrambler */ XXH3_accumulate_512(acc, p, secret + secretSize - XXH_STRIPE_LEN - XXH_SECRET_LASTACC_START); } } } XXH_FORCE_INLINE xxh_u64 XXH3_mix2Accs(const xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT secret) { return XXH3_mul128_fold64( acc[0] ^ XXH_readLE64(secret), acc[1] ^ XXH_readLE64(secret+8) ); } static XXH64_hash_t XXH3_mergeAccs(const xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT secret, xxh_u64 start) { xxh_u64 result64 = start; size_t i = 0; for (i = 0; i < 4; i++) { result64 += XXH3_mix2Accs(acc+2*i, secret + 16*i); #if defined(__clang__) /* Clang */ \ && (defined(__arm__) || defined(__thumb__)) /* ARMv7 */ \ && (defined(__ARM_NEON) || defined(__ARM_NEON__)) /* NEON */ \ && !defined(XXH_ENABLE_AUTOVECTORIZE) /* Define to disable */ /* * UGLY HACK: * Prevent autovectorization on Clang ARMv7-a. Exact same problem as * the one in XXH3_len_129to240_64b. Speeds up shorter keys > 240b. * XXH3_64bits, len == 256, Snapdragon 835: * without hack: 2063.7 MB/s * with hack: 2560.7 MB/s */ XXH_COMPILER_GUARD(result64); #endif } return XXH3_avalanche(result64); } #define XXH3_INIT_ACC { XXH_PRIME32_3, XXH_PRIME64_1, XXH_PRIME64_2, XXH_PRIME64_3, \ XXH_PRIME64_4, XXH_PRIME32_2, XXH_PRIME64_5, XXH_PRIME32_1 } XXH_FORCE_INLINE XXH64_hash_t XXH3_hashLong_64b_internal(const void* XXH_RESTRICT input, size_t len, const void* XXH_RESTRICT secret, size_t secretSize, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble) { XXH_ALIGN(XXH_ACC_ALIGN) xxh_u64 acc[XXH_ACC_NB] = XXH3_INIT_ACC; XXH3_hashLong_internal_loop(acc, (const xxh_u8*)input, len, (const xxh_u8*)secret, secretSize, f_acc, f_scramble); /* converge into final hash */ XXH_STATIC_ASSERT(sizeof(acc) == 64); /* do not align on 8, so that the secret is different from the accumulator */ #define XXH_SECRET_MERGEACCS_START 11 XXH_ASSERT(secretSize >= sizeof(acc) + XXH_SECRET_MERGEACCS_START); return XXH3_mergeAccs(acc, (const xxh_u8*)secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)len * XXH_PRIME64_1); } /* * It's important for performance to transmit secret's size (when it's static) * so that the compiler can properly optimize the vectorized loop. * This makes a big performance difference for "medium" keys (<1 KB) when using AVX instruction set. * When the secret size is unknown, or on GCC 12 where the mix of NO_INLINE and FORCE_INLINE * breaks -Og, this is XXH_NO_INLINE. */ XXH3_WITH_SECRET_INLINE XXH64_hash_t XXH3_hashLong_64b_withSecret(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const xxh_u8* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; return XXH3_hashLong_64b_internal(input, len, secret, secretLen, XXH3_accumulate, XXH3_scrambleAcc); } /* * It's preferable for performance that XXH3_hashLong is not inlined, * as it results in a smaller function for small data, easier to the instruction cache. * Note that inside this no_inline function, we do inline the internal loop, * and provide a statically defined secret size to allow optimization of vector loop. */ XXH_NO_INLINE XXH_PUREF XXH64_hash_t XXH3_hashLong_64b_default(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const xxh_u8* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; (void)secret; (void)secretLen; return XXH3_hashLong_64b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_accumulate, XXH3_scrambleAcc); } /* * XXH3_hashLong_64b_withSeed(): * Generate a custom key based on alteration of default XXH3_kSecret with the seed, * and then use this key for long mode hashing. * * This operation is decently fast but nonetheless costs a little bit of time. * Try to avoid it whenever possible (typically when seed==0). * * It's important for performance that XXH3_hashLong is not inlined. Not sure * why (uop cache maybe?), but the difference is large and easily measurable. */ XXH_FORCE_INLINE XXH64_hash_t XXH3_hashLong_64b_withSeed_internal(const void* input, size_t len, XXH64_hash_t seed, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble, XXH3_f_initCustomSecret f_initSec) { #if XXH_SIZE_OPT <= 0 if (seed == 0) return XXH3_hashLong_64b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), f_acc, f_scramble); #endif { XXH_ALIGN(XXH_SEC_ALIGN) xxh_u8 secret[XXH_SECRET_DEFAULT_SIZE]; f_initSec(secret, seed); return XXH3_hashLong_64b_internal(input, len, secret, sizeof(secret), f_acc, f_scramble); } } /* * It's important for performance that XXH3_hashLong is not inlined. */ XXH_NO_INLINE XXH64_hash_t XXH3_hashLong_64b_withSeed(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed, const xxh_u8* XXH_RESTRICT secret, size_t secretLen) { (void)secret; (void)secretLen; return XXH3_hashLong_64b_withSeed_internal(input, len, seed, XXH3_accumulate, XXH3_scrambleAcc, XXH3_initCustomSecret); } typedef XXH64_hash_t (*XXH3_hashLong64_f)(const void* XXH_RESTRICT, size_t, XXH64_hash_t, const xxh_u8* XXH_RESTRICT, size_t); XXH_FORCE_INLINE XXH64_hash_t XXH3_64bits_internal(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen, XXH3_hashLong64_f f_hashLong) { XXH_ASSERT(secretLen >= XXH3_SECRET_SIZE_MIN); /* * If an action is to be taken if `secretLen` condition is not respected, * it should be done here. * For now, it's a contract pre-condition. * Adding a check and a branch here would cost performance at every hash. * Also, note that function signature doesn't offer room to return an error. */ if (len <= 16) return XXH3_len_0to16_64b((const xxh_u8*)input, len, (const xxh_u8*)secret, seed64); if (len <= 128) return XXH3_len_17to128_64b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); if (len <= XXH3_MIDSIZE_MAX) return XXH3_len_129to240_64b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); return f_hashLong(input, len, seed64, (const xxh_u8*)secret, secretLen); } /* === Public entry point === */ /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits(XXH_NOESCAPE const void* input, size_t length) { return XXH3_64bits_internal(input, length, 0, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_64b_default); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSecret(XXH_NOESCAPE const void* input, size_t length, XXH_NOESCAPE const void* secret, size_t secretSize) { return XXH3_64bits_internal(input, length, 0, secret, secretSize, XXH3_hashLong_64b_withSecret); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSeed(XXH_NOESCAPE const void* input, size_t length, XXH64_hash_t seed) { return XXH3_64bits_internal(input, length, seed, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_64b_withSeed); } XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSecretandSeed(XXH_NOESCAPE const void* input, size_t length, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed) { if (length <= XXH3_MIDSIZE_MAX) return XXH3_64bits_internal(input, length, seed, XXH3_kSecret, sizeof(XXH3_kSecret), NULL); return XXH3_hashLong_64b_withSecret(input, length, seed, (const xxh_u8*)secret, secretSize); } /* === XXH3 streaming === */ #ifndef XXH_NO_STREAM /* * Malloc's a pointer that is always aligned to align. * * This must be freed with `XXH_alignedFree()`. * * malloc typically guarantees 16 byte alignment on 64-bit systems and 8 byte * alignment on 32-bit. This isn't enough for the 32 byte aligned loads in AVX2 * or on 32-bit, the 16 byte aligned loads in SSE2 and NEON. * * This underalignment previously caused a rather obvious crash which went * completely unnoticed due to XXH3_createState() not actually being tested. * Credit to RedSpah for noticing this bug. * * The alignment is done manually: Functions like posix_memalign or _mm_malloc * are avoided: To maintain portability, we would have to write a fallback * like this anyways, and besides, testing for the existence of library * functions without relying on external build tools is impossible. * * The method is simple: Overallocate, manually align, and store the offset * to the original behind the returned pointer. * * Align must be a power of 2 and 8 <= align <= 128. */ static XXH_MALLOCF void* XXH_alignedMalloc(size_t s, size_t align) { XXH_ASSERT(align <= 128 && align >= 8); /* range check */ XXH_ASSERT((align & (align-1)) == 0); /* power of 2 */ XXH_ASSERT(s != 0 && s < (s + align)); /* empty/overflow */ { /* Overallocate to make room for manual realignment and an offset byte */ xxh_u8* base = (xxh_u8*)XXH_malloc(s + align); if (base != NULL) { /* * Get the offset needed to align this pointer. * * Even if the returned pointer is aligned, there will always be * at least one byte to store the offset to the original pointer. */ size_t offset = align - ((size_t)base & (align - 1)); /* base % align */ /* Add the offset for the now-aligned pointer */ xxh_u8* ptr = base + offset; XXH_ASSERT((size_t)ptr % align == 0); /* Store the offset immediately before the returned pointer. */ ptr[-1] = (xxh_u8)offset; return ptr; } return NULL; } } /* * Frees an aligned pointer allocated by XXH_alignedMalloc(). Don't pass * normal malloc'd pointers, XXH_alignedMalloc has a specific data layout. */ static void XXH_alignedFree(void* p) { if (p != NULL) { xxh_u8* ptr = (xxh_u8*)p; /* Get the offset byte we added in XXH_malloc. */ xxh_u8 offset = ptr[-1]; /* Free the original malloc'd pointer */ xxh_u8* base = ptr - offset; XXH_free(base); } } /*! @ingroup XXH3_family */ /*! * @brief Allocate an @ref XXH3_state_t. * * Must be freed with XXH3_freeState(). * @return An allocated XXH3_state_t on success, `NULL` on failure. */ XXH_PUBLIC_API XXH3_state_t* XXH3_createState(void) { XXH3_state_t* const state = (XXH3_state_t*)XXH_alignedMalloc(sizeof(XXH3_state_t), 64); if (state==NULL) return NULL; XXH3_INITSTATE(state); return state; } /*! @ingroup XXH3_family */ /*! * @brief Frees an @ref XXH3_state_t. * * Must be allocated with XXH3_createState(). * @param statePtr A pointer to an @ref XXH3_state_t allocated with @ref XXH3_createState(). * @return XXH_OK. */ XXH_PUBLIC_API XXH_errorcode XXH3_freeState(XXH3_state_t* statePtr) { XXH_alignedFree(statePtr); return XXH_OK; } /*! @ingroup XXH3_family */ XXH_PUBLIC_API void XXH3_copyState(XXH_NOESCAPE XXH3_state_t* dst_state, XXH_NOESCAPE const XXH3_state_t* src_state) { XXH_memcpy(dst_state, src_state, sizeof(*dst_state)); } static void XXH3_reset_internal(XXH3_state_t* statePtr, XXH64_hash_t seed, const void* secret, size_t secretSize) { size_t const initStart = offsetof(XXH3_state_t, bufferedSize); size_t const initLength = offsetof(XXH3_state_t, nbStripesPerBlock) - initStart; XXH_ASSERT(offsetof(XXH3_state_t, nbStripesPerBlock) > initStart); XXH_ASSERT(statePtr != NULL); /* set members from bufferedSize to nbStripesPerBlock (excluded) to 0 */ memset((char*)statePtr + initStart, 0, initLength); statePtr->acc[0] = XXH_PRIME32_3; statePtr->acc[1] = XXH_PRIME64_1; statePtr->acc[2] = XXH_PRIME64_2; statePtr->acc[3] = XXH_PRIME64_3; statePtr->acc[4] = XXH_PRIME64_4; statePtr->acc[5] = XXH_PRIME32_2; statePtr->acc[6] = XXH_PRIME64_5; statePtr->acc[7] = XXH_PRIME32_1; statePtr->seed = seed; statePtr->useSeed = (seed != 0); statePtr->extSecret = (const unsigned char*)secret; XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); statePtr->secretLimit = secretSize - XXH_STRIPE_LEN; statePtr->nbStripesPerBlock = statePtr->secretLimit / XXH_SECRET_CONSUME_RATE; } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset(XXH_NOESCAPE XXH3_state_t* statePtr) { if (statePtr == NULL) return XXH_ERROR; XXH3_reset_internal(statePtr, 0, XXH3_kSecret, XXH_SECRET_DEFAULT_SIZE); return XXH_OK; } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecret(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize) { if (statePtr == NULL) return XXH_ERROR; XXH3_reset_internal(statePtr, 0, secret, secretSize); if (secret == NULL) return XXH_ERROR; if (secretSize < XXH3_SECRET_SIZE_MIN) return XXH_ERROR; return XXH_OK; } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH64_hash_t seed) { if (statePtr == NULL) return XXH_ERROR; if (seed==0) return XXH3_64bits_reset(statePtr); if ((seed != statePtr->seed) || (statePtr->extSecret != NULL)) XXH3_initCustomSecret(statePtr->customSecret, seed); XXH3_reset_internal(statePtr, seed, NULL, XXH_SECRET_DEFAULT_SIZE); return XXH_OK; } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecretandSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed64) { if (statePtr == NULL) return XXH_ERROR; if (secret == NULL) return XXH_ERROR; if (secretSize < XXH3_SECRET_SIZE_MIN) return XXH_ERROR; XXH3_reset_internal(statePtr, seed64, secret, secretSize); statePtr->useSeed = 1; /* always, even if seed64==0 */ return XXH_OK; } /*! * @internal * @brief Processes a large input for XXH3_update() and XXH3_digest_long(). * * Unlike XXH3_hashLong_internal_loop(), this can process data that overlaps a block. * * @param acc Pointer to the 8 accumulator lanes * @param nbStripesSoFarPtr In/out pointer to the number of leftover stripes in the block* * @param nbStripesPerBlock Number of stripes in a block * @param input Input pointer * @param nbStripes Number of stripes to process * @param secret Secret pointer * @param secretLimit Offset of the last block in @p secret * @param f_acc Pointer to an XXH3_accumulate implementation * @param f_scramble Pointer to an XXH3_scrambleAcc implementation * @return Pointer past the end of @p input after processing */ XXH_FORCE_INLINE const xxh_u8 * XXH3_consumeStripes(xxh_u64* XXH_RESTRICT acc, size_t* XXH_RESTRICT nbStripesSoFarPtr, size_t nbStripesPerBlock, const xxh_u8* XXH_RESTRICT input, size_t nbStripes, const xxh_u8* XXH_RESTRICT secret, size_t secretLimit, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble) { const xxh_u8* initialSecret = secret + *nbStripesSoFarPtr * XXH_SECRET_CONSUME_RATE; /* Process full blocks */ if (nbStripes >= (nbStripesPerBlock - *nbStripesSoFarPtr)) { /* Process the initial partial block... */ size_t nbStripesThisIter = nbStripesPerBlock - *nbStripesSoFarPtr; do { /* Accumulate and scramble */ f_acc(acc, input, initialSecret, nbStripesThisIter); f_scramble(acc, secret + secretLimit); input += nbStripesThisIter * XXH_STRIPE_LEN; nbStripes -= nbStripesThisIter; /* Then continue the loop with the full block size */ nbStripesThisIter = nbStripesPerBlock; initialSecret = secret; } while (nbStripes >= nbStripesPerBlock); *nbStripesSoFarPtr = 0; } /* Process a partial block */ if (nbStripes > 0) { f_acc(acc, input, initialSecret, nbStripes); input += nbStripes * XXH_STRIPE_LEN; *nbStripesSoFarPtr += nbStripes; } /* Return end pointer */ return input; } #ifndef XXH3_STREAM_USE_STACK # if XXH_SIZE_OPT <= 0 && !defined(__clang__) /* clang doesn't need additional stack space */ # define XXH3_STREAM_USE_STACK 1 # endif #endif /* * Both XXH3_64bits_update and XXH3_128bits_update use this routine. */ XXH_FORCE_INLINE XXH_errorcode XXH3_update(XXH3_state_t* XXH_RESTRICT const state, const xxh_u8* XXH_RESTRICT input, size_t len, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble) { if (input==NULL) { XXH_ASSERT(len == 0); return XXH_OK; } XXH_ASSERT(state != NULL); { const xxh_u8* const bEnd = input + len; const unsigned char* const secret = (state->extSecret == NULL) ? state->customSecret : state->extSecret; #if defined(XXH3_STREAM_USE_STACK) && XXH3_STREAM_USE_STACK >= 1 /* For some reason, gcc and MSVC seem to suffer greatly * when operating accumulators directly into state. * Operating into stack space seems to enable proper optimization. * clang, on the other hand, doesn't seem to need this trick */ XXH_ALIGN(XXH_ACC_ALIGN) xxh_u64 acc[8]; XXH_memcpy(acc, state->acc, sizeof(acc)); #else xxh_u64* XXH_RESTRICT const acc = state->acc; #endif state->totalLen += len; XXH_ASSERT(state->bufferedSize <= XXH3_INTERNALBUFFER_SIZE); /* small input : just fill in tmp buffer */ if (len <= XXH3_INTERNALBUFFER_SIZE - state->bufferedSize) { XXH_memcpy(state->buffer + state->bufferedSize, input, len); state->bufferedSize += (XXH32_hash_t)len; return XXH_OK; } /* total input is now > XXH3_INTERNALBUFFER_SIZE */ #define XXH3_INTERNALBUFFER_STRIPES (XXH3_INTERNALBUFFER_SIZE / XXH_STRIPE_LEN) XXH_STATIC_ASSERT(XXH3_INTERNALBUFFER_SIZE % XXH_STRIPE_LEN == 0); /* clean multiple */ /* * Internal buffer is partially filled (always, except at beginning) * Complete it, then consume it. */ if (state->bufferedSize) { size_t const loadSize = XXH3_INTERNALBUFFER_SIZE - state->bufferedSize; XXH_memcpy(state->buffer + state->bufferedSize, input, loadSize); input += loadSize; XXH3_consumeStripes(acc, &state->nbStripesSoFar, state->nbStripesPerBlock, state->buffer, XXH3_INTERNALBUFFER_STRIPES, secret, state->secretLimit, f_acc, f_scramble); state->bufferedSize = 0; } XXH_ASSERT(input < bEnd); if (bEnd - input > XXH3_INTERNALBUFFER_SIZE) { size_t nbStripes = (size_t)(bEnd - 1 - input) / XXH_STRIPE_LEN; input = XXH3_consumeStripes(acc, &state->nbStripesSoFar, state->nbStripesPerBlock, input, nbStripes, secret, state->secretLimit, f_acc, f_scramble); XXH_memcpy(state->buffer + sizeof(state->buffer) - XXH_STRIPE_LEN, input - XXH_STRIPE_LEN, XXH_STRIPE_LEN); } /* Some remaining input (always) : buffer it */ XXH_ASSERT(input < bEnd); XXH_ASSERT(bEnd - input <= XXH3_INTERNALBUFFER_SIZE); XXH_ASSERT(state->bufferedSize == 0); XXH_memcpy(state->buffer, input, (size_t)(bEnd-input)); state->bufferedSize = (XXH32_hash_t)(bEnd-input); #if defined(XXH3_STREAM_USE_STACK) && XXH3_STREAM_USE_STACK >= 1 /* save stack accumulators into state */ XXH_memcpy(state->acc, acc, sizeof(acc)); #endif } return XXH_OK; } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_update(XXH_NOESCAPE XXH3_state_t* state, XXH_NOESCAPE const void* input, size_t len) { return XXH3_update(state, (const xxh_u8*)input, len, XXH3_accumulate, XXH3_scrambleAcc); } XXH_FORCE_INLINE void XXH3_digest_long (XXH64_hash_t* acc, const XXH3_state_t* state, const unsigned char* secret) { xxh_u8 lastStripe[XXH_STRIPE_LEN]; const xxh_u8* lastStripePtr; /* * Digest on a local copy. This way, the state remains unaltered, and it can * continue ingesting more input afterwards. */ XXH_memcpy(acc, state->acc, sizeof(state->acc)); if (state->bufferedSize >= XXH_STRIPE_LEN) { /* Consume remaining stripes then point to remaining data in buffer */ size_t const nbStripes = (state->bufferedSize - 1) / XXH_STRIPE_LEN; size_t nbStripesSoFar = state->nbStripesSoFar; XXH3_consumeStripes(acc, &nbStripesSoFar, state->nbStripesPerBlock, state->buffer, nbStripes, secret, state->secretLimit, XXH3_accumulate, XXH3_scrambleAcc); lastStripePtr = state->buffer + state->bufferedSize - XXH_STRIPE_LEN; } else { /* bufferedSize < XXH_STRIPE_LEN */ /* Copy to temp buffer */ size_t const catchupSize = XXH_STRIPE_LEN - state->bufferedSize; XXH_ASSERT(state->bufferedSize > 0); /* there is always some input buffered */ XXH_memcpy(lastStripe, state->buffer + sizeof(state->buffer) - catchupSize, catchupSize); XXH_memcpy(lastStripe + catchupSize, state->buffer, state->bufferedSize); lastStripePtr = lastStripe; } /* Last stripe */ XXH3_accumulate_512(acc, lastStripePtr, secret + state->secretLimit - XXH_SECRET_LASTACC_START); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_digest (XXH_NOESCAPE const XXH3_state_t* state) { const unsigned char* const secret = (state->extSecret == NULL) ? state->customSecret : state->extSecret; if (state->totalLen > XXH3_MIDSIZE_MAX) { XXH_ALIGN(XXH_ACC_ALIGN) XXH64_hash_t acc[XXH_ACC_NB]; XXH3_digest_long(acc, state, secret); return XXH3_mergeAccs(acc, secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)state->totalLen * XXH_PRIME64_1); } /* totalLen <= XXH3_MIDSIZE_MAX: digesting a short input */ if (state->useSeed) return XXH3_64bits_withSeed(state->buffer, (size_t)state->totalLen, state->seed); return XXH3_64bits_withSecret(state->buffer, (size_t)(state->totalLen), secret, state->secretLimit + XXH_STRIPE_LEN); } #endif /* !XXH_NO_STREAM */ /* ========================================== * XXH3 128 bits (a.k.a XXH128) * ========================================== * XXH3's 128-bit variant has better mixing and strength than the 64-bit variant, * even without counting the significantly larger output size. * * For example, extra steps are taken to avoid the seed-dependent collisions * in 17-240 byte inputs (See XXH3_mix16B and XXH128_mix32B). * * This strength naturally comes at the cost of some speed, especially on short * lengths. Note that longer hashes are about as fast as the 64-bit version * due to it using only a slight modification of the 64-bit loop. * * XXH128 is also more oriented towards 64-bit machines. It is still extremely * fast for a _128-bit_ hash on 32-bit (it usually clears XXH64). */ XXH_FORCE_INLINE XXH_PUREF XXH128_hash_t XXH3_len_1to3_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { /* A doubled version of 1to3_64b with different constants. */ XXH_ASSERT(input != NULL); XXH_ASSERT(1 <= len && len <= 3); XXH_ASSERT(secret != NULL); /* * len = 1: combinedl = { input[0], 0x01, input[0], input[0] } * len = 2: combinedl = { input[1], 0x02, input[0], input[1] } * len = 3: combinedl = { input[2], 0x03, input[0], input[1] } */ { xxh_u8 const c1 = input[0]; xxh_u8 const c2 = input[len >> 1]; xxh_u8 const c3 = input[len - 1]; xxh_u32 const combinedl = ((xxh_u32)c1 <<16) | ((xxh_u32)c2 << 24) | ((xxh_u32)c3 << 0) | ((xxh_u32)len << 8); xxh_u32 const combinedh = XXH_rotl32(XXH_swap32(combinedl), 13); xxh_u64 const bitflipl = (XXH_readLE32(secret) ^ XXH_readLE32(secret+4)) + seed; xxh_u64 const bitfliph = (XXH_readLE32(secret+8) ^ XXH_readLE32(secret+12)) - seed; xxh_u64 const keyed_lo = (xxh_u64)combinedl ^ bitflipl; xxh_u64 const keyed_hi = (xxh_u64)combinedh ^ bitfliph; XXH128_hash_t h128; h128.low64 = XXH64_avalanche(keyed_lo); h128.high64 = XXH64_avalanche(keyed_hi); return h128; } } XXH_FORCE_INLINE XXH_PUREF XXH128_hash_t XXH3_len_4to8_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(4 <= len && len <= 8); seed ^= (xxh_u64)XXH_swap32((xxh_u32)seed) << 32; { xxh_u32 const input_lo = XXH_readLE32(input); xxh_u32 const input_hi = XXH_readLE32(input + len - 4); xxh_u64 const input_64 = input_lo + ((xxh_u64)input_hi << 32); xxh_u64 const bitflip = (XXH_readLE64(secret+16) ^ XXH_readLE64(secret+24)) + seed; xxh_u64 const keyed = input_64 ^ bitflip; /* Shift len to the left to ensure it is even, this avoids even multiplies. */ XXH128_hash_t m128 = XXH_mult64to128(keyed, XXH_PRIME64_1 + (len << 2)); m128.high64 += (m128.low64 << 1); m128.low64 ^= (m128.high64 >> 3); m128.low64 = XXH_xorshift64(m128.low64, 35); m128.low64 *= PRIME_MX2; m128.low64 = XXH_xorshift64(m128.low64, 28); m128.high64 = XXH3_avalanche(m128.high64); return m128; } } XXH_FORCE_INLINE XXH_PUREF XXH128_hash_t XXH3_len_9to16_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(9 <= len && len <= 16); { xxh_u64 const bitflipl = (XXH_readLE64(secret+32) ^ XXH_readLE64(secret+40)) - seed; xxh_u64 const bitfliph = (XXH_readLE64(secret+48) ^ XXH_readLE64(secret+56)) + seed; xxh_u64 const input_lo = XXH_readLE64(input); xxh_u64 input_hi = XXH_readLE64(input + len - 8); XXH128_hash_t m128 = XXH_mult64to128(input_lo ^ input_hi ^ bitflipl, XXH_PRIME64_1); /* * Put len in the middle of m128 to ensure that the length gets mixed to * both the low and high bits in the 128x64 multiply below. */ m128.low64 += (xxh_u64)(len - 1) << 54; input_hi ^= bitfliph; /* * Add the high 32 bits of input_hi to the high 32 bits of m128, then * add the long product of the low 32 bits of input_hi and XXH_PRIME32_2 to * the high 64 bits of m128. * * The best approach to this operation is different on 32-bit and 64-bit. */ if (sizeof(void *) < sizeof(xxh_u64)) { /* 32-bit */ /* * 32-bit optimized version, which is more readable. * * On 32-bit, it removes an ADC and delays a dependency between the two * halves of m128.high64, but it generates an extra mask on 64-bit. */ m128.high64 += (input_hi & 0xFFFFFFFF00000000ULL) + XXH_mult32to64((xxh_u32)input_hi, XXH_PRIME32_2); } else { /* * 64-bit optimized (albeit more confusing) version. * * Uses some properties of addition and multiplication to remove the mask: * * Let: * a = input_hi.lo = (input_hi & 0x00000000FFFFFFFF) * b = input_hi.hi = (input_hi & 0xFFFFFFFF00000000) * c = XXH_PRIME32_2 * * a + (b * c) * Inverse Property: x + y - x == y * a + (b * (1 + c - 1)) * Distributive Property: x * (y + z) == (x * y) + (x * z) * a + (b * 1) + (b * (c - 1)) * Identity Property: x * 1 == x * a + b + (b * (c - 1)) * * Substitute a, b, and c: * input_hi.hi + input_hi.lo + ((xxh_u64)input_hi.lo * (XXH_PRIME32_2 - 1)) * * Since input_hi.hi + input_hi.lo == input_hi, we get this: * input_hi + ((xxh_u64)input_hi.lo * (XXH_PRIME32_2 - 1)) */ m128.high64 += input_hi + XXH_mult32to64((xxh_u32)input_hi, XXH_PRIME32_2 - 1); } /* m128 ^= XXH_swap64(m128 >> 64); */ m128.low64 ^= XXH_swap64(m128.high64); { /* 128x64 multiply: h128 = m128 * XXH_PRIME64_2; */ XXH128_hash_t h128 = XXH_mult64to128(m128.low64, XXH_PRIME64_2); h128.high64 += m128.high64 * XXH_PRIME64_2; h128.low64 = XXH3_avalanche(h128.low64); h128.high64 = XXH3_avalanche(h128.high64); return h128; } } } /* * Assumption: `secret` size is >= XXH3_SECRET_SIZE_MIN */ XXH_FORCE_INLINE XXH_PUREF XXH128_hash_t XXH3_len_0to16_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(len <= 16); { if (len > 8) return XXH3_len_9to16_128b(input, len, secret, seed); if (len >= 4) return XXH3_len_4to8_128b(input, len, secret, seed); if (len) return XXH3_len_1to3_128b(input, len, secret, seed); { XXH128_hash_t h128; xxh_u64 const bitflipl = XXH_readLE64(secret+64) ^ XXH_readLE64(secret+72); xxh_u64 const bitfliph = XXH_readLE64(secret+80) ^ XXH_readLE64(secret+88); h128.low64 = XXH64_avalanche(seed ^ bitflipl); h128.high64 = XXH64_avalanche( seed ^ bitfliph); return h128; } } } /* * A bit slower than XXH3_mix16B, but handles multiply by zero better. */ XXH_FORCE_INLINE XXH128_hash_t XXH128_mix32B(XXH128_hash_t acc, const xxh_u8* input_1, const xxh_u8* input_2, const xxh_u8* secret, XXH64_hash_t seed) { acc.low64 += XXH3_mix16B (input_1, secret+0, seed); acc.low64 ^= XXH_readLE64(input_2) + XXH_readLE64(input_2 + 8); acc.high64 += XXH3_mix16B (input_2, secret+16, seed); acc.high64 ^= XXH_readLE64(input_1) + XXH_readLE64(input_1 + 8); return acc; } XXH_FORCE_INLINE XXH_PUREF XXH128_hash_t XXH3_len_17to128_128b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(16 < len && len <= 128); { XXH128_hash_t acc; acc.low64 = len * XXH_PRIME64_1; acc.high64 = 0; #if XXH_SIZE_OPT >= 1 { /* Smaller, but slightly slower. */ unsigned int i = (unsigned int)(len - 1) / 32; do { acc = XXH128_mix32B(acc, input+16*i, input+len-16*(i+1), secret+32*i, seed); } while (i-- != 0); } #else if (len > 32) { if (len > 64) { if (len > 96) { acc = XXH128_mix32B(acc, input+48, input+len-64, secret+96, seed); } acc = XXH128_mix32B(acc, input+32, input+len-48, secret+64, seed); } acc = XXH128_mix32B(acc, input+16, input+len-32, secret+32, seed); } acc = XXH128_mix32B(acc, input, input+len-16, secret, seed); #endif { XXH128_hash_t h128; h128.low64 = acc.low64 + acc.high64; h128.high64 = (acc.low64 * XXH_PRIME64_1) + (acc.high64 * XXH_PRIME64_4) + ((len - seed) * XXH_PRIME64_2); h128.low64 = XXH3_avalanche(h128.low64); h128.high64 = (XXH64_hash_t)0 - XXH3_avalanche(h128.high64); return h128; } } } XXH_NO_INLINE XXH_PUREF XXH128_hash_t XXH3_len_129to240_128b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(128 < len && len <= XXH3_MIDSIZE_MAX); { XXH128_hash_t acc; unsigned i; acc.low64 = len * XXH_PRIME64_1; acc.high64 = 0; /* * We set as `i` as offset + 32. We do this so that unchanged * `len` can be used as upper bound. This reaches a sweet spot * where both x86 and aarch64 get simple agen and good codegen * for the loop. */ for (i = 32; i < 160; i += 32) { acc = XXH128_mix32B(acc, input + i - 32, input + i - 16, secret + i - 32, seed); } acc.low64 = XXH3_avalanche(acc.low64); acc.high64 = XXH3_avalanche(acc.high64); /* * NB: `i <= len` will duplicate the last 32-bytes if * len % 32 was zero. This is an unfortunate necessity to keep * the hash result stable. */ for (i=160; i <= len; i += 32) { acc = XXH128_mix32B(acc, input + i - 32, input + i - 16, secret + XXH3_MIDSIZE_STARTOFFSET + i - 160, seed); } /* last bytes */ acc = XXH128_mix32B(acc, input + len - 16, input + len - 32, secret + XXH3_SECRET_SIZE_MIN - XXH3_MIDSIZE_LASTOFFSET - 16, (XXH64_hash_t)0 - seed); { XXH128_hash_t h128; h128.low64 = acc.low64 + acc.high64; h128.high64 = (acc.low64 * XXH_PRIME64_1) + (acc.high64 * XXH_PRIME64_4) + ((len - seed) * XXH_PRIME64_2); h128.low64 = XXH3_avalanche(h128.low64); h128.high64 = (XXH64_hash_t)0 - XXH3_avalanche(h128.high64); return h128; } } } XXH_FORCE_INLINE XXH128_hash_t XXH3_hashLong_128b_internal(const void* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble) { XXH_ALIGN(XXH_ACC_ALIGN) xxh_u64 acc[XXH_ACC_NB] = XXH3_INIT_ACC; XXH3_hashLong_internal_loop(acc, (const xxh_u8*)input, len, secret, secretSize, f_acc, f_scramble); /* converge into final hash */ XXH_STATIC_ASSERT(sizeof(acc) == 64); XXH_ASSERT(secretSize >= sizeof(acc) + XXH_SECRET_MERGEACCS_START); { XXH128_hash_t h128; h128.low64 = XXH3_mergeAccs(acc, secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)len * XXH_PRIME64_1); h128.high64 = XXH3_mergeAccs(acc, secret + secretSize - sizeof(acc) - XXH_SECRET_MERGEACCS_START, ~((xxh_u64)len * XXH_PRIME64_2)); return h128; } } /* * It's important for performance that XXH3_hashLong() is not inlined. */ XXH_NO_INLINE XXH_PUREF XXH128_hash_t XXH3_hashLong_128b_default(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; (void)secret; (void)secretLen; return XXH3_hashLong_128b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_accumulate, XXH3_scrambleAcc); } /* * It's important for performance to pass @p secretLen (when it's static) * to the compiler, so that it can properly optimize the vectorized loop. * * When the secret size is unknown, or on GCC 12 where the mix of NO_INLINE and FORCE_INLINE * breaks -Og, this is XXH_NO_INLINE. */ XXH3_WITH_SECRET_INLINE XXH128_hash_t XXH3_hashLong_128b_withSecret(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; return XXH3_hashLong_128b_internal(input, len, (const xxh_u8*)secret, secretLen, XXH3_accumulate, XXH3_scrambleAcc); } XXH_FORCE_INLINE XXH128_hash_t XXH3_hashLong_128b_withSeed_internal(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble, XXH3_f_initCustomSecret f_initSec) { if (seed64 == 0) return XXH3_hashLong_128b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), f_acc, f_scramble); { XXH_ALIGN(XXH_SEC_ALIGN) xxh_u8 secret[XXH_SECRET_DEFAULT_SIZE]; f_initSec(secret, seed64); return XXH3_hashLong_128b_internal(input, len, (const xxh_u8*)secret, sizeof(secret), f_acc, f_scramble); } } /* * It's important for performance that XXH3_hashLong is not inlined. */ XXH_NO_INLINE XXH128_hash_t XXH3_hashLong_128b_withSeed(const void* input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen) { (void)secret; (void)secretLen; return XXH3_hashLong_128b_withSeed_internal(input, len, seed64, XXH3_accumulate, XXH3_scrambleAcc, XXH3_initCustomSecret); } typedef XXH128_hash_t (*XXH3_hashLong128_f)(const void* XXH_RESTRICT, size_t, XXH64_hash_t, const void* XXH_RESTRICT, size_t); XXH_FORCE_INLINE XXH128_hash_t XXH3_128bits_internal(const void* input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen, XXH3_hashLong128_f f_hl128) { XXH_ASSERT(secretLen >= XXH3_SECRET_SIZE_MIN); /* * If an action is to be taken if `secret` conditions are not respected, * it should be done here. * For now, it's a contract pre-condition. * Adding a check and a branch here would cost performance at every hash. */ if (len <= 16) return XXH3_len_0to16_128b((const xxh_u8*)input, len, (const xxh_u8*)secret, seed64); if (len <= 128) return XXH3_len_17to128_128b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); if (len <= XXH3_MIDSIZE_MAX) return XXH3_len_129to240_128b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); return f_hl128(input, len, seed64, secret, secretLen); } /* === Public XXH128 API === */ /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits(XXH_NOESCAPE const void* input, size_t len) { return XXH3_128bits_internal(input, len, 0, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_128b_default); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSecret(XXH_NOESCAPE const void* input, size_t len, XXH_NOESCAPE const void* secret, size_t secretSize) { return XXH3_128bits_internal(input, len, 0, (const xxh_u8*)secret, secretSize, XXH3_hashLong_128b_withSecret); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSeed(XXH_NOESCAPE const void* input, size_t len, XXH64_hash_t seed) { return XXH3_128bits_internal(input, len, seed, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_128b_withSeed); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSecretandSeed(XXH_NOESCAPE const void* input, size_t len, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed) { if (len <= XXH3_MIDSIZE_MAX) return XXH3_128bits_internal(input, len, seed, XXH3_kSecret, sizeof(XXH3_kSecret), NULL); return XXH3_hashLong_128b_withSecret(input, len, seed, secret, secretSize); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH128(XXH_NOESCAPE const void* input, size_t len, XXH64_hash_t seed) { return XXH3_128bits_withSeed(input, len, seed); } /* === XXH3 128-bit streaming === */ #ifndef XXH_NO_STREAM /* * All initialization and update functions are identical to 64-bit streaming variant. * The only difference is the finalization routine. */ /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset(XXH_NOESCAPE XXH3_state_t* statePtr) { return XXH3_64bits_reset(statePtr); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecret(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize) { return XXH3_64bits_reset_withSecret(statePtr, secret, secretSize); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH64_hash_t seed) { return XXH3_64bits_reset_withSeed(statePtr, seed); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecretandSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed) { return XXH3_64bits_reset_withSecretandSeed(statePtr, secret, secretSize, seed); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_update(XXH_NOESCAPE XXH3_state_t* state, XXH_NOESCAPE const void* input, size_t len) { return XXH3_64bits_update(state, input, len); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_digest (XXH_NOESCAPE const XXH3_state_t* state) { const unsigned char* const secret = (state->extSecret == NULL) ? state->customSecret : state->extSecret; if (state->totalLen > XXH3_MIDSIZE_MAX) { XXH_ALIGN(XXH_ACC_ALIGN) XXH64_hash_t acc[XXH_ACC_NB]; XXH3_digest_long(acc, state, secret); XXH_ASSERT(state->secretLimit + XXH_STRIPE_LEN >= sizeof(acc) + XXH_SECRET_MERGEACCS_START); { XXH128_hash_t h128; h128.low64 = XXH3_mergeAccs(acc, secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)state->totalLen * XXH_PRIME64_1); h128.high64 = XXH3_mergeAccs(acc, secret + state->secretLimit + XXH_STRIPE_LEN - sizeof(acc) - XXH_SECRET_MERGEACCS_START, ~((xxh_u64)state->totalLen * XXH_PRIME64_2)); return h128; } } /* len <= XXH3_MIDSIZE_MAX : short code */ if (state->seed) return XXH3_128bits_withSeed(state->buffer, (size_t)state->totalLen, state->seed); return XXH3_128bits_withSecret(state->buffer, (size_t)(state->totalLen), secret, state->secretLimit + XXH_STRIPE_LEN); } #endif /* !XXH_NO_STREAM */ /* 128-bit utility functions */ #include /* memcmp, memcpy */ /* return : 1 is equal, 0 if different */ /*! @ingroup XXH3_family */ XXH_PUBLIC_API int XXH128_isEqual(XXH128_hash_t h1, XXH128_hash_t h2) { /* note : XXH128_hash_t is compact, it has no padding byte */ return !(memcmp(&h1, &h2, sizeof(h1))); } /* This prototype is compatible with stdlib's qsort(). * @return : >0 if *h128_1 > *h128_2 * <0 if *h128_1 < *h128_2 * =0 if *h128_1 == *h128_2 */ /*! @ingroup XXH3_family */ XXH_PUBLIC_API int XXH128_cmp(XXH_NOESCAPE const void* h128_1, XXH_NOESCAPE const void* h128_2) { XXH128_hash_t const h1 = *(const XXH128_hash_t*)h128_1; XXH128_hash_t const h2 = *(const XXH128_hash_t*)h128_2; int const hcmp = (h1.high64 > h2.high64) - (h2.high64 > h1.high64); /* note : bets that, in most cases, hash values are different */ if (hcmp) return hcmp; return (h1.low64 > h2.low64) - (h2.low64 > h1.low64); } /*====== Canonical representation ======*/ /*! @ingroup XXH3_family */ XXH_PUBLIC_API void XXH128_canonicalFromHash(XXH_NOESCAPE XXH128_canonical_t* dst, XXH128_hash_t hash) { XXH_STATIC_ASSERT(sizeof(XXH128_canonical_t) == sizeof(XXH128_hash_t)); if (XXH_CPU_LITTLE_ENDIAN) { hash.high64 = XXH_swap64(hash.high64); hash.low64 = XXH_swap64(hash.low64); } XXH_memcpy(dst, &hash.high64, sizeof(hash.high64)); XXH_memcpy((char*)dst + sizeof(hash.high64), &hash.low64, sizeof(hash.low64)); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH128_hashFromCanonical(XXH_NOESCAPE const XXH128_canonical_t* src) { XXH128_hash_t h; h.high64 = XXH_readBE64(src); h.low64 = XXH_readBE64(src->digest + 8); return h; } /* ========================================== * Secret generators * ========================================== */ #define XXH_MIN(x, y) (((x) > (y)) ? (y) : (x)) XXH_FORCE_INLINE void XXH3_combine16(void* dst, XXH128_hash_t h128) { XXH_writeLE64( dst, XXH_readLE64(dst) ^ h128.low64 ); XXH_writeLE64( (char*)dst+8, XXH_readLE64((char*)dst+8) ^ h128.high64 ); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_generateSecret(XXH_NOESCAPE void* secretBuffer, size_t secretSize, XXH_NOESCAPE const void* customSeed, size_t customSeedSize) { #if (XXH_DEBUGLEVEL >= 1) XXH_ASSERT(secretBuffer != NULL); XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); #else /* production mode, assert() are disabled */ if (secretBuffer == NULL) return XXH_ERROR; if (secretSize < XXH3_SECRET_SIZE_MIN) return XXH_ERROR; #endif if (customSeedSize == 0) { customSeed = XXH3_kSecret; customSeedSize = XXH_SECRET_DEFAULT_SIZE; } #if (XXH_DEBUGLEVEL >= 1) XXH_ASSERT(customSeed != NULL); #else if (customSeed == NULL) return XXH_ERROR; #endif /* Fill secretBuffer with a copy of customSeed - repeat as needed */ { size_t pos = 0; while (pos < secretSize) { size_t const toCopy = XXH_MIN((secretSize - pos), customSeedSize); memcpy((char*)secretBuffer + pos, customSeed, toCopy); pos += toCopy; } } { size_t const nbSeg16 = secretSize / 16; size_t n; XXH128_canonical_t scrambler; XXH128_canonicalFromHash(&scrambler, XXH128(customSeed, customSeedSize, 0)); for (n=0; n #include #include "common/debug.h" /**********************************************************************************************************************************/ FN_EXTERN size_t objToLog(const void *const object, ObjToLogFormat formatFunc, char *const buffer, const size_t bufferSize) { StringStatic debugLog = strStcInit(buffer, bufferSize); if (object == NULL) strStcCat(&debugLog, NULL_Z); else formatFunc(object, &debugLog); return strStcResultSize(&debugLog); } /**********************************************************************************************************************************/ FN_EXTERN size_t objNameToLog(const void *object, const char *objectName, char *buffer, size_t bufferSize) { StringStatic debugLog = strStcInit(buffer, bufferSize); if (object == NULL) strStcCat(&debugLog, NULL_Z); else strStcFmt(&debugLog, "{%s}", objectName); return strStcResultSize(&debugLog); } /**********************************************************************************************************************************/ FN_EXTERN size_t ptrToLog(const void *pointer, const char *pointerName, char *buffer, size_t bufferSize) { StringStatic debugLog = strStcInit(buffer, bufferSize); if (pointer == NULL) strStcCat(&debugLog, NULL_Z); else strStcFmt(&debugLog, "(%s)", pointerName); return strStcResultSize(&debugLog); } /**********************************************************************************************************************************/ FN_EXTERN size_t typeToLog(const char *typeName, char *buffer, size_t bufferSize) { StringStatic debugLog = strStcInit(buffer, bufferSize); strStcFmt(&debugLog, "%s", typeName); return strStcResultSize(&debugLog); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/debug.h0000664000175100017510000011347115265101745016727 0ustar00runnerrunner/*********************************************************************************************************************************** Debug Routines ***********************************************************************************************************************************/ #ifndef COMMON_DEBUG_H #define COMMON_DEBUG_H #include "common/assert.h" #include "common/stackTrace.h" #include "common/type/convert.h" #include "common/type/stringStatic.h" /*********************************************************************************************************************************** These functions allow auditing of child mem contexts and allocations that are left in the calling context when a function exits. This helps find leaks, i.e. child mem contexts or allocations created in the calling context (and not freed) accidentally. The FUNCTION_AUDIT_*() macros can be used to annotate functions that do that follow the default behavior, i.e. that a single value is returned and that is the only value created in the calling context. ***********************************************************************************************************************************/ #if defined(DEBUG_MEM) && defined(DEBUG_TEST_TRACE) #include "common/macro.h" #include "common/memContext.h" // Begin the audit #define FUNCTION_TEST_MEM_CONTEXT_AUDIT_BEGIN() \ MemContextAuditState MEM_CONTEXT_AUDIT_param = {.memContext = memContextCurrent()}; \ memContextAuditBegin(&MEM_CONTEXT_AUDIT_param) // End the audit #define FUNCTION_TEST_MEM_CONTEXT_AUDIT_END(returnType) \ memContextAuditEnd(&MEM_CONTEXT_AUDIT_param, returnType) // Allow any new mem contexts or allocations in the calling context. These should be fixed and this macro eventually removed. #define FUNCTION_AUDIT_IF(condition) \ do \ { \ if (!(condition)) \ MEM_CONTEXT_AUDIT_param.returnTypeAny = true; \ } \ while (0) // Callbacks are difficult to audit so ignore them. Eventually they should all be removed. #define FUNCTION_AUDIT_CALLBACK() MEM_CONTEXT_AUDIT_param.returnTypeAny = true // Helper function that creates new mem contexts or allocations in the calling context. These functions should be static (except // for interface helpers) but it is not clear that anything else needs to be done. #define FUNCTION_AUDIT_HELPER() MEM_CONTEXT_AUDIT_param.returnTypeAny = true // Function returns a struct that has new mem contexts or allocations in the calling context. Find a way to fix these. #define FUNCTION_AUDIT_STRUCT() MEM_CONTEXT_AUDIT_param.returnTypeAny = true #else #define FUNCTION_TEST_MEM_CONTEXT_AUDIT_BEGIN() #define FUNCTION_TEST_MEM_CONTEXT_AUDIT_END(returnType) #define FUNCTION_AUDIT_IF(condition) #define FUNCTION_AUDIT_CALLBACK() #define FUNCTION_AUDIT_HELPER() #define FUNCTION_AUDIT_STRUCT() #endif /*********************************************************************************************************************************** Base function debugging macros In debug mode parameters will always be recorded in the stack trace while in production mode they will only be recorded when the log level is set to debug or trace. ***********************************************************************************************************************************/ #define FUNCTION_LOG_LEVEL() \ FUNCTION_LOG_logLevel #ifdef DEBUG_TEST_TRACE #define FUNCTION_LOG_BEGIN_BASE(logLevel) \ LogLevel FUNCTION_LOG_LEVEL() = STACK_TRACE_PUSH(logLevel); \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_BEGIN(); \ \ { \ stackTraceParamLog(); \ stackTraceTestStop() #define FUNCTION_LOG_END_BASE() \ stackTraceTestStart(); \ LOG_FMT(FUNCTION_LOG_LEVEL(), 0, "(%s)", stackTraceParam()); \ } #else #define FUNCTION_LOG_BEGIN_BASE(logLevel) \ LogLevel FUNCTION_LOG_LEVEL() = STACK_TRACE_PUSH(logLevel); \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_BEGIN(); \ \ if (logAny(FUNCTION_LOG_LEVEL())) \ { \ stackTraceParamLog() #define FUNCTION_LOG_END_BASE() \ LOG_FMT(FUNCTION_LOG_LEVEL(), 0, "(%s)", stackTraceParam()); \ } #endif /*********************************************************************************************************************************** General purpose function debugging macros FUNCTION_LOG_VOID() is provided as a shortcut for functions that have no parameters. ***********************************************************************************************************************************/ #define FUNCTION_LOG_BEGIN(logLevel) \ FUNCTION_LOG_BEGIN_BASE(logLevel) #define FUNCTION_LOG_END() \ FUNCTION_LOG_END_BASE() #define FUNCTION_LOG_VOID(logLevel) \ FUNCTION_LOG_BEGIN_BASE(logLevel); \ FUNCTION_LOG_END_BASE() #define FUNCTION_LOG_PARAM(typeMacroPrefix, param) \ stackTraceParamAdd(FUNCTION_LOG_##typeMacroPrefix##_FORMAT(param, stackTraceParamBuffer(#param), STACK_TRACE_PARAM_MAX)) #define FUNCTION_LOG_PARAM_P(typeMacroPrefix, param) \ do \ { \ char *buffer = stackTraceParamBuffer(#param); \ \ if (param == NULL) \ stackTraceParamAdd(typeToLog(NULL_Z, buffer, STACK_TRACE_PARAM_MAX)); \ else \ { \ buffer[0] = '*'; \ stackTraceParamAdd(FUNCTION_LOG_##typeMacroPrefix##_FORMAT(*param, buffer + 1, STACK_TRACE_PARAM_MAX - 1) + 1); \ } \ } \ while (0) #define FUNCTION_LOG_PARAM_PP(typeMacroPrefix, param) \ do \ { \ char *buffer = stackTraceParamBuffer(#param); \ \ if (param == NULL) \ stackTraceParamAdd(typeToLog(NULL_Z, buffer, STACK_TRACE_PARAM_MAX)); \ else if (*param == NULL) \ stackTraceParamAdd(typeToLog("*null", buffer, STACK_TRACE_PARAM_MAX)); \ else \ { \ buffer[0] = '*'; \ buffer[1] = '*'; \ stackTraceParamAdd(FUNCTION_LOG_##typeMacroPrefix##_FORMAT(**param, buffer + 2, STACK_TRACE_PARAM_MAX - 2) + 2); \ } \ } \ while (0) /*********************************************************************************************************************************** Functions and macros to render various data types ***********************************************************************************************************************************/ // Convert object to a zero-terminated string for logging typedef void (*ObjToLogFormat)(const void *object, StringStatic *debugLog); FN_EXTERN size_t objToLog(const void *object, ObjToLogFormat formatFunc, char *buffer, size_t bufferSize); #define FUNCTION_LOG_OBJECT_FORMAT(object, formatFunc, buffer, bufferSize) \ objToLog(object, (ObjToLogFormat)formatFunc, buffer, bufferSize) // Convert object name to a zero-terminated string for logging FN_EXTERN size_t objNameToLog(const void *object, const char *objectName, char *buffer, size_t bufferSize); // Convert pointer to a zero-terminated string for logging FN_EXTERN size_t ptrToLog(const void *pointer, const char *pointerName, char *buffer, size_t bufferSize); // Convert zero-terminated string for logging FN_INLINE_ALWAYS size_t strzToLog(const char *const string, char *const buffer, const size_t bufferSize) { return (size_t)snprintf(buffer, bufferSize, string == NULL ? "%s" : "\"%s\"", string == NULL ? NULL_Z : string); } // Convert a type name to a zero-terminated string for logging FN_EXTERN size_t typeToLog(const char *typeName, char *buffer, size_t bufferSize); #define FUNCTION_LOG_BOOL_TYPE \ bool #define FUNCTION_LOG_BOOL_FORMAT(value, buffer, bufferSize) \ cvtBoolToZ(value, buffer, bufferSize) #define FUNCTION_LOG_CHAR_TYPE \ char #define FUNCTION_LOG_CHAR_FORMAT(value, buffer, bufferSize) \ cvtCharToZ(value, buffer, bufferSize) #define FUNCTION_LOG_CHARDATA_TYPE \ char #define FUNCTION_LOG_CHARDATA_FORMAT(value, buffer, bufferSize) \ typeToLog("(char)", buffer, bufferSize) #define FUNCTION_LOG_CHARPY_TYPE \ char *[] #define FUNCTION_LOG_CHARPY_FORMAT(value, buffer, bufferSize) \ ptrToLog(value, "char *[]", buffer, bufferSize) #define FUNCTION_LOG_INT_TYPE \ int #define FUNCTION_LOG_INT_FORMAT(value, buffer, bufferSize) \ cvtIntToZ(value, buffer, bufferSize) #define FUNCTION_LOG_INT64_TYPE \ int64_t #define FUNCTION_LOG_INT64_FORMAT(value, buffer, bufferSize) \ cvtInt64ToZ(value, buffer, bufferSize) #define FUNCTION_LOG_ENUM_TYPE \ unsigned int #define FUNCTION_LOG_ENUM_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_UINT_FORMAT(value, buffer, bufferSize) #define FUNCTION_LOG_FUNCTIONP_FORMAT(value, buffer, bufferSize) \ ptrToLog(value == NULL ? NULL : (void *)1, "function *", buffer, bufferSize) #define FUNCTION_LOG_MODE_TYPE \ mode_t #define FUNCTION_LOG_MODE_FORMAT(value, buffer, bufferSize) \ cvtModeToZ(value, buffer, bufferSize) #define FUNCTION_LOG_TIMEMSEC_TYPE \ TimeMSec #define FUNCTION_LOG_TIMEMSEC_FORMAT(value, buffer, bufferSize) \ cvtUInt64ToZ(value, buffer, bufferSize) #define FUNCTION_LOG_BYTEDATA_TYPE \ uint8_t #define FUNCTION_LOG_BYTEDATA_FORMAT(value, buffer, bufferSize) \ typeToLog("(uint8_t)", buffer, bufferSize) #define FUNCTION_LOG_SIZE_TYPE \ size_t #define FUNCTION_LOG_SIZE_FORMAT(value, buffer, bufferSize) \ cvtSizeToZ(value, buffer, bufferSize) #define FUNCTION_LOG_TIME_TYPE \ time_t #define FUNCTION_LOG_TIME_FORMAT(value, buffer, bufferSize) \ cvtTimeToZP("%s", value, buffer, bufferSize) #define FUNCTION_LOG_UINT_TYPE \ unsigned int #define FUNCTION_LOG_UINT_FORMAT(value, buffer, bufferSize) \ cvtUIntToZ(value, buffer, bufferSize) #define FUNCTION_LOG_UINT16_TYPE \ uint16_t #define FUNCTION_LOG_UINT16_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_UINT_FORMAT(value, buffer, bufferSize) #define FUNCTION_LOG_UINT32_TYPE \ uint32_t #define FUNCTION_LOG_UINT32_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_UINT_FORMAT(value, buffer, bufferSize) #define FUNCTION_LOG_UINT64_TYPE \ uint64_t #define FUNCTION_LOG_UINT64_FORMAT(value, buffer, bufferSize) \ cvtUInt64ToZ(value, buffer, bufferSize) #define FUNCTION_LOG_VOID_TYPE \ void #define FUNCTION_LOG_VOID_FORMAT(value, buffer, bufferSize) \ typeToLog("void", buffer, bufferSize) #define FUNCTION_LOG_STRINGZ_TYPE \ char * #define FUNCTION_LOG_STRINGZ_FORMAT(value, buffer, bufferSize) \ strzToLog(value, buffer, bufferSize) #define FUNCTION_LOG_SSIZE_TYPE \ ssize_t #define FUNCTION_LOG_SSIZE_FORMAT(value, buffer, bufferSize) \ cvtInt64ToZ(value, buffer, bufferSize) /*********************************************************************************************************************************** Macros to return function results (or void) ***********************************************************************************************************************************/ #define FUNCTION_LOG_RETURN_BASE(typePre, typeMacroPrefix, typePost, ...) \ do \ { \ typePre FUNCTION_LOG_##typeMacroPrefix##_TYPE typePost FUNCTION_LOG_RETURN_result = __VA_ARGS__; \ \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_END(STRINGIFY(FUNCTION_LOG_##typeMacroPrefix##_TYPE)); \ STACK_TRACE_POP(false); \ \ IF_LOG_ANY(FUNCTION_LOG_LEVEL()) \ { \ char buffer[STACK_TRACE_PARAM_MAX]; \ \ FUNCTION_LOG_##typeMacroPrefix##_FORMAT(FUNCTION_LOG_RETURN_result, buffer, sizeof(buffer)); \ LOG_FMT(FUNCTION_LOG_LEVEL(), 0, "=> %s", buffer); \ } \ \ return FUNCTION_LOG_RETURN_result; \ } \ while (0) #define FUNCTION_LOG_RETURN(typeMacroPrefix, ...) \ FUNCTION_LOG_RETURN_BASE(, typeMacroPrefix, , __VA_ARGS__) #define FUNCTION_LOG_RETURN_P(typeMacroPrefix, ...) \ FUNCTION_LOG_RETURN_BASE(, typeMacroPrefix, *, __VA_ARGS__) #define FUNCTION_LOG_RETURN_PP(typeMacroPrefix, ...) \ FUNCTION_LOG_RETURN_BASE(, typeMacroPrefix, **, __VA_ARGS__) #define FUNCTION_LOG_RETURN_CONST(typeMacroPrefix, ...) \ FUNCTION_LOG_RETURN_BASE(const, typeMacroPrefix, , __VA_ARGS__) #define FUNCTION_LOG_RETURN_CONST_P(typeMacroPrefix, ...) \ FUNCTION_LOG_RETURN_BASE(const, typeMacroPrefix, *, __VA_ARGS__) #define FUNCTION_LOG_RETURN_CONST_PP(typeMacroPrefix, ...) \ FUNCTION_LOG_RETURN_BASE(const, typeMacroPrefix, **, __VA_ARGS__) #define FUNCTION_LOG_RETURN_STRUCT(...) \ do \ { \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_END("struct"); \ STACK_TRACE_POP(false); \ \ IF_LOG_ANY(FUNCTION_LOG_LEVEL()) \ LOG_FMT(FUNCTION_LOG_LEVEL(), 0, "=> struct"); \ \ return __VA_ARGS__; \ } \ while (0) #define FUNCTION_LOG_RETURN_VOID() \ do \ { \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_END("void"); \ STACK_TRACE_POP(false); \ \ LOG(FUNCTION_LOG_LEVEL(), 0, "=> void"); \ return; \ } \ while (0) /*********************************************************************************************************************************** Function Test Macros In debug builds these macros will update the stack trace with function names and parameters but will not log. In production builds all test macros are compiled out (except for return statements). Ignore DEBUG_TEST_TRACE_MACRO if DEBUG is not defined because the underlying functions that support the macros will not be present. ***********************************************************************************************************************************/ #ifdef DEBUG #ifdef DEBUG_TEST_TRACE #define DEBUG_TEST_TRACE_MACRO #endif // DEBUG_TEST_TRACE #endif // DEBUG // Annotate functions that do not return so it is clear why there is no FUNCTION_TEST_RETURN*() macro #define FUNCTION_TEST_NO_RETURN() #ifdef DEBUG_TEST_TRACE_MACRO #define FUNCTION_TEST_BEGIN() \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_BEGIN(); \ \ /* Ensure that FUNCTION_LOG_BEGIN() and FUNCTION_TEST_BEGIN() are not both used in a single function by declaring the */ \ /* same variable that FUNCTION_LOG_BEGIN() uses to track logging */ \ LogLevel FUNCTION_LOG_LEVEL(); \ (void)FUNCTION_LOG_LEVEL(); \ \ /* Ensure that FUNCTION_TEST_RETURN*() is not used with FUNCTION_LOG_BEGIN*() by declaring a variable that will be */ \ /* referenced in FUNCTION_TEST_RETURN*() */ \ bool FUNCTION_TEST_BEGIN_exists; \ \ if (stackTraceTest()) \ { \ STACK_TRACE_PUSH(logLevelDebug); \ stackTraceParamLog(); \ stackTraceTestStop() #define FUNCTION_TEST_PARAM(typeMacroPrefix, param) \ FUNCTION_LOG_PARAM(typeMacroPrefix, param) #define FUNCTION_TEST_PARAM_P(typeName, param) \ FUNCTION_LOG_PARAM_P(typeName, param) #define FUNCTION_TEST_PARAM_PP(typeName, param) \ FUNCTION_LOG_PARAM_PP(typeName, param) #define FUNCTION_TEST_END() \ (void)FUNCTION_TEST_BEGIN_exists; /* CHECK for presence of FUNCTION_TEST_BEGIN*() */ \ \ stackTraceTestStart(); \ } #define FUNCTION_TEST_VOID() \ FUNCTION_TEST_BEGIN(); \ FUNCTION_TEST_END(); #define FUNCTION_TEST_RETURN_TYPE_BASE(typePre, type, typePost, ...) \ do \ { \ (void)FUNCTION_TEST_BEGIN_exists; /* CHECK for presence of FUNCTION_TEST_BEGIN*() */ \ \ typePre type typePost FUNCTION_TEST_result = __VA_ARGS__; \ \ STACK_TRACE_POP(true); \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_END(STRINGIFY(type)); \ \ return FUNCTION_TEST_result; \ } \ while (0) #define FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(typePre, typeMacroPrefix, typePost, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(typePre, FUNCTION_LOG_##typeMacroPrefix##_TYPE, typePost, __VA_ARGS__) #define FUNCTION_TEST_RETURN(typeMacroPrefix, ...) \ FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(, typeMacroPrefix, , __VA_ARGS__) #define FUNCTION_TEST_RETURN_P(typeMacroPrefix, ...) \ FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(, typeMacroPrefix, *, __VA_ARGS__) #define FUNCTION_TEST_RETURN_PP(typeMacroPrefix, ...) \ FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(, typeMacroPrefix, **, __VA_ARGS__) #define FUNCTION_TEST_RETURN_CONST(typeMacroPrefix, ...) \ FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(const, typeMacroPrefix, , __VA_ARGS__) #define FUNCTION_TEST_RETURN_CONST_P(typeMacroPrefix, ...) \ FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(const, typeMacroPrefix, *, __VA_ARGS__) #define FUNCTION_TEST_RETURN_CONST_PP(typeMacroPrefix, ...) \ FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(const, typeMacroPrefix, **, __VA_ARGS__) #define FUNCTION_TEST_RETURN_TYPE(type, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(, type, , __VA_ARGS__) #define FUNCTION_TEST_RETURN_TYPE_P(type, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(, type, *, __VA_ARGS__) #define FUNCTION_TEST_RETURN_TYPE_PP(type, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(, type, **, __VA_ARGS__) #define FUNCTION_TEST_RETURN_TYPE_CONST(type, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(const, type, , __VA_ARGS__) #define FUNCTION_TEST_RETURN_TYPE_CONST_P(type, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(const, type, *, __VA_ARGS__) #define FUNCTION_TEST_RETURN_TYPE_CONST_PP(type, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(const, type, **, __VA_ARGS__) #define FUNCTION_TEST_RETURN_VOID() \ do \ { \ (void)FUNCTION_TEST_BEGIN_exists; /* CHECK for presence of FUNCTION_TEST_BEGIN*() */ \ \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_END("void"); \ STACK_TRACE_POP(true); \ return; \ } \ while (0) #else #define FUNCTION_TEST_BEGIN() #define FUNCTION_TEST_PARAM(typeMacroPrefix, param) #define FUNCTION_TEST_PARAM_P(typeMacroPrefix, param) #define FUNCTION_TEST_PARAM_PP(typeMacroPrefix, param) #define FUNCTION_TEST_END() #define FUNCTION_TEST_VOID() #define FUNCTION_TEST_RETURN(typeMacroPrefix, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_P(typeMacroPrefix, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_PP(typeMacroPrefix, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_CONST(typeMacroPrefix, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_CONST_P(typeMacroPrefix, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_CONST_PP(typeMacroPrefix, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_TYPE(type, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_TYPE_P(type, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_TYPE_PP(type, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_TYPE_CONST(type, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_TYPE_CONST_P(type, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_TYPE_CONST_PP(type, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_VOID() \ return #endif // DEBUG_TEST_TRACE_MACRO #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/encode.c0000664000175100017510000005100115265101745017057 0ustar00runnerrunner/*********************************************************************************************************************************** Binary to String Encode/Decode ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/encode.h" /*********************************************************************************************************************************** Assert that encoding type is valid. This needs to be kept up to date with the last item in the enum. ***********************************************************************************************************************************/ #define ASSERT_ENCODE_TYPE_VALID(type) \ ASSERT(type <= encodingHex); /*********************************************************************************************************************************** Base64 encoding/decoding ***********************************************************************************************************************************/ VR_NON_STRING static const char encodeBase64Lookup[64] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; static void encodeToStrBase64(const uint8_t *const source, const size_t sourceSize, char *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(BYTEDATA, source); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_PARAM_P(CHARDATA, destination); FUNCTION_TEST_END(); ASSERT(source != NULL); ASSERT(destination != NULL); unsigned int destinationIdx = 0; // Encode the string from three bytes to four characters for (unsigned int sourceIdx = 0; sourceIdx < sourceSize; sourceIdx += 3) { // First encoded character is always used completely destination[destinationIdx++] = encodeBase64Lookup[source[sourceIdx] >> 2]; // If there is only one byte to encode then the second encoded character is only partly used and the third and fourth // encoded characters are padded. if (sourceSize - sourceIdx == 1) { destination[destinationIdx++] = encodeBase64Lookup[(source[sourceIdx] & 0x03) << 4]; destination[destinationIdx++] = 0x3d; destination[destinationIdx++] = 0x3d; } // Else if more than one byte to encode else { // If there is more than one byte to encode then the second encoded character is used completely destination[destinationIdx++] = encodeBase64Lookup[((source[sourceIdx] & 0x03) << 4) | ((source[sourceIdx + 1] & 0xf0) >> 4)]; // If there are only two bytes to encode then the third encoded character is only partly used and the fourth encoded // character is padded. if (sourceSize - sourceIdx == 2) { destination[destinationIdx++] = encodeBase64Lookup[(source[sourceIdx + 1] & 0x0f) << 2]; destination[destinationIdx++] = 0x3d; } // Else the third and fourth encoded characters are used completely else { destination[destinationIdx++] = encodeBase64Lookup[((source[sourceIdx + 1] & 0x0f) << 2) | ((source[sourceIdx + 2] & 0xc0) >> 6)]; destination[destinationIdx++] = encodeBase64Lookup[source[sourceIdx + 2] & 0x3f]; } } } // Zero-terminate the string destination[destinationIdx] = 0; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static size_t encodeToStrSizeBase64(const size_t sourceSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_END(); // Four characters are needed to encode each 3 byte group (plus up to two bytes of padding) FUNCTION_TEST_RETURN(SIZE, (sourceSize + 2) / 3 * 4); } /**********************************************************************************************************************************/ static const int8_t decodeBase64Lookup[256] = { -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 62, -1, -1, -1, 63, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, -1, -1, -1, -1, -1, -1, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, -1, -1, -1, -1, -1, -1, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, }; static void decodeToBinValidateBase64(const char *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_END(); // Check for the correct length const size_t sourceSize = strlen(source); if (sourceSize % 4 != 0) THROW_FMT(FormatError, "base64 size %zu is not evenly divisible by 4", sourceSize); // Check all characters for (unsigned int sourceIdx = 0; sourceIdx < sourceSize; sourceIdx++) { // Check terminators if (source[sourceIdx] == 0x3d) { // Make sure they are only in the last two positions if (sourceIdx < sourceSize - 2) THROW(FormatError, "base64 '=' character may only appear in last two positions"); // If second to last char is = then last char must also be if (sourceIdx == sourceSize - 2 && source[sourceSize - 1] != 0x3d) THROW(FormatError, "base64 last character must be '=' if second to last is"); } else { // Error on any invalid characters if (decodeBase64Lookup[(int)source[sourceIdx]] == -1) THROW_FMT(FormatError, "base64 invalid character found at position %u", sourceIdx); } } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static void decodeToBinBase64(const char *const source, uint8_t *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_PARAM_P(BYTEDATA, destination); FUNCTION_TEST_END(); // Validate encoded string decodeToBinValidateBase64(source); int destinationIdx = 0; // Decode the binary data from four characters to three bytes for (unsigned int sourceIdx = 0; sourceIdx < strlen(source); sourceIdx += 4) { // Always decode the first character destination[destinationIdx++] = (uint8_t)(decodeBase64Lookup[(int)source[sourceIdx]] << 2 | decodeBase64Lookup[(int)source[sourceIdx + 1]] >> 4); // Second character is optional if (source[sourceIdx + 2] != 0x3d) { destination[destinationIdx++] = (uint8_t) ((decodeBase64Lookup[(int)source[sourceIdx + 1]] << 4) | (decodeBase64Lookup[(int)source[sourceIdx + 2]] >> 2)); } // Third character is optional if (source[sourceIdx + 3] != 0x3d) { destination[destinationIdx++] = (uint8_t) (((decodeBase64Lookup[(int)source[sourceIdx + 2]] << 6) & 0xc0) | decodeBase64Lookup[(int)source[sourceIdx + 3]]); } } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static size_t decodeToBinSizeBase64(const char *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_END(); // Validate encoded string decodeToBinValidateBase64(source); // Start with size calculated directly from source length const size_t sourceSize = strlen(source); size_t destinationSize = sourceSize / 4 * 3; // Subtract last character if it is not present. An empty string is valid base64 and has no last character to check. if (sourceSize != 0 && source[sourceSize - 1] == 0x3d) { destinationSize--; // Subtract second to last character if it is not present if (source[sourceSize - 2] == 0x3d) destinationSize--; } FUNCTION_TEST_RETURN(SIZE, destinationSize); } /*********************************************************************************************************************************** Base64Url encoding ***********************************************************************************************************************************/ VR_NON_STRING static const char encodeBase64LookupUrl[64] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"; static void encodeToStrBase64Url(const uint8_t *const source, const size_t sourceSize, char *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(BYTEDATA, source); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_PARAM_P(CHARDATA, destination); FUNCTION_TEST_END(); ASSERT(source != NULL); ASSERT(destination != NULL); unsigned int destinationIdx = 0; // Encode the string from three bytes to four characters for (unsigned int sourceIdx = 0; sourceIdx < sourceSize; sourceIdx += 3) { // First encoded character is always used completely destination[destinationIdx++] = encodeBase64LookupUrl[source[sourceIdx] >> 2]; // If there is only one byte to encode then the second encoded character is only partly used if (sourceSize - sourceIdx == 1) { destination[destinationIdx++] = encodeBase64LookupUrl[(source[sourceIdx] & 0x03) << 4]; } // Else if more than one byte to encode else { // If there is more than one byte to encode then the second encoded character is used completely destination[destinationIdx++] = encodeBase64LookupUrl[((source[sourceIdx] & 0x03) << 4) | ((source[sourceIdx + 1] & 0xf0) >> 4)]; // If there are only two bytes to encode then the third encoded character is only partly used if (sourceSize - sourceIdx == 2) { destination[destinationIdx++] = encodeBase64LookupUrl[(source[sourceIdx + 1] & 0x0f) << 2]; } // Else the third and fourth encoded characters are used completely else { destination[destinationIdx++] = encodeBase64LookupUrl[((source[sourceIdx + 1] & 0x0f) << 2) | ((source[sourceIdx + 2] & 0xc0) >> 6)]; destination[destinationIdx++] = encodeBase64LookupUrl[source[sourceIdx + 2] & 0x3f]; } } } // Zero-terminate the string destination[destinationIdx] = 0; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static size_t encodeToStrSizeBase64Url(const size_t sourceSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_END(); // Four characters are needed to encode each 3 byte group, three characters for 2 bytes, and two characters for 1 byte FUNCTION_TEST_RETURN(SIZE, sourceSize / 3 * 4 + (sourceSize % 3 == 0 ? 0 : sourceSize % 3 + 1)); } /*********************************************************************************************************************************** Hex encoding/decoding ***********************************************************************************************************************************/ VR_NON_STRING static const char encodeHexLookup[512] = "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f" "202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f" "404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f" "606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f" "808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9f" "a0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebf" "c0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedf" "e0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff"; static void encodeToStrHex(const uint8_t *const source, const size_t sourceSize, char *destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(BYTEDATA, source); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_PARAM_P(CHARDATA, destination); FUNCTION_TEST_END(); ASSERT(source != NULL); ASSERT(destination != NULL); // Encode the string from one bytes to two characters for (unsigned int sourceIdx = 0; sourceIdx < sourceSize; sourceIdx += 1) { memcpy(destination, &encodeHexLookup[source[sourceIdx] * 2], 2); destination += 2; } // Zero-terminate the string *destination = 0; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static size_t encodeToStrSizeHex(const size_t sourceSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(SIZE, sourceSize * 2); } /**********************************************************************************************************************************/ // {uncrustify_off - array alignment} static const int8_t decodeHexLookup[256] = { -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -1, -1, -1, -1, -1, -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, }; // {uncrustify_on} static void decodeToBinValidateHex(const char *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_END(); // Check for the correct length const size_t sourceSize = strlen(source); if (sourceSize % 2 != 0) THROW_FMT(FormatError, "hex size %zu is not evenly divisible by 2", sourceSize); // Check all characters for (unsigned int sourceIdx = 0; sourceIdx < sourceSize; sourceIdx++) { if (decodeHexLookup[(int)source[sourceIdx]] == -1) THROW_FMT(FormatError, "hex invalid character found at position %u", sourceIdx); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static void decodeToBinHex(const char *const source, uint8_t *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_PARAM_P(BYTEDATA, destination); FUNCTION_TEST_END(); // Validate encoded string decodeToBinValidateHex(source); int destinationIdx = 0; // Decode the binary data from two characters to one byte for (unsigned int sourceIdx = 0; sourceIdx < strlen(source); sourceIdx += 2) { destination[destinationIdx++] = (uint8_t)(decodeHexLookup[(int)source[sourceIdx]] << 4 | decodeHexLookup[(int)source[sourceIdx + 1]]); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static size_t decodeToBinSizeHex(const char *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_END(); // Validate encoded string decodeToBinValidateHex(source); FUNCTION_TEST_RETURN(SIZE, strlen(source) / 2); } /*********************************************************************************************************************************** Generic encoding/decoding ***********************************************************************************************************************************/ FN_EXTERN void encodeToStr(const EncodingType type, const uint8_t *const source, const size_t sourceSize, char *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM_P(BYTEDATA, source); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_PARAM_P(CHARDATA, destination); FUNCTION_TEST_END(); ASSERT_ENCODE_TYPE_VALID(type); switch (type) { case encodingBase64: encodeToStrBase64(source, sourceSize, destination); break; case encodingBase64Url: encodeToStrBase64Url(source, sourceSize, destination); break; case encodingHex: encodeToStrHex(source, sourceSize, destination); break; } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN size_t encodeToStrSize(const EncodingType type, const size_t sourceSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_END(); ASSERT_ENCODE_TYPE_VALID(type); size_t destinationSize = SIZE_MAX; switch (type) { case encodingBase64: destinationSize = encodeToStrSizeBase64(sourceSize); break; case encodingBase64Url: destinationSize = encodeToStrSizeBase64Url(sourceSize); break; case encodingHex: destinationSize = encodeToStrSizeHex(sourceSize); break; } FUNCTION_TEST_RETURN(SIZE, destinationSize); } /**********************************************************************************************************************************/ FN_EXTERN void decodeToBin(const EncodingType type, const char *const source, uint8_t *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_PARAM_P(BYTEDATA, destination); FUNCTION_TEST_END(); ASSERT_ENCODE_TYPE_VALID(type); switch (type) { case encodingBase64: decodeToBinBase64(source, destination); break; case encodingHex: decodeToBinHex(source, destination); break; default: ASSERT_MSG("unsupported"); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN size_t decodeToBinSize(const EncodingType type, const char *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_END(); ASSERT_ENCODE_TYPE_VALID(type); size_t destinationSize = SIZE_MAX; switch (type) { case encodingBase64: destinationSize = decodeToBinSizeBase64(source); break; case encodingHex: destinationSize = decodeToBinSizeHex(source); break; default: ASSERT_MSG("unsupported"); } FUNCTION_TEST_RETURN(SIZE, destinationSize); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/encode.h0000664000175100017510000000301415265101745017065 0ustar00runnerrunner/*********************************************************************************************************************************** Binary to String Encode/Decode ***********************************************************************************************************************************/ #ifndef COMMON_ENCODE_H #define COMMON_ENCODE_H #include /*********************************************************************************************************************************** Encoding types ***********************************************************************************************************************************/ typedef enum { encodingBase64, encodingBase64Url, encodingHex, } EncodingType; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Encode binary data to a printable string FN_EXTERN void encodeToStr(EncodingType type, const uint8_t *source, size_t sourceSize, char *destination); // Size of the string returned by encodeToStr() FN_EXTERN size_t encodeToStrSize(EncodingType type, size_t sourceSize); // Decode a string to binary data FN_EXTERN void decodeToBin(EncodingType type, const char *source, uint8_t *destination); // Size of the binary data returned by decodeToBin() FN_EXTERN size_t decodeToBinSize(EncodingType type, const char *source); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/error/0000775000175100017510000000000015265101745016612 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/error/error.auto.c.inc0000664000175100017510000001462715265101745021640 0ustar00runnerrunner/*********************************************************************************************************************************** Error Type Definition Automatically generated by 'build.py error' -- do not modify directly. ***********************************************************************************************************************************/ /*********************************************************************************************************************************** Error type definitions ***********************************************************************************************************************************/ ERROR_DEFINE( 25, AssertError, true, RuntimeError); ERROR_DEFINE( 26, ChecksumError, false, RuntimeError); ERROR_DEFINE( 27, ConfigError, false, RuntimeError); ERROR_DEFINE( 28, FileInvalidError, false, RuntimeError); ERROR_DEFINE( 29, FormatError, false, RuntimeError); ERROR_DEFINE( 30, CommandRequiredError, false, RuntimeError); ERROR_DEFINE( 31, OptionInvalidError, false, RuntimeError); ERROR_DEFINE( 32, OptionInvalidValueError, false, RuntimeError); ERROR_DEFINE( 37, OptionRequiredError, false, RuntimeError); ERROR_DEFINE( 38, PgRunningError, false, RuntimeError); ERROR_DEFINE( 39, ProtocolError, false, RuntimeError); ERROR_DEFINE( 40, PathNotEmptyError, false, RuntimeError); ERROR_DEFINE( 41, FileOpenError, false, RuntimeError); ERROR_DEFINE( 42, FileReadError, false, RuntimeError); ERROR_DEFINE( 43, ParamRequiredError, false, RuntimeError); ERROR_DEFINE( 44, ArchiveMismatchError, false, RuntimeError); ERROR_DEFINE( 45, ArchiveDuplicateError, false, RuntimeError); ERROR_DEFINE( 46, VersionNotSupportedError, false, RuntimeError); ERROR_DEFINE( 47, PathCreateError, false, RuntimeError); ERROR_DEFINE( 48, CommandInvalidError, false, RuntimeError); ERROR_DEFINE( 49, HostConnectError, false, RuntimeError); ERROR_DEFINE( 50, LockAcquireError, false, RuntimeError); ERROR_DEFINE( 51, BackupMismatchError, false, RuntimeError); ERROR_DEFINE( 52, FileSyncError, false, RuntimeError); ERROR_DEFINE( 53, PathOpenError, false, RuntimeError); ERROR_DEFINE( 54, PathSyncError, false, RuntimeError); ERROR_DEFINE( 55, FileMissingError, false, RuntimeError); ERROR_DEFINE( 56, DbConnectError, false, RuntimeError); ERROR_DEFINE( 57, DbQueryError, false, RuntimeError); ERROR_DEFINE( 58, DbMismatchError, false, RuntimeError); ERROR_DEFINE( 60, FileRemoveError, false, RuntimeError); ERROR_DEFINE( 61, PathRemoveError, false, RuntimeError); ERROR_DEFINE( 62, StopError, false, RuntimeError); ERROR_DEFINE( 63, TermError, false, RuntimeError); ERROR_DEFINE( 64, FileWriteError, false, RuntimeError); ERROR_DEFINE( 67, FeatureNotSupportedError, false, RuntimeError); ERROR_DEFINE( 68, ArchiveCommandInvalidError, false, RuntimeError); ERROR_DEFINE( 69, LinkExpectedError, false, RuntimeError); ERROR_DEFINE( 70, LinkDestinationError, false, RuntimeError); ERROR_DEFINE( 72, HostInvalidError, false, RuntimeError); ERROR_DEFINE( 73, PathMissingError, false, RuntimeError); ERROR_DEFINE( 74, FileMoveError, false, RuntimeError); ERROR_DEFINE( 75, BackupSetInvalidError, false, RuntimeError); ERROR_DEFINE( 76, TablespaceMapError, false, RuntimeError); ERROR_DEFINE( 78, LinkMapError, false, RuntimeError); ERROR_DEFINE( 79, FileCloseError, false, RuntimeError); ERROR_DEFINE( 80, DbMissingError, false, RuntimeError); ERROR_DEFINE( 81, DbInvalidError, false, RuntimeError); ERROR_DEFINE( 82, ArchiveTimeoutError, false, RuntimeError); ERROR_DEFINE( 83, FileModeError, false, RuntimeError); ERROR_DEFINE( 87, ArchiveDisabledError, false, RuntimeError); ERROR_DEFINE( 88, FileOwnerError, false, RuntimeError); ERROR_DEFINE( 89, UserMissingError, false, RuntimeError); ERROR_DEFINE( 93, FileExistsError, false, RuntimeError); ERROR_DEFINE( 94, MemoryError, true, RuntimeError); ERROR_DEFINE( 95, CryptoError, false, RuntimeError); ERROR_DEFINE( 96, ParamInvalidError, false, RuntimeError); ERROR_DEFINE( 97, PathCloseError, false, RuntimeError); ERROR_DEFINE( 98, FileInfoError, false, RuntimeError); ERROR_DEFINE( 99, JsonFormatError, false, RuntimeError); ERROR_DEFINE(100, KernelError, false, RuntimeError); ERROR_DEFINE(101, ServiceError, false, RuntimeError); ERROR_DEFINE(102, ExecuteError, false, RuntimeError); ERROR_DEFINE(103, RepoInvalidError, false, RuntimeError); ERROR_DEFINE(104, CommandError, false, RuntimeError); ERROR_DEFINE(105, AccessError, false, RuntimeError); ERROR_DEFINE(106, ClockError, false, RuntimeError); ERROR_DEFINE(122, RuntimeError, false, RuntimeError); ERROR_DEFINE(124, UnhandledError, false, RuntimeError); ERROR_DEFINE(125, UnknownError, false, RuntimeError); /*********************************************************************************************************************************** Error type array ***********************************************************************************************************************************/ static const ErrorType *const errorTypeList[] = { &AssertError, &ChecksumError, &ConfigError, &FileInvalidError, &FormatError, &CommandRequiredError, &OptionInvalidError, &OptionInvalidValueError, &OptionRequiredError, &PgRunningError, &ProtocolError, &PathNotEmptyError, &FileOpenError, &FileReadError, &ParamRequiredError, &ArchiveMismatchError, &ArchiveDuplicateError, &VersionNotSupportedError, &PathCreateError, &CommandInvalidError, &HostConnectError, &LockAcquireError, &BackupMismatchError, &FileSyncError, &PathOpenError, &PathSyncError, &FileMissingError, &DbConnectError, &DbQueryError, &DbMismatchError, &FileRemoveError, &PathRemoveError, &StopError, &TermError, &FileWriteError, &FeatureNotSupportedError, &ArchiveCommandInvalidError, &LinkExpectedError, &LinkDestinationError, &HostInvalidError, &PathMissingError, &FileMoveError, &BackupSetInvalidError, &TablespaceMapError, &LinkMapError, &FileCloseError, &DbMissingError, &DbInvalidError, &ArchiveTimeoutError, &FileModeError, &ArchiveDisabledError, &FileOwnerError, &UserMissingError, &FileExistsError, &MemoryError, &CryptoError, &ParamInvalidError, &PathCloseError, &FileInfoError, &JsonFormatError, &KernelError, &ServiceError, &ExecuteError, &RepoInvalidError, &CommandError, &AccessError, &ClockError, &RuntimeError, &UnhandledError, &UnknownError, NULL, }; ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/error/error.auto.h0000664000175100017510000000564415265101745021074 0ustar00runnerrunner/*********************************************************************************************************************************** Error Type Definition Automatically generated by 'build.py error' -- do not modify directly. ***********************************************************************************************************************************/ #ifndef COMMON_ERROR_ERROR_AUTO_H #define COMMON_ERROR_ERROR_AUTO_H /*********************************************************************************************************************************** Error type declarations ***********************************************************************************************************************************/ ERROR_DECLARE(AssertError); ERROR_DECLARE(ChecksumError); ERROR_DECLARE(ConfigError); ERROR_DECLARE(FileInvalidError); ERROR_DECLARE(FormatError); ERROR_DECLARE(CommandRequiredError); ERROR_DECLARE(OptionInvalidError); ERROR_DECLARE(OptionInvalidValueError); ERROR_DECLARE(OptionRequiredError); ERROR_DECLARE(PgRunningError); ERROR_DECLARE(ProtocolError); ERROR_DECLARE(PathNotEmptyError); ERROR_DECLARE(FileOpenError); ERROR_DECLARE(FileReadError); ERROR_DECLARE(ParamRequiredError); ERROR_DECLARE(ArchiveMismatchError); ERROR_DECLARE(ArchiveDuplicateError); ERROR_DECLARE(VersionNotSupportedError); ERROR_DECLARE(PathCreateError); ERROR_DECLARE(CommandInvalidError); ERROR_DECLARE(HostConnectError); ERROR_DECLARE(LockAcquireError); ERROR_DECLARE(BackupMismatchError); ERROR_DECLARE(FileSyncError); ERROR_DECLARE(PathOpenError); ERROR_DECLARE(PathSyncError); ERROR_DECLARE(FileMissingError); ERROR_DECLARE(DbConnectError); ERROR_DECLARE(DbQueryError); ERROR_DECLARE(DbMismatchError); ERROR_DECLARE(FileRemoveError); ERROR_DECLARE(PathRemoveError); ERROR_DECLARE(StopError); ERROR_DECLARE(TermError); ERROR_DECLARE(FileWriteError); ERROR_DECLARE(FeatureNotSupportedError); ERROR_DECLARE(ArchiveCommandInvalidError); ERROR_DECLARE(LinkExpectedError); ERROR_DECLARE(LinkDestinationError); ERROR_DECLARE(HostInvalidError); ERROR_DECLARE(PathMissingError); ERROR_DECLARE(FileMoveError); ERROR_DECLARE(BackupSetInvalidError); ERROR_DECLARE(TablespaceMapError); ERROR_DECLARE(LinkMapError); ERROR_DECLARE(FileCloseError); ERROR_DECLARE(DbMissingError); ERROR_DECLARE(DbInvalidError); ERROR_DECLARE(ArchiveTimeoutError); ERROR_DECLARE(FileModeError); ERROR_DECLARE(ArchiveDisabledError); ERROR_DECLARE(FileOwnerError); ERROR_DECLARE(UserMissingError); ERROR_DECLARE(FileExistsError); ERROR_DECLARE(MemoryError); ERROR_DECLARE(CryptoError); ERROR_DECLARE(ParamInvalidError); ERROR_DECLARE(PathCloseError); ERROR_DECLARE(FileInfoError); ERROR_DECLARE(JsonFormatError); ERROR_DECLARE(KernelError); ERROR_DECLARE(ServiceError); ERROR_DECLARE(ExecuteError); ERROR_DECLARE(RepoInvalidError); ERROR_DECLARE(CommandError); ERROR_DECLARE(AccessError); ERROR_DECLARE(ClockError); ERROR_DECLARE(RuntimeError); ERROR_DECLARE(UnhandledError); ERROR_DECLARE(UnknownError); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/error/error.c0000664000175100017510000004327115265101745020116 0ustar00runnerrunner/*********************************************************************************************************************************** Error Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "common/error/error.h" #include "common/macro.h" #include "common/stackTrace.h" /*********************************************************************************************************************************** Represents an error type ***********************************************************************************************************************************/ struct ErrorType { const int code; const bool fatal; const char *name; const struct ErrorType *parentType; }; // Macro for defining new error types #define ERROR_DEFINE(code, name, fatal, parentType) \ VR_EXTERN_DEFINE const ErrorType name = {code, fatal, #name, &parentType} // Define test error #ifdef DEBUG ERROR_DEFINE(1, TestError, false, RuntimeError); #endif // Include error type definitions #include "common/error/error.auto.c.inc" /*********************************************************************************************************************************** Maximum allowed number of nested try blocks ***********************************************************************************************************************************/ #define ERROR_TRY_MAX 64 /*********************************************************************************************************************************** States for each try ***********************************************************************************************************************************/ typedef enum {errorStateTry, errorStateCatch, errorStateEnd} ErrorState; /*********************************************************************************************************************************** Track error handling ***********************************************************************************************************************************/ typedef struct Error { const ErrorType *errorType; // Error type const char *fileName; // Source file where the error occurred const char *functionName; // Function where the error occurred int fileLine; // Source file line where the error occurred const char *message; // Description of the error const char *stackTrace; // Stack trace } Error; static struct { // Array of jump buffers jmp_buf jumpList[ERROR_TRY_MAX]; // Handler list const ErrorHandlerFunction *handlerList; unsigned int handlerTotal; // State of each try int tryTotal; struct { ErrorState state; bool uncaught; } tryList[ERROR_TRY_MAX + 1]; // Last error Error error; } errorContext; /*********************************************************************************************************************************** Message buffer and buffer size The message buffer is statically allocated so there is some space to store error messages. Not being able to allocate such a small amount of memory seems pretty unlikely so just keep the code simple and let the loader deal with massively constrained memory situations. The temp buffer is required because the error message being passed might be the error already stored in the message buffer. ***********************************************************************************************************************************/ #ifndef ERROR_MESSAGE_BUFFER_SIZE #define ERROR_MESSAGE_BUFFER_SIZE 8192 #endif static char messageBuffer[ERROR_MESSAGE_BUFFER_SIZE]; static char messageBufferTemp[ERROR_MESSAGE_BUFFER_SIZE]; static char stackTraceBuffer[ERROR_MESSAGE_BUFFER_SIZE]; /**********************************************************************************************************************************/ FN_EXTERN void errorHandlerSet(const ErrorHandlerFunction *const list, const unsigned int total) { assert(total == 0 || list != NULL); errorContext.handlerList = list; errorContext.handlerTotal = total; } /**********************************************************************************************************************************/ FN_EXTERN int errorTypeCode(const ErrorType *const errorType) { return errorType->code; } /**********************************************************************************************************************************/ FN_EXTERN bool errorTypeFatal(const ErrorType *const errorType) { return errorType->fatal; } /**********************************************************************************************************************************/ FN_EXTERN const ErrorType * errorTypeFromCode(const int code) { // Search for error type by code int errorTypeIdx = 0; const ErrorType *result = errorTypeList[errorTypeIdx]; while (result != NULL) { if (result->code == code) break; result = errorTypeList[++errorTypeIdx]; } // Error if type was not found if (result == NULL) result = &UnknownError; return result; } /**********************************************************************************************************************************/ FN_EXTERN const char * errorTypeName(const ErrorType *const errorType) { return errorType->name; } /**********************************************************************************************************************************/ FN_EXTERN const ErrorType * errorTypeParent(const ErrorType *const errorType) { return errorType->parentType; } /**********************************************************************************************************************************/ FN_EXTERN unsigned int errorTryDepth(void) { return (unsigned int)errorContext.tryTotal; } /**********************************************************************************************************************************/ FN_EXTERN bool errorTypeExtends(const ErrorType *const child, const ErrorType *const parent) { const ErrorType *find = child; do { find = errorTypeParent(find); // Parent was found if (find == parent) return true; } while (find != errorTypeParent(find)); // Parent was not found return false; } /**********************************************************************************************************************************/ FN_EXTERN const ErrorType * errorType(void) { assert(errorContext.error.errorType != NULL); return errorContext.error.errorType; } /**********************************************************************************************************************************/ FN_EXTERN int errorCode(void) { return errorTypeCode(errorType()); } /**********************************************************************************************************************************/ FN_EXTERN bool errorFatal(void) { return errorTypeFatal(errorType()); } /**********************************************************************************************************************************/ FN_EXTERN const char * errorFileName(void) { assert(errorContext.error.fileName != NULL); return errorContext.error.fileName; } /**********************************************************************************************************************************/ FN_EXTERN const char * errorFunctionName(void) { assert(errorContext.error.functionName != NULL); return errorContext.error.functionName; } /**********************************************************************************************************************************/ FN_EXTERN int errorFileLine(void) { assert(errorContext.error.fileLine != 0); return errorContext.error.fileLine; } /**********************************************************************************************************************************/ FN_EXTERN const char * errorMessage(void) { assert(errorContext.error.message != NULL); return errorContext.error.message; } /**********************************************************************************************************************************/ FN_EXTERN const char * errorName(void) { return errorTypeName(errorType()); } /**********************************************************************************************************************************/ FN_EXTERN const char * errorStackTrace(void) { assert(errorContext.error.stackTrace != NULL); return errorContext.error.stackTrace; } /**********************************************************************************************************************************/ FN_EXTERN bool errorInstanceOf(const ErrorType *const errorTypeTest) { return errorType() == errorTypeTest || errorTypeExtends(errorType(), errorTypeTest); } /*********************************************************************************************************************************** Return current error context state ***********************************************************************************************************************************/ static ErrorState errorInternalState(void) { return errorContext.tryList[errorContext.tryTotal].state; } /**********************************************************************************************************************************/ FN_EXTERN void errorInternalTryBegin(const char *const fileName, const char *const functionName, const int fileLine) { // If try total has been exceeded then throw an error if (errorContext.tryTotal >= ERROR_TRY_MAX) errorInternalThrowFmt(&AssertError, fileName, functionName, fileLine, "too many nested try blocks"); // Increment try total errorContext.tryTotal++; // Setup try errorContext.tryList[errorContext.tryTotal].state = errorStateTry; errorContext.tryList[errorContext.tryTotal].uncaught = false; } /**********************************************************************************************************************************/ FN_EXTERN jmp_buf * errorInternalJump(void) { return &errorContext.jumpList[errorContext.tryTotal - 1]; } /**********************************************************************************************************************************/ FN_EXTERN bool errorInternalCatch(const ErrorType *const errorTypeCatch, const bool fatalCatch) { assert(fatalCatch || !errorTypeFatal(errorTypeCatch)); // If just entering error state clean up the stack if (errorInternalState() == errorStateTry) { for (unsigned int handlerIdx = 0; handlerIdx < errorContext.handlerTotal; handlerIdx++) errorContext.handlerList[handlerIdx](errorTryDepth(), fatalCatch); errorContext.tryList[errorContext.tryTotal].state++; } if (errorInternalState() == errorStateCatch && errorInstanceOf(errorTypeCatch) && (fatalCatch || !errorFatal())) { errorContext.tryList[errorContext.tryTotal].uncaught = false; errorContext.tryList[errorContext.tryTotal].state++; return true; } return false; } /**********************************************************************************************************************************/ FN_EXTERN void errorInternalPropagate(void) { assert(errorType() != NULL); // Mark the error as uncaught errorContext.tryList[errorContext.tryTotal].uncaught = true; // If there is a parent try then jump to it if (errorContext.tryTotal > 0) longjmp(errorContext.jumpList[errorContext.tryTotal - 1], 1); // If there was no try to catch this error then output to stderr fprintf(stderr, "\nUncaught %s: %s\n thrown at %s:%d\n\n", errorName(), errorMessage(), errorFileName(), errorFileLine()); fflush(stderr); // Exit with failure exit(UnhandledError.code); } /**********************************************************************************************************************************/ FN_EXTERN void errorInternalTryEnd(void) { // Any catch blocks have been processed and none of them called RETHROW() so clear the error if (errorInternalState() == errorStateEnd && !errorContext.tryList[errorContext.tryTotal].uncaught) errorContext.error = (Error){0}; // Remove the try errorContext.tryTotal--; // If not caught in the last try then propagate if (errorContext.tryList[errorContext.tryTotal + 1].uncaught) errorInternalPropagate(); } /**********************************************************************************************************************************/ FN_EXTERN void errorInternalThrow( const ErrorType *const errorType, const char *const fileName, const char *const functionName, const int fileLine, const char *const message, const char *const stackTrace) { // Setup error data errorContext.error.errorType = errorType; errorContext.error.fileName = fileName; errorContext.error.functionName = functionName; errorContext.error.fileLine = fileLine; // Assign message to the error. If errorMessage() is passed as the message there is no need to make a copy. if (message != messageBuffer) snprintf(messageBuffer, sizeof(messageBuffer), "%s", message); errorContext.error.message = (const char *)messageBuffer; // If a stack trace was provided if (stackTrace != NULL) snprintf(stackTraceBuffer, sizeof(stackTraceBuffer), "%s", stackTrace); // Else generate the stack trace for the error else if ( stackTraceToZ( stackTraceBuffer, sizeof(stackTraceBuffer), errorFileName(), errorFunctionName(), (unsigned int)fileLine) >= sizeof(stackTraceBuffer)) { // Indicate that the stack trace was truncated } errorContext.error.stackTrace = (const char *)stackTraceBuffer; // Propagate the error errorInternalPropagate(); } FN_EXTERN void errorInternalThrowFmt( const ErrorType *const errorType, const char *const fileName, const char *const functionName, const int fileLine, const char *const format, ...) { // Format message va_list argument; va_start(argument, format); vsnprintf(messageBufferTemp, ERROR_MESSAGE_BUFFER_SIZE, format, argument); va_end(argument); errorInternalThrow(errorType, fileName, functionName, fileLine, messageBufferTemp, NULL); } /**********************************************************************************************************************************/ FN_EXTERN void errorInternalThrowSys( const int errNo, const ErrorType *const errorType, const char *const fileName, const char *const functionName, const int fileLine, const char *const message) { // Format message with system message appended if (errNo == 0) snprintf(messageBufferTemp, ERROR_MESSAGE_BUFFER_SIZE, "%s", message); else snprintf(messageBufferTemp, ERROR_MESSAGE_BUFFER_SIZE, "%s: [%d] %s", message, errNo, strerror(errNo)); errorInternalThrow(errorType, fileName, functionName, fileLine, messageBufferTemp, NULL); } #ifdef DEBUG_COVERAGE FN_EXTERN void errorInternalThrowOnSys( const bool error, const int errNo, const ErrorType *const errorType, const char *const fileName, const char *const functionName, const int fileLine, const char *const message) { if (error) errorInternalThrowSys(errNo, errorType, fileName, functionName, fileLine, message); } #endif FN_EXTERN void errorInternalThrowSysFmt( const int errNo, const ErrorType *const errorType, const char *const fileName, const char *const functionName, const int fileLine, const char *const format, ...) { // Format message va_list argument; va_start(argument, format); size_t messageSize = (size_t)vsnprintf(messageBufferTemp, ERROR_MESSAGE_BUFFER_SIZE, format, argument); va_end(argument); // vsnprintf() returns the untruncated length; limit it so the appended system message stays in bounds if (messageSize > ERROR_MESSAGE_BUFFER_SIZE - 1) messageSize = ERROR_MESSAGE_BUFFER_SIZE - 1; // Append the system message if (errNo != 0) snprintf(messageBufferTemp + messageSize, ERROR_MESSAGE_BUFFER_SIZE - messageSize, ": [%d] %s", errNo, strerror(errNo)); errorInternalThrow(errorType, fileName, functionName, fileLine, messageBufferTemp, NULL); } #ifdef DEBUG_COVERAGE FN_EXTERN void errorInternalThrowOnSysFmt( const bool error, const int errNo, const ErrorType *const errorType, const char *const fileName, const char *const functionName, const int fileLine, const char *const format, ...) { if (error) { // Format message va_list argument; va_start(argument, format); size_t messageSize = (size_t)vsnprintf(messageBufferTemp, ERROR_MESSAGE_BUFFER_SIZE, format, argument); va_end(argument); // vsnprintf() returns the untruncated length; limit it so the appended system message stays in bounds if (messageSize > ERROR_MESSAGE_BUFFER_SIZE - 1) messageSize = ERROR_MESSAGE_BUFFER_SIZE - 1; // Append the system message if (errNo != 0) { snprintf( messageBufferTemp + messageSize, ERROR_MESSAGE_BUFFER_SIZE - messageSize, ": [%d] %s", errNo, strerror(errNo)); } errorInternalThrow(errorType, fileName, functionName, fileLine, messageBufferTemp, NULL); } } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/error/error.h0000664000175100017510000005712415265101745020125 0ustar00runnerrunner/*********************************************************************************************************************************** Error Handler Implement a try ... catch ... finally error handler. TRY_BEGIN() { } CATCH(Error1) { } CATCH(Error2) { } CATCH_ANY() { } FINALLY() { } TRY_END(); The CATCH() and FINALLY() blocks are optional but at least one must be specified. There is no need for a TRY block by itself because errors will automatically be propagated to the nearest try block in the call stack. IMPORTANT: If a local variable of the function containing a TRY block is modified in the TRY_BEGIN() block and used later in the function after an error is thrown, that variable must be declared "volatile" if the preserving the value is important. Beware that gcc's -Wclobbered warnings are almost entirely useless for catching such issues. IMPORTANT: Never call return from within any of the error-handling blocks. ***********************************************************************************************************************************/ #ifndef COMMON_ERROR_ERROR_H #define COMMON_ERROR_ERROR_H #include #include #include #include "common/macro.h" /*********************************************************************************************************************************** Error type object ***********************************************************************************************************************************/ typedef struct ErrorType ErrorType; // Macro for declaring new error types #define ERROR_DECLARE(name) \ VR_EXTERN_DECLARE const ErrorType name // Include error type declarations #include "common/error/error.auto.h" // Declare test error #ifdef DEBUG ERROR_DECLARE(TestError); #else // Must always be defined since it might be needed to compile (though not used) during profiling #define TestError AssertError #endif /*********************************************************************************************************************************** Functions to get information about a generic error type ***********************************************************************************************************************************/ // Error type code FN_EXTERN int errorTypeCode(const ErrorType *errorType); // Is the error type fatal FN_EXTERN bool errorTypeFatal(const ErrorType *errorType); // Get error type using a code FN_EXTERN const ErrorType *errorTypeFromCode(int code); // Error type name FN_EXTERN const char *errorTypeName(const ErrorType *errorType); // Error type parent FN_EXTERN const ErrorType *errorTypeParent(const ErrorType *errorType); // Does the child error type extend the parent error type? FN_EXTERN bool errorTypeExtends(const ErrorType *child, const ErrorType *parent); /*********************************************************************************************************************************** Functions to get information about the current error within a CATCH() block. Invalid outside a CATCH() block. ***********************************************************************************************************************************/ // Error type FN_EXTERN const ErrorType *errorType(void); // Error code (pulled from error type) FN_EXTERN int errorCode(void); // Is the error fatal? FN_EXTERN bool errorFatal(void); // Error filename FN_EXTERN const char *errorFileName(void); // Error function name FN_EXTERN const char *errorFunctionName(void); // Error file line number FN_EXTERN int errorFileLine(void); // Is this error an instance of the error type? FN_EXTERN bool errorInstanceOf(const ErrorType *errorTypeTest); // Error message FN_EXTERN const char *errorMessage(void); // Error name (pulled from error type) FN_EXTERN const char *errorName(void); // Error stack trace FN_EXTERN const char *errorStackTrace(void); /*********************************************************************************************************************************** Try stack getters/setters ***********************************************************************************************************************************/ // Get the depth of the current try statement (0 if none) FN_EXTERN unsigned int errorTryDepth(void); // Add a handler to be called when an error occurs typedef void (*const ErrorHandlerFunction)(unsigned int, bool); FN_EXTERN void errorHandlerSet(const ErrorHandlerFunction *list, unsigned int total); /*********************************************************************************************************************************** Begin a block where errors can be thrown ***********************************************************************************************************************************/ #define TRY_BEGIN() \ do \ { \ volatile bool TRY_finally = false; \ (void)TRY_finally; /* Will be unused if there is no finally block */ \ \ errorInternalTryBegin(__FILE__, __func__, __LINE__); \ \ if (setjmp(*errorInternalJump()) == 0) \ { /*********************************************************************************************************************************** Catch a specific error thrown in the try block. Fatal errors cannot be caught with this block. ***********************************************************************************************************************************/ #define CATCH(errorTypeCatch) \ } \ else if (errorInternalCatch(&errorTypeCatch, false)) \ { /*********************************************************************************************************************************** Catch any non-fatal error thrown in the try block ***********************************************************************************************************************************/ #define CATCH_ANY() \ } \ else if (errorInternalCatch(&RuntimeError, false)) \ { /*********************************************************************************************************************************** Catch any error thrown in the try block including fatal errors. To maximize the chance of being able to report the error caught, memory contexts will not be freed and mem context callbacks will not be called. If the TRY block is enclosed in a mem context then child mem contexts will be freed and child mem context callbacks called when the enclosing mem context is freed. ***********************************************************************************************************************************/ #define CATCH_FATAL() \ } \ else if (errorInternalCatch(&RuntimeError, true)) \ { /*********************************************************************************************************************************** Code to run whether the try block was successful or not ***********************************************************************************************************************************/ #define FINALLY() \ } \ \ if (!TRY_finally) \ { \ TRY_finally = true; /*********************************************************************************************************************************** End the try block ***********************************************************************************************************************************/ #define TRY_END() \ } \ \ errorInternalTryEnd(); \ } \ while (0) /*********************************************************************************************************************************** Throw an error Errors can be thrown any time, but if there is no TRY block somewhere in the call stack then the program will exit and print the error information to stderr. The seldom used "THROWP" variants allow an error to be thrown with a pointer to the error type. ***********************************************************************************************************************************/ #define THROW(errorType, message) \ errorInternalThrow(&errorType, __FILE__, __func__, __LINE__, message, NULL) #define THROW_FMT(errorType, ...) \ errorInternalThrowFmt(&errorType, __FILE__, __func__, __LINE__, __VA_ARGS__) #define THROWP(errorType, message) \ errorInternalThrow(errorType, __FILE__, __func__, __LINE__, message, NULL) #define THROWP_FMT(errorType, ...) \ errorInternalThrowFmt(errorType, __FILE__, __func__, __LINE__, __VA_ARGS__) #define THROW_CODE(errorCode, message) \ errorInternalThrow(errorTypeFromCode(errorCode), __FILE__, __func__, __LINE__, message, NULL) #define THROW_CODE_FMT(errorCode, ...) \ errorInternalThrowFmt(errorTypeFromCode(errorCode), __FILE__, __func__, __LINE__, __VA_ARGS__) /*********************************************************************************************************************************** Throw an error when a system call fails ***********************************************************************************************************************************/ #define THROW_SYS_ERROR(errorType, message) \ errorInternalThrowSys(errno, &errorType, __FILE__, __func__, __LINE__, message) #define THROW_SYS_ERROR_FMT(errorType, ...) \ errorInternalThrowSysFmt(errno, &errorType, __FILE__, __func__, __LINE__, __VA_ARGS__) #define THROWP_SYS_ERROR(errorType, message) \ errorInternalThrowSys(errno, errorType, __FILE__, __func__, __LINE__, message) #define THROWP_SYS_ERROR_FMT(errorType, ...) \ errorInternalThrowSysFmt(errno, errorType, __FILE__, __func__, __LINE__, __VA_ARGS__) #define THROW_SYS_ERROR_CODE(errNo, errorType, message) \ errorInternalThrowSys(errNo, &errorType, __FILE__, __func__, __LINE__, message) #define THROW_SYS_ERROR_CODE_FMT(errNo, errorType, ...) \ errorInternalThrowSysFmt(errNo, &errorType, __FILE__, __func__, __LINE__, __VA_ARGS__) #define THROWP_SYS_ERROR_CODE(errNo, errorType, message) \ errorInternalThrowSys(errNo, errorType, __FILE__, __func__, __LINE__, message) #define THROWP_SYS_ERROR_CODE_FMT(errNo, errorType, ...) \ errorInternalThrowSysFmt(errNo, errorType, __FILE__, __func__, __LINE__, __VA_ARGS__) // When coverage testing, define special versions of the macros that don't contain branches. These macros are less efficient because // they need to call errorInternalThrowOnSys*() before determining if there is an error or not, but they allow coverage testing for // THROW*_ON*() calls that contain conditionals. #ifdef DEBUG_COVERAGE // The expression can't be passed directly to errorInternalThrowSys*() because we need to be sure it is evaluated before passing // errno. Depending on optimization that might not happen. #define THROW_ON_SYS_ERROR(expression, errorType, message) \ do \ { \ bool error = expression; \ errorInternalThrowOnSys(error, errno, &errorType, __FILE__, __func__, __LINE__, message); \ } while (0) #define THROW_ON_SYS_ERROR_FMT(expression, errorType, ...) \ do \ { \ bool error = expression; \ errorInternalThrowOnSysFmt(error, errno, &errorType, __FILE__, __func__, __LINE__, __VA_ARGS__); \ } while (0) #define THROWP_ON_SYS_ERROR(expression, errorType, message) \ do \ { \ bool error = expression; \ errorInternalThrowOnSys(error, errno, errorType, __FILE__, __func__, __LINE__, message); \ } while (0) #define THROWP_ON_SYS_ERROR_FMT(expression, errorType, ...) \ do \ { \ bool error = expression; \ errorInternalThrowOnSysFmt(error, errno, errorType, __FILE__, __func__, __LINE__, __VA_ARGS__); \ } while (0) // Else define the normal macros which check for an error first #else #define THROW_ON_SYS_ERROR(expression, errorType, message) \ do \ { \ if (UNLIKELY(expression)) \ errorInternalThrowSys(errno, &errorType, __FILE__, __func__, __LINE__, message); \ } while (0) #define THROW_ON_SYS_ERROR_FMT(expression, errorType, ...) \ do \ { \ if (UNLIKELY(expression)) \ errorInternalThrowSysFmt(errno, &errorType, __FILE__, __func__, __LINE__, __VA_ARGS__); \ } while (0) #define THROWP_ON_SYS_ERROR(expression, errorType, message) \ do \ { \ if (UNLIKELY(expression)) \ errorInternalThrowSys(errno, errorType, __FILE__, __func__, __LINE__, message); \ } while (0) #define THROWP_ON_SYS_ERROR_FMT(expression, errorType, ...) \ do \ { \ if (UNLIKELY(expression)) \ errorInternalThrowSysFmt(errno, errorType, __FILE__, __func__, __LINE__, __VA_ARGS__); \ } while (0) #endif /*********************************************************************************************************************************** Rethrow the current error ***********************************************************************************************************************************/ #define RETHROW() \ errorInternalPropagate() /*********************************************************************************************************************************** Internal functions These functions are used by the macros to implement the error handler and should never be called independently. ***********************************************************************************************************************************/ // Begin the try block FN_EXTERN void errorInternalTryBegin(const char *fileName, const char *functionName, int fileLine); // Return jump buffer for current try FN_EXTERN jmp_buf *errorInternalJump(void); // True when in catch state and the expected error matches FN_EXTERN bool errorInternalCatch(const ErrorType *errorTypeCatch, bool fatalCatch); // Propagate the error up so it can be caught FN_EXTERN noreturn void errorInternalPropagate(void); // End the try block FN_EXTERN void errorInternalTryEnd(void); // Throw an error FN_EXTERN noreturn void errorInternalThrow( const ErrorType *errorType, const char *fileName, const char *functionName, int fileLine, const char *message, const char *stackTrace); FN_EXTERN noreturn FN_PRINTF(5, 6) void errorInternalThrowFmt( const ErrorType *errorType, const char *fileName, const char *functionName, int fileLine, const char *format, ...); // Throw a system error FN_EXTERN noreturn void errorInternalThrowSys( int errNo, const ErrorType *errorType, const char *fileName, const char *functionName, int fileLine, const char *message); // Throw a formatted system error FN_EXTERN FN_PRINTF(6, 7) noreturn void errorInternalThrowSysFmt( int errNo, const ErrorType *errorType, const char *fileName, const char *functionName, int fileLine, const char *format, ...); // Versions of the above for coverage testing which checks the error condition inside the function #ifdef DEBUG_COVERAGE FN_EXTERN void errorInternalThrowOnSys( bool error, int errNo, const ErrorType *errorType, const char *fileName, const char *functionName, int fileLine, const char *message); FN_EXTERN FN_PRINTF(7, 8) void errorInternalThrowOnSysFmt( bool error, int errNo, const ErrorType *errorType, const char *fileName, const char *functionName, int fileLine, const char *format, ...); #endif /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_ERROR_TYPE_TYPE \ ErrorType * #define FUNCTION_LOG_ERROR_TYPE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ErrorType", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/error/retry.c0000664000175100017510000001177515265101745020136 0ustar00runnerrunner/*********************************************************************************************************************************** Error Retry Message ***********************************************************************************************************************************/ #include #include "string.h" #include "common/debug.h" #include "common/error/retry.h" #include "common/time.h" #include "common/type/list.h" #include "common/type/string.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct ErrorRetry { ErrorRetryPub pub; // Publicly accessible variables const String *message; // First error message List *list; // List of retry errors TimeMSec timeBegin; // Time error retries began }; // Keep track of retries with the same error typedef struct ErrorRetryItem { String *message; // Error message unsigned int total; // Total errors with this message const ErrorType *type; // Error type TimeMSec retryFirst; // First retry TimeMSec retryLast; // Last Retry } ErrorRetryItem; /**********************************************************************************************************************************/ FN_EXTERN ErrorRetry * errRetryNew(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(ErrorRetry, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (ErrorRetry) { .timeBegin = timeMSec(), .list = lstNewP(sizeof(ErrorRetryItem), .comparator = lstComparatorStr), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(ERROR_RETRY, this); } /**********************************************************************************************************************************/ FN_EXTERN String * errRetryMessage(const ErrorRetry *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ERROR_RETRY, this); FUNCTION_TEST_END(); ASSERT(this->message != NULL); // Always include first message String *const result = strCat(strNew(), this->message); // Add retries, varying the message depending on how many retries there were for (unsigned int listIdx = 0; listIdx < lstSize(this->list); listIdx++) { const ErrorRetryItem *const error = lstGet(this->list, listIdx); strCatFmt(result, "\n [%s] ", errorTypeName(error->type)); if (error->retryFirst == error->retryLast) strCatFmt(result, "on retry at %" PRIu64, error->retryFirst); else strCatFmt(result, "on %u retries from %" PRIu64 "-%" PRIu64, error->total, error->retryFirst, error->retryLast); strCatFmt(result, "ms: %s", strZ(error->message)); } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void errRetryAdd(ErrorRetry *const this, const ErrRetryAddParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ERROR_RETRY, this); FUNCTION_TEST_PARAM(ERROR_TYPE, param.type); FUNCTION_TEST_PARAM(STRING, param.message); FUNCTION_TEST_END(); // Set defaults const ErrorType *const type = param.type == NULL ? errorType() : param.type; const char *const message = param.message == NULL ? errorMessage() : strZ(param.message); // Set type on first error if (this->pub.type == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->pub.type = type; this->message = strNewZ(message); } MEM_CONTEXT_OBJ_END(); } else { // If error is not found then add it const String *const messageFind = STR(message); const TimeMSec retryTime = timeMSec() - this->timeBegin; ErrorRetryItem *const error = lstFind(this->list, &messageFind); if (error == NULL) { MEM_CONTEXT_OBJ_BEGIN(this->list) { const ErrorRetryItem errorNew = { .type = type, .total = 1, .message = strDup(messageFind), .retryFirst = retryTime, .retryLast = retryTime, }; lstAdd(this->list, &errorNew); } MEM_CONTEXT_OBJ_END(); } // Else update the error found else { error->total++; error->retryLast = retryTime; } } FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/error/retry.h0000664000175100017510000000741615265101745020140 0ustar00runnerrunner/*********************************************************************************************************************************** Error Retry Message Accumulate errors during retries to provide a single, coherent error message to the user that includes information about the original error and all the failed retries. Retries with the same error message are aggregated to keep the error message as short as possible. ***********************************************************************************************************************************/ #ifndef COMMON_ERROR_RETRY_H #define COMMON_ERROR_RETRY_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ErrorRetry ErrorRetry; #include "common/error/error.h" #include "common/type/object.h" #include "common/type/string.h" /*********************************************************************************************************************************** Constructor ***********************************************************************************************************************************/ FN_EXTERN ErrorRetry *errRetryNew(void); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct ErrorRetryPub { const ErrorType *type; // Error type } ErrorRetryPub; // Get error type FN_INLINE_ALWAYS const ErrorType * errRetryType(const ErrorRetry *const this) { return THIS_PUB(ErrorRetry)->type; } // Get error message FN_EXTERN String *errRetryMessage(const ErrorRetry *this); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add error typedef struct ErrRetryAddParam { VAR_PARAM_HEADER; const ErrorType *type; // Error type (defaults to errorType()) const String *message; // Error message (defaults to errorMessage()) } ErrRetryAddParam; #define errRetryAddP(this, ...) \ errRetryAdd(this, (ErrRetryAddParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void errRetryAdd(ErrorRetry *const this, ErrRetryAddParam param); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void errRetryFree(ErrorRetry *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_ERROR_RETRY_TYPE \ ErrorRetry * #define FUNCTION_LOG_ERROR_RETRY_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ErrorRetry", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/exec.c0000664000175100017510000004165315265101745016562 0ustar00runnerrunner/*********************************************************************************************************************************** Execute Process ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include "common/debug.h" #include "common/exec.h" #include "common/fork.h" #include "common/io/fdRead.h" #include "common/io/fdWrite.h" #include "common/io/io.h" #include "common/io/read.h" #include "common/io/write.h" #include "common/log.h" #include "common/wait.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Exec { ExecPub pub; // Publicly accessible variables StringList *param; // List of parameters to pass to command const String *name; // Name to display in log/error messages TimeMSec timeout; // Timeout for any i/o operation (read, write, etc.) pid_t processId; // Process id of the child process int fdRead; // Read file descriptor int fdWrite; // Write file descriptor int fdError; // Error file descriptor IoRead *ioReadFd; // File descriptor read interface IoWrite *ioWriteFd; // File descriptor write interface }; /*********************************************************************************************************************************** Macro to close file descriptors after dup2() in the child process If the parent process is daemomized and has closed stdout, stdin, and stderr or some combination of them, then the newly created descriptors might overlap stdout, stdin, or stderr. In that case we don't want to accidentally close the descriptor that we have just copied. Note that this is pretty specific to the way that file descriptors are handled in this module and may not be generally applicable in other code. ***********************************************************************************************************************************/ #define PIPE_DUP2(pipe, pipeIdx, fd) \ do \ { \ dup2(pipe[pipeIdx], fd); \ \ if (pipe[0] != fd) \ close(pipe[0]); \ \ if (pipe[1] != fd) \ close(pipe[1]); \ } \ while (0); /*********************************************************************************************************************************** Free exec file descriptors and ensure process is shut down ***********************************************************************************************************************************/ static void execFreeResource(THIS_VOID) { THIS(Exec); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(EXEC, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close file descriptors close(this->fdRead); close(this->fdWrite); close(this->fdError); // Wait for the child to exit. We don't really care how it exits as long as it does. if (this->processId != 0) { MEM_CONTEXT_TEMP_BEGIN() { int processResult = 0; Wait *const wait = waitNew(this->timeout); do { THROW_ON_SYS_ERROR( (processResult = waitpid(this->processId, NULL, WNOHANG)) == -1, ExecuteError, "unable to wait on child process"); } while (processResult == 0 && waitMore(wait)); // If the process did not exit then error -- else we may end up with a collection of zombie processes if (processResult == 0) THROW_FMT(ExecuteError, "%s did not exit when expected", strZ(this->name)); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Exec * execNew(const String *const command, const StringList *const param, const String *const name, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, command); FUNCTION_LOG_PARAM(STRING_LIST, param); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(name != NULL); ASSERT(timeout > 0); OBJ_NEW_BEGIN(Exec, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (Exec) { .name = strDup(name), .timeout = timeout, // Parameter list is optional but if not specified we need to build one with the command .param = param == NULL ? strLstNew() : strLstDup(param), }; // If command is NULL then the first item in the param list is the command if (command == NULL) { ASSERT(param != NULL && strLstSize(param) != 0); this->pub.command = strLstGet(param, 0); } // Else copy command and insert it as the first parameter else { this->pub.command = strDup(command); strLstInsert(this->param, 0, execCommand(this)); } } OBJ_NEW_END(); FUNCTION_LOG_RETURN(EXEC, this); } /*********************************************************************************************************************************** Check if the process is still running This should be called when anything unexpected happens while reading or writing, including errors and eof. If this function returns then the original error should be rethrown. ***********************************************************************************************************************************/ // Helper to throw status error static void execCheckStatusError(Exec *const this, int status, const String *const output) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(EXEC, this); FUNCTION_TEST_PARAM(INT, status); FUNCTION_TEST_PARAM(STRING, output); FUNCTION_TEST_END(); // Throw the error with as much information as is available THROWP_FMT( errorTypeFromCode(WEXITSTATUS(status)), "%s terminated unexpectedly [%d]%s%s", strZ(this->name), WEXITSTATUS(status), strSize(output) > 0 ? ": " : "", strSize(output) > 0 ? strZ(output) : ""); FUNCTION_TEST_RETURN_VOID(); } // Helper to throw signal error static void execCheckSignalError(Exec *const this, int status) { THROW_FMT(ExecuteError, "%s terminated unexpectedly on signal %d", strZ(this->name), WTERMSIG(status)); } static void execCheck(Exec *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(EXEC, this); FUNCTION_LOG_END(); ASSERT(this != NULL); int processStatus; int processResult; THROW_ON_SYS_ERROR( (processResult = waitpid(this->processId, &processStatus, WNOHANG)) == -1, ExecuteError, "unable to wait on child process"); if (processResult != 0) { // Clear the process id so we don't try to wait for this process on free this->processId = 0; // If the process exited normally if (WIFEXITED(processStatus)) { execCheckStatusError( this, processStatus, strTrim(strNewBuf(ioReadBuf(ioFdReadNewOpen(strNewFmt("%s error", strZ(this->name)), this->fdError, 0))))); } // If the process did not exit normally then it must have been a signal execCheckSignalError(this, processStatus); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Read from the process ***********************************************************************************************************************************/ static size_t execRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(Exec); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(EXEC, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); size_t result; TRY_BEGIN() { result = ioReadInterface(this->ioReadFd)->read(ioReadDriver(this->ioReadFd), buffer, block); } CATCH_ANY() { execCheck(this); RETHROW(); } TRY_END(); FUNCTION_LOG_RETURN(SIZE, result); } /*********************************************************************************************************************************** Write to the process ***********************************************************************************************************************************/ static void execWrite(THIS_VOID, const Buffer *const buffer) { THIS(Exec); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(EXEC, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); TRY_BEGIN() { ioWrite(this->ioWriteFd, buffer); ioWriteFlush(this->ioWriteFd); } CATCH_ANY() { execCheck(this); RETHROW(); } TRY_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is the process eof? ***********************************************************************************************************************************/ static bool execEof(THIS_VOID) { THIS(Exec); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(EXEC, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Check if the process is still running on eof if (ioReadInterface(this->ioReadFd)->eof(ioReadDriver(this->ioReadFd))) execCheck(this); FUNCTION_LOG_RETURN(BOOL, false); } /*********************************************************************************************************************************** Get the read file descriptor ***********************************************************************************************************************************/ static int execFdRead(const THIS_VOID) { THIS(const Exec); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(EXEC, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->fdRead); } /**********************************************************************************************************************************/ FN_EXTERN void execOpen(Exec *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(EXEC, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Create pipes to communicate with the subprocess. The names of the pipes are from the perspective of the parent process since // the child process will use them only briefly before exec'ing. int pipeRead[2]; int pipeWrite[2]; int pipeError[2]; THROW_ON_SYS_ERROR(pipe(pipeRead) == -1, KernelError, "unable to create read pipe"); THROW_ON_SYS_ERROR(pipe(pipeWrite) == -1, KernelError, "unable to create write pipe"); THROW_ON_SYS_ERROR(pipe(pipeError) == -1, KernelError, "unable to create error pipe"); // Fork the subprocess this->processId = forkSafe(); // Exec command in the child process if (this->processId == 0) { // Disable logging and close log file logClose(); // Assign stdout to the input side of the read pipe PIPE_DUP2(pipeRead, 1, STDOUT_FILENO); // Assign stdin to the output side of the write pipe PIPE_DUP2(pipeWrite, 0, STDIN_FILENO); // Assign stderr to the input side of the error pipe PIPE_DUP2(pipeError, 1, STDERR_FILENO); // Execute the binary. This statement will not return if it is successful. execvp() requires non-const parameters because it // modifies them after the fork. #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wcast-qual" execvp(strZ(execCommand(this)), UNCONSTIFY(char **, strLstPtr(this->param))); #pragma GCC diagnostic pop // If we got here then there was an error. We can't use a throw as we normally would because we have already shutdown // logging and we don't want to execute exit paths that might free parent resources which we still have references to. fprintf(stderr, "unable to execute '%s': [%d] %s\n", strZ(execCommand(this)), errno, strerror(errno)); exit(errorTypeCode(&ExecuteError)); } // Close the unused file descriptors close(pipeRead[1]); close(pipeWrite[0]); close(pipeError[1]); // Store the file descriptors we'll use and need to close when the process terminates this->fdRead = pipeRead[0]; this->fdWrite = pipeWrite[1]; this->fdError = pipeError[0]; MEM_CONTEXT_OBJ_BEGIN(this) { // Assign file descriptors to io interfaces this->ioReadFd = ioFdReadNew(strNewFmt("%s read", strZ(this->name)), this->fdRead, this->timeout); this->ioWriteFd = ioFdWriteNewOpen(strNewFmt("%s write", strZ(this->name)), this->fdWrite, this->timeout); // Create wrapper interfaces that check process state this->pub.ioReadExec = ioReadNewP(this, .block = true, .read = execRead, .eof = execEof, .fd = execFdRead); ioReadOpen(execIoRead(this)); this->pub.ioWriteExec = ioWriteNewP(this, .write = execWrite); ioWriteOpen(execIoWrite(this)); } MEM_CONTEXT_OBJ_END(); // Set a callback so the file descriptors will get freed memContextCallbackSet(objMemContext(this), execFreeResource, this); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * execOne(const StringList *const list) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_LIST, list); FUNCTION_LOG_END(); ASSERT(list != NULL); ASSERT(strLstSize(list) != 0); String *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Exec command Exec *const exec = execNew(NULL, list, strLstGet(list, 0), ioTimeoutMs()); execOpen(exec); // Read all output from stdout ioReadOpen(exec->ioReadFd); String *const output = strNewBuf(ioReadBuf(exec->ioReadFd)); // Wait for the process to exit int processStatus; THROW_ON_SYS_ERROR( waitpid(exec->processId, &processStatus, 0) == -1, ExecuteError, "unable to wait on child process"); // Clear the process id so we don't try to wait for this process on free exec->processId = 0; // Check exit status if (WIFEXITED(processStatus)) { if (WEXITSTATUS(processStatus) != 0) execCheckStatusError(exec, processStatus, strTrim(output)); } // Else if the process did not exit normally then it must have been a signal else execCheckSignalError(exec, processStatus); MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(strTrim(output)); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/exec.h0000664000175100017510000001033015265101745016553 0ustar00runnerrunner/*********************************************************************************************************************************** Execute Process Executes a child process and allows the calling process to communicate with it using read/write io. This object is specially tailored to implement the protocol layer and may or may not be generally applicable to general purpose execution. ***********************************************************************************************************************************/ #ifndef COMMON_EXEC_H #define COMMON_EXEC_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Exec Exec; #include "common/io/read.h" #include "common/io/write.h" #include "common/time.h" #include "common/type/object.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Exec *execNew(const String *command, const StringList *param, const String *name, TimeMSec timeout); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct ExecPub { String *command; // Command to execute IoRead *ioReadExec; // Wrapper for file descriptor read interface IoWrite *ioWriteExec; // Wrapper for file descriptor write interface } ExecPub; // Exec command FN_INLINE_ALWAYS const String * execCommand(const Exec *const this) { return THIS_PUB(Exec)->command; } // Read interface FN_INLINE_ALWAYS IoRead * execIoRead(Exec *const this) { return THIS_PUB(Exec)->ioReadExec; } // Write interface FN_INLINE_ALWAYS IoWrite * execIoWrite(Exec *const this) { return THIS_PUB(Exec)->ioWriteExec; } // Exec MemContext FN_INLINE_ALWAYS MemContext * execMemContext(Exec *const this) { return objMemContext(this); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Execute command FN_EXTERN void execOpen(Exec *this); /*********************************************************************************************************************************** Helper Functions ***********************************************************************************************************************************/ // Execute a command with an explicit parameter list and return stdout output. The first item in the list is the command and the // remaining items are parameters. FN_EXTERN String *execOne(const StringList *list); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void execFree(Exec *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_EXEC_TYPE \ Exec * #define FUNCTION_LOG_EXEC_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Exec", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/fork.c0000664000175100017510000000463415265101745016575 0ustar00runnerrunner/*********************************************************************************************************************************** Fork Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/fork.h" #include "common/log.h" /**********************************************************************************************************************************/ FN_EXTERN int forkSafe(void) { FUNCTION_LOG_VOID(logLevelTrace); const int result = fork(); THROW_ON_SYS_ERROR(result == -1, KernelError, "unable to fork"); FUNCTION_LOG_RETURN(INT, result); } /**********************************************************************************************************************************/ FN_EXTERN void forkDetach(void) { FUNCTION_LOG_VOID(logLevelTrace); // Make this process a group leader so the parent process won't block waiting for it to finish THROW_ON_SYS_ERROR(setsid() == -1, KernelError, "unable to create new session group"); // The process should never receive a SIGHUP but ignore it just in case signal(SIGHUP, SIG_IGN); // There should be no way the child process can exit first (after the next fork) but just in case ignore SIGCHLD. This means // that the child process will automatically be reaped by the kernel should it finish first rather than becoming defunct. signal(SIGCHLD, SIG_IGN); // Fork again and let the parent process terminate to ensure that we get rid of the session leading process. Only session // leaders may get a TTY again. if (forkSafe() != 0) exit(0); // Reset SIGCHLD to the default handler so waitpid() calls in the process will work as expected signal(SIGCHLD, SIG_DFL); // Change the working directory to / so there is no dependency on the original working directory THROW_ON_SYS_ERROR(chdir("/") == -1, PathMissingError, "unable to change directory to '/'"); // Close standard file descriptors THROW_ON_SYS_ERROR(close(STDIN_FILENO) == -1, FileCloseError, "unable to close stdin"); THROW_ON_SYS_ERROR(close(STDOUT_FILENO) == -1, FileCloseError, "unable to close stdout"); THROW_ON_SYS_ERROR(close(STDERR_FILENO) == -1, FileCloseError, "unable to close stderr"); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/fork.h0000664000175100017510000000164715265101745016603 0ustar00runnerrunner/*********************************************************************************************************************************** Fork Handler ***********************************************************************************************************************************/ #ifndef COMMON_FORK_H #define COMMON_FORK_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Fork a new process and throw an error if it fails FN_EXTERN int forkSafe(void); // Detach a forked process so it can continue running after the parent process has exited. This is not a typical daemon startup // because the parent process may continue to run and perform work for some time. FN_EXTERN void forkDetach(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/format/0000775000175100017510000000000015265101745016751 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/format/cipherBlockFormat.c0000664000175100017510000006435715265101745022532 0ustar00runnerrunner/*********************************************************************************************************************************** Cipher Block Format ***********************************************************************************************************************************/ #include #include #include #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/format/cipherBlockFormat.h" #include "common/format/format.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/convert.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Format header A file written at a format that has one begins with fixed-size plaintext naming the format it was written with. The header exists because the digest the pass derives with follows the format, and the format is recorded inside the file that the pass encrypts. A reader that could not see the format in advance would have to decrypt to learn what it should have decrypted with. The header takes the place of the salted magic, which is why the content behind it is encrypted raw. Both are eight bytes followed by the salt, so the eight bytes are consumed either way and what follows begins with the salt no matter which was there. It also means a file of either kind is opened with the openssl command-line tool the same way: replace the first eight bytes with the magic that tool expects. A file that begins with the magic rather than the header was written at format 5, the only format there was before the header, so that is not an error when a header was expected. The first three bytes of the header magic are the format and the last is a marker indicating whether a key id follows. The format comes first so it is always in the same place, which is how a version decides whether it can read the file. The marker is only checked once the format turns out to be readable, and it must be valid for this version. If a key id is indicated by the marker, there is first a byte to indicate the length of the key id and then the key id as a string. ***********************************************************************************************************************************/ #define CIPHER_BLOCK_FORMAT_MAGIC "PGBR" #define CIPHER_BLOCK_FORMAT_MAGIC_SIZE (sizeof(CIPHER_BLOCK_FORMAT_MAGIC) - 1) #define CIPHER_BLOCK_FORMAT_MARKER_NONE '_' #define CIPHER_BLOCK_FORMAT_MARKER_KEY 'K' #define CIPHER_BLOCK_FORMAT_SIZE 3 // The key id length is encoded as a single byte with a max of 255 characters #define CIPHER_BLOCK_FORMAT_KEY_SIZE_MAX 255 // Total header length: magic, format, and marker #define CIPHER_BLOCK_FORMAT_HEADER_SIZE (CIPHER_BLOCK_FORMAT_MAGIC_SIZE + CIPHER_BLOCK_FORMAT_SIZE + 1) // The header is written in place of the magic, so the two must add up to exactly the same size or the salt would no longer begin at // the same place static_assert( CIPHER_BLOCK_FORMAT_HEADER_SIZE == CIPHER_BLOCK_MAGIC_SIZE, "cipher header must be the size of the magic it replaces"); // A format too large for the digits it is written in would be truncated to a different format static_assert(REPOSITORY_FORMAT_MAX < 1000, "repository format must fit in the header digits"); static_assert(CIPHER_BLOCK_FORMAT_SIZE == 3, "format digits must match the digits the header is written with"); // Max possible size of the header, the fixed part plus a length byte and the longest key id it can contain #define CIPHER_BLOCK_FORMAT_HEADER_SIZE_MAX \ (CIPHER_BLOCK_FORMAT_HEADER_SIZE + 1 + CIPHER_BLOCK_FORMAT_KEY_SIZE_MAX) /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct CipherBlockFormat { const CipherSpecMap *cipherSpecMap; // Keys to choose from unsigned int formatExpected; // Format the caller expects, zero when any will do unsigned int format; // Format the header gave uint8_t header[CIPHER_BLOCK_FORMAT_HEADER_SIZE_MAX]; // Header bytes held until there are enough to read size_t headerSize; // Header bytes held so far size_t headerSizeExpected; // Total header size, which grows as the header is read IoFilter *cipherBlock; // Block cipher the content behind the header is decrypted by size_t sourceOffset; // Bytes of the current source taken for the header } CipherBlockFormat; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void cipherBlockFormatToLog(const CipherBlockFormat *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{format: %u}", this->format); } #define FUNCTION_LOG_CIPHER_BLOCK_FORMAT_TYPE \ CipherBlockFormat * #define FUNCTION_LOG_CIPHER_BLOCK_FORMAT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, cipherBlockFormatToLog, buffer, bufferSize) /*********************************************************************************************************************************** Read the header and return the format it gives. A file that begins with the magic instead was written at format 5, the only format there was before the header, so that start is not an error here. ***********************************************************************************************************************************/ static unsigned int cipherBlockFormatHeaderRead(const uint8_t *const header) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(BYTEDATA, header); FUNCTION_TEST_END(); ASSERT(header != NULL); const char *const headerZ = (const char *)header; unsigned int result = REPOSITORY_FORMAT_5; if (memcmp(headerZ, CIPHER_BLOCK_FORMAT_MAGIC, CIPHER_BLOCK_FORMAT_MAGIC_SIZE) == 0) { for (unsigned int digitIdx = 0; digitIdx < CIPHER_BLOCK_FORMAT_SIZE; digitIdx++) { if (!isdigit((unsigned char)headerZ[CIPHER_BLOCK_FORMAT_MAGIC_SIZE + digitIdx])) THROW(FormatError, "invalid cipher header"); } result = cvtZSubNToUInt(headerZ, CIPHER_BLOCK_FORMAT_MAGIC_SIZE, CIPHER_BLOCK_FORMAT_SIZE); // Error on a format this version cannot read before anything is decrypted repoFormatValidate(result); // Check that the marker is valid for this version if (headerZ[CIPHER_BLOCK_FORMAT_HEADER_SIZE - 1] != CIPHER_BLOCK_FORMAT_MARKER_NONE && headerZ[CIPHER_BLOCK_FORMAT_HEADER_SIZE - 1] != CIPHER_BLOCK_FORMAT_MARKER_KEY) { THROW(FormatError, "invalid cipher header"); } } // Else the bytes must be the magic, since that is all a file with no header can begin with else if (memcmp(headerZ, CIPHER_BLOCK_MAGIC, CIPHER_BLOCK_MAGIC_SIZE) != 0) THROW(CryptoError, "cipher header invalid"); FUNCTION_TEST_RETURN(UINT, result); } /*********************************************************************************************************************************** Build the block cipher that decrypts the content, which is only possible once the header has been read since the header defines the format and the key id ***********************************************************************************************************************************/ static void cipherBlockFormatCipherNew(CipherBlockFormat *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CIPHER_BLOCK_FORMAT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->cipherBlock == NULL); // Error when the format the caller expected is not the one the file was written with if (this->formatExpected != 0 && this->formatExpected != this->format) THROW_FMT(FormatError, "expected repository format %u but found %u", this->formatExpected, this->format); // Get the key specified in the header or the default key when none is specified const CipherSpec *cipherSpec; MEM_CONTEXT_TEMP_BEGIN() { const String *keyId = CIPHER_SPEC_MAP_ID_DEFAULT_STR; if (this->headerSize > CIPHER_BLOCK_FORMAT_HEADER_SIZE) { keyId = strNewZN( (const char *)this->header + CIPHER_BLOCK_FORMAT_HEADER_SIZE + 1, this->headerSize - CIPHER_BLOCK_FORMAT_HEADER_SIZE - 1); } cipherSpec = cipherSpecMapGet(this->cipherSpecMap, keyId); } MEM_CONTEXT_TEMP_END(); // Decrypt raw since the format header has already been parsed MEM_CONTEXT_OBJ_BEGIN(this) { this->cipherBlock = cipherBlockNewP( cipherModeDecrypt, cipherSpecDupP(cipherSpec, .digestDefault = repoFormatDigest(this->format)), .header = cipherBlockHeaderNone); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Read the header and pass everything behind it through the block cipher the header selected ***********************************************************************************************************************************/ static void cipherBlockFormatProcess(THIS_VOID, const Buffer *const source, Buffer *const destination) { THIS(CipherBlockFormat); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CIPHER_BLOCK_FORMAT, this); FUNCTION_LOG_PARAM(BUFFER, source); FUNCTION_LOG_PARAM(BUFFER, destination); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(destination != NULL); if (source != NULL) { // Read until the header is complete. The total size is not known until the marker and key id length have been read. if (this->cipherBlock == NULL) { this->sourceOffset = 0; do { size_t copySize = this->headerSizeExpected - this->headerSize; if (copySize > bufUsed(source) - this->sourceOffset) copySize = bufUsed(source) - this->sourceOffset; memcpy(this->header + this->headerSize, bufPtrConst(source) + this->sourceOffset, copySize); this->headerSize += copySize; this->sourceOffset += copySize; // Nothing can be decrypted until the header is complete if (this->headerSize < this->headerSizeExpected) FUNCTION_LOG_RETURN_VOID(); // Parse the fixed part first to reject an invalid header before using the marker if (this->headerSizeExpected == CIPHER_BLOCK_FORMAT_HEADER_SIZE) { this->format = cipherBlockFormatHeaderRead(this->header); if (this->header[CIPHER_BLOCK_FORMAT_HEADER_SIZE - 1] == CIPHER_BLOCK_FORMAT_MARKER_KEY) this->headerSizeExpected++; } // Else the length is complete so wait for the key id else if (this->headerSizeExpected == CIPHER_BLOCK_FORMAT_HEADER_SIZE + 1) { const size_t keySize = this->header[CIPHER_BLOCK_FORMAT_HEADER_SIZE]; // Marker indicates a key id but the length is zero if (keySize == 0) THROW(FormatError, "invalid cipher header"); this->headerSizeExpected += keySize; } } while (this->headerSize < this->headerSizeExpected); cipherBlockFormatCipherNew(this); } // Decrypt whatever of the source the header did not take if (this->sourceOffset < bufUsed(source)) { ioFilterProcessInOut( this->cipherBlock, BUF(bufPtrConst(source) + this->sourceOffset, bufUsed(source) - this->sourceOffset), destination); } // The header is stripped from the source only once if (!ioFilterInputSame(this->cipherBlock)) this->sourceOffset = 0; } // Else all the input has been seen, so the block cipher is flushed else { // A file too short to hold a header cannot have one if (this->cipherBlock == NULL) THROW(CryptoError, "cipher header missing"); ioFilterProcessInOut(this->cipherBlock, NULL, destination); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is the filter done? ***********************************************************************************************************************************/ static bool cipherBlockFormatDone(const THIS_VOID) { THIS(const CipherBlockFormat); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CIPHER_BLOCK_FORMAT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->cipherBlock != NULL && ioFilterDone(this->cipherBlock)); } /*********************************************************************************************************************************** Should the same input be provided again? ***********************************************************************************************************************************/ static bool cipherBlockFormatInputSame(const THIS_VOID) { THIS(const CipherBlockFormat); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CIPHER_BLOCK_FORMAT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->cipherBlock != NULL && ioFilterInputSame(this->cipherBlock)); } /*********************************************************************************************************************************** Report the format the header gave ***********************************************************************************************************************************/ static Pack * cipherBlockFormatResultPack(THIS_VOID) { THIS(CipherBlockFormat); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CIPHER_BLOCK_FORMAT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); Pack *result; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteU32P(packWrite, this->format); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * cipherBlockFormatNew(const CipherSpecMap *const cipherSpecMap, const CipherBlockFormatNewParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CIPHER_SPEC_MAP, cipherSpecMap); FUNCTION_LOG_PARAM(UINT, param.format); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(cipherSpecMap != NULL); ASSERT(cipherSpecMapSize(cipherSpecMap) != 0); OBJ_NEW_BEGIN(CipherBlockFormat, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (CipherBlockFormat) { .cipherSpecMap = cipherSpecMapDup(cipherSpecMap), .formatExpected = param.format, .headerSizeExpected = CIPHER_BLOCK_FORMAT_HEADER_SIZE, }; } OBJ_NEW_END(); // Create param list Pack *paramList; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); cipherSpecMapPack(packWrite, cipherSpecMap); pckWriteU32P(packWrite, param.format); pckWriteEndP(packWrite); paramList = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( CIPHER_BLOCK_FORMAT_FILTER_TYPE, this, paramList, .done = cipherBlockFormatDone, .inOut = cipherBlockFormatProcess, .inputSame = cipherBlockFormatInputSame, .result = cipherBlockFormatResultPack)); } FN_EXTERN IoFilter * cipherBlockFormatNewPack(const Pack *const paramList) { IoFilter *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackRead *const paramListPack = pckReadNew(paramList); const CipherSpecMap *const cipherSpecMap = cipherSpecMapNewPack(paramListPack); const unsigned int format = pckReadU32P(paramListPack); result = ioFilterMove(cipherBlockFormatNewP(cipherSpecMap, .format = format), memContextPrior()); } MEM_CONTEXT_TEMP_END(); return result; } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cipherBlockFormatResult(PackRead *const packRead) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, packRead); FUNCTION_TEST_END(); ASSERT(packRead != NULL); FUNCTION_TEST_RETURN(UINT, pckReadU32P(packRead)); } /*********************************************************************************************************************************** Filter to write the header before the content. The header must be plaintext so this filter runs after the block cipher. ***********************************************************************************************************************************/ typedef struct CipherBlockFormatHeader { const Buffer *header; // Header to write before any content size_t headerOffset; // Header bytes written so far size_t sourceOffset; // Content bytes written from the current source bool inputSame; // Is the same input required on the next process call? } CipherBlockFormatHeader; static void cipherBlockFormatHeaderProcess(THIS_VOID, const Buffer *const source, Buffer *const destination) { THIS(CipherBlockFormatHeader); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, this); FUNCTION_LOG_PARAM(BUFFER, source); FUNCTION_LOG_PARAM(BUFFER, destination); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(source != NULL); ASSERT(destination != NULL); // Write the header before any content if (this->headerOffset < bufUsed(this->header)) { size_t copySize = bufUsed(this->header) - this->headerOffset; if (copySize > bufRemains(destination)) copySize = bufRemains(destination); bufCatC(destination, bufPtrConst(this->header), this->headerOffset, copySize); this->headerOffset += copySize; } // Write the content once the header is complete if (this->headerOffset == bufUsed(this->header)) { size_t copySize = bufUsed(source) - this->sourceOffset; if (copySize > bufRemains(destination)) copySize = bufRemains(destination); bufCatC(destination, bufPtrConst(source), this->sourceOffset, copySize); this->sourceOffset += copySize; } this->inputSame = this->headerOffset < bufUsed(this->header) || this->sourceOffset < bufUsed(source); // Reset source offset when source has been completely processed if (!this->inputSame) this->sourceOffset = 0; FUNCTION_LOG_RETURN_VOID(); } static bool cipherBlockFormatHeaderInputSame(const THIS_VOID) { THIS(const CipherBlockFormatHeader); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } static IoFilter * cipherBlockFormatHeaderNew(const Buffer *const header) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, header); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(header != NULL); ASSERT(!bufEmpty(header)); OBJ_NEW_BEGIN(CipherBlockFormatHeader, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (CipherBlockFormatHeader){.header = bufDup(header)}; } OBJ_NEW_END(); // Create param list Pack *paramList; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteBinP(packWrite, header); pckWriteEndP(packWrite); paramList = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN( IO_FILTER, ioFilterNewP( CIPHER_BLOCK_FORMAT_HEADER_FILTER_TYPE, this, paramList, .inOut = cipherBlockFormatHeaderProcess, .inputSame = cipherBlockFormatHeaderInputSame)); } FN_EXTERN IoFilter * cipherBlockFormatHeaderNewPack(const Pack *const paramList) { IoFilter *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { result = ioFilterMove(cipherBlockFormatHeaderNew(pckReadBinP(pckReadNew(paramList))), memContextPrior()); } MEM_CONTEXT_TEMP_END(); return result; } /**********************************************************************************************************************************/ FN_EXTERN void cipherBlockFormatFilterGroupWriteAdd( IoFilterGroup *const filterGroup, const CipherSpec *const cipherSpec, const unsigned int format, const CipherBlockFormatFilterGroupWriteAddParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_PARAM(UINT, format); FUNCTION_LOG_PARAM(STRING, param.keyId); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(filterGroup != NULL); ASSERT(cipherSpec != NULL); ASSERT(format >= REPOSITORY_FORMAT_MIN && format <= REPOSITORY_FORMAT_MAX); ASSERT(param.keyId == NULL || !strEmpty(param.keyId)); ASSERT(param.keyId == NULL || strSize(param.keyId) <= CIPHER_BLOCK_FORMAT_KEY_SIZE_MAX); // Key id is only valid for format >= 6 ASSERT(param.keyId == NULL || format >= REPOSITORY_FORMAT_6); if (cipherSpecType(cipherSpec) != cipherTypeNone) { // Format >= 6 writes the header in place of the magic and the content behind it is encrypted raw const bool header = format >= REPOSITORY_FORMAT_6; ioFilterGroupAdd( filterGroup, cipherBlockNewP( cipherModeEncrypt, cipherSpecDupP(cipherSpec, .digestDefault = repoFormatDigest(format)), .header = header ? cipherBlockHeaderNone : cipherBlockHeaderMagic)); // The filter is added after the block cipher so the plaintext header is placed before the cipher output if (header) { MEM_CONTEXT_TEMP_BEGIN() { char headerZ[CIPHER_BLOCK_FORMAT_HEADER_SIZE + 1]; // Zero-pad the format and add the marker, depending on whether or not there is a key snprintf( headerZ, sizeof(headerZ), CIPHER_BLOCK_FORMAT_MAGIC "%03u%c", format % 1000, param.keyId == NULL ? CIPHER_BLOCK_FORMAT_MARKER_NONE : CIPHER_BLOCK_FORMAT_MARKER_KEY); Buffer *const headerBuffer = bufNew(CIPHER_BLOCK_FORMAT_HEADER_SIZE_MAX); bufCatC(headerBuffer, (const uint8_t *)headerZ, 0, CIPHER_BLOCK_FORMAT_HEADER_SIZE); // Write the key size and id when present if (param.keyId != NULL) { const uint8_t keySize = (uint8_t)strSize(param.keyId); bufCatC(headerBuffer, &keySize, 0, sizeof(keySize)); bufCatC(headerBuffer, (const uint8_t *)strZ(param.keyId), 0, strSize(param.keyId)); } ioFilterGroupAdd(filterGroup, cipherBlockFormatHeaderNew(headerBuffer)); } MEM_CONTEXT_TEMP_END(); } } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN IoFilterGroup * cipherBlockFormatFilterGroupReadAdd(IoFilterGroup *const filterGroup, const CipherSpec *const cipherSpec) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(filterGroup != NULL); ASSERT(cipherSpec != NULL); if (cipherSpecType(cipherSpec) != cipherTypeNone) { MEM_CONTEXT_TEMP_BEGIN() { CipherSpecMap *const cipherSpecMap = cipherSpecMapNew(); cipherSpecMapAdd(cipherSpecMap, CIPHER_SPEC_MAP_ID_DEFAULT_STR, cipherSpec); ioFilterGroupAdd(filterGroup, cipherBlockFormatNewP(cipherSpecMap)); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(IO_FILTER_GROUP, filterGroup); } /**********************************************************************************************************************************/ FN_EXTERN IoFilterGroup * cipherBlockFormatFilterGroupReadAddMap(IoFilterGroup *const filterGroup, const CipherSpecMap *const cipherSpecMap) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_LOG_PARAM(CIPHER_SPEC_MAP, cipherSpecMap); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(filterGroup != NULL); ASSERT(cipherSpecMap != NULL); // An unencrypted repository has no keys, so there is nothing to decrypt if (cipherSpecMapSize(cipherSpecMap) != 0) ioFilterGroupAdd(filterGroup, cipherBlockFormatNewP(cipherSpecMap)); FUNCTION_LOG_RETURN(IO_FILTER_GROUP, filterGroup); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/format/cipherBlockFormat.h0000664000175100017510000001231415265101745022521 0ustar00runnerrunner/*********************************************************************************************************************************** Cipher Block Format Encryption of a file written at a repository format, which is the format header in front of the content and the digest the format derives the key with. Everything the format defines is here so that the block cipher itself has no knowledge of the repository. A writer knows the format, so it writes the header itself and encrypts the content raw behind it. A reader cannot know the format until it has read the header, and a filter cannot be added to a read once it is open, so the read side is a filter that parses the header, works out from it what the content is encrypted with, and passes everything after it through a block cipher of its own. A file whose key cannot be determined from where the file is located stores the key in its header. ***********************************************************************************************************************************/ #ifndef COMMON_FORMAT_CIPHERBLOCKFORMAT_H #define COMMON_FORMAT_CIPHERBLOCKFORMAT_H #include "common/crypto/spec.h" #include "common/format/cipherSpecMap.h" #include "common/io/filter/group.h" /*********************************************************************************************************************************** Filter type constants ***********************************************************************************************************************************/ #define CIPHER_BLOCK_FORMAT_FILTER_TYPE STRID5("cipher-fmt", 0x28d36e45441230) // Filter that writes the format header #define CIPHER_BLOCK_FORMAT_HEADER_FILTER_TYPE STRID5("cipher-hdr", 0x24446e45441230) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct CipherBlockFormatNewParam { VAR_PARAM_HEADER; unsigned int format; // Format the file is expected at, any when zero } CipherBlockFormatNewParam; // Decrypt with the stored key id or with CIPHER_SPEC_MAP_ID_DEFAULT when no key is stored #define cipherBlockFormatNewP(cipherSpecMap, ...) \ cipherBlockFormatNew(cipherSpecMap, (CipherBlockFormatNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN IoFilter *cipherBlockFormatNew(const CipherSpecMap *cipherSpecMap, CipherBlockFormatNewParam param); FN_EXTERN IoFilter *cipherBlockFormatNewPack(const Pack *paramList); // Create filter that writes the format header from its param list FN_EXTERN IoFilter *cipherBlockFormatHeaderNewPack(const Pack *paramList); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // The format the header gave, which is what the header was read for FN_EXTERN unsigned int cipherBlockFormatResult(PackRead *packRead); /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Write the header the format requires into the buffer the content will be written into and add encryption to the filter group. A // key id is stored in the header when one is specified. No header is added when the repository is not encrypted. typedef struct CipherBlockFormatFilterGroupWriteAddParam { VAR_PARAM_HEADER; const String *keyId; // Key id to store in the header, NULL to store no id } CipherBlockFormatFilterGroupWriteAddParam; #define cipherBlockFormatFilterGroupWriteAddP(filterGroup, cipherSpec, format, ...) \ cipherBlockFormatFilterGroupWriteAdd( \ filterGroup, cipherSpec, format, (CipherBlockFormatFilterGroupWriteAddParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void cipherBlockFormatFilterGroupWriteAdd( IoFilterGroup *filterGroup, const CipherSpec *cipherSpec, unsigned int format, CipherBlockFormatFilterGroupWriteAddParam param); // Add decryption to the filter group for a file with format yet to be determined. The key is stored under // CIPHER_SPEC_MAP_ID_DEFAULT, so the file must not contain a key id. Nothing is added when the repository is not encrypted. FN_EXTERN IoFilterGroup *cipherBlockFormatFilterGroupReadAdd(IoFilterGroup *filterGroup, const CipherSpec *cipherSpec); // Add decryption to the filter group for a file that may contain a key id. An empty map means the repository is not encrypted and // nothing is added. FN_EXTERN IoFilterGroup *cipherBlockFormatFilterGroupReadAddMap(IoFilterGroup *filterGroup, const CipherSpecMap *cipherSpecMap); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/format/cipherSpecMap.c0000664000175100017510000001652415265101745021650 0ustar00runnerrunner/*********************************************************************************************************************************** Cipher Spec Map ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/format/cipherSpecMap.h" #include "common/log.h" STRING_EXTERN(CIPHER_SPEC_MAP_ID_DEFAULT_STR, CIPHER_SPEC_MAP_ID_DEFAULT); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct CipherSpecMap { CipherSpecMapPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN CipherSpecMap * cipherSpecMapNew(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(CipherSpecMap, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (CipherSpecMap) { .pub = {.list = lstNewP(sizeof(CipherSpecMapItem), .comparator = lstComparatorStr)}, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(CIPHER_SPEC_MAP, this); } /**********************************************************************************************************************************/ FN_EXTERN CipherSpecMap * cipherSpecMapNewPack(PackRead *const packRead) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, packRead); FUNCTION_TEST_END(); ASSERT(packRead != NULL); CipherSpecMap *const this = cipherSpecMapNew(); MEM_CONTEXT_OBJ_BEGIN(this) { const unsigned int total = pckReadU32P(packRead); for (unsigned int keyIdx = 0; keyIdx < total; keyIdx++) { const CipherSpecMapItem item = { .id = pckReadStrP(packRead), .cipherSpec = cipherSpecNewPack(packRead), }; lstAdd(this->pub.list, &item); } lstSort(this->pub.list, sortOrderAsc); this->pub.idCurrent = pckReadStrP(packRead); } MEM_CONTEXT_OBJ_END(); // The key with no id is never current ASSERT(this->pub.idCurrent == NULL || !strEqZ(this->pub.idCurrent, CIPHER_SPEC_MAP_ID_DEFAULT)); FUNCTION_TEST_RETURN(CIPHER_SPEC_MAP, this); } /**********************************************************************************************************************************/ FN_EXTERN const CipherSpec * cipherSpecMapGet(const CipherSpecMap *const this, const String *const id) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CIPHER_SPEC_MAP, this); FUNCTION_TEST_PARAM(STRING, id); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(id != NULL); const CipherSpecMapItem *const item = lstFind(this->pub.list, &id); // Error when request key was not found if (item == NULL) THROW_FMT(CryptoError, "unable to find cipher key '%s'", strZ(id)); FUNCTION_TEST_RETURN_CONST(CIPHER_SPEC, item->cipherSpec); } /**********************************************************************************************************************************/ FN_EXTERN void cipherSpecMapAdd(CipherSpecMap *const this, const String *const id, const CipherSpec *const cipherSpec) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CIPHER_SPEC_MAP, this); FUNCTION_TEST_PARAM(STRING, id); FUNCTION_TEST_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(id != NULL); ASSERT(!strEmpty(id)); ASSERT(cipherSpec != NULL); ASSERT(lstFind(this->pub.list, &id) == NULL); MEM_CONTEXT_OBJ_BEGIN(this) { const CipherSpecMapItem item = {.id = strDup(id), .cipherSpec = cipherSpecDupP(cipherSpec)}; lstAdd(this->pub.list, &item); lstSort(this->pub.list, sortOrderAsc); // The default key id is for files that contain no id so it is never current if (!strEqZ(id, CIPHER_SPEC_MAP_ID_DEFAULT)) this->pub.idCurrent = item.id; } MEM_CONTEXT_OBJ_END(); // The key with no id is never current and every other key is current once added ASSERT(strEqZ(id, CIPHER_SPEC_MAP_ID_DEFAULT) || strEq(cipherSpecMapIdCurrent(this), id)); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cipherSpecMapIdCurrentSet(CipherSpecMap *const this, const String *const id) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CIPHER_SPEC_MAP, this); FUNCTION_TEST_PARAM(STRING, id); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(id != NULL); ASSERT(!strEqZ(id, CIPHER_SPEC_MAP_ID_DEFAULT)); const CipherSpecMapItem *const item = lstFind(this->pub.list, &id); ASSERT(item != NULL); // Point at the id stored with the key rather than duplicating it this->pub.idCurrent = item->id; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN CipherSpecMap * cipherSpecMapDup(const CipherSpecMap *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CIPHER_SPEC_MAP, this); FUNCTION_TEST_END(); ASSERT(this != NULL); CipherSpecMap *const result = cipherSpecMapNew(); for (unsigned int keyIdx = 0; keyIdx < cipherSpecMapSize(this); keyIdx++) { const CipherSpecMapItem *const item = cipherSpecMapGetIdx(this, keyIdx); cipherSpecMapAdd(result, item->id, item->cipherSpec); } if (cipherSpecMapIdCurrent(this) != NULL) cipherSpecMapIdCurrentSet(result, cipherSpecMapIdCurrent(this)); FUNCTION_TEST_RETURN(CIPHER_SPEC_MAP, result); } /**********************************************************************************************************************************/ FN_EXTERN void cipherSpecMapPack(PackWrite *const packWrite, const CipherSpecMap *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, packWrite); FUNCTION_TEST_PARAM(CIPHER_SPEC_MAP, this); FUNCTION_TEST_END(); ASSERT(packWrite != NULL); ASSERT(this != NULL); pckWriteU32P(packWrite, cipherSpecMapSize(this)); for (unsigned int keyIdx = 0; keyIdx < cipherSpecMapSize(this); keyIdx++) { const CipherSpecMapItem *const item = cipherSpecMapGetIdx(this, keyIdx); pckWriteStrP(packWrite, item->id); cipherSpecPack(packWrite, item->cipherSpec); } pckWriteStrP(packWrite, cipherSpecMapIdCurrent(this)); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cipherSpecMapToLog(const CipherSpecMap *const this, StringStatic *const debugLog) { // List only the ids (never the keys themselves) strStcCat(debugLog, "{ids: ["); for (unsigned int keyIdx = 0; keyIdx < cipherSpecMapSize(this); keyIdx++) { if (keyIdx != 0) strStcCat(debugLog, ", "); strStcFmt(debugLog, "%s", strZ(cipherSpecMapGetIdx(this, keyIdx)->id)); } strStcCat(debugLog, "]}"); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/format/cipherSpecMap.h0000664000175100017510000001336115265101745021651 0ustar00runnerrunner/*********************************************************************************************************************************** Cipher Spec Map The keys used to encrypt a repository, stored by id. A file identifies its key by id, either in the header or by where the file is stored. The meaning of an id is set by the caller. archive.info numbers its keys and backup.info uses the backup set label. One key is current and new files are encrypted with it. Adding a key makes it current, except for the key stored under CIPHER_SPEC_MAP_ID_DEFAULT. ***********************************************************************************************************************************/ #ifndef COMMON_FORMAT_CIPHERSPECMAP_H #define COMMON_FORMAT_CIPHERSPECMAP_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct CipherSpecMap CipherSpecMap; #include "common/crypto/spec.h" #include "common/type/list.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Id for the key required by a file that contains no key id. In archive.info this is the key preserved by migration, which is what files written before key ids existed were encrypted with. ***********************************************************************************************************************************/ #define CIPHER_SPEC_MAP_ID_DEFAULT "0" STRING_DECLARE(CIPHER_SPEC_MAP_ID_DEFAULT_STR); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN CipherSpecMap *cipherSpecMapNew(void); // Create from a pack written by cipherSpecMapPack() FN_EXTERN CipherSpecMap *cipherSpecMapNewPack(PackRead *packRead); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct CipherSpecMapItem { const String *id; // Id associated with the key const CipherSpec *cipherSpec; // Key and the digest it derives with } CipherSpecMapItem; typedef struct CipherSpecMapPub { List *list; // Keys sorted by id const String *idCurrent; // Current key id } CipherSpecMapPub; // Total keys FN_INLINE_ALWAYS unsigned int cipherSpecMapSize(const CipherSpecMap *const this) { return lstSize(THIS_PUB(CipherSpecMap)->list); } // Key by index FN_INLINE_ALWAYS const CipherSpecMapItem * cipherSpecMapGetIdx(const CipherSpecMap *const this, const unsigned int idx) { return lstGet(THIS_PUB(CipherSpecMap)->list, idx); } // Key by id. Errors when the id is not found. FN_EXTERN const CipherSpec *cipherSpecMapGet(const CipherSpecMap *this, const String *id); // Current key id, NULL when the map has only the default key with no id FN_INLINE_ALWAYS const String * cipherSpecMapIdCurrent(const CipherSpecMap *const this) { return THIS_PUB(CipherSpecMap)->idCurrent; } // Set the current key id. The id must already be in the map. FN_EXTERN void cipherSpecMapIdCurrentSet(CipherSpecMap *this, const String *id); // Duplicate FN_EXTERN CipherSpecMap *cipherSpecMapDup(const CipherSpecMap *this); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add a key and make it current, except for a key stored under CIPHER_SPEC_MAP_ID_DEFAULT. The id must be unique. FN_EXTERN void cipherSpecMapAdd(CipherSpecMap *this, const String *id, const CipherSpec *cipherSpec); // Write to a pack so it can be passed over a protocol FN_EXTERN void cipherSpecMapPack(PackWrite *packWrite, const CipherSpecMap *this); // Move to a new parent mem context FN_INLINE_ALWAYS CipherSpecMap * cipherSpecMapMove(CipherSpecMap *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void cipherSpecMapFree(CipherSpecMap *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void cipherSpecMapToLog(const CipherSpecMap *this, StringStatic *debugLog); #define FUNCTION_LOG_CIPHER_SPEC_MAP_TYPE \ CipherSpecMap * #define FUNCTION_LOG_CIPHER_SPEC_MAP_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, cipherSpecMapToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/format/format.c0000664000175100017510000000401015265101745020400 0ustar00runnerrunner/*********************************************************************************************************************************** Repository Format ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/format/format.h" #include "version.h" /**********************************************************************************************************************************/ FN_EXTERN void repoFormatValidate(const unsigned int format) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, format); FUNCTION_TEST_END(); // A format newer than this version can read requires an upgrade. Do not suggest a version since this version cannot know which // version added the format. if (format > REPOSITORY_FORMAT_MAX) { THROW_FMT( FormatError, "repository format %u requires a newer version of " PROJECT_NAME "\n" "HINT: " PROJECT_NAME " " PROJECT_VERSION " supports repository format %d to %d.", format, REPOSITORY_FORMAT_MIN, REPOSITORY_FORMAT_MAX); } // A format older than this version can read requires an older version to migrate the repository if (format < REPOSITORY_FORMAT_MIN) { THROW_FMT( FormatError, "repository format %u is no longer supported by " PROJECT_NAME "\n" "HINT: " PROJECT_NAME " " PROJECT_VERSION " supports repository format %d to %d.", format, REPOSITORY_FORMAT_MIN, REPOSITORY_FORMAT_MAX); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN HashType repoFormatDigest(const unsigned int format) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, format); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(STRING_ID, format >= REPOSITORY_FORMAT_6 ? hashTypeSha256 : hashTypeSha1); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/format/format.h0000664000175100017510000000475315265101745020423 0ustar00runnerrunner/*********************************************************************************************************************************** Repository Format Defines the format for info and manifest files as well as on-disk structure. Each info file and manifest stores the format it was written with, so a repository may contain more than one format while older backups and archives expire. A constant is defined for each format so that code which varies by format can be explicit about the format it applies to. REPOSITORY_FORMAT_MIN/MAX are the range that can be read. The allow list for repo-format in build/config.yaml must be kept in sync with MIN/MAX and its default is the format used for new repositories. ***********************************************************************************************************************************/ #ifndef COMMON_FORMAT_FORMAT_H #define COMMON_FORMAT_FORMAT_H #include "common/crypto/common.h" /*********************************************************************************************************************************** Format numbers ***********************************************************************************************************************************/ #define REPOSITORY_FORMAT_5 5 #define REPOSITORY_FORMAT_6 6 #define REPOSITORY_FORMAT_MIN REPOSITORY_FORMAT_5 #define REPOSITORY_FORMAT_MAX REPOSITORY_FORMAT_6 /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Error when a format cannot be read by this version. The format in a cipher header is checked before anything is decrypted, since // decrypting requires knowing what the format expects and this version does not know what a newer format expects. FN_EXTERN void repoFormatValidate(unsigned int format); // Digest a pass stored in a file at this format derives the key with. SHA-1 is what every repository used before format 6 and is // kept for those, so a repository that has not been migrated is read and written exactly as it was. A pass is generated with the // digest of the file it will be stored in, since that is what a reader will derive it with. FN_EXTERN HashType repoFormatDigest(unsigned int format); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/ini.c0000664000175100017510000003106015265101745016404 0ustar00runnerrunner/*********************************************************************************************************************************** Ini Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/ini.h" #include "common/log.h" #include "common/type/json.h" #include "common/type/keyValue.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Ini { KeyValue *store; // Key value store that contains the ini data IoRead *read; // Read object for ini data IniValue value; // Current value unsigned int lineIdx; // Current line used for error reporting bool strict; // Expect all values to be JSON and do not trim }; /**********************************************************************************************************************************/ FN_EXTERN Ini * iniNew(IoRead *const read, const IniNewParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(BOOL, param.strict); FUNCTION_LOG_PARAM(BOOL, param.store); FUNCTION_LOG_END(); OBJ_NEW_BEGIN(Ini, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (Ini) { .read = read, .value = { .section = strNew(), .key = strNew(), .value = strNew(), }, .strict = param.strict, }; ioReadOpen(this->read); // Build KeyValue store if (param.store) { this->store = kvNew(); MEM_CONTEXT_TEMP_RESET_BEGIN() { const IniValue *value = iniValueNext(this); // Add values while not done while (value != NULL) { const Variant *const sectionKey = VARSTR(value->section); KeyValue *sectionKv = varKv(kvGet(this->store, sectionKey)); if (sectionKv == NULL) sectionKv = kvPutKv(this->store, sectionKey); kvAdd(sectionKv, VARSTR(value->key), VARSTR(value->value)); MEM_CONTEXT_TEMP_RESET(1000); value = iniValueNext(this); } } MEM_CONTEXT_TEMP_END(); } } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INI, this); } /**********************************************************************************************************************************/ FN_EXTERN const IniValue * iniValueNext(Ini *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_END(); ASSERT(this != NULL); const IniValue *result = NULL; MEM_CONTEXT_TEMP_RESET_BEGIN() { // Read until a value is found or eof while (result == NULL && !ioReadEof(this->read)) { // Read the line and trim if needed String *const line = ioReadLineParam(this->read, true); if (!this->strict) strTrim(line); const char *const linePtr = strZ(line); this->lineIdx++; // Only interested in lines that are not blank or comments if (strSize(line) > 0 && (this->strict || linePtr[0] != '#')) { // The line is a section. Since the value must be valid JSON this means that the value must never be an array. if (linePtr[0] == '[' && (!this->strict || linePtr[strSize(line) - 1] == ']')) { // Make sure the section ends with ] if (!this->strict && linePtr[strSize(line) - 1] != ']') THROW_FMT(FormatError, "ini section should end with ] at line %u: %s", this->lineIdx, linePtr); // Store the section strCatZN(strTrunc(this->value.section), linePtr + 1, strSize(line) - 2); if (strEmpty(this->value.section)) THROW_FMT(FormatError, "invalid empty section at line %u: %s", this->lineIdx, linePtr); } // Else it is a key/value else { if (strEmpty(this->value.section)) THROW_FMT(FormatError, "key/value found outside of section at line %u: %s", this->lineIdx, linePtr); // Find the = const char *lineEqual = strstr(linePtr, "="); if (lineEqual == NULL) THROW_FMT(FormatError, "missing '=' in key/value at line %u: %s", this->lineIdx, linePtr); // Extract the key/value. For strict this may require some retries if the key includes an = character since this // is also the separator. We know the value must be valid JSON so if it isn't then add the characters up to the // next = to the key and try to parse the value as JSON again. If the value never becomes valid JSON then an // error is thrown. bool retry; do { retry = false; // Get key/value strCatZN(strTrunc(this->value.key), linePtr, (size_t)(lineEqual - linePtr)); strCatZ(strTrunc(this->value.value), lineEqual + 1); // Value is expected to be valid JSON if (this->strict) { TRY_BEGIN() { jsonValidate(this->value.value); } CATCH(JsonFormatError) { // If value is not valid JSON look for another =. If not found then nothing to retry. lineEqual = strstr(lineEqual + 1, "="); if (lineEqual == NULL) { THROW_FMT( FormatError, "invalid JSON value at line %u '%s': %s", this->lineIdx, linePtr, errorMessage()); } // Try again with = in new position retry = true; } TRY_END(); } // Else just trim else { strTrim(this->value.key); strTrim(this->value.value); } } while (retry); // Key may not be zero-length if (strSize(this->value.key) == 0) THROW_FMT(FormatError, "key is zero-length at line %u: %s", this->lineIdx, linePtr); // A value was found so return it result = &this->value; } } MEM_CONTEXT_TEMP_RESET(1000); } // If no more values found then close if (result == NULL) ioReadClose(this->read); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_CONST(INI_VALUE, result); } /**********************************************************************************************************************************/ FN_EXTERN void iniValid(Ini *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_END(); while (iniValueNext(this) != NULL); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Internal function to get an ini value ***********************************************************************************************************************************/ static const Variant * iniGetInternal(const Ini *const this, const String *const section, const String *const key, const bool required) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(BOOL, required); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->store != NULL); ASSERT(section != NULL); ASSERT(key != NULL); const Variant *result = NULL; // Get the section const KeyValue *const sectionKv = varKv(kvGet(this->store, VARSTR(section))); // Section must exist to get the value if (sectionKv != NULL) result = kvGet(sectionKv, VARSTR(key)); // If value is null and required then error if (result == NULL && required) THROW_FMT(FormatError, "section '%s', key '%s' does not exist", strZ(section), strZ(key)); FUNCTION_TEST_RETURN_CONST(VARIANT, result); } /**********************************************************************************************************************************/ FN_EXTERN const String * iniGet(const Ini *const this, const String *const section, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, varStr(iniGetInternal(this, section, key, true))); } /**********************************************************************************************************************************/ FN_EXTERN StringList * iniGetList(const Ini *const this, const String *const section, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); // Get the value const Variant *const result = iniGetInternal(this, section, key, false); FUNCTION_TEST_RETURN(STRING_LIST, result == NULL ? NULL : strLstNewVarLst(varVarLst(result))); } /**********************************************************************************************************************************/ FN_EXTERN bool iniSectionKeyIsList(const Ini *const this, const String *const section, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); // Get the value const Variant *const result = iniGetInternal(this, section, key, true); FUNCTION_TEST_RETURN(BOOL, varType(result) == varTypeVariantList); } /**********************************************************************************************************************************/ FN_EXTERN StringList * iniSectionList(const Ini *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->store != NULL); FUNCTION_TEST_RETURN(STRING_LIST, strLstNewVarLst(kvKeyList(this->store))); } /**********************************************************************************************************************************/ FN_EXTERN StringList * iniSectionKeyList(const Ini *const this, const String *const section) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->store != NULL); ASSERT(section != NULL); StringList *result; MEM_CONTEXT_TEMP_BEGIN() { // Get the section const KeyValue *const sectionKv = varKv(kvGet(this->store, VARSTR(section))); // Return key list if the section exists if (sectionKv != NULL) { result = strLstNewVarLst(kvKeyList(sectionKv)); } // Otherwise return an empty list else result = strLstNew(); strLstMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING_LIST, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/ini.h0000664000175100017510000001146515265101745016420 0ustar00runnerrunner/*********************************************************************************************************************************** Ini Handler ***********************************************************************************************************************************/ #ifndef COMMON_INI_H #define COMMON_INI_H /*********************************************************************************************************************************** Ini object ***********************************************************************************************************************************/ typedef struct Ini Ini; #include "common/io/read.h" #include "common/io/write.h" #include "common/type/object.h" #include "common/type/variant.h" /*********************************************************************************************************************************** Ini section/key/value ***********************************************************************************************************************************/ typedef struct IniValue { String *section; String *key; String *value; } IniValue; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct IniNewParam { VAR_PARAM_HEADER; bool strict; // Expect all values to be JSON and do not trim bool store; // Store in KeyValue so functions can be used to query } IniNewParam; #define iniNewP(read, ...) \ iniNew(read, (IniNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN Ini *iniNew(IoRead *read, IniNewParam param); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Move to a new parent mem context FN_INLINE_ALWAYS Ini * iniMove(Ini *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Check that the ini is valid FN_EXTERN void iniValid(Ini *this); // Get the next value in the ini file. Note that members of IniValue are reused between calls so the members of a previous call may // change after the next call. Any members that need to be preserved should be copied before the next call. FN_EXTERN const IniValue *iniValueNext(Ini *this); /*********************************************************************************************************************************** Getters ***********************************************************************************************************************************/ // Get an ini value -- error if it does not exist FN_EXTERN const String *iniGet(const Ini *this, const String *section, const String *key); // Ini key list FN_EXTERN StringList *iniGetList(const Ini *this, const String *section, const String *key); // The key's value is a list FN_EXTERN bool iniSectionKeyIsList(const Ini *this, const String *section, const String *key); // List of sections FN_EXTERN StringList *iniSectionList(const Ini *const this); // List of keys for a section FN_EXTERN StringList *iniSectionKeyList(const Ini *this, const String *section); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void iniFree(Ini *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_INI_TYPE \ Ini * #define FUNCTION_LOG_INI_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Ini", buffer, bufferSize) #define FUNCTION_LOG_INI_VALUE_TYPE \ IniValue * #define FUNCTION_LOG_INI_VALUE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IniValue", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/0000775000175100017510000000000015265101745016070 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/bufferRead.c0000664000175100017510000001010515265101745020276 0ustar00runnerrunner/*********************************************************************************************************************************** Buffer IO Read ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/read.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoBufferRead { const Buffer *read; // Buffer to read data from size_t readPos; // Current position in the read buffer bool eof; // Has the end of the buffer been reached? } IoBufferRead; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_BUFFER_READ_TYPE \ IoBufferRead * #define FUNCTION_LOG_IO_BUFFER_READ_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoBufferRead", buffer, bufferSize) /*********************************************************************************************************************************** Read data from the buffer ***********************************************************************************************************************************/ static size_t ioBufferRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(IoBufferRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_BUFFER_READ, this); FUNCTION_LOG_PARAM(BUFFER, buffer); (void)block; // Blocking is handled by the caller FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); size_t actualBytes = 0; if (!this->eof) { // Determine how many bytes can actually be read from the buffer (if all bytes will be read then EOF) actualBytes = bufUsed(this->read) - this->readPos; if (actualBytes > bufRemains(buffer)) actualBytes = bufRemains(buffer); else this->eof = true; // Copy bytes to buffer bufCatSub(buffer, this->read, this->readPos, actualBytes); this->readPos += actualBytes; } FUNCTION_LOG_RETURN(SIZE, actualBytes); } /*********************************************************************************************************************************** Have all bytes been read from the buffer? ***********************************************************************************************************************************/ static bool ioBufferReadEof(THIS_VOID) { THIS(IoBufferRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_BUFFER_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(BOOL, this->eof); } /**********************************************************************************************************************************/ FN_EXTERN IoRead * ioBufferReadNew(const Buffer *const buffer) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(buffer != NULL); OBJ_NEW_BEGIN(IoBufferRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoBufferRead) { .read = buffer, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_READ, ioReadNewP(this, .eof = ioBufferReadEof, .read = ioBufferRead)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/bufferRead.h0000664000175100017510000000172415265101745020312 0ustar00runnerrunner/*********************************************************************************************************************************** Buffer IO Read Read from a Buffer object using the IoRead interface. ***********************************************************************************************************************************/ #ifndef COMMON_IO_BUFFERREAD_H #define COMMON_IO_BUFFERREAD_H #include "common/io/read.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoRead *ioBufferReadNew(const Buffer *buffer); // Construct and open buffer read FN_INLINE_ALWAYS IoRead * ioBufferReadNewOpen(const Buffer *const buffer) { IoRead *const result = ioBufferReadNew(buffer); ioReadOpen(result); return result; } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/bufferWrite.c0000664000175100017510000000531315265101745020522 0ustar00runnerrunner/*********************************************************************************************************************************** Buffer IO Write ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/bufferWrite.h" #include "common/io/write.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoBufferWrite { Buffer *write; // Buffer to write into } IoBufferWrite; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_BUFFER_WRITE_TYPE \ IoBufferWrite * #define FUNCTION_LOG_IO_BUFFER_WRITE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoBufferWrite", buffer, bufferSize) /*********************************************************************************************************************************** Write to the buffer ***********************************************************************************************************************************/ static void ioBufferWrite(THIS_VOID, const Buffer *const buffer) { THIS(IoBufferWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_BUFFER_WRITE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); bufCat(this->write, buffer); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN IoWrite * ioBufferWriteNew(Buffer *const buffer) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(buffer != NULL); OBJ_NEW_BEGIN(IoBufferWrite, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoBufferWrite) { .write = buffer, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_WRITE, ioWriteNewP(this, .write = ioBufferWrite)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/bufferWrite.h0000664000175100017510000000172415265101745020531 0ustar00runnerrunner/*********************************************************************************************************************************** Buffer IO Write Write to a Buffer object using the IoWrite interface. ***********************************************************************************************************************************/ #ifndef COMMON_IO_BUFFERWRITE_H #define COMMON_IO_BUFFERWRITE_H #include "common/io/write.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoWrite *ioBufferWriteNew(Buffer *buffer); // Construct and open buffer write FN_INLINE_ALWAYS IoWrite * ioBufferWriteNewOpen(Buffer *const buffer) { IoWrite *const result = ioBufferWriteNew(buffer); ioWriteOpen(result); return result; } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/client.c0000664000175100017510000000433215265101745017514 0ustar00runnerrunner/*********************************************************************************************************************************** Io Client Interface ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/client.h" #include "common/log.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoClient { IoClientPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN IoClient * ioClientNew(void *const driver, const IoClientInterface *const interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(IO_CLIENT_INTERFACE, interface); FUNCTION_LOG_END(); ASSERT(driver != NULL); ASSERT(interface != NULL); ASSERT(interface->type != 0); ASSERT(interface->name != NULL); ASSERT(interface->open != NULL); ASSERT(interface->toLog != NULL); OBJ_NEW_BEGIN(IoClient, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoClient) { .pub = { .driver = objMoveToInterface(driver, this, memContextPrior()), .interface = interface, }, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_CLIENT, this); } /**********************************************************************************************************************************/ FN_EXTERN void ioClientToLog(const IoClient *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "); strStcResultSizeInc(debugLog, strIdToLog(this->pub.interface->type, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", driver: "); this->pub.interface->toLog(this->pub.driver, debugLog); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/client.h0000664000175100017510000000635015265101745017523 0ustar00runnerrunner/*********************************************************************************************************************************** Io Client Interface Create sessions for protocol clients. For example, a TLS client can be created with tlsClientNewP() and then new TLS sessions can be opened with ioClientOpen(). ***********************************************************************************************************************************/ #ifndef COMMON_IO_CLIENT_H #define COMMON_IO_CLIENT_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoClient IoClient; #include "common/io/client.intern.h" #include "common/io/session.h" #include "common/type/object.h" /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoClientPub { void *driver; // Driver object const IoClientInterface *interface; // Driver interface } IoClientPub; // Name that identifies the client FN_INLINE_ALWAYS const String * ioClientName(const IoClient *const this) { return THIS_PUB(IoClient)->interface->name(THIS_PUB(IoClient)->driver); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Move to a new parent mem context FN_INLINE_ALWAYS IoClient * ioClientMove(IoClient *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Open session FN_INLINE_ALWAYS IoSession * ioClientOpen(IoClient *const this) { return THIS_PUB(IoClient)->interface->open(THIS_PUB(IoClient)->driver); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioClientFree(IoClient *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void ioClientToLog(const IoClient *this, StringStatic *debugLog); #define FUNCTION_LOG_IO_CLIENT_TYPE \ IoClient * #define FUNCTION_LOG_IO_CLIENT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioClientToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/client.intern.h0000664000175100017510000000412715265101745021021 0ustar00runnerrunner/*********************************************************************************************************************************** Io Client Interface Internal ***********************************************************************************************************************************/ #ifndef COMMON_IO_CLIENT_INTERN_H #define COMMON_IO_CLIENT_INTERN_H #include "common/io/client.h" #include "common/io/session.h" #include "common/type/string.h" /*********************************************************************************************************************************** Interface ***********************************************************************************************************************************/ typedef struct IoClientInterface { // Type used to identify the client StringId type; // Client name, usually host:port or some other unique identifier const String *(*name)(void *driver); // Open a session IoSession *(*open)(void *driver); // Driver log function void (*toLog)(const void *driver, StringStatic *debugLog); } IoClientInterface; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoClient *ioClientNew(void *driver, const IoClientInterface *interface); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_CLIENT_INTERFACE_TYPE \ IoClientInterface * #define FUNCTION_LOG_IO_CLIENT_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "IoClientInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/fd.c0000664000175100017510000000640715265101745016634 0ustar00runnerrunner/*********************************************************************************************************************************** File Descriptor Functions ***********************************************************************************************************************************/ #include #ifdef __sun__ // Illumos needs sys/siginfo for sigset_t inside poll.h #include #endif #include #include "common/debug.h" #include "common/io/fd.h" #include "common/log.h" /*********************************************************************************************************************************** Use poll() to determine when data is ready to read/write on a socket. Retry after EINTR with whatever time is left on the timer. ***********************************************************************************************************************************/ // Helper to determine when poll() should be retried static bool fdReadyRetry(const int pollResult, const int errNo, const bool first, TimeMSec *const timeout, const TimeMSec timeEnd) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, pollResult); FUNCTION_TEST_PARAM(INT, errNo); FUNCTION_TEST_PARAM(BOOL, first); FUNCTION_TEST_PARAM_P(TIME_MSEC, timeout); FUNCTION_TEST_PARAM(TIME_MSEC, timeEnd); FUNCTION_TEST_END(); ASSERT(timeout != NULL); // No retry by default bool result = false; // Process errors if (pollResult == -1) { // Don't error on an interrupt. If the interrupt lasts long enough there may be a timeout, though. if (errNo != EINTR) THROW_SYS_ERROR_CODE(errNo, KernelError, "unable to poll socket"); // Always retry on the first iteration if (first) { result = true; } // Else retry if there is time left else { TimeMSec timeCurrent = timeMSec(); if (timeEnd > timeCurrent) { *timeout = timeEnd - timeCurrent; result = true; } } } FUNCTION_TEST_RETURN(BOOL, result); } FN_EXTERN bool fdReady(const int fd, const bool read, const bool write, TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INT, fd); FUNCTION_LOG_PARAM(BOOL, read); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(fd >= 0); ASSERT(read || write); ASSERT(timeout < INT_MAX); // Poll settings struct pollfd inputFd = {.fd = fd}; if (read) inputFd.events |= POLLIN; if (write) inputFd.events |= POLLOUT; // Wait for ready or timeout TimeMSec timeEnd = timeMSec() + timeout; bool first = true; // Initialize result and errno to look like a retryable error. We have no good way to test this function with interrupts so this // at least ensures that the condition is retried. int result = -1; int errNo = EINTR; while (fdReadyRetry(result, errNo, first, &timeout, timeEnd)) { result = poll(&inputFd, 1, (int)timeout); errNo = errno; first = false; } FUNCTION_LOG_RETURN(BOOL, result > 0); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/fd.h0000664000175100017510000000221215265101745016627 0ustar00runnerrunner/*********************************************************************************************************************************** File Descriptor Functions ***********************************************************************************************************************************/ #ifndef COMMON_IO_FD_H #define COMMON_IO_FD_H #include "common/time.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Wait until the file descriptor is ready to read/write or timeout FN_EXTERN bool fdReady(int fd, bool read, bool write, TimeMSec timeout); // Wait until the file descriptor is ready to read or timeout FN_INLINE_ALWAYS bool fdReadyRead(const int fd, const TimeMSec timeout) { return fdReady(fd, true, false, timeout); } // Wait until the file descriptor is ready to write or timeout FN_INLINE_ALWAYS bool fdReadyWrite(const int fd, const TimeMSec timeout) { return fdReady(fd, false, true, timeout); } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/fdRead.c0000664000175100017510000001422615265101745017426 0ustar00runnerrunner/*********************************************************************************************************************************** File Descriptor Io Read ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/fd.h" #include "common/io/fdRead.h" #include "common/io/read.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoFdRead { const String *name; // File descriptor name for error messages int fd; // File descriptor to read data from TimeMSec timeout; // Timeout for read operation bool eof; // Has the end of the stream been reached? } IoFdRead; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_FD_READ_TYPE \ IoFdRead * #define FUNCTION_LOG_IO_FD_READ_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoFdRead", buffer, bufferSize) /*********************************************************************************************************************************** Are there bytes ready to read immediately? ***********************************************************************************************************************************/ static bool ioFdReadReady(THIS_VOID, const bool error) { THIS(IoFdRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FD_READ, this); FUNCTION_LOG_PARAM(BOOL, error); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = true; // Check if the file descriptor is ready to read if (!fdReadyRead(this->fd, this->timeout)) { // Error if requested if (error) THROW_FMT(FileReadError, "timeout after %" PRIu64 "ms waiting for read from '%s'", this->timeout, strZ(this->name)); // File descriptor is not ready to read result = false; } FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Read data from the file descriptor ***********************************************************************************************************************************/ static size_t ioFdRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(IoFdRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FD_READ, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(!bufFull(buffer)); ssize_t actualBytes = 0; // If blocking keep reading until buffer is full or eof if (!this->eof) { do { // Check if there is data to be read and error if not ioFdReadReady(this, true); // Read and handle errors THROW_ON_SYS_ERROR_FMT( (actualBytes = read(this->fd, bufRemainsPtr(buffer), bufRemains(buffer))) == -1, FileReadError, "unable to read from %s", strZ(this->name)); // Update amount of buffer used bufUsedInc(buffer, (size_t)actualBytes); // If zero bytes were returned then eof if (actualBytes == 0) this->eof = true; } while (bufRemains(buffer) > 0 && !this->eof && block); } FUNCTION_LOG_RETURN(SIZE, (size_t)actualBytes); } /*********************************************************************************************************************************** Have all bytes been read from the buffer? ***********************************************************************************************************************************/ static bool ioFdReadEof(THIS_VOID) { THIS(IoFdRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FD_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(BOOL, this->eof); } /*********************************************************************************************************************************** Get file descriptor ***********************************************************************************************************************************/ static int ioFdReadFd(const THIS_VOID) { THIS(const IoFdRead); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FD_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->fd); } /**********************************************************************************************************************************/ FN_EXTERN IoRead * ioFdReadNew(const String *const name, const int fd, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(INT, fd); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(fd != -1); OBJ_NEW_BEGIN(IoFdRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoFdRead) { .name = strDup(name), .fd = fd, .timeout = timeout, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_READ, ioReadNewP(this, .block = true, .eof = ioFdReadEof, .fd = ioFdReadFd, .read = ioFdRead, .ready = ioFdReadReady)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/fdRead.h0000664000175100017510000000204415265101745017426 0ustar00runnerrunner/*********************************************************************************************************************************** File Descriptor Read Read from a file descriptor using the IoRead interface. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FDREAD_H #define COMMON_IO_FDREAD_H #include "common/io/read.h" #include "common/time.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoRead *ioFdReadNew(const String *name, int fd, TimeMSec timeout); // Construct and open read fd FN_INLINE_ALWAYS IoRead * ioFdReadNewOpen(const String *const name, const int fd, const TimeMSec timeout) { IoRead *const result = ioFdReadNew(name, fd, timeout); ioReadOpen(result); return result; } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/fdWrite.c0000664000175100017510000001475215265101745017651 0ustar00runnerrunner/*********************************************************************************************************************************** File Descriptor Io Write ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/fd.h" #include "common/io/fdWrite.h" #include "common/io/write.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoFdWrite { const String *name; // File descriptor name for error messages int fd; // File descriptor to write to TimeMSec timeout; // Timeout for write operation } IoFdWrite; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_FD_WRITE_TYPE \ IoFdWrite * #define FUNCTION_LOG_IO_FD_WRITE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoFdWrite", buffer, bufferSize) /*********************************************************************************************************************************** // Can bytes be written immediately? ***********************************************************************************************************************************/ static bool ioFdWriteReady(THIS_VOID, const bool error) { THIS(IoFdWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FD_WRITE, this); FUNCTION_LOG_PARAM(BOOL, error); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = true; // Check if the file descriptor is ready to write if (!fdReadyWrite(this->fd, this->timeout)) { // Error if requested if (error) THROW_FMT(FileWriteError, "timeout after %" PRIu64 "ms waiting for write to '%s'", this->timeout, strZ(this->name)); // File descriptor is not ready to write result = false; } FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Write to the file descriptor ***********************************************************************************************************************************/ // Helper wrapper around write() system call for unit testing static ssize_t ioFdWriteInternal(const int fd, const void *const buffer, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, fd); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(fd >= 0); ASSERT(buffer != NULL); FUNCTION_TEST_RETURN(SSIZE, write(fd, buffer, size)); } static void ioFdWrite(THIS_VOID, const Buffer *const buffer) { THIS(IoFdWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FD_WRITE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); // Handle partial writes and non-blocking socket operations size_t totalWritten = 0; size_t bufferRemaining = bufUsed(buffer); while (bufferRemaining > 0) { const ssize_t result = ioFdWriteInternal(this->fd, bufPtrConst(buffer) + totalWritten, bufferRemaining); if (result == -1) { // Handle non-blocking socket case where write buffer is full if (errno == EAGAIN) { // Wait for socket to become writable (will throw on timeout) ioFdWriteReady(this, true); continue; } // Treat all other errors as fatal THROW_SYS_ERROR_FMT( FileWriteError, "unable to finish write to %s (wrote %zu/%zu bytes)", strZ(this->name), totalWritten, bufUsed(buffer)); } totalWritten += (size_t)result; bufferRemaining -= (size_t)result; } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get file descriptor ***********************************************************************************************************************************/ static int ioFdWriteFd(const THIS_VOID) { THIS(const IoFdWrite); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FD_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->fd); } /**********************************************************************************************************************************/ FN_EXTERN IoWrite * ioFdWriteNew(const String *const name, const int fd, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(INT, fd); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); OBJ_NEW_BEGIN(IoFdWrite, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoFdWrite) { .name = strDup(name), .fd = fd, .timeout = timeout, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_WRITE, ioWriteNewP(this, .fd = ioFdWriteFd, .ready = ioFdWriteReady, .write = ioFdWrite)); } /**********************************************************************************************************************************/ FN_EXTERN void ioFdWriteOneStr(const int fd, const String *const string) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INT, fd); FUNCTION_LOG_PARAM(STRING, string); FUNCTION_LOG_END(); ASSERT(string != NULL); if (write(fd, strZ(string), strSize(string)) != (int)strSize(string)) THROW_SYS_ERROR(FileWriteError, "unable to write to fd"); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/fdWrite.h0000664000175100017510000000270015265101745017644 0ustar00runnerrunner/*********************************************************************************************************************************** File Descriptor Io Write Write to a file descriptor using the IoWrite interface. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FDWRITE_H #define COMMON_IO_FDWRITE_H #include "common/io/write.h" #include "common/time.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoWrite *ioFdWriteNew(const String *name, int fd, TimeMSec timeout); // Construct and open write fd FN_INLINE_ALWAYS IoWrite * ioFdWriteNewOpen(const String *const name, const int fd, const TimeMSec timeout) { IoWrite *const result = ioFdWriteNew(name, fd, timeout); ioWriteOpen(result); return result; } /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Write a string to the specified file descriptor FN_EXTERN void ioFdWriteOneStr(int fd, const String *string); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/filter/0000775000175100017510000000000015265101745017355 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/filter/buffer.c0000664000175100017510000001133315265101745020773 0ustar00runnerrunner/*********************************************************************************************************************************** IO Buffer Filter ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/filter/buffer.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define BUFFER_FILTER_TYPE STRID5("buffer", 0x24531aa20) /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoBuffer { size_t inputPos; // Position in input buffer bool inputSame; // Is the same input required again? } IoBuffer; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void ioBufferToLog(const IoBuffer *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{inputSame: %s, inputPos: %zu}", cvtBoolToConstZ(this->inputSame), this->inputPos); } #define FUNCTION_LOG_IO_BUFFER_TYPE \ IoBuffer * #define FUNCTION_LOG_IO_BUFFER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioBufferToLog, buffer, bufferSize) /*********************************************************************************************************************************** Move data from the input buffer to the output buffer ***********************************************************************************************************************************/ static void ioBufferProcess(THIS_VOID, const Buffer *const input, Buffer *const output) { THIS(IoBuffer); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_BUFFER, this); FUNCTION_LOG_PARAM(BUFFER, input); FUNCTION_LOG_PARAM(BUFFER, output); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(input != NULL); ASSERT(output != NULL); // Determine how much data needs to be copied and reduce if there is not enough space in the output size_t copySize = bufUsed(input) - this->inputPos; if (copySize > bufRemains(output)) copySize = bufRemains(output); // Copy data to the output buffer bufCatSub(output, input, this->inputPos, copySize); // If all data was copied then reset inputPos and allow new input if (this->inputPos + copySize == bufUsed(input)) { this->inputSame = false; this->inputPos = 0; } // Else update inputPos and indicate that the same input should be passed again else { this->inputSame = true; this->inputPos += copySize; } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is the same input required again? If the remaining space in the output buffer is smaller than the used space in the input buffer then the same input must be provided again. ***********************************************************************************************************************************/ static bool ioBufferInputSame(const THIS_VOID) { THIS(const IoBuffer); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_BUFFER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * ioBufferNew(void) { FUNCTION_LOG_VOID(logLevelTrace); OBJ_NEW_BEGIN(IoBuffer) { *this = (IoBuffer){0}; } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP(BUFFER_FILTER_TYPE, this, NULL, .inOut = ioBufferProcess, .inputSame = ioBufferInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/filter/buffer.h0000664000175100017510000000175615265101745021010 0ustar00runnerrunner/*********************************************************************************************************************************** IO Buffer Filter Move data from the input buffer to the output buffer without overflowing the output buffer. Automatically used as the last filter in a FilterGroup if the last filter is not already an InOut filter, so there is no reason to add it manually to a FilterGroup. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FILTER_BUFFER_H #define COMMON_IO_FILTER_BUFFER_H #include "common/io/filter/filter.h" #include "common/type/buffer.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *ioBufferNew(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/filter/filter.c0000664000175100017510000001474615265101745021022 0ustar00runnerrunner/*********************************************************************************************************************************** IO Filter Interface ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/memContext.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoFilter { IoFilterPub pub; // Publicly accessible variables bool flushing; // Has the filter started flushing? }; /*********************************************************************************************************************************** Allocations will be in the memory context of the caller. ***********************************************************************************************************************************/ FN_EXTERN IoFilter * ioFilterNew(const StringId type, void *const driver, Pack *const paramList, const IoFilterInterface interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(PACK, paramList); FUNCTION_LOG_PARAM(IO_FILTER_INTERFACE, interface); FUNCTION_LOG_END(); ASSERT(type != 0); ASSERT(driver != NULL); // One of processIn or processInOut must be set ASSERT(interface.in != NULL || interface.inOut != NULL); // But not both of them ASSERT(!(interface.in != NULL && interface.inOut != NULL)); // If the filter does not produce output then it should produce a result ASSERT(interface.in == NULL || (interface.result != NULL && interface.done == NULL && interface.inputSame == NULL)); OBJ_NEW_BEGIN(IoFilter, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoFilter) { .pub = { .type = type, .driver = objMoveToInterface(driver, this, memContextPrior()), .paramList = pckMove(paramList, objMemContext(this)), .interface = interface, }, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_FILTER, this); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterProcessIn(IoFilter *const this, const Buffer *const input) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER, this); FUNCTION_TEST_PARAM(BUFFER, input); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->pub.interface.in != NULL); CHECK(AssertError, input == NULL || !bufEmpty(input), "expected data or flush"); CHECK(AssertError, !this->flushing || input == NULL, "no data allowed after flush"); if (input == NULL) this->flushing = true; else this->pub.interface.in(this->pub.driver, input); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterProcessInOut(IoFilter *const this, const Buffer *const input, Buffer *const output) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER, this); FUNCTION_TEST_PARAM(BUFFER, input); FUNCTION_TEST_PARAM(BUFFER, output); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(output != NULL); ASSERT(this->pub.interface.inOut != NULL); CHECK(AssertError, input == NULL || !bufEmpty(input), "expected data or flush"); CHECK(AssertError, !this->flushing || input == NULL, "no data allowed after flush"); if (input == NULL && !this->flushing) this->flushing = true; if (!ioFilterDone(this)) this->pub.interface.inOut(this->pub.driver, input, output); // If input same is requested then there must be some output otherwise there is no point in requesting the same input CHECK(AssertError, !ioFilterInputSame(this) || !bufEmpty(output), "expected input to be consumed or some output"); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** If done is not defined by the filter then check inputSame. If inputSame is true then the filter is not done. Even if the filter is done the interface will not report done until the interface is flushing. ***********************************************************************************************************************************/ FN_EXTERN bool ioFilterDone(const IoFilter *this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); bool result = false; if (this->flushing) result = this->pub.interface.done != NULL ? this->pub.interface.done(this->pub.driver) : !ioFilterInputSame(this); FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN bool ioFilterInputSame(const IoFilter *this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->pub.interface.inputSame != NULL ? this->pub.interface.inputSame(this->pub.driver) : false); } /**********************************************************************************************************************************/ FN_EXTERN Pack * ioFilterResult(const IoFilter *this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(PACK, this->pub.interface.result ? this->pub.interface.result(this->pub.driver) : NULL); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterToLog(const IoFilter *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "); strStcResultSizeInc(debugLog, strIdToLog(ioFilterType(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/filter/filter.h0000664000175100017510000000537615265101745021026 0ustar00runnerrunner/*********************************************************************************************************************************** IO Filter Interface Filters can modify an i/o stream (e.g. GzCompress, GzDecompress), generate a result (e.g. IoSize, CryptoHash), or even do both. A filter is created using a constructor implemented by each filter (e.g. ioBufferNew). Filter processing is managed by IoFilterGroup so the only user facing functions are ioFilterResult() and ioFilterType(). Information on implementing a filter is in filter.internal.h. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FILTER_FILTER_H #define COMMON_IO_FILTER_FILTER_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoFilter IoFilter; #include "common/io/filter/filter.intern.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Get filter result FN_EXTERN Pack *ioFilterResult(const IoFilter *this); // Identifies the filter and is used when pulling results from the filter group FN_INLINE_ALWAYS StringId ioFilterType(const IoFilter *const this) { return THIS_PUB(IoFilter)->type; } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioFilterFree(IoFilter *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void ioFilterToLog(const IoFilter *this, StringStatic *debugLog); #define FUNCTION_LOG_IO_FILTER_TYPE \ IoFilter * #define FUNCTION_LOG_IO_FILTER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioFilterToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/filter/filter.intern.h0000664000175100017510000001503215265101745022312 0ustar00runnerrunner/*********************************************************************************************************************************** IO Filter Interface Internal Two types of filters are implemented using this interface: In and InOut. In filters accept input and produce a result, but do not modify the input. An example is the IoSize filter which counts all bytes that pass through it. InOut filters accept input and produce output (and perhaps a result). Because the input/output buffers may not be the same size the filter must be prepared to accept the same input again (by implementing IoFilterInputSame) if the output buffer is too small to accept all processed data. If the filter holds state even when inputSame is false then it may also implement IoFilterDone to indicate that the filter should be flushed (by passing NULL inputs) after all input has been processed. InOut filters should strive to fill the output buffer as much as possible, i.e., if the output buffer is not full after processing then inputSame should be false. An example is the IoBuffer filter which buffers data between unequally sized input/output buffers. Each filter has a type that allows it to be identified in the filter list. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FILTER_FILTER_INTERN_H #define COMMON_IO_FILTER_FILTER_INTERN_H #include "common/type/buffer.h" #include "common/type/pack.h" #include "common/type/stringId.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct IoFilterInterface { // Indicates that filter processing is done. This is used for filters that have additional data to be flushed even after all // input has been processed. Compression and encryption filters will usually need to implement done. If done is not implemented // then it will always return true if all input has been consumed, i.e. if inputSame returns false. bool (*done)(const void *driver); // Processing function for filters that do not produce output. Note that result must be implemented in this case (or else what // would be the point. void (*in)(void *driver, const Buffer *); // Processing function for filters that produce output. InOut filters will typically implement inputSame and may also implement // done. void (*inOut)(void *driver, const Buffer *, Buffer *); // InOut filters must be prepared for an output buffer that is too small to accept all the processed output. In this case the // filter must implement inputSame and set it to true when there is more output to be produced for a given input. On the next // call to inOut the same input will be passed along with a fresh output buffer with space for more processed output. bool (*inputSame)(const void *driver); // If the filter produces a result then this function must be implemented to return the result. A result can be anything that // is not processed output, e.g. a count of total bytes or a cryptographic hash. The returned buffer must be a pack containing // the result. Pack *(*result)(void *driver); } IoFilterInterface; #define ioFilterNewP(type, driver, paramList, ...) \ ioFilterNew(type, driver, paramList, (IoFilterInterface){__VA_ARGS__}) FN_EXTERN IoFilter *ioFilterNew(StringId type, void *driver, Pack *paramList, IoFilterInterface); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoFilterPub { StringId type; // Filter type IoFilterInterface interface; // Filter interface void *driver; // Filter driver const Pack *paramList; // Filter parameters } IoFilterPub; // Is the filter done? FN_EXTERN bool ioFilterDone(const IoFilter *this); // Driver for the filter FN_INLINE_ALWAYS void * ioFilterDriver(IoFilter *const this) { return THIS_PUB(IoFilter)->driver; } // Does the filter need the same input again? If the filter cannot get all its output into the output buffer then it may need access // to the same input again. FN_EXTERN bool ioFilterInputSame(const IoFilter *this); // Interface for the filter FN_INLINE_ALWAYS const IoFilterInterface * ioFilterInterface(const IoFilter *const this) { return &THIS_PUB(IoFilter)->interface; } // Does filter produce output? All In filters produce output. FN_INLINE_ALWAYS bool ioFilterOutput(const IoFilter *const this) { return THIS_PUB(IoFilter)->interface.inOut != NULL; } // List of filter parameters FN_INLINE_ALWAYS const Pack * ioFilterParamList(const IoFilter *const this) { return THIS_PUB(IoFilter)->paramList; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Filter input only (a result is expected) FN_EXTERN void ioFilterProcessIn(IoFilter *this, const Buffer *input); // Filter input and produce output FN_EXTERN void ioFilterProcessInOut(IoFilter *this, const Buffer *input, Buffer *output); // Move filter to a new parent mem context FN_INLINE_ALWAYS IoFilter * ioFilterMove(IoFilter *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_FILTER_INTERFACE_TYPE \ IoFilterInterface * #define FUNCTION_LOG_IO_FILTER_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "IoFilterInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/filter/group.c0000664000175100017510000005112215265101745020656 0ustar00runnerrunner/*********************************************************************************************************************************** IO Filter Group ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/filter/buffer.h" #include "common/io/filter/filter.h" #include "common/io/filter/group.h" #include "common/io/io.h" #include "common/log.h" #include "common/type/list.h" /*********************************************************************************************************************************** Filter and buffer structure Contains the filter object and inout/output buffers. ***********************************************************************************************************************************/ typedef struct IoFilterData { const Buffer *const *input; // Input buffer for filter Buffer *inputLocal; // Non-null if a locally created buffer that can be cleared IoFilter *filter; // Filter to apply Buffer *output; // Output buffer for filter } IoFilterData; // Macros for logging #define FUNCTION_LOG_IO_FILTER_DATA_TYPE \ IoFilterData * #define FUNCTION_LOG_IO_FILTER_DATA_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoFilterData", buffer, bufferSize) /*********************************************************************************************************************************** Filter results ***********************************************************************************************************************************/ typedef struct IoFilterResult { StringId type; // Filter type Pack *result; // Filter result } IoFilterResult; /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoFilterGroup { IoFilterGroupPub pub; // Publicly accessible variables const Buffer *input; // Input buffer passed in for processing List *filterResult; // Filter results (if any) #ifdef DEBUG bool flushing; // Is output being flushed? #endif }; /**********************************************************************************************************************************/ FN_EXTERN IoFilterGroup * ioFilterGroupNew(void) { FUNCTION_LOG_VOID(logLevelTrace); OBJ_NEW_BEGIN(IoFilterGroup, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoFilterGroup) { .pub = { .done = false, .filterList = lstNewP(sizeof(IoFilterData)), }, .filterResult = lstNewP(sizeof(IoFilterResult)), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_FILTER_GROUP, this); } /**********************************************************************************************************************************/ FN_EXTERN IoFilterGroup * ioFilterGroupAdd(IoFilterGroup *const this, IoFilter *const filter) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_PARAM(IO_FILTER, filter); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->pub.opened && !this->pub.closed); ASSERT(filter != NULL); // Move the filter to this object's mem context ioFilterMove(filter, objMemContext(this)); // Add the filter IoFilterData filterData = {.filter = filter}; lstAdd(this->pub.filterList, &filterData); FUNCTION_LOG_RETURN(IO_FILTER_GROUP, this); } /**********************************************************************************************************************************/ FN_EXTERN IoFilterGroup * ioFilterGroupInsert(IoFilterGroup *const this, const unsigned int listIdx, IoFilter *const filter) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_PARAM(IO_FILTER, filter); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->pub.opened && !this->pub.closed); ASSERT(filter != NULL); // Move the filter to this object's mem context ioFilterMove(filter, objMemContext(this)); // Add the filter IoFilterData filterData = {.filter = filter}; lstInsert(this->pub.filterList, listIdx, &filterData); FUNCTION_LOG_RETURN(IO_FILTER_GROUP, this); } /*********************************************************************************************************************************** Get a filter ***********************************************************************************************************************************/ static IoFilterData * ioFilterGroupGet(const IoFilterGroup *const this, const unsigned int filterIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER_GROUP, this); FUNCTION_TEST_PARAM(UINT, filterIdx); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(IO_FILTER_DATA, (IoFilterData *)lstGet(this->pub.filterList, filterIdx)); } /**********************************************************************************************************************************/ FN_EXTERN IoFilterGroup * ioFilterGroupClear(IoFilterGroup *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->pub.opened); for (unsigned int filterIdx = 0; filterIdx < ioFilterGroupSize(this); filterIdx++) ioFilterFree(ioFilterGroupGet(this, filterIdx)->filter); lstClear(this->pub.filterList); FUNCTION_LOG_RETURN(IO_FILTER_GROUP, this); } /*********************************************************************************************************************************** Setup the filter group and allocate any required buffers ***********************************************************************************************************************************/ FN_EXTERN void ioFilterGroupOpen(IoFilterGroup *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_OBJ_BEGIN(this) { // If the last filter is not an output filter then add a filter to buffer/copy data. Input filters won't copy to an output // buffer so we need some way to get the data to the output buffer. if (ioFilterGroupSize(this) == 0 || !ioFilterOutput((ioFilterGroupGet(this, ioFilterGroupSize(this) - 1))->filter)) { ioFilterGroupAdd(this, ioBufferNew()); } // Create filter input/output buffers. Input filters do not get an output buffer since they don't produce output. Buffer **lastOutputBuffer = NULL; for (unsigned int filterIdx = 0; filterIdx < ioFilterGroupSize(this); filterIdx++) { IoFilterData *const filterData = ioFilterGroupGet(this, filterIdx); // If there is no last output buffer yet, then use the input buffer that will be provided by the caller if (lastOutputBuffer == NULL) { filterData->input = &this->input; } // Else assign the last output buffer to the input else { // This cast is required because the compiler can't guarantee the const-ness of this object, i.e. it could be // modified in other parts of the code. This is actually expected and the only reason we need this const is to match // the const-ness of the input buffer provided by the caller. filterData->input = (const Buffer *const *)lastOutputBuffer; filterData->inputLocal = *lastOutputBuffer; } // If this is not the last output filter then create a new output buffer for it. The output buffer for the last filter // will be provided to the process function. ASSERT(ioFilterGroupSize(this) != 0); if (ioFilterOutput(filterData->filter) && filterIdx < ioFilterGroupSize(this) - 1) { filterData->output = bufNew(ioBufferSize()); lastOutputBuffer = &filterData->output; } } } MEM_CONTEXT_OBJ_END(); // Filter group is open #ifdef DEBUG this->pub.opened = true; #endif FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterGroupProcess(IoFilterGroup *const this, const Buffer *const input, Buffer *const output) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_PARAM(BUFFER, input); FUNCTION_LOG_PARAM(BUFFER, output); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.opened && !this->pub.closed); ASSERT(input == NULL || !bufEmpty(input)); ASSERT(!this->flushing || input == NULL); ASSERT(output != NULL); ASSERT(bufRemains(output) > 0); // Once input goes to NULL then flushing has started #ifdef DEBUG if (input == NULL) this->flushing = true; #endif // Assign input and output buffers this->input = input; (ioFilterGroupGet(this, ioFilterGroupSize(this) - 1))->output = output; // do { // Start from the first filter by default unsigned int filterIdx = 0; // Search from the end of the list for a filter that needs the same input. This indicates that the filter was not able to // empty the input buffer on the last call. Maybe it won't this time either -- we can but try. if (ioFilterGroupInputSame(this)) { this->pub.inputSame = false; filterIdx = ioFilterGroupSize(this); do { filterIdx--; if (ioFilterInputSame((ioFilterGroupGet(this, filterIdx))->filter)) { this->pub.inputSame = true; break; } } while (filterIdx != 0); // If no filter is found that needs the same input that means we are done with the current input. So end the loop and // get some more input. if (!ioFilterGroupInputSame(this)) break; } // Process forward from the filter that has input to process. This may be a filter that needs the same input or it may be // new input for the first filter. for (; filterIdx < ioFilterGroupSize(this); filterIdx++) { IoFilterData *const filterData = ioFilterGroupGet(this, filterIdx); // Process the filter if it is not done if (!ioFilterDone(filterData->filter)) { // If the filter produces output if (ioFilterOutput(filterData->filter)) { ioFilterProcessInOut(filterData->filter, *filterData->input, filterData->output); // If inputSame is set then the output buffer for this filter is full and it will need to be re-processed with // the same input once the output buffer is cleared if (ioFilterInputSame(filterData->filter)) { this->pub.inputSame = true; } // Else clear the buffer if it was locally allocated. If the input buffer was passed in then the caller is // responsible for clearing it. else if (filterData->inputLocal != NULL) bufUsedZero(filterData->inputLocal); // If the output buffer is not full and the filter is not done then more data is required if (!bufFull(filterData->output) && !ioFilterDone(filterData->filter)) break; } // Else the filter does not produce output else ioFilterProcessIn(filterData->filter, *filterData->input); } // If the filter is done and has no more output then null the output buffer. Downstream filters have a pointer to this // buffer so their inputs will also change to null and they'll flush. if (filterData->output != NULL && ioFilterDone(filterData->filter) && bufUsed(filterData->output) == 0) filterData->output = NULL; } } while (!bufFull(output) && ioFilterGroupInputSame(this)); // Scan the filter list to determine if inputSame is set or done is not set for any filter. We can't trust // ioFilterGroupInputSame() when it is true without going through the loop above again. We need to scan to set this->pub.done // anyway so set this->pub.inputSame in the same loop. this->pub.done = true; this->pub.inputSame = false; for (unsigned int filterIdx = 0; filterIdx < ioFilterGroupSize(this); filterIdx++) { const IoFilterData *const filterData = ioFilterGroupGet(this, filterIdx); // When inputSame then this->pub.done = false and we can exit the loop immediately if (ioFilterInputSame(filterData->filter)) { this->pub.done = false; this->pub.inputSame = true; break; } // Set this->pub.done = false if any filter is not done if (!ioFilterDone(filterData->filter)) this->pub.done = false; } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterGroupClose(IoFilterGroup *this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Gather results from the filters for (unsigned int filterIdx = 0; filterIdx < ioFilterGroupSize(this); filterIdx++) { const IoFilter *const filter = ioFilterGroupGet(this, filterIdx)->filter; MEM_CONTEXT_BEGIN(lstMemContext(this->filterResult)) { lstAdd(this->filterResult, &(IoFilterResult){.type = ioFilterType(filter), .result = ioFilterResult(filter)}); } MEM_CONTEXT_END(); } // Filter group is open #ifdef DEBUG this->pub.closed = true; #endif FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Pack * ioFilterGroupParamAll(const IoFilterGroup *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->pub.opened); ASSERT(this->pub.filterList != NULL); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); for (unsigned int filterIdx = 0; filterIdx < ioFilterGroupSize(this); filterIdx++) { IoFilter *filter = ioFilterGroupGet(this, filterIdx)->filter; pckWriteStrIdP(packWrite, ioFilterType(filter)); pckWritePackP(packWrite, ioFilterParamList(filter)); } pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN const Pack * ioFilterGroupResultPack(const IoFilterGroup *const this, const StringId filterType, const IoFilterGroupResultParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_PARAM(STRING_ID, filterType); FUNCTION_LOG_PARAM(UINT, param.idx); FUNCTION_LOG_END(); ASSERT(this->pub.opened); ASSERT(filterType != 0); const Pack *result = NULL; // Search for the result unsigned int foundIdx = 0; for (unsigned int filterResultIdx = 0; filterResultIdx < lstSize(this->filterResult); filterResultIdx++) { const IoFilterResult *const filterResult = lstGet(this->filterResult, filterResultIdx); // If the filter matches check the index if (filterResult->type == filterType) { // If the index matches return the result if (foundIdx == param.idx) { result = filterResult->result; break; } // Increment the index and keep searching foundIdx++; } } FUNCTION_LOG_RETURN_CONST(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * ioFilterGroupResult(const IoFilterGroup *const this, const StringId filterType, const IoFilterGroupResultParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_PARAM(STRING_ID, filterType); FUNCTION_LOG_PARAM(UINT, param.idx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN(PACK_READ, pckReadNew(ioFilterGroupResultPack(this, filterType, param))); } /**********************************************************************************************************************************/ FN_EXTERN Pack * ioFilterGroupResultAll(const IoFilterGroup *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.closed); Pack *result = NULL; // Pack the result list MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); for (unsigned int filterResultIdx = 0; filterResultIdx < lstSize(this->filterResult); filterResultIdx++) { const IoFilterResult *const filterResult = lstGet(this->filterResult, filterResultIdx); pckWriteStrIdP(packWrite, filterResult->type); pckWritePackP(packWrite, filterResult->result); } pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterGroupResultAllSet(IoFilterGroup *const this, const Pack *const filterResultPack) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_PARAM(PACK, filterResultPack); FUNCTION_LOG_END(); ASSERT(this != NULL); if (filterResultPack != NULL) { // Unpack the results into a list MEM_CONTEXT_TEMP_BEGIN() { PackRead *const packRead = pckReadNew(filterResultPack); MEM_CONTEXT_BEGIN(lstMemContext(this->filterResult)) { while (!pckReadNullP(packRead)) { const StringId type = pckReadStrIdP(packRead); Pack *const result = pckReadPackP(packRead); lstAdd(this->filterResult, &(IoFilterResult){.type = type, .result = result}); } } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterGroupToLog(const IoFilterGroup *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, done: %s" #ifdef DEBUG ", opened %s, flushing %s, closed %s" #endif "}", cvtBoolToConstZ(this->pub.inputSame), cvtBoolToConstZ(this->pub.done) #ifdef DEBUG , cvtBoolToConstZ(this->pub.opened), cvtBoolToConstZ(this->flushing), cvtBoolToConstZ(this->pub.closed) #endif ); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/filter/group.h0000664000175100017510000001460515265101745020670 0ustar00runnerrunner/*********************************************************************************************************************************** IO Filter Group Process data through an arbitrary group of filters in the order added by the user using ioFilterGroupAdd(). After processing results can be gathered using ioFilterGroupResult() for any filters that produce results. Processing is complex and asymmetric for read/write so should be done via the IoRead and IoWrite objects. General users need only call ioFilterGroupNew(), ioFilterGroupAdd(), and ioFilterGroupResult(). ***********************************************************************************************************************************/ #ifndef COMMON_IO_FILTER_GROUP_H #define COMMON_IO_FILTER_GROUP_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoFilterGroup IoFilterGroup; #include "common/io/filter/filter.h" #include "common/type/list.h" #include "common/type/object.h" #include "common/type/pack.h" #include "common/type/stringId.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilterGroup *ioFilterGroupNew(void); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoFilterGroupPub { List *filterList; // List of filters to apply bool inputSame; // Same input required again? bool done; // Is processing done? #ifdef DEBUG bool opened; // Has the filter set been opened? bool closed; // Has the filter set been closed? #endif } IoFilterGroupPub; // Is the filter group done processing? FN_INLINE_ALWAYS bool ioFilterGroupDone(const IoFilterGroup *const this) { ASSERT_INLINE(THIS_PUB(IoFilterGroup)->opened && !THIS_PUB(IoFilterGroup)->closed); return THIS_PUB(IoFilterGroup)->done; } // Should the same input be passed again? A buffer of input can produce multiple buffers of output, e.g. when a file containing all // zeroes is being decompressed. FN_INLINE_ALWAYS bool ioFilterGroupInputSame(const IoFilterGroup *const this) { ASSERT_INLINE(THIS_PUB(IoFilterGroup)->opened && !THIS_PUB(IoFilterGroup)->closed); return THIS_PUB(IoFilterGroup)->inputSame; } // Get all filters and their parameters so they can be passed to a remote FN_EXTERN Pack *ioFilterGroupParamAll(const IoFilterGroup *this); // Get filter results. If the same filter was used more than once then idx can be used to specify which one to get. typedef struct IoFilterGroupResultParam { VAR_PARAM_HEADER; unsigned int idx; } IoFilterGroupResultParam; #define ioFilterGroupResultP(this, filterType, ...) \ ioFilterGroupResult(this, filterType, (IoFilterGroupResultParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackRead *ioFilterGroupResult(const IoFilterGroup *this, StringId filterType, IoFilterGroupResultParam param); #define ioFilterGroupResultPackP(this, filterType, ...) \ ioFilterGroupResultPack(this, filterType, (IoFilterGroupResultParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN const Pack *ioFilterGroupResultPack(const IoFilterGroup *this, StringId filterType, IoFilterGroupResultParam param); // Get/set all filter results FN_EXTERN Pack *ioFilterGroupResultAll(const IoFilterGroup *this); FN_EXTERN void ioFilterGroupResultAllSet(IoFilterGroup *this, const Pack *filterResult); // Return total number of filters FN_INLINE_ALWAYS unsigned int ioFilterGroupSize(const IoFilterGroup *const this) { return lstSize(THIS_PUB(IoFilterGroup)->filterList); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add a filter FN_EXTERN IoFilterGroup *ioFilterGroupAdd(IoFilterGroup *this, IoFilter *filter); // Insert a filter before an index FN_EXTERN IoFilterGroup *ioFilterGroupInsert(IoFilterGroup *this, unsigned int listIdx, IoFilter *filter); // Clear filters FN_EXTERN IoFilterGroup *ioFilterGroupClear(IoFilterGroup *this); // Open filter group FN_EXTERN void ioFilterGroupOpen(IoFilterGroup *this); // Process filters FN_EXTERN void ioFilterGroupProcess(IoFilterGroup *this, const Buffer *input, Buffer *output); // Close filter group and gather results FN_EXTERN void ioFilterGroupClose(IoFilterGroup *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioFilterGroupFree(IoFilterGroup *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void ioFilterGroupToLog(const IoFilterGroup *this, StringStatic *debugLog); #define FUNCTION_LOG_IO_FILTER_GROUP_TYPE \ IoFilterGroup * #define FUNCTION_LOG_IO_FILTER_GROUP_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioFilterGroupToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/filter/sink.c0000664000175100017510000000471115265101745020470 0ustar00runnerrunner/*********************************************************************************************************************************** IO Sink Filter ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/io/filter/sink.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoSink { bool dummy; // Struct requires one member } IoSink; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_SINK_TYPE \ IoSink * #define FUNCTION_LOG_IO_SINK_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoSink", buffer, bufferSize) /*********************************************************************************************************************************** Discard all input ***********************************************************************************************************************************/ static void ioSinkProcess(THIS_VOID, const Buffer *const input, Buffer *const output) { THIS(IoSink); FUNCTION_LOG_BEGIN(logLevelTrace); (void)this; (void)input; (void)output; FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(input != NULL); ASSERT(output != NULL); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * ioSinkNew(void) { FUNCTION_LOG_VOID(logLevelTrace); OBJ_NEW_BEGIN(IoSink) { } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_FILTER, ioFilterNewP(SINK_FILTER_TYPE, this, NULL, .inOut = ioSinkProcess)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/filter/sink.h0000664000175100017510000000233215265101745020472 0ustar00runnerrunner/*********************************************************************************************************************************** IO Sink Filter Consume all bytes sent to the filter without passing any on. This filter is useful when running size/hash filters on a remote when no data should be returned. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FILTER_SINK_H #define COMMON_IO_FILTER_SINK_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define SINK_FILTER_TYPE STRID5("sink", 0x5b9330) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *ioSinkNew(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/filter/size.c0000664000175100017510000000702215265101745020474 0ustar00runnerrunner/*********************************************************************************************************************************** IO Size Filter ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/io/filter/size.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoSize { uint64_t size; // Total size of all input } IoSize; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void ioSizeToLog(const IoSize *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{size: %" PRIu64 "}", this->size); } #define FUNCTION_LOG_IO_SIZE_TYPE \ IoSize * #define FUNCTION_LOG_IO_SIZE_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioSizeToLog, buffer, bufferSize) /*********************************************************************************************************************************** Count bytes in the input ***********************************************************************************************************************************/ static void ioSizeProcess(THIS_VOID, const Buffer *const input) { THIS(IoSize); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_SIZE, this); FUNCTION_LOG_PARAM(BUFFER, input); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(input != NULL); this->size += bufUsed(input); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Return filter result ***********************************************************************************************************************************/ static Pack * ioSizeResult(THIS_VOID) { THIS(IoSize); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_SIZE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteU64P(packWrite, this->size); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * ioSizeNew(void) { FUNCTION_LOG_VOID(logLevelTrace); OBJ_NEW_BEGIN(IoSize) { *this = (IoSize){0}; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_FILTER, ioFilterNewP(SIZE_FILTER_TYPE, this, NULL, .in = ioSizeProcess, .result = ioSizeResult)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/filter/size.h0000664000175100017510000000233415265101745020502 0ustar00runnerrunner/*********************************************************************************************************************************** IO Size Filter Count all bytes that pass through the filter. Useful for getting file/IO size if added first in a FilterGroup with IoRead or last in a FilterGroup with IoWrite. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FILTER_SIZE_H #define COMMON_IO_FILTER_SIZE_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define SIZE_FILTER_TYPE STRID5("size", 0x2e9330) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *ioSizeNew(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/0000775000175100017510000000000015265101745017047 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/client.c0000664000175100017510000001101215265101745020464 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Client ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/client.h" #include "common/io/http/client.h" #include "common/log.h" #include "common/stat.h" #include "common/type/object.h" /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_EXTERN(HTTP_STAT_CLIENT_STR, HTTP_STAT_CLIENT); STRING_EXTERN(HTTP_STAT_CLOSE_STR, HTTP_STAT_CLOSE); STRING_EXTERN(HTTP_STAT_REQUEST_STR, HTTP_STAT_REQUEST); STRING_EXTERN(HTTP_STAT_RETRY_STR, HTTP_STAT_RETRY); STRING_EXTERN(HTTP_STAT_SESSION_STR, HTTP_STAT_SESSION); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpClient { HttpClientPub pub; // Publicly accessible variables IoClient *ioClient; // Io client (e.g. TLS or socket client) List *sessionReuseList; // List of HTTP sessions that can be reused }; /**********************************************************************************************************************************/ FN_EXTERN HttpClient * httpClientNew(IoClient *const ioClient, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_CLIENT, ioClient); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(ioClient != NULL); OBJ_NEW_BEGIN(HttpClient, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpClient) { .pub = { .timeout = timeout, }, .ioClient = ioClient, .sessionReuseList = lstNewP(sizeof(HttpSession *)), }; statInc(HTTP_STAT_CLIENT_STR); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(HTTP_CLIENT, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpSession * httpClientOpen(HttpClient *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); HttpSession *result = NULL; // Check if there is a reusable session if (!lstEmpty(this->sessionReuseList)) { // Remove session from reusable list result = *(HttpSession **)lstGet(this->sessionReuseList, 0); lstRemoveIdx(this->sessionReuseList, 0); // Move session to the calling context httpSessionMove(result, memContextCurrent()); } // Else create a new session else { result = httpSessionNew(this, ioClientOpen(this->ioClient)); statInc(HTTP_STAT_SESSION_STR); } FUNCTION_LOG_RETURN(HTTP_SESSION, result); } /**********************************************************************************************************************************/ FN_EXTERN void httpClientReuse(HttpClient *const this, HttpSession *const session) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_CLIENT, this); FUNCTION_LOG_PARAM(HTTP_SESSION, session); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(session != NULL); httpSessionMove(session, lstMemContext(this->sessionReuseList)); lstAdd(this->sessionReuseList, &session); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void httpClientToLog(const HttpClient *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{ioClient: "); ioClientToLog(this->ioClient, debugLog); strStcFmt(debugLog, ", reusable: %u, timeout: %" PRIu64 "}", lstSize(this->sessionReuseList), httpClientTimeout(this)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/client.h0000664000175100017510000001124715265101745020503 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Client A robust HTTP client with connection reuse and automatic retries. Using a single object to make multiple requests is more efficient because connections are reused whenever possible. Requests are automatically retried when the connection has been closed by the server. Any 5xx response is also retried. Only the HTTPS protocol is currently supported. IMPORTANT NOTE: HttpClient should have a longer lifetime than any active HttpSession objects. This does not apply to HttpSession objects that are freed, i.e. if an error occurs it does not matter in what order HttpClient and HttpSession objects are destroyed, or HttpSession objects that have been returned to the client with httpClientReuse(). The danger is when an active HttpResponse completes and tries to call httpClientReuse() on an HttpClient that has been freed thus causing a segfault. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_CLIENT_H #define COMMON_IO_HTTP_CLIENT_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpClient HttpClient; #include "common/io/client.h" #include "common/io/http/session.h" #include "common/time.h" /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ #define HTTP_STAT_CLIENT "http.client" // Clients created STRING_DECLARE(HTTP_STAT_CLIENT_STR); #define HTTP_STAT_CLOSE "http.close" // Closes forced by server STRING_DECLARE(HTTP_STAT_CLOSE_STR); #define HTTP_STAT_REQUEST "http.request" // Requests (i.e. calls to httpRequestNew()) STRING_DECLARE(HTTP_STAT_REQUEST_STR); #define HTTP_STAT_RETRY "http.retry" // Request retries STRING_DECLARE(HTTP_STAT_RETRY_STR); #define HTTP_STAT_SESSION "http.session" // Sessions created STRING_DECLARE(HTTP_STAT_SESSION_STR); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN HttpClient *httpClientNew(IoClient *ioClient, TimeMSec timeout); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct HttpClientPub { TimeMSec timeout; // Request timeout } HttpClientPub; FN_INLINE_ALWAYS TimeMSec httpClientTimeout(const HttpClient *const this) { return THIS_PUB(HttpClient)->timeout; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Open a new session FN_EXTERN HttpSession *httpClientOpen(HttpClient *this); // Request/response finished cleanly so session can be reused FN_EXTERN void httpClientReuse(HttpClient *this, HttpSession *session); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void httpClientToLog(const HttpClient *this, StringStatic *debugLog); #define FUNCTION_LOG_HTTP_CLIENT_TYPE \ HttpClient * #define FUNCTION_LOG_HTTP_CLIENT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, httpClientToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/common.c0000664000175100017510000001151415265101745020505 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Common ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/http/common.h" #include "common/time.h" /*********************************************************************************************************************************** Convert the time using the format specified in https://tools.ietf.org/html/rfc7231#section-7.1.1.1 which is used by HTTP 1.1 (the only version we support). ***********************************************************************************************************************************/ static const char *const httpCommonMonthList[] = { "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" }; static const char *const httpCommonDayList[] = {"Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"}; FN_EXTERN time_t httpDateToTime(const String *const lastModified) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, lastModified); FUNCTION_TEST_END(); // Find the month const char *const month = strZ(lastModified) + 8; unsigned int monthIdx = 0; for (; monthIdx < LENGTH_OF(httpCommonMonthList); monthIdx++) { if (strncmp(month, httpCommonMonthList[monthIdx], 3) == 0) break; } if (monthIdx == LENGTH_OF(httpCommonMonthList)) THROW_FMT(FormatError, "invalid month '%.3s'", month); // Convert to time_t FUNCTION_TEST_RETURN( TIME, epochFromParts( cvtZSubNToInt(strZ(lastModified), 12, 4), (int)monthIdx + 1, cvtZSubNToInt(strZ(lastModified), 5, 2), cvtZSubNToInt(strZ(lastModified), 17, 2), cvtZSubNToInt(strZ(lastModified), 20, 2), cvtZSubNToInt(strZ(lastModified), 23, 2), 0)); } FN_EXTERN String * httpDateFromTime(const time_t time) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TIME, time); FUNCTION_TEST_END(); struct tm timePart; gmtime_r(&time, &timePart); FUNCTION_TEST_RETURN( STRING, strNewFmt( "%s, %02d %s %04d %02d:%02d:%02d GMT", httpCommonDayList[timePart.tm_wday], timePart.tm_mday, httpCommonMonthList[timePart.tm_mon], timePart.tm_year + 1900, timePart.tm_hour, timePart.tm_min, timePart.tm_sec)); } /**********************************************************************************************************************************/ FN_EXTERN String * httpUriDecode(const String *const uri) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, uri); FUNCTION_TEST_END(); String *result = NULL; // Decode if the string is not null if (uri != NULL) { result = strNew(); // Iterate all characters in the string for (unsigned uriIdx = 0; uriIdx < strSize(uri); uriIdx++) { char uriChar = strZ(uri)[uriIdx]; // Convert escaped characters if (uriChar == '%') { // Sequence must be exactly three characters (% and two hex digits) if (strSize(uri) - uriIdx < 3) THROW_FMT(FormatError, "invalid escape sequence length in '%s'", strZ(uri)); // Convert hex digits uriChar = (char)cvtZSubNToUIntBase(strZ(uri), uriIdx + 1, 2, 16); // Skip to next character or escape uriIdx += 2; } strCatChr(result, uriChar); } } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * httpUriEncode(const String *const uri, const bool path) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, uri); FUNCTION_TEST_PARAM(BOOL, path); FUNCTION_TEST_END(); String *result = NULL; // Encode if the string is not null if (uri != NULL) { result = strNew(); // Iterate all characters in the string for (unsigned uriIdx = 0; uriIdx < strSize(uri); uriIdx++) { const char uriChar = strZ(uri)[uriIdx]; // These characters are reproduced verbatim if ((uriChar >= 'A' && uriChar <= 'Z') || (uriChar >= 'a' && uriChar <= 'z') || (uriChar >= '0' && uriChar <= '9') || uriChar == '_' || uriChar == '-' || uriChar == '~' || uriChar == '.' || (path && uriChar == '/')) { strCatChr(result, uriChar); } // All other characters are hex-encoded else strCatFmt(result, "%%%02X", (uint8_t)uriChar); } } FUNCTION_TEST_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/common.h0000664000175100017510000000214115265101745020506 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Common ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_COMMON_H #define COMMON_IO_HTTP_COMMON_H #include #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Convert HTTP date to time_t and vice versa FN_EXTERN time_t httpDateToTime(const String *lastModified); FN_EXTERN String *httpDateFromTime(time_t time); // Encode string to conform with URI specifications. If a path is being encoded then / characters won't be encoded. FN_EXTERN String *httpUriEncode(const String *uri, bool path); // Decode string that conforms to URI specifications FN_EXTERN String *httpUriDecode(const String *uri); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/header.c0000664000175100017510000001646415265101745020456 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Header ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/header.h" #include "common/io/http/request.h" #include "common/type/keyValue.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpHeader { const StringList *redactList; // List of headers to redact during logging KeyValue *kv; // KeyValue store }; /**********************************************************************************************************************************/ FN_EXTERN HttpHeader * httpHeaderNew(const StringList *const redactList) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(HttpHeader, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpHeader) { .redactList = strLstDup(redactList), .kv = kvNew(), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_HEADER, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpHeader * httpHeaderDup(const HttpHeader *const header, const StringList *const redactList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, header); FUNCTION_TEST_PARAM(STRING_LIST, redactList); FUNCTION_TEST_END(); if (header == NULL) FUNCTION_TEST_RETURN(HTTP_HEADER, NULL); OBJ_NEW_BEGIN(HttpHeader, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpHeader) { .redactList = redactList == NULL ? strLstDup(header->redactList) : strLstDup(redactList), .kv = kvDup(header->kv), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_HEADER, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpHeader * httpHeaderAdd(HttpHeader *const this, const String *const key, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); ASSERT(value != NULL); // Check if the key already exists const Variant *const keyVar = VARSTR(key); const Variant *const valueVar = kvGet(this->kv, keyVar); // If the key exists then append the new value. The HTTP spec (RFC 2616, Section 4.2) says that if a header appears more than // once then it is equivalent to a single comma-separated header. There appear to be a few exceptions such as Set-Cookie, but // they should not be of concern to us here. if (valueVar != NULL) { MEM_CONTEXT_TEMP_BEGIN() { String *const valueAppend = strCat(strNew(), varStr(valueVar)); strCatZ(valueAppend, ", "); strCatZ(valueAppend, strZ(value)); kvPut(this->kv, keyVar, VARSTR(valueAppend)); } MEM_CONTEXT_TEMP_END(); } // Else store the key else kvPut(this->kv, keyVar, VARSTR(value)); FUNCTION_TEST_RETURN(HTTP_HEADER, this); } /**********************************************************************************************************************************/ FN_EXTERN const String * httpHeaderGet(const HttpHeader *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN_CONST(STRING, varStr(kvGet(this->kv, VARSTR(key)))); } /**********************************************************************************************************************************/ FN_EXTERN StringList * httpHeaderList(const HttpHeader *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(STRING_LIST, strLstSort(strLstNewVarLst(kvKeyList(this->kv)), sortOrderAsc)); } /**********************************************************************************************************************************/ FN_EXTERN HttpHeader * httpHeaderPut(HttpHeader *const this, const String *const key, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); ASSERT(value != NULL); // Store the key kvPut(this->kv, VARSTR(key), VARSTR(value)); FUNCTION_TEST_RETURN(HTTP_HEADER, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpHeader * httpHeaderPutRange(HttpHeader *const this, const uint64_t offset, const Variant *const limit) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, this); FUNCTION_TEST_PARAM(UINT64, offset); FUNCTION_TEST_PARAM(VARIANT, limit); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(limit == NULL || varType(limit) == varTypeUInt64); if (offset != 0 || limit != NULL) { String *const range = strCatFmt(strNew(), HTTP_HEADER_RANGE_BYTES "=%" PRIu64 "-", offset); if (limit != NULL) strCatFmt(range, "%" PRIu64, offset + varUInt64(limit) - 1); httpHeaderPut(this, HTTP_HEADER_RANGE_STR, range); strFree(range); } FUNCTION_TEST_RETURN(HTTP_HEADER, this); } /**********************************************************************************************************************************/ FN_EXTERN bool httpHeaderRedact(const HttpHeader *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN(BOOL, this->redactList != NULL && strLstExists(this->redactList, key)); } /**********************************************************************************************************************************/ FN_EXTERN void httpHeaderToLog(const HttpHeader *const this, StringStatic *const debugLog) { const VariantList *const keyList = kvKeyList(this->kv); strStcCatChr(debugLog, '{'); for (unsigned int keyIdx = 0; keyIdx < varLstSize(keyList); keyIdx++) { const String *const key = varStr(varLstGet(keyList, keyIdx)); if (keyIdx != 0) strStcCat(debugLog, ", "); if (httpHeaderRedact(this, key)) strStcFmt(debugLog, "%s: ", strZ(key)); else strStcFmt(debugLog, "%s: '%s'", strZ(key), strZ(httpHeaderGet(this, key))); } strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/header.h0000664000175100017510000000664315265101745020461 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Header Object to track HTTP headers. Headers can be marked as redacted so they are not logged. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_HEADER_H #define COMMON_IO_HTTP_HEADER_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpHeader HttpHeader; #include "common/type/object.h" #include "common/type/stringList.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN HttpHeader *httpHeaderNew(const StringList *redactList); FN_EXTERN HttpHeader *httpHeaderDup(const HttpHeader *header, const StringList *redactList); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add a header FN_EXTERN HttpHeader *httpHeaderAdd(HttpHeader *this, const String *key, const String *value); // Get a value using the key FN_EXTERN const String *httpHeaderGet(const HttpHeader *this, const String *key); // Get list of keys FN_EXTERN StringList *httpHeaderList(const HttpHeader *this); // Move to a new parent mem context FN_INLINE_ALWAYS HttpHeader * httpHeaderMove(HttpHeader *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Put a header FN_EXTERN HttpHeader *httpHeaderPut(HttpHeader *this, const String *header, const String *value); // Put range header when needed FN_EXTERN HttpHeader *httpHeaderPutRange(HttpHeader *this, uint64_t offset, const Variant *limit); // Should the header be redacted when logging? FN_EXTERN bool httpHeaderRedact(const HttpHeader *this, const String *key); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void httpHeaderFree(HttpHeader *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void httpHeaderToLog(const HttpHeader *this, StringStatic *debugLog); #define FUNCTION_LOG_HTTP_HEADER_TYPE \ HttpHeader * #define FUNCTION_LOG_HTTP_HEADER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, httpHeaderToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/query.c0000664000175100017510000002327215265101745020366 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Query ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/common.h" #include "common/io/http/query.h" #include "common/type/keyValue.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpQuery { KeyValue *kv; // KeyValue store const StringList *redactList; // List of keys to redact values for }; /**********************************************************************************************************************************/ FN_EXTERN HttpQuery * httpQueryNew(const HttpQueryNewParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, param.kv); FUNCTION_TEST_PARAM(STRING_LIST, param.redactList); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(HttpQuery, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpQuery) { .kv = param.kv != NULL ? kvDup(param.kv) : kvNew(), .redactList = strLstDup(param.redactList), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_QUERY, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpQuery * httpQueryNewStr(const String *query) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, query); FUNCTION_TEST_END(); ASSERT(query != NULL); OBJ_NEW_BEGIN(HttpQuery, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpQuery) { .kv = kvNew(), }; MEM_CONTEXT_TEMP_BEGIN() { // Remove initial ? when present if (strBeginsWithZ(query, "?")) query = strSub(query, 1); // Split query into individual key value pairs const StringList *const keyValueList = strLstNewSplitZ(query, "&"); for (unsigned int keyValueIdx = 0; keyValueIdx < strLstSize(keyValueList); keyValueIdx++) { // Add each key/value pair StringList *keyValue = strLstNewSplitZ(strLstGet(keyValueList, keyValueIdx), "="); if (strLstSize(keyValue) != 2) { THROW_FMT( FormatError, "invalid key/value '%s' in query '%s'", strZ(strLstGet(keyValueList, keyValueIdx)), strZ(query)); } httpQueryAdd(this, httpUriDecode(strLstGet(keyValue, 0)), httpUriDecode(strLstGet(keyValue, 1))); } } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_QUERY, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpQuery * httpQueryDup(const HttpQuery *const query, const HttpQueryDupParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, query); FUNCTION_TEST_PARAM(STRING_LIST, param.redactList); FUNCTION_TEST_END(); if (query == NULL) FUNCTION_TEST_RETURN(HTTP_QUERY, NULL); OBJ_NEW_BEGIN(HttpQuery, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpQuery) { .kv = kvDup(query->kv), .redactList = param.redactList != NULL ? strLstDup(param.redactList) : strLstDup(query->redactList), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_QUERY, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpQuery * httpQueryAdd(HttpQuery *const this, const String *const key, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); ASSERT(value != NULL); // Make sure the key does not already exist const Variant *keyVar = VARSTR(key); if (kvGet(this->kv, keyVar) != NULL) THROW_FMT(AssertError, "key '%s' already exists", strZ(key)); // Store the key kvPut(this->kv, keyVar, VARSTR(value)); FUNCTION_TEST_RETURN(HTTP_QUERY, this); } /**********************************************************************************************************************************/ FN_EXTERN const String * httpQueryGet(const HttpQuery *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN_CONST(STRING, varStr(kvGet(this->kv, VARSTR(key)))); } /**********************************************************************************************************************************/ FN_EXTERN StringList * httpQueryList(const HttpQuery *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(STRING_LIST, strLstSort(strLstNewVarLst(kvKeyList(this->kv)), sortOrderAsc)); } /**********************************************************************************************************************************/ FN_EXTERN HttpQuery * httpQueryMerge(HttpQuery *const this, const HttpQuery *const query) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_PARAM(HTTP_QUERY, query); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(query != NULL); MEM_CONTEXT_TEMP_BEGIN() { const VariantList *const keyList = kvKeyList(query->kv); for (unsigned int keyIdx = 0; keyIdx < varLstSize(keyList); keyIdx++) { const Variant *const key = varLstGet(keyList, keyIdx); httpQueryAdd(this, varStr(key), varStr(kvGet(query->kv, key))); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(HTTP_QUERY, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpQuery * httpQueryPut(HttpQuery *const this, const String *const key, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); ASSERT(value != NULL); // Store the key kvPut(this->kv, VARSTR(key), VARSTR(value)); FUNCTION_TEST_RETURN(HTTP_QUERY, this); } /**********************************************************************************************************************************/ FN_EXTERN bool httpQueryRedact(const HttpQuery *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN(BOOL, this->redactList != NULL && strLstExists(this->redactList, key)); } /**********************************************************************************************************************************/ FN_EXTERN String * httpQueryRender(const HttpQuery *const this, const HttpQueryRenderParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_PARAM(BOOL, param.redact); FUNCTION_TEST_END(); String *result = NULL; if (this != NULL) { MEM_CONTEXT_TEMP_BEGIN() { const StringList *const keyList = httpQueryList(this); if (!strLstEmpty(keyList)) { MEM_CONTEXT_PRIOR_BEGIN() { result = strNew(); } MEM_CONTEXT_PRIOR_END(); for (unsigned int keyIdx = 0; keyIdx < strLstSize(keyList); keyIdx++) { const String *const key = strLstGet(keyList, keyIdx); if (strSize(result) != 0) strCatZ(result, "&"); strCatFmt( result, "%s=%s", strZ(httpUriEncode(key, false)), param.redact && httpQueryRedact(this, key) ? "" : strZ(httpUriEncode(httpQueryGet(this, key), false))); } } } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void httpQueryToLog(const HttpQuery *const this, StringStatic *const debugLog) { const VariantList *const keyList = kvKeyList(this->kv); strStcCatChr(debugLog, '{'); for (unsigned int keyIdx = 0; keyIdx < varLstSize(keyList); keyIdx++) { const String *const key = varStr(varLstGet(keyList, keyIdx)); if (keyIdx != 0) strStcCat(debugLog, ", "); if (httpQueryRedact(this, key)) strStcFmt(debugLog, "%s: ", strZ(key)); else strStcFmt(debugLog, "%s: '%s'", strZ(key), strZ(httpQueryGet(this, key))); } strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/query.h0000664000175100017510000001163015265101745020366 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Query Object to track HTTP queries and output them with proper escaping. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_QUERY_H #define COMMON_IO_HTTP_QUERY_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpQuery HttpQuery; #include "common/type/object.h" #include "common/type/stringList.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct HttpQueryNewParam { VAR_PARAM_HEADER; const KeyValue *kv; // Initial query key/value list const StringList *redactList; // List of keys to redact values for } HttpQueryNewParam; #define httpQueryNewP(...) \ httpQueryNew((HttpQueryNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpQuery *httpQueryNew(HttpQueryNewParam param); // New from encoded query string FN_EXTERN HttpQuery *httpQueryNewStr(const String *query); // Duplicate typedef struct HttpQueryDupParam { VAR_PARAM_HEADER; const StringList *redactList; // List of keys to redact values for } HttpQueryDupParam; #define httpQueryDupP(query, ...) \ httpQueryDup(query, (HttpQueryDupParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpQuery *httpQueryDup(const HttpQuery *query, HttpQueryDupParam param); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add a query item FN_EXTERN HttpQuery *httpQueryAdd(HttpQuery *this, const String *key, const String *value); // Get a value using the key FN_EXTERN const String *httpQueryGet(const HttpQuery *this, const String *key); // Get list of keys FN_EXTERN StringList *httpQueryList(const HttpQuery *this); // Merge the contents of another query into this one FN_EXTERN HttpQuery *httpQueryMerge(HttpQuery *this, const HttpQuery *query); // Move to a new parent mem context FN_INLINE_ALWAYS HttpQuery * httpQueryMove(HttpQuery *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Put a query item FN_EXTERN HttpQuery *httpQueryPut(HttpQuery *this, const String *header, const String *value); // Should the query key be redacted when logging? FN_EXTERN bool httpQueryRedact(const HttpQuery *this, const String *key); // Render the query for inclusion in an HTTP request typedef struct HttpQueryRenderParam { VAR_PARAM_HEADER; bool redact; // Redact user-visible query string } HttpQueryRenderParam; #define httpQueryRenderP(this, ...) \ httpQueryRender(this, (HttpQueryRenderParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN String *httpQueryRender(const HttpQuery *this, HttpQueryRenderParam param); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void httpQueryFree(HttpQuery *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void httpQueryToLog(const HttpQuery *this, StringStatic *debugLog); #define FUNCTION_LOG_HTTP_QUERY_TYPE \ HttpQuery * #define FUNCTION_LOG_HTTP_QUERY_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, httpQueryToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/request.c0000664000175100017510000005316715265101745020717 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Request ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/error/retry.h" #include "common/io/http/common.h" #include "common/io/http/request.h" #include "common/log.h" #include "common/stat.h" #include "common/wait.h" #include "version.h" /*********************************************************************************************************************************** HTTP constants ***********************************************************************************************************************************/ STRING_EXTERN(HTTP_VERSION_STR, HTTP_VERSION); STRING_EXTERN(HTTP_VERSION_10_STR, HTTP_VERSION_10); STRING_EXTERN(HTTP_VERB_DELETE_STR, HTTP_VERB_DELETE); STRING_EXTERN(HTTP_VERB_GET_STR, HTTP_VERB_GET); STRING_EXTERN(HTTP_VERB_HEAD_STR, HTTP_VERB_HEAD); STRING_EXTERN(HTTP_VERB_POST_STR, HTTP_VERB_POST); STRING_EXTERN(HTTP_VERB_PUT_STR, HTTP_VERB_PUT); STRING_EXTERN(HTTP_HEADER_AUTHORIZATION_STR, HTTP_HEADER_AUTHORIZATION); STRING_EXTERN(HTTP_HEADER_CONTENT_ID_STR, HTTP_HEADER_CONTENT_ID); STRING_EXTERN(HTTP_HEADER_CONTENT_LENGTH_STR, HTTP_HEADER_CONTENT_LENGTH); STRING_EXTERN(HTTP_HEADER_CONTENT_MD5_STR, HTTP_HEADER_CONTENT_MD5); STRING_EXTERN(HTTP_HEADER_CONTENT_RANGE_STR, HTTP_HEADER_CONTENT_RANGE); STRING_EXTERN(HTTP_HEADER_CONTENT_TYPE_STR, HTTP_HEADER_CONTENT_TYPE); STRING_EXTERN(HTTP_HEADER_CONTENT_TYPE_APP_FORM_URL_STR, HTTP_HEADER_CONTENT_TYPE_APP_FORM_URL); STRING_EXTERN(HTTP_HEADER_CONTENT_TYPE_HTTP_STR, HTTP_HEADER_CONTENT_TYPE_HTTP); STRING_EXTERN(HTTP_HEADER_CONTENT_TYPE_JSON_STR, HTTP_HEADER_CONTENT_TYPE_JSON); STRING_EXTERN(HTTP_HEADER_CONTENT_TYPE_XML_STR, HTTP_HEADER_CONTENT_TYPE_XML); STRING_EXTERN(HTTP_HEADER_ETAG_STR, HTTP_HEADER_ETAG); STRING_EXTERN(HTTP_HEADER_DATE_STR, HTTP_HEADER_DATE); STRING_EXTERN(HTTP_HEADER_HOST_STR, HTTP_HEADER_HOST); STRING_EXTERN(HTTP_HEADER_LAST_MODIFIED_STR, HTTP_HEADER_LAST_MODIFIED); STRING_EXTERN(HTTP_HEADER_RANGE_STR, HTTP_HEADER_RANGE); #define HTTP_HEADER_USER_AGENT "user-agent" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpRequest { HttpRequestPub pub; // Publicly accessible variables HttpClient *client; // HTTP client const Buffer *content; // HTTP content HttpSession *session; // Session for async requests }; struct HttpRequestMulti { List *contentList; // Multipart content size_t contentSize; // Size of all content (excluding boundaries) Buffer *boundaryRaw; // Multipart boundary (not yet formatted for output) }; /*********************************************************************************************************************************** Format the request (excluding content) ***********************************************************************************************************************************/ static String * httpRequestFmt( const String *const verb, const String *const path, const HttpQuery *const query, const HttpHeader *const header, const bool agent, const bool headerSpace) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, verb); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(HTTP_QUERY, query); FUNCTION_TEST_PARAM(HTTP_HEADER, header); FUNCTION_TEST_PARAM(BOOL, agent); FUNCTION_TEST_PARAM(BOOL, headerSpace); FUNCTION_TEST_END(); ASSERT(verb != NULL); ASSERT(path != NULL); ASSERT(header != NULL); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { // Add verb, path, and query strCatFmt( result, "%s %s%s%s " HTTP_VERSION "\r\n", strZ(verb), strZ(path), query == NULL ? "" : "?", query == NULL ? "" : strZ(httpQueryRenderP(query))); // Add user agent if (agent) strCatZ(result, HTTP_HEADER_USER_AGENT ":" PROJECT_NAME "/" PROJECT_VERSION "\r\n"); // Add headers. A space after the colon is used for multipart sub-requests since Azure's batch parser requires the canonical // "Name: value" form, but is omitted for normal requests to preserve existing behavior. StringList *const headerList = httpHeaderList(header); for (unsigned int headerIdx = 0; headerIdx < strLstSize(headerList); headerIdx++) { const String *const headerKey = strLstGet(headerList, headerIdx); strCatFmt( result, headerSpace ? "%s: %s\r\n" : "%s:%s\r\n", strZ(headerKey), strZ(httpHeaderGet(header, headerKey))); } // Add blank line to end of headers strCat(result, CRLF_STR); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Process the request ***********************************************************************************************************************************/ static HttpResponse * httpRequestProcess(HttpRequest *const this, const bool waitForResponse, const bool contentCache) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_REQUEST, this); FUNCTION_LOG_PARAM(BOOL, waitForResponse); FUNCTION_LOG_PARAM(BOOL, contentCache); FUNCTION_LOG_END(); ASSERT(this != NULL); // HTTP Response HttpResponse *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { bool retry; ErrorRetry *const errRetry = errRetryNew(); Wait *const wait = waitNew(httpClientTimeout(this->client)); do { // Create a session to process the request. This is done outside exception handling to avoid multiplying retries. HttpSession *session = NULL; bool send = true; // If a session is saved then the request was already successfully sent if (this->session != NULL) { session = httpSessionMove(this->session, memContextCurrent()); this->session = NULL; send = false; } // Else the request has not been sent yet or this is a retry else session = httpClientOpen(this->client); // Assume there will be no retry retry = false; TRY_BEGIN() { MEM_CONTEXT_TEMP_BEGIN() { // Send the request if (send) { // Write the request as a buffer so secrets do not show up in logs ioWrite( httpSessionIoWrite(session), BUFSTR( httpRequestFmt( httpRequestVerb(this), httpRequestPath(this), httpRequestQuery(this), httpRequestHeader(this), true, false))); // Write out content if any if (this->content != NULL) ioWrite(httpSessionIoWrite(session), this->content); // Flush all writes ioWriteFlush(httpSessionIoWrite(session)); // If not waiting for the response then move the session to the object context if (!waitForResponse) this->session = httpSessionMove(session, objMemContext(this)); } // Wait for response if (waitForResponse) { result = httpResponseNew(session, httpRequestVerb(this), contentCache); // Error on response codes that should be retried to engage the retry logic if (httpResponseCodeRetry(result)) THROW_FMT(ServiceError, "[%u] %s", httpResponseCode(result), strZ(httpResponseReason(result))); // Move response to outer temp context httpResponseMove(result, memContextPrior()); } } MEM_CONTEXT_TEMP_END(); } CATCH_ANY() { // Add the error retry info errRetryAddP(errRetry); // Sleep and then retry unless the total wait time has expired if (waitMore(wait)) { LOG_DEBUG_FMT("retry %s: %s", errorTypeName(errorType()), errorMessage()); retry = true; statInc(HTTP_STAT_RETRY_STR); } else THROWP(errRetryType(errRetry), strZ(errRetryMessage(errRetry))); } TRY_END(); } while (retry); // Move response to calling context httpResponseMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } /**********************************************************************************************************************************/ FN_EXTERN HttpRequest * httpRequestNew(HttpClient *const client, const String *const verb, const String *const path, const HttpRequestNewParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_CLIENT, client); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_END(); ASSERT(verb != NULL); ASSERT(path != NULL); OBJ_NEW_BEGIN(HttpRequest, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpRequest) { .pub = { .verb = strDup(verb), .path = strDup(path), .query = httpQueryDupP(param.query), .header = param.header == NULL ? httpHeaderNew(NULL) : httpHeaderDup(param.header, NULL), }, .client = client, .content = param.content == NULL ? NULL : bufDup(param.content), }; } OBJ_NEW_END(); // Send the request httpRequestProcess(this, false, false); statInc(HTTP_STAT_REQUEST_STR); FUNCTION_LOG_RETURN(HTTP_REQUEST, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpResponse * httpRequestResponse(HttpRequest *const this, const bool contentCache) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_REQUEST, this); FUNCTION_LOG_PARAM(BOOL, contentCache); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(HTTP_RESPONSE, httpRequestProcess(this, true, contentCache)); } /**********************************************************************************************************************************/ FN_EXTERN void httpRequestError(const HttpRequest *const this, HttpResponse *const response) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_REQUEST, this); FUNCTION_LOG_PARAM(HTTP_RESPONSE, response); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(response != NULL); // Error code String *const error = strCatFmt(strNew(), "HTTP request failed with %u", httpResponseCode(response)); // Add reason when present if (strSize(httpResponseReason(response)) > 0) strCatFmt(error, " (%s)", strZ(httpResponseReason(response))); // Output path/query strCatZ(error, ":\n*** Path/Query ***:"); strCatFmt(error, "\n%s %s", strZ(httpRequestVerb(this)), strZ(httpRequestPath(this))); if (httpRequestQuery(this) != NULL) strCatFmt(error, "?%s", strZ(httpQueryRenderP(httpRequestQuery(this), .redact = true))); // Output request headers const StringList *const requestHeaderList = httpHeaderList(httpRequestHeader(this)); if (!strLstEmpty(requestHeaderList)) { strCatZ(error, "\n*** Request Headers ***:"); for (unsigned int requestHeaderIdx = 0; requestHeaderIdx < strLstSize(requestHeaderList); requestHeaderIdx++) { const String *const key = strLstGet(requestHeaderList, requestHeaderIdx); strCatFmt( error, "\n%s: %s", strZ(key), httpHeaderRedact(httpRequestHeader(this), key) ? "" : strZ(httpHeaderGet(httpRequestHeader(this), key))); } } // Output response headers const HttpHeader *const responseHeader = httpResponseHeader(response); const StringList *const responseHeaderList = httpHeaderList(responseHeader); if (!strLstEmpty(responseHeaderList)) { strCatZ(error, "\n*** Response Headers ***:"); for (unsigned int responseHeaderIdx = 0; responseHeaderIdx < strLstSize(responseHeaderList); responseHeaderIdx++) { const String *const key = strLstGet(responseHeaderList, responseHeaderIdx); strCatFmt(error, "\n%s: %s", strZ(key), strZ(httpHeaderGet(responseHeader, key))); } } // Add response content, if any if (!bufEmpty(httpResponseContent(response))) { strCatZ(error, "\n*** Response Content ***:\n"); strCat(error, strNewBuf(httpResponseContent(response))); } THROW(ProtocolError, strZ(error)); } /**********************************************************************************************************************************/ #define HTTP_MULTIPART_BOUNDARY_INIT_SIZE (sizeof(HTTP_MULTIPART_BOUNDARY_INIT) - 1) FN_EXTERN HttpRequestMulti * httpRequestMultiNew(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(HttpRequestMulti, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpRequestMulti) { .boundaryRaw = bufNewC(HTTP_MULTIPART_BOUNDARY_INIT, HTTP_MULTIPART_BOUNDARY_INIT_SIZE), .contentList = lstNewP(sizeof(Buffer *)), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_REQUEST_MULTI, this); } /**********************************************************************************************************************************/ #define HTTP_MULTIPART_BOUNDARY_EXTRA_SIZE (sizeof(HTTP_MULTIPART_BOUNDARY_EXTRA) - 1) FN_EXTERN void httpRequestMultiAdd( HttpRequestMulti *const this, const String *const contentId, const String *const verb, const String *const path, const HttpRequestNewParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_REQUEST_MULTI, this); FUNCTION_TEST_PARAM(STRING, contentId); FUNCTION_TEST_PARAM(STRING, verb); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(HTTP_QUERY, param.query); FUNCTION_TEST_PARAM(HTTP_HEADER, param.header); FUNCTION_TEST_PARAM(BUFFER, param.content); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(contentId != NULL); ASSERT(verb != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Construct request header. The MIME part headers use the canonical "Name: value" form (capitalized name, space after the // colon) since Azure's batch parser is stricter than the MIME/OData spec and rejects other forms. String *const request = strNew(); strCatZ(request, "Content-Type: " HTTP_HEADER_CONTENT_TYPE_HTTP "\r\n"); strCatZ(request, "Content-Transfer-Encoding: " HTTP_HEADER_CONTENT_TRANSFER_ENCODING_BINARY "\r\n"); strCatFmt(request, "Content-ID: %s\r\n\r\n", strZ(contentId)); strCat(request, httpRequestFmt(verb, path, param.query, param.header, false, true)); MEM_CONTEXT_OBJ_BEGIN(this->contentList) { // Add content Buffer *const content = bufNew(strSize(request) + (param.content == NULL ? 0 : bufUsed(param.content))); bufCat(content, BUFSTR(request)); bufCat(content, param.content); // Find a boundary that is not used in the content while (bufFindP(content, this->boundaryRaw) != NULL) { if (bufUsed(this->boundaryRaw) == HTTP_MULTIPART_BOUNDARY_INIT_SIZE + HTTP_MULTIPART_BOUNDARY_EXTRA_SIZE) THROW(AssertError, "unable to construct unique boundary"); bufCatC( this->boundaryRaw, (const uint8_t *)HTTP_MULTIPART_BOUNDARY_EXTRA, bufUsed(this->boundaryRaw) - HTTP_MULTIPART_BOUNDARY_INIT_SIZE, HTTP_MULTIPART_BOUNDARY_NEXT); } // Add to list lstAdd(this->contentList, &content); this->contentSize += bufUsed(content); } MEM_CONTEXT_OBJ_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ #define HTTP_MULTIPART_BOUNDARY_PRE_SIZE (sizeof(HTTP_MULTIPART_BOUNDARY_PRE) - 1) #define HTTP_MULTIPART_BOUNDARY_POST_SIZE (sizeof(HTTP_MULTIPART_BOUNDARY_POST) - 1) #define HTTP_MULTIPART_BOUNDARY_POST_LAST_SIZE (sizeof(HTTP_MULTIPART_BOUNDARY_POST_LAST) - 1) FN_EXTERN Buffer * httpRequestMultiContent(HttpRequestMulti *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_REQUEST_MULTI, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(!lstEmpty(this->contentList)); const size_t boundarySize = bufUsed(this->boundaryRaw) + HTTP_MULTIPART_BOUNDARY_PRE_SIZE; Buffer *const result = bufNew( this->contentSize + (lstSize(this->contentList) * (boundarySize + HTTP_MULTIPART_BOUNDARY_POST_SIZE)) + HTTP_MULTIPART_BOUNDARY_POST_LAST_SIZE); MEM_CONTEXT_TEMP_BEGIN() { // Generate boundary. The leading CRLF is conceptually part of the delimiter that precedes each subsequent part, but the // opening delimiter is emitted without it below since Azure's batch parser requires the body to begin with the delimiter. Buffer *const boundary = bufNew(boundarySize); bufCat(boundary, BUFSTRDEF(HTTP_MULTIPART_BOUNDARY_PRE)); bufCat(boundary, this->boundaryRaw); // Add first boundary (without the leading CRLF so the body begins with the delimiter) bufCat(result, BUFSTRDEF(HTTP_MULTIPART_BOUNDARY_OPEN)); bufCat(result, this->boundaryRaw); bufCat(result, BUFSTRDEF(HTTP_MULTIPART_BOUNDARY_POST)); // Add content and boundaries for (unsigned int contentIdx = 0; contentIdx < lstSize(this->contentList); contentIdx++) { bufCat(result, *(Buffer **)lstGet(this->contentList, contentIdx)); bufCat(result, boundary); if (contentIdx < lstSize(this->contentList) - 1) bufCat(result, BUFSTRDEF(HTTP_MULTIPART_BOUNDARY_POST)); else bufCat(result, BUFSTRDEF(HTTP_MULTIPART_BOUNDARY_POST_LAST)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(BUFFER, result); } /**********************************************************************************************************************************/ FN_EXTERN HttpHeader * httpRequestMultiHeaderAdd(HttpRequestMulti *const this, HttpHeader *const header) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_REQUEST_MULTI, this); FUNCTION_TEST_PARAM(HTTP_HEADER, header); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(header != NULL); MEM_CONTEXT_TEMP_BEGIN() { httpHeaderAdd( header, HTTP_HEADER_CONTENT_TYPE_STR, strNewFmt(HTTP_HEADER_CONTENT_TYPE_MULTIPART "%s", strZ(strNewBuf(this->boundaryRaw)))); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(HTTP_HEADER, header); } /**********************************************************************************************************************************/ FN_EXTERN void httpRequestToLog(const HttpRequest *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{verb: %s, path: %s, contentSize: %zu, query: ", strZ(httpRequestVerb(this)), strZ(httpRequestPath(this)), this->content == NULL ? 0 : bufUsed(this->content)); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT( httpRequestQuery(this), httpQueryToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", header: "), httpHeaderToLog(httpRequestHeader(this), debugLog); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/request.h0000664000175100017510000002470415265101745020717 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Request Send a request to an HTTP server and get a response. The interface is natively asynchronous, i.e. httpRequestNew() sends a request and httpRequestResponse() waits for a response. These can be called together for synchronous behavior or separately for asynchronous behavior. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_REQUEST_H #define COMMON_IO_HTTP_REQUEST_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpRequest HttpRequest; typedef struct HttpRequestMulti HttpRequestMulti; #include "common/io/http/header.h" #include "common/io/http/query.h" #include "common/io/http/response.h" #include "common/type/object.h" /*********************************************************************************************************************************** HTTP Constants ***********************************************************************************************************************************/ #define HTTP_VERSION "HTTP/1.1" STRING_DECLARE(HTTP_VERSION_STR); #define HTTP_VERSION_10 "HTTP/1.0" STRING_DECLARE(HTTP_VERSION_10_STR); #define HTTP_VERB_DELETE "DELETE" STRING_DECLARE(HTTP_VERB_DELETE_STR); #define HTTP_VERB_GET "GET" STRING_DECLARE(HTTP_VERB_GET_STR); #define HTTP_VERB_HEAD "HEAD" STRING_DECLARE(HTTP_VERB_HEAD_STR); #define HTTP_VERB_POST "POST" STRING_DECLARE(HTTP_VERB_POST_STR); #define HTTP_VERB_PUT "PUT" STRING_DECLARE(HTTP_VERB_PUT_STR); #define HTTP_HEADER_AUTHORIZATION "authorization" STRING_DECLARE(HTTP_HEADER_AUTHORIZATION_STR); #define HTTP_HEADER_CONTENT_ID "content-id" STRING_DECLARE(HTTP_HEADER_CONTENT_ID_STR); #define HTTP_HEADER_CONTENT_LENGTH "content-length" STRING_DECLARE(HTTP_HEADER_CONTENT_LENGTH_STR); #define HTTP_HEADER_CONTENT_MD5 "content-md5" STRING_DECLARE(HTTP_HEADER_CONTENT_MD5_STR); #define HTTP_HEADER_CONTENT_RANGE "content-range" STRING_DECLARE(HTTP_HEADER_CONTENT_RANGE_STR); #define HTTP_HEADER_CONTENT_TRANSFER_ENCODING "content-transfer-encoding" #define HTTP_HEADER_CONTENT_TRANSFER_ENCODING_BINARY "binary" #define HTTP_HEADER_CONTENT_TYPE "content-type" STRING_DECLARE(HTTP_HEADER_CONTENT_TYPE_STR); #define HTTP_HEADER_CONTENT_TYPE_APP_FORM_URL "application/x-www-form-urlencoded" STRING_DECLARE(HTTP_HEADER_CONTENT_TYPE_APP_FORM_URL_STR); #define HTTP_HEADER_CONTENT_TYPE_HTTP "application/http" STRING_DECLARE(HTTP_HEADER_CONTENT_TYPE_HTTP_STR); #define HTTP_HEADER_CONTENT_TYPE_JSON "application/json" STRING_DECLARE(HTTP_HEADER_CONTENT_TYPE_JSON_STR); #define HTTP_HEADER_CONTENT_TYPE_MULTIPART "multipart/mixed; boundary=" #define HTTP_HEADER_CONTENT_TYPE_XML "application/xml" STRING_DECLARE(HTTP_HEADER_CONTENT_TYPE_XML_STR); #define HTTP_HEADER_CONTENT_RANGE_BYTES "bytes" #define HTTP_HEADER_DATE "date" STRING_DECLARE(HTTP_HEADER_DATE_STR); #define HTTP_HEADER_ETAG "etag" STRING_DECLARE(HTTP_HEADER_ETAG_STR); #define HTTP_HEADER_HOST "host" STRING_DECLARE(HTTP_HEADER_HOST_STR); #define HTTP_HEADER_LAST_MODIFIED "last-modified" STRING_DECLARE(HTTP_HEADER_LAST_MODIFIED_STR); #define HTTP_HEADER_RANGE "range" STRING_DECLARE(HTTP_HEADER_RANGE_STR); #define HTTP_HEADER_RANGE_BYTES "bytes" #define HTTP_MULTIPART_BOUNDARY_INIT "QKX4EYg4" #define HTTP_MULTIPART_BOUNDARY_NEXT 4 #define HTTP_MULTIPART_BOUNDARY_EXTRA "LARJ52gF4F239iNVrrC5w5aYskcGrWCXIFlMp5IxswggIhcX2A0gF9nrgN8q" #define HTTP_MULTIPART_BOUNDARY_OPEN "--" #define HTTP_MULTIPART_BOUNDARY_PRE "\r\n--" #define HTTP_MULTIPART_BOUNDARY_POST "\r\n" #define HTTP_MULTIPART_BOUNDARY_POST_LAST "--\r\n" /*********************************************************************************************************************************** Request Constructors ***********************************************************************************************************************************/ typedef struct HttpRequestNewParam { VAR_PARAM_HEADER; const HttpQuery *query; const HttpHeader *header; const Buffer *content; } HttpRequestNewParam; #define httpRequestNewP(client, verb, path, ...) \ httpRequestNew(client, verb, path, (HttpRequestNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpRequest *httpRequestNew(HttpClient *client, const String *verb, const String *path, HttpRequestNewParam param); /*********************************************************************************************************************************** Request Getters/Setters ***********************************************************************************************************************************/ typedef struct HttpRequestPub { const String *verb; // HTTP verb (GET, POST, etc.) const String *path; // HTTP path const HttpQuery *query; // HTTP query const HttpHeader *header; // HTTP headers } HttpRequestPub; // Request path FN_INLINE_ALWAYS const String * httpRequestPath(const HttpRequest *const this) { return THIS_PUB(HttpRequest)->path; } // Request query FN_INLINE_ALWAYS const HttpQuery * httpRequestQuery(const HttpRequest *const this) { return THIS_PUB(HttpRequest)->query; } // Request headers FN_INLINE_ALWAYS const HttpHeader * httpRequestHeader(const HttpRequest *const this) { return THIS_PUB(HttpRequest)->header; } // Request verb FN_INLINE_ALWAYS const String * httpRequestVerb(const HttpRequest *const this) { return THIS_PUB(HttpRequest)->verb; } /*********************************************************************************************************************************** Request Functions ***********************************************************************************************************************************/ // Wait for a response from the request FN_EXTERN HttpResponse *httpRequestResponse(HttpRequest *this, bool contentCache); // Throw an error if the request failed FN_EXTERN noreturn void httpRequestError(const HttpRequest *this, HttpResponse *response); // Move to a new parent mem context FN_INLINE_ALWAYS HttpRequest * httpRequestMove(HttpRequest *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Request Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void httpRequestFree(HttpRequest *const this) { objFree(this); } /*********************************************************************************************************************************** Request Multi Constructors ***********************************************************************************************************************************/ FN_EXTERN HttpRequestMulti *httpRequestMultiNew(void); /*********************************************************************************************************************************** Request Multi Functions ***********************************************************************************************************************************/ // Add a request to multipart content #define httpRequestMultiAddP(this, contentId, verb, path, ...) \ httpRequestMultiAdd(this, contentId, verb, path, (HttpRequestNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void httpRequestMultiAdd( HttpRequestMulti *this, const String *contentId, const String *verb, const String *path, HttpRequestNewParam param); // Concatenate multipart content FN_EXTERN Buffer *httpRequestMultiContent(HttpRequestMulti *this); // Add multipart header FN_EXTERN HttpHeader *httpRequestMultiHeaderAdd(HttpRequestMulti *this, HttpHeader *header); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void httpRequestToLog(const HttpRequest *this, StringStatic *debugLog); #define FUNCTION_LOG_HTTP_REQUEST_TYPE \ HttpRequest * #define FUNCTION_LOG_HTTP_REQUEST_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, httpRequestToLog, buffer, bufferSize) #define FUNCTION_LOG_HTTP_REQUEST_MULTI_TYPE \ HttpRequestMulti * #define FUNCTION_LOG_HTTP_REQUEST_MULTI_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "HttpRequestMulti", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/response.c0000664000175100017510000006106515265101745021061 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Response ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/bufferWrite.h" #include "common/io/http/client.h" #include "common/io/http/common.h" #include "common/io/http/request.h" #include "common/io/http/response.h" #include "common/io/io.h" #include "common/io/read.h" #include "common/log.h" #include "common/stat.h" #include "common/wait.h" /*********************************************************************************************************************************** HTTP constants ***********************************************************************************************************************************/ #define HTTP_HEADER_CONNECTION "connection" STRING_STATIC(HTTP_HEADER_CONNECTION_STR, HTTP_HEADER_CONNECTION); #define HTTP_HEADER_TRANSFER_ENCODING "transfer-encoding" STRING_STATIC(HTTP_HEADER_TRANSFER_ENCODING_STR, HTTP_HEADER_TRANSFER_ENCODING); #define HTTP_VALUE_CONNECTION_CLOSE "close" STRING_STATIC(HTTP_VALUE_CONNECTION_CLOSE_STR, HTTP_VALUE_CONNECTION_CLOSE); #define HTTP_VALUE_TRANSFER_ENCODING_CHUNKED "chunked" STRING_STATIC(HTTP_VALUE_TRANSFER_ENCODING_CHUNKED_STR, HTTP_VALUE_TRANSFER_ENCODING_CHUNKED); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpResponse { HttpResponsePub pub; // Publicly accessible variables HttpSession *session; // HTTP session bool contentChunked; // Is the response content chunked? bool contentSizeSet; // Was content size specified? uint64_t contentSize; // Content size (ignored for chunked) uint64_t contentRemaining; // Content remaining (per chunk if chunked) bool closeOnContentEof; // Will server close after content is sent? bool contentExists; // Does content exist? bool contentEof; // Has all content been read? Buffer *content; // Caches content once requested }; struct HttpResponseMulti { const Buffer *content; // Response content Buffer *boundary; // Multipart boundary const uint8_t *boundaryLast; // Last boundary location }; /*********************************************************************************************************************************** When response is done close/reuse the connection ***********************************************************************************************************************************/ static void httpResponseDone(HttpResponse *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_RESPONSE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->session != NULL); // If close was requested by the server then free the session if (this->closeOnContentEof) { httpSessionFree(this->session); // Only update the close stats after a successful response so it is not counted if there was an error/retry statInc(HTTP_STAT_CLOSE_STR); } // Else return it to the client so it can be reused else httpSessionDone(this->session); this->session = NULL; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Read content ***********************************************************************************************************************************/ // Helper to determine if it is OK to accept unexpected EOF, e.g. the server closed the socket with properly closing the TLS // connection. The idea is that since these types of responses can be validated we should be able to detect a short read. static bool httpResponseReadIgnoreUnexpectedEof(const HttpResponse *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_RESPONSE, this); FUNCTION_LOG_END(); const String *const contentType = httpHeaderGet(this->pub.header, HTTP_HEADER_CONTENT_TYPE_STR); FUNCTION_LOG_RETURN( BOOL, strEq(contentType, HTTP_HEADER_CONTENT_TYPE_XML_STR) || strEq(contentType, HTTP_HEADER_CONTENT_TYPE_JSON_STR)); } static size_t httpResponseRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(HttpResponse); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_RESPONSE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); (void)block; // Blocking is handled by the caller FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(!bufFull(buffer)); ASSERT(this->contentEof || this->session != NULL); // Read if EOF has not been reached size_t actualBytes = 0; if (!this->contentEof) { MEM_CONTEXT_TEMP_BEGIN() { // If close was requested and no content specified then the server may send content up until the eof if (this->closeOnContentEof && !this->contentChunked && this->contentSize == 0) { IoRead *const rawRead = httpSessionIoReadP( this->session, .ignoreUnexpectedEof = httpResponseReadIgnoreUnexpectedEof(this)); ioRead(rawRead, buffer); this->contentEof = ioReadEof(rawRead); } // Else read using specified encoding or size else { IoRead *const rawRead = httpSessionIoReadP(this->session); do { // If chunked content and no content remaining if (this->contentChunked && this->contentRemaining == 0) { // Read length of next chunk const String *chunkHeader = strTrim(ioReadLine(rawRead)); const int chunkExtPos = strChr(chunkHeader, ';'); if (chunkExtPos != -1) chunkHeader = strTrim(strSubN(chunkHeader, 0, (size_t)chunkExtPos)); this->contentRemaining = cvtZToUInt64Base(strZ(chunkHeader), 16); // A zero-size chunk terminates the response if (this->contentRemaining == 0) { // Consume chunk trailers and the blank line that terminates them. while (strSize(strTrim(ioReadLine(rawRead))) != 0); this->contentEof = true; } } // Read if there is content remaining if (this->contentRemaining > 0) { // If the buffer is larger than the content that needs to be read then limit the buffer size so the read // won't block or read too far. Casting to size_t is safe on 32-bit because we know the max buffer size is // defined as less than 2^32 so content remaining can't be more than that. if (bufRemains(buffer) > this->contentRemaining) bufLimitSet(buffer, bufSize(buffer) - (bufRemains(buffer) - (size_t)this->contentRemaining)); actualBytes = bufRemains(buffer); this->contentRemaining -= ioRead(rawRead, buffer); // Error if EOF but content read is not complete if (ioReadEof(rawRead)) THROW(FileReadError, "unexpected EOF reading HTTP content"); // Clear limit (this works even if the limit was not set and it is easier than checking) bufLimitClear(buffer); } // If no content remaining if (this->contentRemaining == 0) { // If chunked then consume the blank line that follows every chunk. There might be more chunk data so loop // back around to check. if (this->contentChunked && !this->contentEof) { ioReadLine(rawRead); } // If total content size was provided then this is eof else this->contentEof = true; } } while (!bufFull(buffer) && !this->contentEof); } // If all content has been read if (this->contentEof) httpResponseDone(this); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(SIZE, (size_t)actualBytes); } /*********************************************************************************************************************************** Has all content been read? ***********************************************************************************************************************************/ static bool httpResponseEof(THIS_VOID) { THIS(HttpResponse); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_RESPONSE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(BOOL, this->contentEof); } /*********************************************************************************************************************************** Read response status ***********************************************************************************************************************************/ static void httpResponseStatusRead(HttpResponse *const this, IoRead *const read) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_RESPONSE, this); FUNCTION_TEST_PARAM(IO_READ, read); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { // Read status String *status = ioReadLine(read); // Check status ends with a CR and remove it to make error formatting easier and more accurate if (!strEndsWith(status, CR_STR)) THROW_FMT(FormatError, "HTTP response status '%s' should be CR-terminated", strZ(status)); status = strSubN(status, 0, strSize(status) - 1); // Check status is at least the minimum required length to avoid harder to interpret errors later on if (strSize(status) < sizeof(HTTP_VERSION) + 4) THROW_FMT(FormatError, "HTTP response '%s' has invalid length", strZ(strTrim(status))); // If HTTP/1.0 then the connection will be closed on content eof since connections are not reused by default if (strBeginsWith(status, HTTP_VERSION_10_STR)) { this->closeOnContentEof = true; } // Else check that the version is the default (1.1) else if (!strBeginsWith(status, HTTP_VERSION_STR)) THROW_FMT(FormatError, "HTTP version of response '%s' must be " HTTP_VERSION " or " HTTP_VERSION_10, strZ(status)); // Read status code status = strSub(status, sizeof(HTTP_VERSION)); int spacePos = strChr(status, ' '); if (spacePos != 3) THROW_FMT(FormatError, "response status '%s' must have a space after the status code", strZ(status)); this->pub.code = cvtZSubNToUInt(strZ(status), 0, (size_t)spacePos); // Read reason phrase. A missing reason phrase will be represented as an empty string. MEM_CONTEXT_OBJ_BEGIN(this) { this->pub.reason = strSub(status, (size_t)spacePos + 1); } MEM_CONTEXT_OBJ_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Read response headers ***********************************************************************************************************************************/ static void httpResponseHeaderRead(HttpResponse *const this, IoRead *const read, const bool allowEof) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_RESPONSE, this); FUNCTION_TEST_PARAM(IO_READ, read); FUNCTION_TEST_PARAM(BOOL, allowEof); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { do { // Read the next header. In a multipart part the headers may be terminated by the boundary rather than a blank line (the // boundary's leading CRLF doubles as the terminator), so allow eof to end the header block in that case. String *const header = strTrim(ioReadLineParam(read, allowEof)); // If the header is empty then we have reached the end of the headers if (strSize(header) == 0) break; // Split the header and store it int colonPos = strChr(header, ':'); if (colonPos < 0) THROW_FMT(FormatError, "header '%s' missing colon", strZ(strTrim(header))); const String *const headerKey = strLower(strTrim(strSubN(header, 0, (size_t)colonPos))); String *const headerValue = strTrim(strSub(header, (size_t)colonPos + 1)); // Read transfer encoding (only chunked is supported) if (strEq(headerKey, HTTP_HEADER_TRANSFER_ENCODING_STR)) { // Error if transfer encoding is not chunked if (!strEq(strLower(headerValue), HTTP_VALUE_TRANSFER_ENCODING_CHUNKED_STR)) { THROW_FMT( FormatError, "only '%s' is supported for '%s' header", HTTP_VALUE_TRANSFER_ENCODING_CHUNKED, HTTP_HEADER_TRANSFER_ENCODING); } this->contentChunked = true; } // Read content size if (strEq(headerKey, HTTP_HEADER_CONTENT_LENGTH_STR)) { this->contentSizeSet = true; this->contentSize = cvtZToUInt64(strZ(headerValue)); this->contentRemaining = this->contentSize; } // If the server notified of a closed connection then close the client connection after reading content. This // prevents doing a retry on the next request when using the closed connection. if (strEq(headerKey, HTTP_HEADER_CONNECTION_STR) && strEq(strLower(headerValue), HTTP_VALUE_CONNECTION_CLOSE_STR)) this->closeOnContentEof = true; // Add after header checks in case the value was modified httpHeaderAdd(this->pub.header, headerKey, headerValue); } while (1); // Error if transfer encoding and content length are both set if (this->contentChunked && this->contentSizeSet) { THROW_FMT( FormatError, "'%s' and '%s' headers are both set", HTTP_HEADER_TRANSFER_ENCODING, HTTP_HEADER_CONTENT_LENGTH); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ // Constructor helper static HttpResponse * httpResponseNewInternal(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(HttpResponse, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpResponse) { .pub = { .header = httpHeaderNew(NULL), }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_RESPONSE, this); } FN_EXTERN HttpResponse * httpResponseNew(HttpSession *const session, const String *const verb, const bool contentCache) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_SESSION, session); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(BOOL, contentCache); FUNCTION_LOG_END(); ASSERT(session != NULL); ASSERT(verb != NULL); HttpResponse *const this = httpResponseNewInternal(); this->session = httpSessionMove(session, objMemContext(this)); // Read status httpResponseStatusRead(this, httpSessionIoReadP(this->session)); // Read headers httpResponseHeaderRead(this, httpSessionIoReadP(this->session), false); // Was content returned in the response? HEAD will report content but not actually return any. this->contentExists = (this->contentChunked || this->contentSize > 0 || this->closeOnContentEof) && !strEq(verb, HTTP_VERB_HEAD_STR); this->contentEof = !this->contentExists; // Create an io object, even if there is no content. This makes the logic for readers easier -- they can just check eof // rather than also checking if the io object exists. MEM_CONTEXT_OBJ_BEGIN(this) { this->pub.contentRead = ioReadNewP(this, .eof = httpResponseEof, .read = httpResponseRead); ioReadOpen(httpResponseIoRead(this)); } MEM_CONTEXT_OBJ_END(); // If there is no content then we are done with the client if (!this->contentExists) { httpResponseDone(this); } // Else cache content when requested or on error else if (contentCache || !httpResponseCodeOk(this)) { MEM_CONTEXT_OBJ_BEGIN(this) { httpResponseContent(this); } MEM_CONTEXT_OBJ_END(); } FUNCTION_LOG_RETURN(HTTP_RESPONSE, this); } /**********************************************************************************************************************************/ FN_EXTERN const Buffer * httpResponseContent(HttpResponse *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_RESPONSE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); if (this->content == NULL) { this->content = bufNew(0); if (this->contentExists) { do { bufResize(this->content, bufSize(this->content) + ioBufferSize()); httpResponseRead(this, this->content, true); } while (!httpResponseEof(this)); bufResize(this->content, bufUsed(this->content)); } } FUNCTION_TEST_RETURN(BUFFER, this->content); } /**********************************************************************************************************************************/ FN_EXTERN bool httpResponseCodeRetry(const HttpResponse *const this) { return httpResponseCode(this) / 100 == HTTP_RESPONSE_CODE_CLASS_RETRY || httpResponseCode(this) == HTTP_RESPONSE_CODE_REQUEST_TIMEOUT || httpResponseCode(this) == HTTP_RESPONSE_CODE_TOO_MANY_REQUESTS; } /**********************************************************************************************************************************/ FN_EXTERN HttpResponseMulti * httpResponseMultiNew(const Buffer *const content, const String *const contentType) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, content); FUNCTION_LOG_PARAM(STRING, contentType); FUNCTION_LOG_END(); ASSERT(content != NULL); OBJ_NEW_BEGIN(HttpResponseMulti, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpResponseMulti) { .content = content, }; MEM_CONTEXT_TEMP_BEGIN() { // Extract boundary from content type header if (contentType == NULL || !strBeginsWith(contentType, STRDEF(HTTP_HEADER_CONTENT_TYPE_MULTIPART))) THROW(FormatError, "expected multipart content type"); const String *boundary = strSub(contentType, sizeof(HTTP_HEADER_CONTENT_TYPE_MULTIPART) - 1); if (strZ(boundary)[0] == '"') boundary = strSubN(boundary, 1, strSize(boundary) - 2); const String *const boundaryNext = strNewFmt(HTTP_MULTIPART_BOUNDARY_PRE "%s", strZ(boundary)); MEM_CONTEXT_PRIOR_BEGIN() { this->boundary = bufNewC(strZ(boundaryNext), strSize(boundaryNext)); } MEM_CONTEXT_PRIOR_END(); // Find first boundary const String *const boundaryFirst = strNewFmt("--%s" HTTP_MULTIPART_BOUNDARY_POST, strZ(boundary)); ASSERT(strSize(boundaryFirst) == strSize(boundaryNext)); this->boundaryLast = bufFindP(this->content, BUFSTR(boundaryFirst)); if (this->boundaryLast == NULL) THROW_FMT(FormatError, "multipart boundary '%s' not found", strZ(boundary)); } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(HTTP_RESPONSE_MULTI, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpResponse * httpResponseMultiNext(HttpResponseMulti *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_RESPONSE_MULTI, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Return NULL when all parts have been read so the function may safely be called again if (this->boundaryLast == NULL) FUNCTION_LOG_RETURN(HTTP_RESPONSE, NULL); HttpResponse *result = NULL; // Find next boundary this->boundaryLast += bufSize(this->boundary); const uint8_t *const boundaryNext = bufFindP(this->content, this->boundary, .begin = this->boundaryLast); if (boundaryNext != NULL) { result = httpResponseNewInternal(); MEM_CONTEXT_TEMP_BEGIN() { // Create a buffer for the response Buffer *const response = bufNewC(this->boundaryLast, (size_t)(boundaryNext - this->boundaryLast)); IoRead *const responseIo = ioBufferReadNewOpen(response); // Read multipart headers httpResponseHeaderRead(result, responseIo, true); CHECK( FormatError, strEq(httpHeaderGet(httpResponseHeader(result), HTTP_HEADER_CONTENT_TYPE_STR), HTTP_HEADER_CONTENT_TYPE_HTTP_STR), "expected '" HTTP_HEADER_CONTENT_TYPE ":" HTTP_HEADER_CONTENT_TYPE_HTTP "'"); // Read status httpResponseStatusRead(result, responseIo); // Read headers (eof is allowed since the boundary, not a blank line, may terminate the embedded response headers) httpResponseHeaderRead(result, responseIo, true); // Read content CHECK(FormatError, !result->contentChunked, "chunked encoding not supported in multipart"); MEM_CONTEXT_OBJ_BEGIN(result) { result->content = bufNew((size_t)result->contentSize); } MEM_CONTEXT_OBJ_END(); IoWrite *const write = ioBufferWriteNewOpen(result->content); ioCopyP(responseIo, write); ioWriteClose(write); // Set last boundary this->boundaryLast = boundaryNext + 2; } MEM_CONTEXT_TEMP_END(); } // Else all parts have been read else this->boundaryLast = NULL; FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } /**********************************************************************************************************************************/ FN_EXTERN void httpResponseToLog(const HttpResponse *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{code: %u, reason: %s, contentChunked: %s, contentSize: %" PRIu64 ", contentRemaining: %" PRIu64 ", closeOnContentEof: %s" ", contentExists: %s, contentEof: %s, contentCached: %s}", httpResponseCode(this), httpResponseReason(this) == NULL ? NULL_Z : strZ(httpResponseReason(this)), cvtBoolToConstZ(this->contentChunked), this->contentSize, this->contentRemaining, cvtBoolToConstZ(this->closeOnContentEof), cvtBoolToConstZ(this->contentExists), cvtBoolToConstZ(this->contentEof), cvtBoolToConstZ(this->content != NULL)); strStcCat(debugLog, ", header: "), httpHeaderToLog(httpResponseHeader(this), debugLog); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/response.h0000664000175100017510000001566015265101745021066 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Response Response created after a successful request. Once the content is read the underlying connection may be recycled but the headers, cached content, etc. will still be available for the lifetime of the object. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_RESPONSE_H #define COMMON_IO_HTTP_RESPONSE_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpResponse HttpResponse; typedef struct HttpResponseMulti HttpResponseMulti; #include "common/io/http/header.h" #include "common/io/http/session.h" #include "common/io/read.h" #include "common/type/object.h" /*********************************************************************************************************************************** HTTP Response Constants ***********************************************************************************************************************************/ #define HTTP_RESPONSE_CODE_PERMANENT_REDIRECT 308 #define HTTP_RESPONSE_CODE_FORBIDDEN 403 #define HTTP_RESPONSE_CODE_NOT_FOUND 404 #define HTTP_RESPONSE_CODE_REQUEST_TIMEOUT 408 #define HTTP_RESPONSE_CODE_TOO_MANY_REQUESTS 429 // 2xx indicates success #define HTTP_RESPONSE_CODE_CLASS_OK 2 // 5xx errors that should always be retried #define HTTP_RESPONSE_CODE_CLASS_RETRY 5 /*********************************************************************************************************************************** Response Constructors ***********************************************************************************************************************************/ FN_EXTERN HttpResponse *httpResponseNew(HttpSession *session, const String *verb, bool contentCache); /*********************************************************************************************************************************** Response Getters/Setters ***********************************************************************************************************************************/ typedef struct HttpResponsePub { IoRead *contentRead; // Read interface for response content unsigned int code; // Response code (e.g. 200, 404) HttpHeader *header; // Response headers String *reason; // Response reason e.g. (OK, Not Found) } HttpResponsePub; // Read interface used to get the response content. This is intended for reading content that may be very large and will not be held // in memory all at once. If the content must be loaded completely for processing (e.g. XML) then httpResponseContent() is simpler. FN_INLINE_ALWAYS IoRead * httpResponseIoRead(HttpResponse *const this) { return THIS_PUB(HttpResponse)->contentRead; } // Response code FN_INLINE_ALWAYS unsigned int httpResponseCode(const HttpResponse *const this) { return THIS_PUB(HttpResponse)->code; } // Response headers FN_INLINE_ALWAYS const HttpHeader * httpResponseHeader(const HttpResponse *const this) { return THIS_PUB(HttpResponse)->header; } // Response reason FN_INLINE_ALWAYS const String * httpResponseReason(const HttpResponse *const this) { return THIS_PUB(HttpResponse)->reason; } /*********************************************************************************************************************************** Response Functions ***********************************************************************************************************************************/ // Is this response code OK, i.e. 2XX? FN_INLINE_ALWAYS bool httpResponseCodeOk(const HttpResponse *const this) { return httpResponseCode(this) / 100 == HTTP_RESPONSE_CODE_CLASS_OK; } // Should the request be retried? FN_EXTERN bool httpResponseCodeRetry(const HttpResponse *this); // Fetch all response content. Content will be cached so it can be retrieved again without additional cost. FN_EXTERN const Buffer *httpResponseContent(HttpResponse *this); // Move to a new parent mem context FN_INLINE_ALWAYS HttpResponse * httpResponseMove(HttpResponse *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Response Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void httpResponseFree(HttpResponse *const this) { objFree(this); } /*********************************************************************************************************************************** Response Multi Constructors ***********************************************************************************************************************************/ FN_EXTERN HttpResponseMulti *httpResponseMultiNew(const Buffer *content, const String *contentType); /*********************************************************************************************************************************** Response Multi Functions ***********************************************************************************************************************************/ // Get next response FN_EXTERN HttpResponse *httpResponseMultiNext(HttpResponseMulti *this); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void httpResponseToLog(const HttpResponse *this, StringStatic *debugLog); #define FUNCTION_LOG_HTTP_RESPONSE_TYPE \ HttpResponse * #define FUNCTION_LOG_HTTP_RESPONSE_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, httpResponseToLog, buffer, bufferSize) #define FUNCTION_LOG_HTTP_RESPONSE_MULTI_TYPE \ HttpResponseMulti * #define FUNCTION_LOG_HTTP_RESPONSE_MULTI_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "HttpResponseMulti", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/session.c0000664000175100017510000000570015265101745020700 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Session ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/session.h" #include "common/io/io.h" #include "common/log.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpSession { HttpClient *httpClient; // HTTP client IoSession *ioSession; // IO session }; /**********************************************************************************************************************************/ FN_EXTERN HttpSession * httpSessionNew(HttpClient *const httpClient, IoSession *const ioSession) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_CLIENT, httpClient); FUNCTION_LOG_PARAM(IO_SESSION, ioSession); FUNCTION_LOG_END(); ASSERT(httpClient != NULL); ASSERT(ioSession != NULL); OBJ_NEW_BEGIN(HttpSession, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpSession) { .httpClient = httpClient, .ioSession = ioSessionMove(ioSession, memContextCurrent()), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(HTTP_SESSION, this); } /**********************************************************************************************************************************/ FN_EXTERN void httpSessionDone(HttpSession *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); httpClientReuse(this->httpClient, this); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN IoRead * httpSessionIoRead(HttpSession *const this, const HttpSessionIoReadParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_SESSION, this); FUNCTION_TEST_PARAM(BOOL, param.ignoreUnexpectedEof); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(IO_READ, ioSessionIoReadP(this->ioSession, .ignoreUnexpectedEof = param.ignoreUnexpectedEof)); } /**********************************************************************************************************************************/ FN_EXTERN IoWrite * httpSessionIoWrite(HttpSession *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(IO_WRITE, ioSessionIoWrite(this->ioSession)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/session.h0000664000175100017510000000714315265101745020710 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Session HTTP sessions are created by calling httpClientOpen(), which is currently done exclusively by the HttpRequest object. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_SESSION_H #define COMMON_IO_HTTP_SESSION_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpSession HttpSession; #include "common/io/http/client.h" #include "common/io/read.h" #include "common/io/session.h" #include "common/io/write.h" #include "common/type/object.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN HttpSession *httpSessionNew(HttpClient *client, IoSession *session); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Move to a new parent mem context FN_INLINE_ALWAYS HttpSession * httpSessionMove(HttpSession *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Work with the session has finished cleanly and it can be reused FN_EXTERN void httpSessionDone(HttpSession *this); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Read interface typedef struct HttpSessionIoReadParam { VAR_PARAM_HEADER; bool ignoreUnexpectedEof; } HttpSessionIoReadParam; #define httpSessionIoReadP(this, ...) \ httpSessionIoRead(this, (HttpSessionIoReadParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN IoRead *httpSessionIoRead(HttpSession *this, HttpSessionIoReadParam param); // Write interface FN_EXTERN IoWrite *httpSessionIoWrite(HttpSession *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void httpSessionFree(HttpSession *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_HTTP_SESSION_TYPE \ HttpSession * #define FUNCTION_LOG_HTTP_SESSION_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "HttpSession", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/url.c0000664000175100017510000001723715265101745020027 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP URL ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/url.h" #include "common/regExp.h" #include "common/type/stringList.h" /*********************************************************************************************************************************** Regular expression for URLs. This is not intended to be completely comprehensive, e.g. it is still possible to enter bad hostnames. The goal is to make sure the syntax is correct enough for the rest of the parsing to succeed. ***********************************************************************************************************************************/ STRING_STATIC( HTTP_URL_REGEXP_STR, "^(http[s]{0,1}:\\/\\/){0,1}" // Optional protocol (http or https) "([^\\[\\:\\/?]+|\\[[a-fA-F0-9:]+\\])" // host/ipv4/ipv6 "(:[1-9][0-9]{0,4}){0,1}" // Optional port "(\\/[^?\\/]*)*$"); // Optional path /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpUrl { HttpUrlPub pub; }; /*********************************************************************************************************************************** Convert protocol type to a string ***********************************************************************************************************************************/ STRING_STATIC(HTTP_PROTOCOL_HTTP_STR, "http"); STRING_STATIC(HTTP_PROTOCOL_HTTPS_STR, "https"); static const String * httpProtocolTypeStr(const HttpProtocolType type) { switch (type) { case httpProtocolTypeHttp: return HTTP_PROTOCOL_HTTP_STR; case httpProtocolTypeHttps: return HTTP_PROTOCOL_HTTPS_STR; default: return NULL; } } /**********************************************************************************************************************************/ FN_EXTERN HttpUrl * httpUrlNewParse(const String *const url, const HttpUrlNewParseParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, url); FUNCTION_TEST_PARAM(ENUM, param.type); FUNCTION_TEST_PARAM(ENUM, param.defaultType); FUNCTION_TEST_END(); ASSERT(url != NULL); OBJ_NEW_BEGIN(HttpUrl, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpUrl) { .pub = { .url = strDup(url), }, }; MEM_CONTEXT_TEMP_BEGIN() { // Check that URL format is one we accept if (!regExpMatchOne(HTTP_URL_REGEXP_STR, url)) THROW_FMT(FormatError, "invalid URL '%s'", strZ(url)); // Determine whether the first part is protocol or host StringList *const splitUrl = strLstNewSplitZ(url, "/"); if (strEqZ(strLstGet(splitUrl, 0), "http:")) this->pub.type = httpProtocolTypeHttp; else if (strEqZ(strLstGet(splitUrl, 0), "https:")) this->pub.type = httpProtocolTypeHttps; // If no protocol found then the first part is the host if (this->pub.type == httpProtocolTypeAny) { // Must have either an explicitly set protocol via the type parameter or a default protocol if the type is any ASSERT(param.type != httpProtocolTypeAny || param.defaultType != httpProtocolTypeAny); this->pub.type = param.type != httpProtocolTypeAny ? param.type : param.defaultType; } // Else protocol was found else { // Protocol must match expected if (param.type != httpProtocolTypeAny && this->pub.type != param.type) THROW_FMT(FormatError, "expected protocol '%s' in URL '%s'", strZ(httpProtocolTypeStr(param.type)), strZ(url)); // Remove protocol parts from split strLstRemoveIdx(splitUrl, 0); strLstRemoveIdx(splitUrl, 0); } // Get host const String *host = strLstGet(splitUrl, 0); const String *port = NULL; // If an IPv6 address if (strBeginsWithZ(host, "[")) { // Split closing bracket const StringList *const splitHost = strLstNewSplitZ(host, "]"); ASSERT(strLstSize(splitHost) == 2); // Remove opening bracket host = strSub(strLstGet(splitHost, 0), 1); // Get port if specified if (strSize(strLstGet(splitHost, 1)) > 0) port = strSub(strLstGet(splitHost, 1), 1); } // Else IPv4 or host name else { // Split on colon const StringList *const splitHost = strLstNewSplitZ(host, ":"); ASSERT(strLstSize(splitHost) != 0); // First part is the host host = strLstGet(splitHost, 0); // Second part is the port, if it exists if (strLstSize(splitHost) > 1) { ASSERT(strLstSize(splitHost) == 2); port = strLstGet(splitHost, 1); } } // Copy host into object context MEM_CONTEXT_PRIOR_BEGIN() { this->pub.host = strDup(host); } MEM_CONTEXT_PRIOR_END(); // Get port if specified if (port != NULL) { this->pub.port = cvtZToUInt(strZ(port)); } // Else set default port based on the protocol else { ASSERT(this->pub.type != httpProtocolTypeAny); if (this->pub.type == httpProtocolTypeHttp) this->pub.port = 80; else this->pub.port = 443; } // Check for path if (strLstSize(splitUrl) > 1) { // Remove host part so it is easier to construct the path strLstRemoveIdx(splitUrl, 0); // Construct path and copy into local context const String *const path = strLstJoin(splitUrl, "/"); MEM_CONTEXT_PRIOR_BEGIN() { this->pub.path = strNewFmt("/%s", strZ(path)); } MEM_CONTEXT_PRIOR_END(); } // Else default path is / else this->pub.path = FSLASH_STR; } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_URL, this); } /**********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN void httpUrlToLog(const HttpUrl *const this, StringStatic *const debugLog) { // Is IPv6 address? bool ipv6 = strChr(this->pub.host, ':') != -1; strStcFmt( debugLog, "{%s://%s%s%s:%u%s}", strZ(httpProtocolTypeStr(this->pub.type)), ipv6 ? "[" : "", strZ(this->pub.host), ipv6 ? "]" : "", this->pub.port, strZ(this->pub.path)); } #endif // DEBUG ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/http/url.h0000664000175100017510000001075215265101745020027 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP URL Parse a URL into component parts. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_URL_H #define COMMON_IO_HTTP_URL_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpUrl HttpUrl; #include "common/type/object.h" #include "common/type/param.h" #include "common/type/string.h" /*********************************************************************************************************************************** HTTP protocol type ***********************************************************************************************************************************/ typedef enum { httpProtocolTypeAny = 0, httpProtocolTypeHttp = 1, httpProtocolTypeHttps = 2, } HttpProtocolType; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct HttpUrlNewParseParam { VAR_PARAM_HEADER; HttpProtocolType type; // Expected protocol type (httpProtocolTypeAny if any) HttpProtocolType defaultType; // Default protocol type if explicitly set type is any } HttpUrlNewParseParam; #define httpUrlNewParseP(url, ...) \ httpUrlNewParse(url, (HttpUrlNewParseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpUrl *httpUrlNewParse(const String *const url, HttpUrlNewParseParam param); /*********************************************************************************************************************************** Getters/setters ***********************************************************************************************************************************/ typedef struct HttpUrlPub { const String *url; // Original URL HttpProtocolType type; // Protocol type, e.g. http const String *host; // Host unsigned int port; // Port const String *path; // Path } HttpUrlPub; // Protocol type FN_INLINE_ALWAYS HttpProtocolType httpUrlProtocolType(const HttpUrl *const this) { return THIS_PUB(HttpUrl)->type; } // Host FN_INLINE_ALWAYS const String * httpUrlHost(const HttpUrl *const this) { return THIS_PUB(HttpUrl)->host; } // Path FN_INLINE_ALWAYS const String * httpUrlPath(const HttpUrl *const this) { return THIS_PUB(HttpUrl)->path; } // Port FN_INLINE_ALWAYS unsigned int httpUrlPort(const HttpUrl *const this) { return THIS_PUB(HttpUrl)->port; } // URL (exactly as originally passed) FN_INLINE_ALWAYS const String * httpUrl(const HttpUrl *const this) { return THIS_PUB(HttpUrl)->url; } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void httpUrlFree(HttpUrl *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN void httpUrlToLog(const HttpUrl *this, StringStatic *debugLog); #define FUNCTION_LOG_HTTP_URL_TYPE \ HttpUrl * #define FUNCTION_LOG_HTTP_URL_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, httpUrlToLog, buffer, bufferSize) #endif // DEBUG #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/io.c0000664000175100017510000001224115265101745016643 0ustar00runnerrunner/*********************************************************************************************************************************** IO Functions ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/filter/sink.h" #include "common/io/io.h" #include "common/log.h" /*********************************************************************************************************************************** Buffer size This buffer size will be used for all IO operations that require buffers not passed by the caller. Initially it is set to a conservative default with the expectation that it will be changed to a new value after options have been loaded. In general callers should set their buffer size using ioBufferSize() but there may be cases where an alternative buffer size makes sense. ***********************************************************************************************************************************/ #define IO_BUFFER_BLOCK_SIZE (8 * 1024) static size_t bufferSize = (8 * IO_BUFFER_BLOCK_SIZE); // I/O timeout in milliseconds static TimeMSec timeoutMs = 60000; /**********************************************************************************************************************************/ FN_EXTERN size_t ioBufferSize(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(SIZE, bufferSize); } FN_EXTERN void ioBufferSizeSet(const size_t bufferSizeParam) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, bufferSizeParam); FUNCTION_TEST_END(); bufferSize = bufferSizeParam; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN TimeMSec ioTimeoutMs(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(TIME_MSEC, timeoutMs); } FN_EXTERN void ioTimeoutMsSet(const TimeMSec timeoutMsParam) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TIME_MSEC, timeoutMsParam); FUNCTION_TEST_END(); timeoutMs = timeoutMsParam; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * ioReadBuf(IoRead *const read) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_READ, read); FUNCTION_TEST_END(); ASSERT(read != NULL); Buffer *const result = bufNew(0); MEM_CONTEXT_TEMP_BEGIN() { // Read IO into the buffer do { bufResize(result, bufSize(result) + ioBufferSize()); ioRead(read, result); } while (!ioReadEof(read)); // Resize the buffer and move to prior context bufResize(result, bufUsed(result)); bufMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(BUFFER, result); } /**********************************************************************************************************************************/ FN_EXTERN void ioCopy(IoRead *const source, IoWrite *const destination, const IoCopyParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_READ, source); FUNCTION_TEST_PARAM(IO_WRITE, destination); FUNCTION_TEST_PARAM(VARIANT, param.limit); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { Buffer *const buffer = bufNew(ioBufferSize()); uint64_t copied = 0; bool limitReached = false; do { // If a limit was specified then limit the buffer size on the last iteration if (param.limit != NULL) { const uint64_t bufferLimit = varUInt64(param.limit) - copied; if (bufferLimit < bufSize(buffer)) { bufLimitSet(buffer, (size_t)bufferLimit); limitReached = true; } } // Copy bytes ioRead(source, buffer); ioWrite(destination, buffer); // Update bytes copied and clear the buffer copied += bufUsed(buffer); bufUsedZero(buffer); } while (!limitReached && !ioReadEof(source)); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool ioReadDrain(IoRead *const read) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_READ, read); FUNCTION_TEST_END(); ASSERT(read != NULL); // Add a sink filter so we only need one read ioFilterGroupAdd(ioReadFilterGroup(read), ioSinkNew()); // Check if the IO can be opened bool result = ioReadOpen(read); if (result) { MEM_CONTEXT_TEMP_BEGIN() { // A single read that returns zero bytes ioRead(read, bufNew(1)); ASSERT(ioReadEof(read)); // Close the IO ioReadClose(read); } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN(BOOL, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/io.h0000664000175100017510000000424115265101745016651 0ustar00runnerrunner/*********************************************************************************************************************************** IO Functions Common IO functions. ***********************************************************************************************************************************/ #ifndef COMMON_IO_IO_H #define COMMON_IO_IO_H #include #include #include #include /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Copy data from source to destination (both must be open and neither will be closed) typedef struct IoCopyParam { VAR_PARAM_HEADER; const Variant *limit; // Limit bytes to copy from source } IoCopyParam; #define ioCopyP(source, destination, ...) \ ioCopy(source, destination, (IoCopyParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void ioCopy(IoRead *source, IoWrite *destination, IoCopyParam param); // Read all IO into a buffer FN_EXTERN Buffer *ioReadBuf(IoRead *read); // Read all IO but don't store it. Useful for calculating checksums, size, etc. FN_EXTERN bool ioReadDrain(IoRead *read); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Buffer size FN_EXTERN size_t ioBufferSize(void); FN_EXTERN void ioBufferSizeSet(size_t bufferSize); // I/O timeout in milliseconds. Used to timeout on connections and read/write operations. Note that an *entire* read/write operation // does not need to take place within this timeout but at least some progress needs to be made, even if it is only a byte. FN_EXTERN TimeMSec ioTimeoutMs(void); FN_EXTERN void ioTimeoutMsSet(TimeMSec timeout); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/limitRead.c0000664000175100017510000000773415265101745020161 0ustar00runnerrunner/*********************************************************************************************************************************** Read Chunked I/O ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/limitRead.h" #include "common/io/read.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoLimitRead { IoRead *read; // IoRead to read data from uint64_t limit; // Limit of data to read uint64_t current; // Current out of data read } IoLimitRead; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_LIMIT_READ_TYPE \ IoLimitRead * #define FUNCTION_LOG_IO_LIMIT_READ_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoLimitRead", buffer, bufferSize) /*********************************************************************************************************************************** Read next chunk or partial chunk ***********************************************************************************************************************************/ static size_t ioLimitRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(IoLimitRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_LIMIT_READ, this); FUNCTION_LOG_PARAM(BUFFER, buffer); (void)block; // Blocking is handled by the caller FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); // Continue reading until limit const size_t result = this->limit - this->current >= bufRemains(buffer) ? bufRemains(buffer) : (size_t)(this->limit - this->current); bufLimitSet(buffer, bufUsed(buffer) + result); ioRead(this->read, buffer); // Update current read this->current += result; FUNCTION_LOG_RETURN(SIZE, result); } /*********************************************************************************************************************************** Have all chunks been read? ***********************************************************************************************************************************/ static bool ioLimitReadEof(THIS_VOID) { THIS(IoLimitRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_LIMIT_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(BOOL, this->limit == this->current); } /**********************************************************************************************************************************/ FN_EXTERN IoRead * ioLimitReadNew(IoRead *const read, const uint64_t limit) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(UINT64, limit); FUNCTION_LOG_END(); ASSERT(read != NULL); OBJ_NEW_BEGIN(IoLimitRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoLimitRead) { .read = read, .limit = limit, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_READ, ioReadNewP(this, .eof = ioLimitReadEof, .read = ioLimitRead)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/limitRead.h0000664000175100017510000000137315265101745020157 0ustar00runnerrunner/*********************************************************************************************************************************** Read Limited Data Read only as much data as specified. ***********************************************************************************************************************************/ #ifndef COMMON_IO_LIMITREAD_H #define COMMON_IO_LIMITREAD_H #include "common/io/read.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoRead *ioLimitReadNew(IoRead *read, uint64_t limit); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/read.c0000664000175100017510000004263415265101745017160 0ustar00runnerrunner/*********************************************************************************************************************************** IO Read Interface ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/io.h" #include "common/io/read.h" #include "common/log.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoRead { IoReadPub pub; // Publicly accessible variables Buffer *input; // Input buffer Buffer *output; // Internal output buffer (extra output from buffered reads) size_t outputPos; // Current position in the internal output buffer }; /**********************************************************************************************************************************/ FN_EXTERN IoRead * ioReadNew(void *const driver, const IoReadInterface interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(IO_READ_INTERFACE, interface); FUNCTION_LOG_END(); ASSERT(driver != NULL); ASSERT(interface.read != NULL); OBJ_NEW_BEGIN(IoRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoRead) { .pub = { .driver = objMoveToInterface(driver, this, memContextPrior()), .interface = interface, .filterGroup = ioFilterGroupNew(), }, .input = bufNew(ioBufferSize()), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_READ, this); } /**********************************************************************************************************************************/ FN_EXTERN bool ioReadOpen(IoRead *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->pub.opened && !this->pub.closed); ASSERT(ioFilterGroupSize(this->pub.filterGroup) == 0 || !ioReadBlock(this)); // Open if the driver has an open function const bool result = ioReadInterface(this)->open != NULL ? ioReadInterface(this)->open(ioReadDriver(this)) : true; // Only open the filter group if the read was opened if (result) ioFilterGroupOpen(this->pub.filterGroup); #ifdef DEBUG this->pub.opened = result; #endif FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Is the driver at EOF? This is different from the overall eof because filters may still be holding buffered data. ***********************************************************************************************************************************/ static bool ioReadEofDriver(const IoRead *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.opened && !this->pub.closed); FUNCTION_LOG_RETURN(BOOL, ioReadInterface(this)->eof != NULL ? ioReadInterface(this)->eof(ioReadDriver(this)) : false); } /**********************************************************************************************************************************/ static void ioReadInternal(IoRead *const this, Buffer *const buffer, const bool block) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Loop until EOF or the output buffer is full const size_t bufferUsedBegin = bufUsed(buffer); while (!ioReadEof(this) && bufRemains(buffer) > 0) { // Process input buffer again to get more output if (ioFilterGroupInputSame(this->pub.filterGroup)) { ioFilterGroupProcess(this->pub.filterGroup, this->input, buffer); } // Else new input can be accepted else { // Read if not EOF if (this->input != NULL) { if (!ioReadEofDriver(this)) { bufUsedZero(this->input); // If blocking then limit the amount of data requested if (ioReadBlock(this) && bufRemains(this->input) > bufRemains(buffer)) bufLimitSet(this->input, bufRemains(buffer)); ioReadInterface(this)->read(ioReadDriver(this), this->input, block); bufLimitClear(this->input); } // Set input to NULL and flush (no need to actually free the buffer here as it will be freed with the mem context) else this->input = NULL; } // Process the input buffer (or flush if NULL) if (this->input == NULL || !bufEmpty(this->input)) ioFilterGroupProcess(this->pub.filterGroup, this->input, buffer); } // Stop if not blocking -- we don't need to fill the buffer as long as we got some data. Eof is set on a later read that // gets no data. if (!block && bufUsed(buffer) > bufferUsedBegin) break; // Eof when no more input and the filter group is done this->pub.eofAll = ioReadEofDriver(this) && ioFilterGroupDone(this->pub.filterGroup); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Read data and use buffered line read output when present ***********************************************************************************************************************************/ FN_EXTERN size_t ioRead(IoRead *const this, Buffer *const buffer) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BUFFER, this->output); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Store size of remaining portion of buffer to calculate total read at the end const size_t outputRemains = bufRemains(buffer); // Copy any data in the internal output buffer if (this->output != NULL && bufUsed(this->output) - this->outputPos > 0 && bufRemains(buffer) > 0) { // Internal output buffer remains taking into account the position const size_t outputInternalRemains = bufUsed(this->output) - this->outputPos; // Determine how much data should be copied const size_t size = outputInternalRemains > bufRemains(buffer) ? bufRemains(buffer) : outputInternalRemains; // Copy data to the output buffer bufCatSub(buffer, this->output, this->outputPos, size); this->outputPos += size; } // Read data ioReadInternal(this, buffer, true); FUNCTION_LOG_RETURN(SIZE, outputRemains - bufRemains(buffer)); } /**********************************************************************************************************************************/ FN_EXTERN size_t ioReadSmall(IoRead *const this, Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_READ, this); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Allocate the internal output buffer if it has not already been allocated if (this->output == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->output = bufNew(ioBufferSize()); } MEM_CONTEXT_OBJ_END(); } // Store size of remaining portion of buffer to calculate total read at the end size_t outputRemains = bufRemains(buffer); do { // Internal output buffer remains taking into account the position const size_t outputInternalRemains = bufUsed(this->output) - this->outputPos; // Use any data in the internal output buffer if (outputInternalRemains > 0) { // Determine how much data should be copied const size_t size = outputInternalRemains > bufRemains(buffer) ? bufRemains(buffer) : outputInternalRemains; // Copy data to the output buffer bufCatSub(buffer, this->output, this->outputPos, size); this->outputPos += size; } // If more data is required if (!bufFull(buffer)) { // If the data required is the same size as the internal output buffer then just read into the external buffer if (bufRemains(buffer) >= bufSize(this->output)) { ioReadInternal(this, buffer, true); } // Else read as much data as is available. If it is not enough we will try again later. else { // Clear the internal output buffer since all data was copied already bufUsedZero(this->output); this->outputPos = 0; ioReadInternal(this, this->output, false); } } } while (!bufFull(buffer) && !ioReadEof(this)); FUNCTION_TEST_RETURN(SIZE, outputRemains - bufRemains(buffer)); } /*********************************************************************************************************************************** The entire string to search for must fit within a single buffer. ***********************************************************************************************************************************/ FN_EXTERN String * ioReadLineParam(IoRead *const this, const bool allowEof) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_PARAM(BOOL, allowEof); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Allocate the output buffer if it has not already been allocated. This buffer is not allocated at object creation because it // is not always used. if (this->output == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->output = bufNew(ioBufferSize()); } MEM_CONTEXT_OBJ_END(); } // Search for a linefeed String *result = NULL; do { // Internal output buffer remains taking into account the position size_t outputInternalRemains = bufUsed(this->output) - this->outputPos; if (outputInternalRemains > 0) { // Internal output buffer pointer taking into account the position const char *const outputPtr = (char *)bufPtr(this->output) + this->outputPos; // Search for a linefeed in the buffer const char *const linefeed = memchr(outputPtr, '\n', outputInternalRemains); // A linefeed was found so get the string if (linefeed != NULL) { // Get the string size const size_t size = (size_t)(linefeed - outputPtr); // Create the string result = strNewZN(outputPtr, size); this->outputPos += size + 1; } } // Read data if no linefeed was found in the existing buffer if (result == NULL) { // If there is remaining data left in the internal output buffer then trim off the used data if (outputInternalRemains > 0) { memmove( bufPtr(this->output), bufPtr(this->output) + (bufUsed(this->output) - outputInternalRemains), outputInternalRemains); } // Set used bytes and reset position bufUsedSet(this->output, outputInternalRemains); this->outputPos = 0; // If the buffer is full then the linefeed (if it exists) is outside the buffer if (bufFull(this->output)) THROW_FMT(FileReadError, "unable to find line in %zu byte buffer", bufUsed(this->output)); if (ioReadEof(this)) { if (allowEof) result = strNewZN((char *)bufPtr(this->output), bufUsed(this->output)); else THROW(FileReadError, "unexpected eof while reading line"); } else ioReadInternal(this, this->output, false); } } while (result == NULL); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN uint64_t ioReadVarIntU64(IoRead *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_END(); // Allocate the internal output buffer if it has not already been allocated if (this->output == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->output = bufNew(ioBufferSize()); } MEM_CONTEXT_OBJ_END(); } uint64_t result = 0; uint8_t byte; // Convert bytes from varint-128 encoding to a uint64 for (unsigned int bufferIdx = 0; bufferIdx < CVT_VARINT128_BUFFER_SIZE; bufferIdx++) { // Get more bytes if needed if (bufUsed(this->output) - this->outputPos == 0) { // Clear the internal output buffer since all data was copied already bufUsedZero(this->output); this->outputPos = 0; // Read into the internal output buffer ioReadInternal(this, this->output, false); // Error on eof if (bufUsed(this->output) == 0) THROW(FileReadError, "unexpected eof"); } // Get the next encoded byte byte = bufPtr(this->output)[this->outputPos++]; // Shift the lower order 7 encoded bits into the uint64 in reverse order result |= (uint64_t)(byte & 0x7f) << (7 * bufferIdx); // Done if the high order bit is not set to indicate more data if (byte < 0x80) break; } // By this point all bytes should have been read so error if this is not the case. This could be due to a coding error or // corrupton in the data stream. if (byte >= 0x80) THROW(FormatError, "unterminated base-128 integer"); FUNCTION_LOG_RETURN(UINT64, result); } /**********************************************************************************************************************************/ FN_EXTERN bool ioReadReady(IoRead *const this, const IoReadReadyParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_PARAM(BOOL, param.error); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = true; if (ioReadInterface(this)->ready != NULL) result = ioReadInterface(this)->ready(ioReadDriver(this), param.error); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN uint64_t ioReadFlush(IoRead *const this, const IoReadFlushParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_PARAM(BOOL, param.errorOnBytes); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Flush remaining data uint64_t result = 0; if (!ioReadEof(this)) { Buffer *const buffer = bufNew(ioBufferSize()); do { result += ioRead(this, buffer); bufUsedZero(buffer); } while (!ioReadEof(this)); bufFree(buffer); } // Error when bytes found and error requested if (result != 0 && param.errorOnBytes) THROW_FMT(FileReadError, "expected EOF but flushed %" PRIu64 " byte(s)", result); FUNCTION_LOG_RETURN(UINT64, result); } /**********************************************************************************************************************************/ FN_EXTERN void ioReadClose(IoRead *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Close the filter group and gather results ioFilterGroupClose(this->pub.filterGroup); // Close the driver if there is a close function if (ioReadInterface(this)->close != NULL) ioReadInterface(this)->close(ioReadDriver(this)); #ifdef DEBUG this->pub.closed = true; #endif FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN int ioReadFd(const IoRead *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(INT, ioReadInterface(this)->fd == NULL ? -1 : ioReadInterface(this)->fd(ioReadDriver(this))); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/read.h0000664000175100017510000001251415265101745017157 0ustar00runnerrunner/*********************************************************************************************************************************** IO Read Interface Objects that read from some IO source (file, socket, etc.) are implemented using this interface. All objects are required to implement IoReadProcess and can optionally implement IoReadOpen, IoReadClose, or IoReadEof. IoReadOpen and IoReadClose can be used to allocate/open or deallocate/free resources. If IoReadEof is not implemented then ioReadEof() will always return false. An example of an IoRead object is IoBufferRead. ***********************************************************************************************************************************/ #ifndef COMMON_IO_READ_H #define COMMON_IO_READ_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoRead IoRead; #include "common/io/filter/group.h" #include "common/io/read.intern.h" #include "common/type/buffer.h" #include "common/type/object.h" /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Do reads block when more bytes are requested than are available to read? FN_INLINE_ALWAYS bool ioReadBlock(const IoRead *const this) { return THIS_PUB(IoRead)->interface.block; } // Is IO at EOF? All driver reads are complete and all data has been flushed from the filters (if any). FN_INLINE_ALWAYS bool ioReadEof(const IoRead *const this) { ASSERT_INLINE(THIS_PUB(IoRead)->opened && !THIS_PUB(IoRead)->closed); return THIS_PUB(IoRead)->eofAll; } // Get filter group if filters need to be added FN_INLINE_ALWAYS IoFilterGroup * ioReadFilterGroup(IoRead *const this) { return THIS_PUB(IoRead)->filterGroup; } // File descriptor for the read object. Not all read objects have a file descriptor and -1 will be returned in that case. FN_EXTERN int ioReadFd(const IoRead *this); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Open the IO FN_EXTERN bool ioReadOpen(IoRead *this); // Read data from IO and process filters FN_EXTERN size_t ioRead(IoRead *this, Buffer *buffer); // Same as ioRead() but optimized for small reads (intended for making repetitive reads that are smaller than ioBufferSize()) FN_EXTERN size_t ioReadSmall(IoRead *this, Buffer *buffer); // Read linefeed-terminated string and optionally error on eof FN_EXTERN String *ioReadLineParam(IoRead *this, bool allowEof); // Read linefeed-terminated string FN_INLINE_ALWAYS String * ioReadLine(IoRead *const this) { return ioReadLineParam(this, false); } // Read varint-128 encoding FN_EXTERN uint64_t ioReadVarIntU64(IoRead *this); // Are there bytes ready to read immediately? There are no guarantees on how much data is available to read but it must be at least // one byte. typedef struct IoReadReadyParam { VAR_PARAM_HEADER; bool error; // Error when read not ready } IoReadReadyParam; #define ioReadReadyP(this, ...) \ ioReadReady(this, (IoReadReadyParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool ioReadReady(IoRead *this, IoReadReadyParam param); // Flush all remaining bytes and return bytes flushed. Optionally error when bytes are flushed. typedef struct IoReadFlushParam { VAR_PARAM_HEADER; bool errorOnBytes; // Error when any bytes are found during flush } IoReadFlushParam; #define ioReadFlushP(this, ...) \ ioReadFlush(this, (IoReadFlushParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN uint64_t ioReadFlush(IoRead *this, IoReadFlushParam param); // Close the IO FN_EXTERN void ioReadClose(IoRead *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioReadFree(IoRead *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_READ_TYPE \ IoRead * #define FUNCTION_LOG_IO_READ_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoRead", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/read.intern.h0000664000175100017510000000712515265101745020457 0ustar00runnerrunner/*********************************************************************************************************************************** IO Read Interface Internal ***********************************************************************************************************************************/ #ifndef COMMON_IO_READ_INTERN_H #define COMMON_IO_READ_INTERN_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ #include "common/io/read.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct IoReadInterface { bool block; // Do reads block when buffer is larger than available bytes? bool (*eof)(void *driver); void (*close)(void *driver); bool (*open)(void *driver); int (*fd)(const void *driver); size_t (*read)(void *driver, Buffer *buffer, bool block); // Are there bytes ready to read immediately? There are no guarantees on how much data is available to read but it must be at // least one byte. Optionally error when read is not ready. bool (*ready)(void *driver, bool error); } IoReadInterface; #define ioReadNewP(driver, ...) \ ioReadNew(driver, (IoReadInterface){__VA_ARGS__}) FN_EXTERN IoRead *ioReadNew(void *driver, IoReadInterface interface); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoReadPub { void *driver; // Driver object IoReadInterface interface; // Driver interface IoFilterGroup *filterGroup; // IO filters bool eofAll; // Is the read done (read and filters complete)? #ifdef DEBUG bool opened; // Has the io been opened? bool closed; // Has the io been closed? #endif } IoReadPub; // Driver for the read object FN_INLINE_ALWAYS void * ioReadDriver(const IoRead *const this) { return THIS_PUB(IoRead)->driver; } // Interface for the read object FN_INLINE_ALWAYS const IoReadInterface * ioReadInterface(const IoRead *const this) { return &THIS_PUB(IoRead)->interface; } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_READ_INTERFACE_TYPE \ IoReadInterface #define FUNCTION_LOG_IO_READ_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "IoReadInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/server.c0000664000175100017510000000437315265101745017551 0ustar00runnerrunner/*********************************************************************************************************************************** Io Server Interface ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/server.h" #include "common/log.h" #include "common/memContext.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoServer { IoServerPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN IoServer * ioServerNew(void *const driver, const IoServerInterface *const interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(IO_SERVER_INTERFACE, interface); FUNCTION_LOG_END(); ASSERT(driver != NULL); ASSERT(interface != NULL); ASSERT(interface->type != 0); ASSERT(interface->name != NULL); ASSERT(interface->accept != NULL); ASSERT(interface->toLog != NULL); OBJ_NEW_BEGIN(IoServer, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoServer) { .pub = { .driver = objMoveToInterface(driver, this, memContextPrior()), .interface = interface, }, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_SERVER, this); } /**********************************************************************************************************************************/ FN_EXTERN void ioServerToLog(const IoServer *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "); strStcResultSizeInc(debugLog, strIdToLog(this->pub.interface->type, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", driver: "); this->pub.interface->toLog(this->pub.driver, debugLog); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/server.h0000664000175100017510000000642215265101745017553 0ustar00runnerrunner/*********************************************************************************************************************************** Io Server Interface Create sessions for protocol servers. For example, a TLS server can be created with tlsServerNew() and then new TLS sessions can be accepted with ioServerAccept(). ***********************************************************************************************************************************/ #ifndef COMMON_IO_SERVER_H #define COMMON_IO_SERVER_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoServer IoServer; #include "common/io/server.intern.h" #include "common/io/session.h" #include "common/type/object.h" /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoServerPub { void *driver; // Driver object const IoServerInterface *interface; // Driver interface } IoServerPub; // Name that identifies the server FN_INLINE_ALWAYS const String * ioServerName(const IoServer *const this) { return THIS_PUB(IoServer)->interface->name(THIS_PUB(IoServer)->driver); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Move to a new parent mem context FN_INLINE_ALWAYS IoServer * ioServerMove(IoServer *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Open session FN_INLINE_ALWAYS IoSession * ioServerAccept(IoServer *const this, IoSession *const session) { return THIS_PUB(IoServer)->interface->accept(THIS_PUB(IoServer)->driver, session); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioServerFree(IoServer *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void ioServerToLog(const IoServer *this, StringStatic *debugLog); #define FUNCTION_LOG_IO_SERVER_TYPE \ IoServer * #define FUNCTION_LOG_IO_SERVER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioServerToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/server.intern.h0000664000175100017510000000416215265101745021050 0ustar00runnerrunner/*********************************************************************************************************************************** Io Server Interface Internal ***********************************************************************************************************************************/ #ifndef COMMON_IO_SERVER_INTERN_H #define COMMON_IO_SERVER_INTERN_H #include "common/io/server.h" #include "common/io/session.h" #include "common/type/string.h" /*********************************************************************************************************************************** Interface ***********************************************************************************************************************************/ typedef struct IoServerInterface { // Type used to identify the server StringId type; // Server name, usually address:port or some other unique identifier const String *(*name)(void *driver); // Accept a session IoSession *(*accept)(void *driver, IoSession *session); // Driver log function void (*toLog)(const void *driver, StringStatic *debugLog); } IoServerInterface; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoServer *ioServerNew(void *driver, const IoServerInterface *interface); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_SERVER_INTERFACE_TYPE \ IoServerInterface * #define FUNCTION_LOG_IO_SERVER_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "IoServerInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/session.c0000664000175100017510000001142115265101745017716 0ustar00runnerrunner/*********************************************************************************************************************************** Io Session Interface ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/session.h" #include "common/log.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoSession { IoSessionPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN IoSession * ioSessionNew(void *const driver, const IoSessionInterface *const interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(IO_SESSION_INTERFACE, interface); FUNCTION_LOG_END(); ASSERT(driver != NULL); ASSERT(interface != NULL); ASSERT(interface->type != 0); ASSERT(interface->close != NULL); ASSERT(interface->ioRead != NULL); ASSERT(interface->ioWrite != NULL); ASSERT(interface->role != NULL); ASSERT(interface->toLog != NULL); OBJ_NEW_BEGIN(IoSession, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoSession) { .pub = { .driver = objMoveToInterface(driver, this, memContextPrior()), .interface = interface, }, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_SESSION, this); } /**********************************************************************************************************************************/ FN_EXTERN void ioSessionAuthenticatedSet(IoSession *const this, const bool authenticated) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_SESSION, this); FUNCTION_TEST_PARAM(BOOL, authenticated); FUNCTION_TEST_END(); this->pub.authenticated = authenticated; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN int ioSessionFd(IoSession *this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->pub.interface->fd == NULL ? -1 : this->pub.interface->fd(this->pub.driver)); } /**********************************************************************************************************************************/ FN_EXTERN void ioSessionPeerNameSet(IoSession *const this, const String *const peerName) // {vm_covered} { FUNCTION_TEST_BEGIN(); // {vm_covered} FUNCTION_TEST_PARAM(IO_SESSION, this); // {vm_covered} FUNCTION_TEST_PARAM(STRING, peerName); // {vm_covered} FUNCTION_TEST_END(); // {vm_covered} MEM_CONTEXT_OBJ_BEGIN(this) // {vm_covered} { this->pub.peerName = strDup(peerName); // {vm_covered} } MEM_CONTEXT_OBJ_END(); // {vm_covered} FUNCTION_TEST_RETURN_VOID(); // {vm_covered} } // {vm_covered} /**********************************************************************************************************************************/ FN_EXTERN void ioSessionToLog(const IoSession *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "); strStcResultSizeInc(debugLog, strIdToLog(this->pub.interface->type, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", role: "); strStcResultSizeInc(debugLog, strIdToLog(ioSessionRole(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", driver: "); this->pub.interface->toLog(this->pub.driver, debugLog); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/session.h0000664000175100017510000001267515265101745017737 0ustar00runnerrunner/*********************************************************************************************************************************** Io Session Interface Provides access to IoRead and IoWrite interfaces for interacting with the session returned by ioClientOpen(). Sessions should always be closed when work with them is done but they also contain destructors to do cleanup if there is an error. ***********************************************************************************************************************************/ #ifndef COMMON_IO_SESSION_H #define COMMON_IO_SESSION_H #include "common/type/stringId.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoSession IoSession; /*********************************************************************************************************************************** Session roles ***********************************************************************************************************************************/ typedef enum { ioSessionRoleClient = STRID5("client", 0x28e2a5830), // Client session ioSessionRoleServer = STRID5("server", 0x245b48b30), // Server session } IoSessionRole; #include "common/io/read.h" #include "common/io/session.intern.h" #include "common/io/write.h" #include "common/type/object.h" /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoSessionPub { void *driver; // Driver object const IoSessionInterface *interface; // Driver interface const String *peerName; // Name of peer (exact meaning depends on driver) bool authenticated; // Is the session authenticated? } IoSessionPub; // Is the session authenticated? The exact meaning of "authenticated" will vary by driver type. FN_INLINE_ALWAYS bool ioSessionAuthenticated(const IoSession *const this) { return THIS_PUB(IoSession)->authenticated; } // Session file descriptor, -1 if none FN_EXTERN int ioSessionFd(IoSession *this); // Read interface typedef struct IoSessionIoReadParam { VAR_PARAM_HEADER; bool ignoreUnexpectedEof; // Allow unexpected EOF, e.g. TLS session improperly terminated } IoSessionIoReadParam; #define ioSessionIoReadP(this, ...) \ ioSessionIoRead(this, (IoSessionIoReadParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_INLINE_ALWAYS IoRead * ioSessionIoRead(IoSession *const this, const IoSessionIoReadParam param) { return THIS_PUB(IoSession)->interface->ioRead(THIS_PUB(IoSession)->driver, param.ignoreUnexpectedEof); } // Write interface FN_INLINE_ALWAYS IoWrite * ioSessionIoWrite(IoSession *const this) { return THIS_PUB(IoSession)->interface->ioWrite(THIS_PUB(IoSession)->driver); } // The peer name, if any. The exact meaning will vary by driver type, for example the peer name will be the client CN for the // TlsSession driver. FN_INLINE_ALWAYS const String * ioSessionPeerName(const IoSession *const this) { return THIS_PUB(IoSession)->peerName; } // Session role FN_INLINE_ALWAYS IoSessionRole ioSessionRole(const IoSession *const this) { return THIS_PUB(IoSession)->interface->role(THIS_PUB(IoSession)->driver); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Close the session FN_INLINE_ALWAYS void ioSessionClose(IoSession *const this) { return THIS_PUB(IoSession)->interface->close(THIS_PUB(IoSession)->driver); } // Move to a new parent mem context FN_INLINE_ALWAYS IoSession * ioSessionMove(IoSession *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioSessionFree(IoSession *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void ioSessionToLog(const IoSession *this, StringStatic *debugLog); #define FUNCTION_LOG_IO_SESSION_TYPE \ IoSession * #define FUNCTION_LOG_IO_SESSION_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioSessionToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/session.intern.h0000664000175100017510000000540615265101745021227 0ustar00runnerrunner/*********************************************************************************************************************************** Io Session Interface Internal ***********************************************************************************************************************************/ #ifndef COMMON_IO_SESSION_INTERN_H #define COMMON_IO_SESSION_INTERN_H #include "common/io/session.h" #include "common/io/write.h" /*********************************************************************************************************************************** Interface ***********************************************************************************************************************************/ typedef struct IoSessionInterface { // Type used to identify the session StringId type; // Close the session void (*close)(void *driver); // Session file descriptor, if any int (*fd)(void *driver); // IoRead interface for the session IoRead *(*ioRead)(void *driver, bool ignoreUnexpectedEof); // IoWrite interface for the session IoWrite *(*ioWrite)(void *driver); // Session role IoSessionRole (*role)(const void *driver); // Driver log function void (*toLog)(const void *driver, StringStatic *debugLog); } IoSessionInterface; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoSession *ioSessionNew(void *driver, const IoSessionInterface *interface); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Has the session been authenticated? FN_EXTERN void ioSessionAuthenticatedSet(IoSession *this, bool authenticated); // Set the peer name FN_EXTERN void ioSessionPeerNameSet(IoSession *this, const String *peerName); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_SESSION_INTERFACE_TYPE \ IoSessionInterface * #define FUNCTION_LOG_IO_SESSION_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "IoSessionInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/socket/0000775000175100017510000000000015265101745017360 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/socket/address.c0000664000175100017510000003271615265101745021162 0ustar00runnerrunner/*********************************************************************************************************************************** Address Info ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/io/socket/address.h" #include "common/log.h" #include "common/memContext.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct AddressInfo { AddressInfoPub pub; // Publicly accessible variables struct addrinfo *info; // Linked list of addresses }; /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ // Store preferred addresses for hosts typedef struct AddressInfoPreference { const String *const host; // Host String *address; // Preferred address for host } AddressInfoPreference; static struct AddressInfoLocal { MemContext *memContext; // Mem context List *prefList; // List of preferred addresses for hosts } addressInfoLocal; /*********************************************************************************************************************************** Free addrinfo linked list allocated by getaddrinfo() ***********************************************************************************************************************************/ static void addrInfoFreeResource(THIS_VOID) { THIS(AddressInfo); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ADDRESS_INFO, this); FUNCTION_LOG_END(); ASSERT(this != NULL); freeaddrinfo(this->info); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void addrInfoSort(AddressInfo *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ADDRESS_INFO, this); FUNCTION_TEST_END(); // Only sort if there is more than one address if (lstSize(this->pub.list) > 1) { // By default start with IPv6 and first address int family = AF_INET6; unsigned int addrIdx = 0; // If a preferred address is in the list then move it to the first position and update family const AddressInfoPreference *const addrPref = lstFind(addressInfoLocal.prefList, &this->pub.host); if (addrPref != NULL) // {vm_covered} { AddressInfoItem *const addrFindItem = lstFind(this->pub.list, &addrPref->address); if (addrFindItem != NULL) // {vm_covered} { // Swap with first address if the address is not already first AddressInfoItem *const addrFirstItem = lstGet(this->pub.list, 0); if (addrFirstItem != addrFindItem) // {vm_covered} { const AddressInfoItem addrCopyItem = *addrFirstItem; *addrFirstItem = *addrFindItem; *addrFindItem = addrCopyItem; } // Set family and skip first address family = addrFirstItem->info->ai_family == AF_INET6 ? AF_INET : AF_INET6; // {vm_covered} addrIdx++; } } // Alternate IPv6 and IPv4 addresses for (; addrIdx < lstSize(this->pub.list); addrIdx++) { AddressInfoItem *const addrItem = lstGet(this->pub.list, addrIdx); // If not the family we expect, search for one if (addrItem->info->ai_family != family) // {vm_covered} { unsigned int addrFindIdx = addrIdx + 1; // {vm_covered} for (; addrFindIdx < lstSize(this->pub.list); addrFindIdx++) // {vm_covered} { AddressInfoItem *const addrFindItem = lstGet(this->pub.list, addrFindIdx); // {vm_covered} if (addrFindItem->info->ai_family == family) // {vm_covered} { // Swap addresses const AddressInfoItem addrCopyItem = *addrItem; // {vm_covered} *addrItem = *addrFindItem; // {vm_covered} *addrFindItem = addrCopyItem; // {vm_covered} // Move on to next address break; // {vm_covered} } } // Address of the required family not found so sorting is done if (addrFindIdx == lstSize(this->pub.list)) // {vm_covered} break; // {vm_covered} } // Switch family family = family == AF_INET6 ? AF_INET : AF_INET6; // {vm_covered} } } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN AddressInfo * addrInfoNew(const String *const host, const unsigned int port) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_END(); ASSERT(host != NULL); ASSERT(port != 0); // Initialize mem context and preference list if (addressInfoLocal.memContext == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(AddressInfo, .childQty = MEM_CONTEXT_QTY_MAX) { addressInfoLocal.memContext = MEM_CONTEXT_NEW(); addressInfoLocal.prefList = lstNewP(sizeof(AddressInfoPreference), .comparator = lstComparatorStr); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); } OBJ_NEW_BEGIN(AddressInfo, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (AddressInfo) { .pub = { .host = strDup(host), .port = port, .list = lstNewP(sizeof(AddressInfoItem), .comparator = lstComparatorStr), }, }; MEM_CONTEXT_TEMP_BEGIN() { // Set hints that narrow the type of address we are looking for -- we'll take ipv4 or ipv6 struct addrinfo hints = (struct addrinfo) { .ai_family = AF_UNSPEC, .ai_flags = AI_PASSIVE | AI_NUMERICSERV, .ai_socktype = SOCK_STREAM, .ai_protocol = IPPROTO_TCP, }; // Convert the port to a zero-terminated string for use with getaddrinfo() char portZ[CVT_BASE10_BUFFER_SIZE]; cvtUIntToZ(port, portZ, sizeof(portZ)); // Do the lookup int error; if ((error = getaddrinfo(strZ(host), portZ, &hints, &this->info)) != 0) THROW_FMT(HostConnectError, "unable to get address for '%s': [%d] %s", strZ(host), error, gai_strerror(error)); // Convert address linked list to list MEM_CONTEXT_OBJ_BEGIN(this->pub.list) { struct addrinfo *info = this->info; lstAdd(this->pub.list, &(AddressInfoItem){.name = addrInfoToStr(info), .info = info}); // Set free callback to ensure address info is freed memContextCallbackSet(objMemContext(this), addrInfoFreeResource, this); while (info->ai_next != NULL) { info = info->ai_next; lstAdd(this->pub.list, &(AddressInfoItem){.name = addrInfoToStr(info), .info = info}); } } MEM_CONTEXT_OBJ_END(); } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(ADDRESS_INFO, this); } /**********************************************************************************************************************************/ FN_EXTERN void addrInfoPrefer(AddressInfo *const this, const unsigned int index) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(ADDRESS_INFO, this); FUNCTION_LOG_PARAM(UINT, index); FUNCTION_LOG_END(); AddressInfoPreference *const addrPref = lstFind(addressInfoLocal.prefList, &this->pub.host); const String *const address = addrInfoGet(this, index)->name; MEM_CONTEXT_OBJ_BEGIN(addressInfoLocal.prefList) { // If no preference set yet if (addrPref == NULL) { lstAdd(addressInfoLocal.prefList, &(AddressInfoPreference){.host = strDup(this->pub.host), .address = strDup(address)}); lstSort(addressInfoLocal.prefList, sortOrderAsc); } // Else update address else { // Free the old address strFree(addrPref->address); // Assign new address addrPref->address = strDup(address); } } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Convert address to a zero-terminated string ***********************************************************************************************************************************/ #define ADDR_INFO_STR_BUFFER_SIZE 48 #define ADDR_INFO_STR_INVALID "invalid" static void addrInfoToZ(const struct addrinfo *const addrInfo, char *const address, const size_t addressSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, addrInfo); FUNCTION_TEST_PARAM_P(CHAR, address); FUNCTION_TEST_PARAM(SIZE, addressSize); FUNCTION_TEST_END(); ASSERT(addrInfo != NULL); ASSERT(address != NULL); ASSERT(addressSize >= sizeof(ADDR_INFO_STR_INVALID)); if (getnameinfo( // {vm_covered} addrInfo->ai_addr, addrInfo->ai_addrlen, address, (socklen_t)addressSize, 0, 0, NI_NUMERICHOST) != 0) { memcpy(address, ADDR_INFO_STR_INVALID, sizeof(ADDR_INFO_STR_INVALID)); // {vm_covered} } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * addrInfoToName(const String *const host, const unsigned int port, const struct addrinfo *const addrInfo) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, host); FUNCTION_TEST_PARAM(UINT, port); FUNCTION_TEST_PARAM_P(VOID, addrInfo); FUNCTION_TEST_END(); String *const result = strCatFmt(strNew(), "%s:%u", strZ(host), port); String *const address = addrInfoToStr(addrInfo); if (!strEq(host, address)) strCatFmt(result, " (%s)", strZ(address)); strFree(address); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * addrInfoToStr(const struct addrinfo *const addrInfo) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, addrInfo); FUNCTION_TEST_END(); char address[ADDR_INFO_STR_BUFFER_SIZE]; addrInfoToZ(addrInfo, address, sizeof(address)); FUNCTION_TEST_RETURN(STRING, strNewZ(address)); } /**********************************************************************************************************************************/ FN_EXTERN void addrInfoToLog(const AddressInfo *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{host: "); strToLog(addrInfoHost(this), debugLog); strStcFmt(debugLog, ", port: %u, list: [", addrInfoPort(this)); for (unsigned int listIdx = 0; listIdx < addrInfoSize(this); listIdx++) { const AddressInfoItem *const addrItem = addrInfoGet(this, listIdx); if (listIdx != 0) strStcCat(debugLog, ", "); strStcCat(debugLog, strZ(addrItem->name)); } strStcCat(debugLog, "]}"); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/socket/address.h0000664000175100017510000001157615265101745021170 0ustar00runnerrunner/*********************************************************************************************************************************** Address Info Get address info for a host/address. If the input is a host then there may be multiple addresses that map to the host. ***********************************************************************************************************************************/ #ifndef COMMON_IO_SOCKET_ADDRESSINFO_H #define COMMON_IO_SOCKET_ADDRESSINFO_H #include /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct AddressInfo AddressInfo; #include "common/type/list.h" #include "common/type/object.h" #include "common/type/string.h" /*********************************************************************************************************************************** Constructor ***********************************************************************************************************************************/ FN_EXTERN AddressInfo *addrInfoNew(const String *const host, unsigned int port); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct AddressInfoItem { String *name; // Address as a string struct addrinfo *info; // Full address info } AddressInfoItem; typedef struct AddressInfoPub { const String *host; // Host for address info lookup unsigned int port; // Port for address info lookup List *list; // List of addresses for the host } AddressInfoPub; // Get address FN_INLINE_ALWAYS const AddressInfoItem * addrInfoGet(const AddressInfo *const this, const unsigned int index) { return (const AddressInfoItem *)lstGet(THIS_PUB(AddressInfo)->list, index); } // Get lookup host FN_INLINE_ALWAYS const String * addrInfoHost(const AddressInfo *const this) { return THIS_PUB(AddressInfo)->host; } // Get lookup port FN_INLINE_ALWAYS unsigned int addrInfoPort(const AddressInfo *const this) { return THIS_PUB(AddressInfo)->port; } // Size of address list FN_INLINE_ALWAYS unsigned int addrInfoSize(const AddressInfo *const this) { return lstSize(THIS_PUB(AddressInfo)->list); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Sort addresses alternating between IPv6 and IPv4 FN_EXTERN void addrInfoSort(AddressInfo *this); // Set preferred address for host, which will be preferred in future lookups FN_EXTERN void addrInfoPrefer(AddressInfo *this, unsigned int index); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void addrInfoFree(AddressInfo *const this) { objFree(this); } /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Convert address info to string FN_EXTERN String *addrInfoToStr(const struct addrinfo *addrInfo); // Generate name for the host/port/address combination (address is omitted if it equals host) FN_EXTERN String *addrInfoToName(const String *host, unsigned int port, const struct addrinfo *addrInfo); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void addrInfoToLog(const AddressInfo *this, StringStatic *debugLog); #define FUNCTION_LOG_ADDRESS_INFO_TYPE \ AddressInfo * #define FUNCTION_LOG_ADDRESS_INFO_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, addrInfoToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/socket/client.c0000664000175100017510000003650215265101745021010 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Client ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/debug.h" #include "common/error/retry.h" #include "common/io/client.h" #include "common/io/fd.h" #include "common/io/socket/address.h" #include "common/io/socket/client.h" #include "common/io/socket/common.h" #include "common/io/socket/session.h" #include "common/log.h" #include "common/stat.h" #include "common/type/object.h" #include "common/wait.h" /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_EXTERN(SOCKET_STAT_CLIENT_STR, SOCKET_STAT_CLIENT); STRING_EXTERN(SOCKET_STAT_RETRY_STR, SOCKET_STAT_RETRY); STRING_EXTERN(SOCKET_STAT_SESSION_STR, SOCKET_STAT_SESSION); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct SocketClient { String *host; // Hostname or IP address unsigned int port; // Port to connect to host on String *name; // Socket name (host:port) TimeMSec timeoutConnect; // Timeout for connection TimeMSec timeoutSession; // Timeout passed to session } SocketClient; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void sckClientToLog(const THIS_VOID, StringStatic *const debugLog) { THIS(const SocketClient); strStcFmt( debugLog, "{host: %s, port: %u, timeoutConnect: %" PRIu64 ", timeoutSession: %" PRIu64 "}", strZ(this->host), this->port, this->timeoutConnect, this->timeoutSession); } #define FUNCTION_LOG_SOCKET_CLIENT_TYPE \ SocketClient * #define FUNCTION_LOG_SOCKET_CLIENT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, sckClientToLog, buffer, bufferSize) /*********************************************************************************************************************************** Connections are established using the "happy eyeballs" approach from RFC 8305, i.e. new addresses (if available) are tried if the prior address has already had a reasonable time to connect. This prevents waiting too long on a failed connection but does not try all the addresses at once. Prior connections that are still waiting are rechecked periodically if no subsequent connection is successful. ***********************************************************************************************************************************/ typedef struct SckClientOpenData { const struct addrinfo *const address; // IP address const char *name; // Combination of host, address, port int fd; // File descriptor for socket int errNo; // Current error (may just indicate to keep waiting) } SckClientOpenData; // Helper to determine if non-blocking connection has been established static bool sckClientOpenWait(SckClientOpenData *const openData, const TimeMSec timeout) { ASSERT(openData != NULL); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRINGZ, openData->name); FUNCTION_LOG_PARAM(INT, openData->fd); FUNCTION_LOG_PARAM(INT, openData->errNo); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(openData->name != NULL); ASSERT(openData->fd > 0); ASSERT(openData->errNo != 0); bool result = true; // The connection has started but since we are in non-blocking mode it has not completed yet if (openData->errNo == EINPROGRESS || openData->errNo == EINTR) // {vm_covered} { // Wait for write-ready if (fdReadyWrite(openData->fd, timeout)) { // Check for success or error. If the connection was successful this will set errNo to 0. socklen_t errNoLen = sizeof(openData->errNo); THROW_ON_SYS_ERROR_FMT( getsockopt(openData->fd, SOL_SOCKET, SO_ERROR, &openData->errNo, &errNoLen) == -1, HostConnectError, "unable to get socket error for '%s'", openData->name); } // Else still waiting on connection else result = false; } // Throw error if it is still set if (result && openData->errNo != 0) THROW_SYS_ERROR_CODE_FMT(openData->errNo, HostConnectError, "unable to connect to '%s'", openData->name); FUNCTION_LOG_RETURN(BOOL, result); } static IoSession * sckClientOpen(THIS_VOID) { THIS(SocketClient); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SOCKET_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); IoSession *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { Wait *const wait = waitNew(this->timeoutConnect); ErrorRetry *const errRetry = errRetryNew(); bool retry; // Get an address list for the host and sort it AddressInfo *const addrInfo = addrInfoNew(this->host, this->port); addrInfoSort(addrInfo); // Try addresses until success or timeout List *const openDataList = lstNewP(sizeof(SckClientOpenData)); volatile unsigned int addrInfoIdx = 0; do { // Assume there will be no retry SckClientOpenData *volatile openData = NULL; const ErrorType *errLastType = NULL; String *errLastMessage = NULL; retry = false; // Try connection TRY_BEGIN() { // Iterate waiting connections and see if any of them have completed for (unsigned int openDataIdx = 0; openDataIdx < lstSize(openDataList); openDataIdx++) { SckClientOpenData *const openDataWait = lstGet(openDataList, openDataIdx); if (openDataWait->fd != 0) { // Set so the connection will be closed on error openData = openDataWait; // Check if the connection has completed without waiting if (sckClientOpenWait(openData, 0)) // {vm_covered} break; // {vm_covered} // Reset to NULL if the connection is still waiting so another connection can be attempted openData = NULL; } } // Try or retry a connection since none of the waiting connections completed if (openData == NULL) // {vm_covered} { // If connection does not exist yet then create it if (addrInfoIdx == lstSize(openDataList)) { openData = lstAdd(openDataList, &(SckClientOpenData){.address = addrInfoGet(addrInfo, addrInfoIdx)->info}); openData->name = strZ(addrInfoToName(this->host, this->port, openData->address)); } // Else search for a connection that is not waiting else { for (; addrInfoIdx < lstSize(openDataList); addrInfoIdx++) { SckClientOpenData *const openDataNoWait = lstGet(openDataList, addrInfoIdx); if (openDataNoWait->fd == 0) { openData = openDataNoWait; break; } } } // Attempt connection to host if (openData != NULL) { openData->fd = socket( openData->address->ai_family, openData->address->ai_socktype, openData->address->ai_protocol); THROW_ON_SYS_ERROR(openData->fd == -1, HostConnectError, "unable to create socket"); sckOptionSet(openData->fd); if (connect(openData->fd, openData->address->ai_addr, openData->address->ai_addrlen) == -1) { // Save the error and wait for connection. The initial wait time will be fixed at 250ms, the default // value recommended by RFC 8305. openData->errNo = errno; sckClientOpenWait(openData, 250); } } } // Connection was successful so open session and set result if (openData != NULL && openData->errNo == 0) { // Create the session MEM_CONTEXT_PRIOR_BEGIN() { result = sckSessionNew(ioSessionRoleClient, openData->fd, this->host, this->port, this->timeoutSession); } MEM_CONTEXT_PRIOR_END(); // Update client name to include address strTrunc(this->name); strCatZ(this->name, openData->name); // Set preferred address addrInfoPrefer(addrInfo, addrInfoIdx); // Clear socket so it will not be freed later openData->fd = 0; } } CATCH_ANY() { // Cast is required to prevent Coverity from complaining about side effects of using volatile in an assertion ASSERT((SckClientOpenData *)openData != NULL); // Close socket close(openData->fd); // Clear socket so the connection can be retried openData->fd = 0; openData->errNo = 0; // Add the error retry info errRetryAddP(errRetry); // Store error info for later logging errLastType = errorType(); errLastMessage = strNewZ(errorMessage()); } TRY_END(); // If the connection was not completed then wait and/or retry if (result == NULL) { // Increment address info index and reset if the end has been reached. When the end has been reached sleep for a bit // to hopefully have better chance at succeeding, otherwise continue right to the next address. addrInfoIdx++; if (addrInfoIdx >= addrInfoSize(addrInfo)) { addrInfoIdx = 0; retry = waitMore(wait); } else retry = true; // General timeout errors when no retries remain if (!retry) { for (unsigned int openDataIdx = 0; openDataIdx < lstSize(openDataList); openDataIdx++) { SckClientOpenData *const openDataTimeout = lstGet(openDataList, openDataIdx); if (openDataTimeout->fd != 0) { errRetryAddP( errRetry, .type = &HostConnectError, .message = strNewFmt("timeout connecting to '%s'", openDataTimeout->name)); } } } // Log retry when there was an error if (errLastType != NULL) { LOG_DEBUG_FMT("retry %s: %s", errorTypeName(errLastType), strZ(errLastMessage)); strFree(errLastMessage); statInc(SOCKET_STAT_RETRY_STR); } } } while (retry); // Free any connections that were in progress but never completed for (unsigned int openDataIdx = 0; openDataIdx < lstSize(openDataList); openDataIdx++) { SckClientOpenData *const openDataFree = lstGet(openDataList, openDataIdx); if (openDataFree->fd != 0) close(openDataFree->fd); } // Error when no result if (result == NULL) THROWP(errRetryType(errRetry), strZ(errRetryMessage(errRetry))); statInc(SOCKET_STAT_SESSION_STR); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(IO_SESSION, result); } /**********************************************************************************************************************************/ static const String * sckClientName(THIS_VOID) { THIS(SocketClient); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SOCKET_CLIENT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN_CONST(STRING, this->name); } /**********************************************************************************************************************************/ static const IoClientInterface sckClientInterface = { .type = IO_CLIENT_SOCKET_TYPE, .name = sckClientName, .open = sckClientOpen, .toLog = sckClientToLog, }; FN_EXTERN IoClient * sckClientNew(const String *const host, const unsigned int port, const TimeMSec timeoutConnect, const TimeMSec timeoutSession) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(TIME_MSEC, timeoutConnect); FUNCTION_LOG_PARAM(TIME_MSEC, timeoutSession); FUNCTION_LOG_END(); ASSERT(host != NULL); OBJ_NEW_BEGIN(SocketClient, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (SocketClient) { .host = strDup(host), .port = port, .name = strCatFmt(strNew(), "%s:%u", strZ(host), port), .timeoutConnect = timeoutConnect, .timeoutSession = timeoutSession, }; } OBJ_NEW_END(); statInc(SOCKET_STAT_CLIENT_STR); FUNCTION_LOG_RETURN(IO_CLIENT, ioClientNew(this, &sckClientInterface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/socket/client.h0000664000175100017510000000376015265101745021015 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Client A simple socket client intended to allow access to services that are exposed via a socket. ***********************************************************************************************************************************/ #ifndef COMMON_IO_SOCKET_CLIENT_H #define COMMON_IO_SOCKET_CLIENT_H #include "common/io/client.h" #include "common/time.h" /*********************************************************************************************************************************** Io client type ***********************************************************************************************************************************/ #define IO_CLIENT_SOCKET_TYPE STRID5("socket", 0x28558df30) /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ #define SOCKET_STAT_CLIENT "socket.client" // Clients created STRING_DECLARE(SOCKET_STAT_CLIENT_STR); #define SOCKET_STAT_RETRY "socket.retry" // Connection retries STRING_DECLARE(SOCKET_STAT_RETRY_STR); #define SOCKET_STAT_SESSION "socket.session" // Sessions created STRING_DECLARE(SOCKET_STAT_SESSION_STR); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoClient *sckClientNew(const String *host, unsigned int port, TimeMSec timeoutConnect, TimeMSec timeoutSession); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/socket/common.c0000664000175100017510000001234215265101745021016 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Common Functions ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/debug.h" #include "common/io/socket/address.h" #include "common/io/socket/common.h" #include "common/log.h" #include "common/wait.h" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct SocketLocal { bool init; // sckInit() has been called bool block; // Use blocking mode socket bool keepAlive; // Is socket keep-alive enabled? int tcpKeepAliveCount; // TCP keep alive count (0 disables) int tcpKeepAliveIdle; // TCP keep alive idle (0 disables) int tcpKeepAliveInterval; // TCP keep alive interval (0 disables) } socketLocal; /**********************************************************************************************************************************/ FN_EXTERN void sckInit( const bool block, const bool keepAlive, const int tcpKeepAliveCount, const int tcpKeepAliveIdle, const int tcpKeepAliveInterval) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_PARAM(BOOL, keepAlive); FUNCTION_LOG_PARAM(INT, tcpKeepAliveCount); FUNCTION_LOG_PARAM(INT, tcpKeepAliveIdle); FUNCTION_LOG_PARAM(INT, tcpKeepAliveInterval); FUNCTION_LOG_END(); ASSERT(tcpKeepAliveCount >= 0); ASSERT(tcpKeepAliveIdle >= 0); ASSERT(tcpKeepAliveInterval >= 0); socketLocal.init = true; socketLocal.block = block; socketLocal.keepAlive = keepAlive; socketLocal.tcpKeepAliveCount = tcpKeepAliveCount; socketLocal.tcpKeepAliveIdle = tcpKeepAliveIdle; socketLocal.tcpKeepAliveInterval = tcpKeepAliveInterval; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void sckOptionSet(const int fd) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, fd); FUNCTION_TEST_END(); ASSERT(socketLocal.init); // Disable the Nagle algorithm. This means that segments are always sent as soon as possible, even if there is only a small // amount of data. Our internal buffering minimizes the benefit of this optimization so lower latency is preferred. #ifdef TCP_NODELAY int socketValue = 1; THROW_ON_SYS_ERROR( setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &socketValue, sizeof(int)) == -1, ProtocolError, "unable to set TCP_NODELAY"); #endif // Put the socket in non-blocking mode if (!socketLocal.block) { int flags; THROW_ON_SYS_ERROR((flags = fcntl(fd, F_GETFL)) == -1, ProtocolError, "unable to get flags"); THROW_ON_SYS_ERROR(fcntl(fd, F_SETFL, flags | O_NONBLOCK) == -1, ProtocolError, "unable to set O_NONBLOCK"); } // Automatically close the socket (in the child process) on a successful execve() call. Connections are never shared between // processes so there is no reason to leave them open. #ifdef F_SETFD THROW_ON_SYS_ERROR(fcntl(fd, F_SETFD, FD_CLOEXEC) == -1, ProtocolError, "unable to set FD_CLOEXEC"); #endif // Enable TCP keepalives if (socketLocal.keepAlive) { int socketValue = 1; THROW_ON_SYS_ERROR( setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &socketValue, sizeof(int)) == -1, ProtocolError, "unable to set SO_KEEPALIVE"); // Set TCP_KEEPCNT when available #ifdef TCP_KEEPIDLE if (socketLocal.tcpKeepAliveCount > 0) { socketValue = socketLocal.tcpKeepAliveCount; THROW_ON_SYS_ERROR( setsockopt(fd, IPPROTO_TCP, TCP_KEEPCNT, &socketValue, sizeof(int)) == -1, ProtocolError, "unable to set TCP_KEEPCNT"); } #endif // Set TCP_KEEPIDLE when available #ifdef TCP_KEEPIDLE if (socketLocal.tcpKeepAliveIdle > 0) { socketValue = socketLocal.tcpKeepAliveIdle; THROW_ON_SYS_ERROR( setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE, &socketValue, sizeof(int)) == -1, ProtocolError, "unable to set SO_KEEPIDLE"); } #endif // Set TCP_KEEPINTVL when available #ifdef TCP_KEEPIDLE if (socketLocal.tcpKeepAliveInterval > 0) { socketValue = socketLocal.tcpKeepAliveInterval; THROW_ON_SYS_ERROR( setsockopt(fd, IPPROTO_TCP, TCP_KEEPINTVL, &socketValue, sizeof(int)) == -1, ProtocolError, "unable to set SO_KEEPINTVL"); } #endif } FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/socket/common.h0000664000175100017510000000173015265101745021022 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Common Functions ***********************************************************************************************************************************/ #ifndef COMMON_IO_SOCKET_COMMON_H #define COMMON_IO_SOCKET_COMMON_H #include #include "common/time.h" #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize settings for socket connections (some are used only for TCP) FN_EXTERN void sckInit(bool block, bool keepAlive, int tcpKeepAliveCount, int tcpKeepAliveIdle, int tcpKeepAliveInterval); // Set options on a socket FN_EXTERN void sckOptionSet(int fd); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/socket/server.c0000664000175100017510000002056415265101745021041 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Server ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/debug.h" #include "common/io/server.h" #include "common/io/socket/address.h" #include "common/io/socket/common.h" #include "common/io/socket/server.h" #include "common/io/socket/session.h" #include "common/log.h" #include "common/memContext.h" #include "common/stat.h" #include "common/type/object.h" #include "common/wait.h" /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_EXTERN(SOCKET_STAT_SERVER_STR, SOCKET_STAT_SERVER); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct SocketServer { MemContext *memContext; // Mem context String *address; // Address to listen on unsigned int port; // Port to listen on int socket; // Socket used to listen for new connections String *name; // Socket name (address:port) TimeMSec timeout; // Timeout for any i/o operation (connect, read, etc.) } SocketServer; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void sckServerToLog(const THIS_VOID, StringStatic *const debugLog) { THIS(const SocketServer); strStcFmt(debugLog, "{address: %s, port: %u, timeout: %" PRIu64 "}", strZ(this->address), this->port, this->timeout); } #define FUNCTION_LOG_SOCKET_SERVER_TYPE \ SocketServer * #define FUNCTION_LOG_SOCKET_SERVER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, sckServerToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free connection ***********************************************************************************************************************************/ static void sckServerFreeResource(THIS_VOID) { THIS(SocketServer); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SOCKET_SERVER, this); FUNCTION_LOG_END(); ASSERT(this != NULL); close(this->socket); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static IoSession * sckServerAccept(THIS_VOID, IoSession *const session) { THIS(SocketServer); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SOCKET_SERVER, this); (void)session; // Not used by this server FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(session == NULL); IoSession *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Accept the socket connection struct sockaddr_in addr; unsigned int len = sizeof(addr); int serverSocket = accept(this->socket, (struct sockaddr *)&addr, &len); if (serverSocket != -1) { // Create socket session sckOptionSet(serverSocket); MEM_CONTEXT_PRIOR_BEGIN() { result = sckSessionNew(ioSessionRoleServer, serverSocket, this->address, this->port, this->timeout); } MEM_CONTEXT_PRIOR_END(); statInc(SOCKET_STAT_SESSION_STR); } else THROW_ON_SYS_ERROR(errno != EINTR, FileOpenError, "unable to accept socket"); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(IO_SESSION, result); } /**********************************************************************************************************************************/ static const String * sckServerName(THIS_VOID) // {vm_covered} { THIS(SocketServer); // {vm_covered} FUNCTION_TEST_BEGIN(); // {vm_covered} FUNCTION_TEST_PARAM(SOCKET_SERVER, this); // {vm_covered} FUNCTION_TEST_END(); // {vm_covered} ASSERT(this != NULL); // {vm_covered} FUNCTION_TEST_RETURN_CONST(STRING, this->name); // {vm_covered} } /**********************************************************************************************************************************/ static const IoServerInterface sckServerInterface = { .type = IO_SERVER_SOCKET_TYPE, .name = sckServerName, .accept = sckServerAccept, .toLog = sckServerToLog, }; FN_EXTERN IoServer * sckServerNew(const String *const address, const unsigned int port, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, address); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(address != NULL); ASSERT(port > 0); OBJ_NEW_BEGIN(SocketServer, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (SocketServer) { .address = strDup(address), .port = port, .name = strCatFmt(strNew(), "%s:%u", strZ(address), port), .timeout = timeout, }; // Lookup address const struct addrinfo *const addressFound = addrInfoGet(addrInfoNew(this->address, this->port), 0)->info; // Create socket THROW_ON_SYS_ERROR( (this->socket = socket(addressFound->ai_family, SOCK_STREAM, 0)) == -1, FileOpenError, "unable to create socket"); // Set the address as reusable so we can bind again quickly after a restart or crash int reuseAddr = 1; THROW_ON_SYS_ERROR( setsockopt(this->socket, SOL_SOCKET, SO_REUSEADDR, &reuseAddr, sizeof(reuseAddr)) == -1, ProtocolError, "unable to set SO_REUSEADDR"); // Ensure file descriptor is closed memContextCallbackSet(objMemContext(this), sckServerFreeResource, this); // Update server name to include address strTrunc(this->name); strCat(this->name, addrInfoToName(this->address, this->port, addressFound)); // Bind the address Wait *const wait = waitNew(timeout); int result; do { result = bind(this->socket, addressFound->ai_addr, addressFound->ai_addrlen); if (result == -1 && !waitMore(wait)) THROW_SYS_ERROR(FileOpenError, "unable to bind socket"); } while (result == -1); // Listen for client connections. It might be a good idea to make the backlog configurable but this value seems OK for now. THROW_ON_SYS_ERROR(listen(this->socket, 100) == -1, FileOpenError, "unable to listen on socket"); } OBJ_NEW_END(); statInc(SOCKET_STAT_SERVER_STR); FUNCTION_LOG_RETURN(IO_SERVER, ioServerNew(this, &sckServerInterface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/socket/server.h0000664000175100017510000000325515265101745021044 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Server A simple socket server intended to expose services by accepting a socket. ***********************************************************************************************************************************/ #ifndef COMMON_IO_SOCKET_SERVER_H #define COMMON_IO_SOCKET_SERVER_H #include "common/io/server.h" #include "common/io/socket/client.h" #include "common/time.h" /*********************************************************************************************************************************** Io server type ***********************************************************************************************************************************/ #define IO_SERVER_SOCKET_TYPE IO_CLIENT_SOCKET_TYPE /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ #define SOCKET_STAT_SERVER "socket.server" // Servers created STRING_DECLARE(SOCKET_STAT_SERVER_STR); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoServer *sckServerNew(const String *address, unsigned int port, TimeMSec timeout); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/socket/session.c0000664000175100017510000001550615265101745021216 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Session ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/fdRead.h" #include "common/io/fdWrite.h" #include "common/io/session.h" #include "common/io/socket/client.h" #include "common/io/socket/session.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct SocketSession { IoSessionRole role; // Role (server or client) int fd; // File descriptor String *host; // Hostname or IP address unsigned int port; // Port to connect to host on TimeMSec timeout; // Timeout for any i/o operation (read, write, etc.) IoRead *read; // IoRead interface to the file descriptor IoWrite *write; // IoWrite interface to the file descriptor } SocketSession; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void sckSessionToLog(const THIS_VOID, StringStatic *const debugLog) { THIS(const SocketSession); strStcFmt( debugLog, "{host: %s, port: %u, fd: %d, timeout: %" PRIu64 "}", strZ(this->host), this->port, this->fd, this->timeout); } #define FUNCTION_LOG_SOCKET_SESSION_TYPE \ SocketSession * #define FUNCTION_LOG_SOCKET_SESSION_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, sckSessionToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free connection ***********************************************************************************************************************************/ static void sckSessionFreeResource(THIS_VOID) { THIS(SocketSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SOCKET_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); close(this->fd); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static void sckSessionClose(THIS_VOID) { THIS(SocketSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SOCKET_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // If not already closed if (this->fd != -1) { // Clear the callback to close the socket memContextCallbackClear(objMemContext(this)); // Close the socket close(this->fd); this->fd = -1; } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static int sckSessionFd(THIS_VOID) { THIS(SocketSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SOCKET_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->fd); } /**********************************************************************************************************************************/ static IoRead * sckSessionIoRead(THIS_VOID, const bool ignoreUnexpectedEof) { THIS(SocketSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SOCKET_SESSION, this); (void)ignoreUnexpectedEof; // Unused FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(IO_READ, this->read); } /**********************************************************************************************************************************/ static IoWrite * sckSessionIoWrite(THIS_VOID) { THIS(SocketSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SOCKET_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(IO_WRITE, this->write); } /**********************************************************************************************************************************/ static IoSessionRole sckSessionRole(const THIS_VOID) { THIS(const SocketSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SOCKET_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN_TYPE(IoSessionRole, this->role); } /**********************************************************************************************************************************/ static const IoSessionInterface sckSessionInterface = { .type = IO_CLIENT_SOCKET_TYPE, .close = sckSessionClose, .fd = sckSessionFd, .ioRead = sckSessionIoRead, .ioWrite = sckSessionIoWrite, .role = sckSessionRole, .toLog = sckSessionToLog, }; FN_EXTERN IoSession * sckSessionNew(const IoSessionRole role, const int fd, const String *const host, const unsigned int port, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, role); FUNCTION_LOG_PARAM(INT, fd); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(fd != -1); ASSERT(host != NULL); OBJ_NEW_BEGIN(SocketSession, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { String *const name = strNewFmt("%s:%u", strZ(host), port); *this = (SocketSession) { .role = role, .fd = fd, .host = strDup(host), .port = port, .timeout = timeout, .read = ioFdReadNewOpen(name, fd, timeout), .write = ioFdWriteNewOpen(name, fd, timeout), }; strFree(name); // Ensure file descriptor is closed memContextCallbackSet(objMemContext(this), sckSessionFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_SESSION, ioSessionNew(this, &sckSessionInterface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/socket/session.h0000664000175100017510000000161615265101745021220 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Session A simple socket session intended to allow access to services that are exposed via a socket. ***********************************************************************************************************************************/ #ifndef COMMON_IO_SOCKET_SESSION_H #define COMMON_IO_SOCKET_SESSION_H #include "common/io/session.h" #include "common/time.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoSession *sckSessionNew(IoSessionRole role, int fd, const String *host, unsigned int port, TimeMSec timeout); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/tls/0000775000175100017510000000000015265101745016672 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/tls/client.c0000664000175100017510000005267115265101745020327 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Client ***********************************************************************************************************************************/ #include // {uncrustify_off - header order required for FreeBSD} #include #include #include #include // {uncrustify_on} #include #include "common/crypto/common.h" #include "common/debug.h" #include "common/io/client.h" #include "common/io/io.h" #include "common/io/tls/client.h" #include "common/io/tls/common.h" #include "common/io/tls/session.h" #include "common/log.h" #include "common/stat.h" #include "common/type/object.h" #include "common/wait.h" /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_EXTERN(TLS_STAT_CLIENT_STR, TLS_STAT_CLIENT); STRING_EXTERN(TLS_STAT_RETRY_STR, TLS_STAT_RETRY); STRING_EXTERN(TLS_STAT_SESSION_STR, TLS_STAT_SESSION); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct TlsClient { const String *host; // Host to use for peer verification TimeMSec timeoutConnect; // Timeout for connection TimeMSec timeoutSession; // Timeout passed to session bool verifyPeer; // Should the peer (server) certificate be verified? IoClient *ioClient; // Underlying client (usually a SocketClient) SSL_CTX *context; // TLS context } TlsClient; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void tlsClientToLog(const THIS_VOID, StringStatic *const debugLog) { THIS(const TlsClient); strStcCat(debugLog, "{ioClient: "); ioClientToLog(this->ioClient, debugLog); strStcFmt( debugLog, ", timeoutConnect: %" PRIu64 ", timeoutSession: %" PRIu64 ", verifyPeer: %s}", this->timeoutConnect, this->timeoutSession, cvtBoolToConstZ(this->verifyPeer)); } #define FUNCTION_LOG_TLS_CLIENT_TYPE \ TlsClient * #define FUNCTION_LOG_TLS_CLIENT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, tlsClientToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free connection ***********************************************************************************************************************************/ static void tlsClientFreeResource(THIS_VOID) { THIS(TlsClient); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); SSL_CTX_free(this->context); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Check if a name from the server certificate matches the hostname Matching is always case-insensitive since DNS is case insensitive. ***********************************************************************************************************************************/ static bool tlsClientHostVerifyName(const String *const host, const String *const name) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_END(); ASSERT(host != NULL); ASSERT(name != NULL); // Check for NULLs in the name tlsCertNameVerify(name); bool result = false; // Try an exact match if (strcasecmp(strZ(name), strZ(host)) == 0) // {vm_covered} { result = true; // {vm_covered} } // Else check if a wildcard certificate matches the host name // // The rules are: // 1. Only match the '*' character as wildcard // 2. Only match wildcards at the start of the string // 3. The '*' character does *not* match '.', meaning that we match only a single pathname component. // 4. Don't support more than one '*' in a single pattern. // // This is roughly in line with RFC2818, but contrary to what most browsers appear to be implementing (point 3 being the // difference) else if (strZ(name)[0] == '*' && strZ(name)[1] == '.' && strSize(name) > 2 && // {vm_covered} strSize(name) < strSize(host) && // {vm_covered} strcasecmp(strZ(name) + 1, strZ(host) + strSize(host) - strSize(name) + 1) == 0 && // {vm_covered} strChr(host, '.') >= (int)(strSize(host) - strSize(name))) // {vm_covered} { result = true; // {vm_covered} } FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Check if an IP address from the server certificate matches the hostname Adapted from libpq/fe-secure-common.c. ***********************************************************************************************************************************/ static bool tlsClientHostVerifyIPAddr(const String *const host, const Buffer *const address) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(BUFFER, address); FUNCTION_LOG_END(); ASSERT(host != NULL); ASSERT(address != NULL); bool result = false; // The data from the certificate is in network byte order. Convert our host string to network-ordered bytes as well for // comparison. The host string is not guaranteed to be an IP address so if this conversion fails consider it a mismatch rather // than an error. Use the size of the address parameter to determine ipv4 or ipv6 checking. const size_t addrSize = bufSize(address); // IPv4 if (addrSize == sizeof(struct in_addr)) { struct in_addr addr; if (inet_pton(AF_INET, strZ(host), &addr) == 1 && // {vm_covered} memcmp(bufPtrConst(address), &addr.s_addr, addrSize) == 0) // {vm_covered} { result = true; // {vm_covered} } } // Else IPv6 else if (addrSize == sizeof(struct in6_addr)) { struct in6_addr addr; if (inet_pton(AF_INET6, strZ(host), &addr) == 1 && // {vm_covered} memcmp(bufPtrConst(address), &addr.s6_addr, addrSize) == 0) // {vm_covered} { result = true; // {vm_covered} } } FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Verify that the server certificate matches the hostname we connected to The certificate's Common Name and Subject Alternative Names are considered. ***********************************************************************************************************************************/ static bool tlsClientHostVerify(const String *const host, X509 *const certificate) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM_P(VOID, certificate); FUNCTION_LOG_END(); ASSERT(host != NULL); bool result = false; // Error if the certificate is NULL if (certificate == NULL) // {vm_covered} THROW(CryptoError, "No certificate presented by the TLS server"); // {vm_covered} MEM_CONTEXT_TEMP_BEGIN() // {vm_covered} { // First get the subject alternative names from the certificate and compare them against the hostname STACK_OF(GENERAL_NAME) *const altNameStack = (STACK_OF(GENERAL_NAME) *)X509_get_ext_d2i( // {vm_covered} certificate, NID_subject_alt_name, NULL, NULL); // {vm_covered} bool altNameFound = false; // {vm_covered} if (altNameStack) // {vm_covered} { for (int altNameIdx = 0; altNameIdx < sk_GENERAL_NAME_num(altNameStack); altNameIdx++) // {vm_covered} { const GENERAL_NAME *const name = sk_GENERAL_NAME_value(altNameStack, altNameIdx); // {vm_covered} altNameFound = true; // {vm_covered} if (name->type == GEN_DNS) // {vm_covered} result = tlsClientHostVerifyName(host, tlsAsn1ToStr(name->d.dNSName)); // {vm_covered} else if (name->type == GEN_IPADD) // {vm_covered} result = tlsClientHostVerifyIPAddr(host, tlsAsn1ToBuf(name->d.dNSName)); // {vm_covered} if (result != false) // {vm_covered} break; // {vm_covered} } sk_GENERAL_NAME_pop_free(altNameStack, GENERAL_NAME_free); // {vm_covered} } // If no subject alternative name was found then check the common name. Per RFC 2818 and RFC 6125, if the subjectAltName // extension of type dNSName is present the CN must be ignored. if (!altNameFound) // {vm_covered} result = tlsClientHostVerifyName(host, tlsCertCommonName(certificate)); // {vm_covered} } MEM_CONTEXT_TEMP_END(); // {vm_covered} FUNCTION_LOG_RETURN(BOOL, result); // {vm_covered} } /*********************************************************************************************************************************** Authenticate server Adapted from PostgreSQL open_client_SSL() in src/interfaces/libpq/fe-secure-openssl.c. ***********************************************************************************************************************************/ static bool tlsClientAuth(const TlsClient *const this, SSL *const tlsSession) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_CLIENT, this); FUNCTION_LOG_PARAM_P(VOID, tlsSession); FUNCTION_LOG_END(); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { // Verify that the certificate presented by the server is valid if (this->verifyPeer) // {vm_covered} { // Verify that the chain of trust leads to a valid CA const long int verifyResult = SSL_get_verify_result(tlsSession); // {vm_covered} if (verifyResult != X509_V_OK) // {vm_covered} { THROW_FMT( // {vm_covered} CryptoError, "unable to verify certificate presented by '%s': [%ld] %s", // {vm_covered} strZ(ioClientName(this->ioClient)), verifyResult, // {vm_covered} X509_verify_cert_error_string(verifyResult)); // {vm_covered} } // Verify that the hostname appears in the certificate X509 *const certificate = SSL_get_peer_certificate(tlsSession); // {vm_covered} const bool nameResult = tlsClientHostVerify(this->host, certificate); // {vm_covered} X509_free(certificate); // {vm_covered} if (!nameResult) // {vm_covered} { THROW_FMT( // {vm_covered} CryptoError, // {vm_covered} "unable to find hostname '%s' in certificate common name or subject alternative names", // {vm_covered} strZ(this->host)); // {vm_covered} } } result = true; } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Open TLS session on a socket ***********************************************************************************************************************************/ static IoSession * tlsClientOpen(THIS_VOID) { THIS(TlsClient); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); IoSession *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { bool retry; Wait *const wait = waitNew(this->timeoutConnect); SSL *tlsSession = NULL; do { // Assume there will be no retry retry = false; // Create the TLS session tlsSession = SSL_new(this->context); cryptoError(tlsSession == NULL, "unable to create TLS session"); // Set server host name used for validation. The exception here is necessary for MacOS which for some reason defines the // host name parameter as void * rather than const char * as on most platforms. #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wcast-qual" cryptoError(SSL_set_tlsext_host_name(tlsSession, strZ(this->host)) != 1, "unable to set TLS host name"); #pragma GCC diagnostic pop // Open the underlying session first since this is mostly likely to fail. This is done outside exception handling to // avoid multiplying retries. IoSession *ioSession = NULL; TRY_BEGIN() { ioSession = ioClientOpen(this->ioClient); } CATCH_ANY() { SSL_free(tlsSession); RETHROW(); } TRY_END(); // Open TLS session TRY_BEGIN() { result = tlsSessionNew(tlsSession, ioSession, this->timeoutSession); } CATCH_ANY() { result = NULL; // Retry if wait time has not expired if (waitMore(wait)) { LOG_DEBUG_FMT("retry %s: %s", errorTypeName(errorType()), errorMessage()); retry = true; statInc(TLS_STAT_RETRY_STR); } else RETHROW(); } TRY_END(); } while (retry); // Authenticate TLS session ASSERT(result != NULL); ioSessionAuthenticatedSet(result, tlsClientAuth(this, tlsSession)); // Move session ioSessionMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); statInc(TLS_STAT_SESSION_STR); FUNCTION_LOG_RETURN(IO_SESSION, result); } /**********************************************************************************************************************************/ static const String * tlsClientName(THIS_VOID) { THIS(TlsClient); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TLS_CLIENT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN_CONST(STRING, ioClientName(this->ioClient)); } /*********************************************************************************************************************************** Initialize TLS session with all required security features Adapted from PostgreSQL initialize_SSL() in src/interfaces/libpq/fe-secure-openssl.c. ***********************************************************************************************************************************/ static const IoClientInterface tlsClientInterface = { .type = IO_CLIENT_TLS_TYPE, .name = tlsClientName, .open = tlsClientOpen, .toLog = tlsClientToLog, }; FN_EXTERN IoClient * tlsClientNew( IoClient *const ioClient, const String *const host, const TimeMSec timeoutConnect, const TimeMSec timeoutSession, const bool verifyPeer, const TlsClientNewParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_CLIENT, ioClient); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(TIME_MSEC, timeoutConnect); FUNCTION_LOG_PARAM(TIME_MSEC, timeoutSession); FUNCTION_LOG_PARAM(BOOL, verifyPeer); FUNCTION_LOG_PARAM(STRING, param.caFile); FUNCTION_LOG_PARAM(STRING, param.caPath); FUNCTION_LOG_PARAM(STRING, param.certFile); FUNCTION_LOG_PARAM(STRING, param.keyFile); FUNCTION_LOG_END(); ASSERT(ioClient != NULL); OBJ_NEW_BEGIN(TlsClient, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (TlsClient) { .ioClient = ioClientMove(ioClient, objMemContext(this)), .host = strDup(host), .timeoutConnect = timeoutConnect, .timeoutSession = timeoutSession, .verifyPeer = verifyPeer, .context = tlsContext(verifyPeer), }; // Set callback to free context memContextCallbackSet(objMemContext(this), tlsClientFreeResource, this); // Enable safe compatibility options SSL_CTX_set_options(this->context, SSL_OP_ALL); // Set location of CA certificates if the server certificate will be verified if (this->verifyPeer) { // If the user specified a location if (param.caFile != NULL || param.caPath != NULL) // {vm_covered} { cryptoError( // {vm_covered} SSL_CTX_load_verify_locations( // {vm_covered} this->context, strZNull(param.caFile), strZNull(param.caPath)) != 1, // {vm_covered} "unable to set user-defined CA certificate location"); // {vm_covered} } // Else use the defaults else { cryptoError( SSL_CTX_set_default_verify_paths(this->context) != 1, "unable to set default CA certificate location"); } } // Load certificate and key, if specified tlsCertKeyLoad(this->context, param.certFile, param.keyFile); } OBJ_NEW_END(); statInc(TLS_STAT_CLIENT_STR); FUNCTION_LOG_RETURN(IO_CLIENT, ioClientNew(this, &tlsClientInterface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/tls/client.h0000664000175100017510000000577615265101745020340 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Client A simple, secure TLS client intended to allow access to services that are exposed via HTTPS. We call it TLS instead of SSL because SSL methods are disabled so only TLS connections are allowed. This object is intended to be used for multiple TLS sessions so ioClientOpen() can be called each time a new session is needed. ***********************************************************************************************************************************/ #ifndef COMMON_IO_TLS_CLIENT_H #define COMMON_IO_TLS_CLIENT_H #include "common/io/client.h" #include "common/time.h" /*********************************************************************************************************************************** Io client type ***********************************************************************************************************************************/ #define IO_CLIENT_TLS_TYPE STRID5("tls", 0x4d940) /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ #define TLS_STAT_CLIENT "tls.client" // Clients created STRING_DECLARE(TLS_STAT_CLIENT_STR); #define TLS_STAT_RETRY "tls.retry" // Connection retries STRING_DECLARE(TLS_STAT_RETRY_STR); #define TLS_STAT_SESSION "tls.session" // Sessions created STRING_DECLARE(TLS_STAT_SESSION_STR); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct TlsClientNewParam { VAR_PARAM_HEADER; const String *caFile; const String *caPath; const String *certFile; const String *keyFile; } TlsClientNewParam; #define tlsClientNewP(ioClient, host, timeoutConnect, timeoutSession, verifyPeer, ...) \ tlsClientNew(ioClient, host, timeoutConnect, timeoutSession, verifyPeer, (TlsClientNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN IoClient *tlsClientNew( IoClient *ioClient, const String *host, TimeMSec timeoutConnect, TimeMSec timeoutSession, bool verifyPeer, TlsClientNewParam param); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Statistics as a formatted string String *tlsClientStatStr(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/tls/common.c0000664000175100017510000003147115265101745020334 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Common ***********************************************************************************************************************************/ #include #include #include #include #include "common/crypto/common.h" #include "common/debug.h" #include "common/io/tls/common.h" #include "common/user.h" #include "storage/posix/storage.h" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct TlsCommonLocal { MemContext *memContext; // Mem context for TLS common String *tlsCipher12; // Allowed ciphers for TLSv1.2 String *tlsCipher13; // Allowed ciphers for TLSv1.3 } tlsCommonLocal; /**********************************************************************************************************************************/ FN_EXTERN Buffer * tlsAsn1ToBuf(const ASN1_STRING *const nameAsn1) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, nameAsn1); FUNCTION_TEST_END(); // The name should not be null if (nameAsn1 == NULL) // {vm_covered} THROW(CryptoError, "TLS certificate name entry is missing"); FUNCTION_TEST_RETURN( // {vm_covered} BUFFER, bufNewC( (const char *)ASN1_STRING_get0_data(nameAsn1), (size_t)ASN1_STRING_length(nameAsn1))); } /**********************************************************************************************************************************/ FN_EXTERN String * tlsAsn1ToStr(const ASN1_STRING *const nameAsn1) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, nameAsn1); FUNCTION_TEST_END(); // The name should not be null if (nameAsn1 == NULL) // {vm_covered} THROW(CryptoError, "TLS certificate name entry is missing"); FUNCTION_TEST_RETURN( // {vm_covered} STRING, strNewZN( (const char *)ASN1_STRING_get0_data(nameAsn1), (size_t)ASN1_STRING_length(nameAsn1))); } /**********************************************************************************************************************************/ FN_EXTERN void tlsCertNameVerify(const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); // Check for embedded NULLs if (strlen(strZ(name)) != strSize(name)) THROW(CryptoError, "TLS certificate name contains embedded null"); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * tlsCertCommonName(X509 *const certificate) // {vm_covered} { FUNCTION_TEST_BEGIN(); // {vm_covered} FUNCTION_TEST_PARAM_P(VOID, certificate); // {vm_covered} FUNCTION_TEST_END(); // {vm_covered} const X509_NAME *const subjectName = X509_get_subject_name(certificate); // {vm_covered} CHECK(FormatError, subjectName != NULL, "subject name is missing"); // {vm_covered} // OpenSSL < 3 requires a non-const name #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wcast-qual" const int commonNameIndex = X509_NAME_get_index_by_NID( // {vm_covered} UNCONSTIFY(X509_NAME *, subjectName), NID_commonName, -1); // {vm_covered} #pragma GCC diagnostic pop CHECK(FormatError, commonNameIndex >= 0, "common name is missing"); // {vm_covered} String *result = tlsAsn1ToStr(X509_NAME_ENTRY_get_data(X509_NAME_get_entry(subjectName, commonNameIndex))); // {vm_covered} // Check for NULLs in the name tlsCertNameVerify(result); // {vm_covered} FUNCTION_TEST_RETURN(STRING, result); // {vm_covered} } /**********************************************************************************************************************************/ // Callback to process cert passwords static int tlsCertPwd(char *const buffer, const int size, const int rwFlag, void *const userData) { (void)rwFlag; (void)userData; CHECK(ServiceError, size > 0, "buffer has zero size"); // No password is currently supplied buffer[0] = '\0'; return 0; } FN_EXTERN void tlsCertKeyLoad(SSL_CTX *const context, const String *const certFile, const String *const keyFile) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, context); FUNCTION_TEST_PARAM(STRING, certFile); FUNCTION_TEST_PARAM(STRING, keyFile); FUNCTION_TEST_END(); ASSERT(context != NULL); ASSERT((certFile == NULL && keyFile == NULL) || (certFile != NULL && keyFile != NULL)); if (certFile != NULL) { userInit(); MEM_CONTEXT_TEMP_BEGIN() { // Set cert password callback SSL_CTX_set_default_passwd_cb(context, tlsCertPwd); // Load certificate cryptoError( SSL_CTX_use_certificate_chain_file(context, strZ(certFile)) != 1, zNewFmt("unable to load cert file '%s'", strZ(certFile))); // Check that key has the correct permissions const StorageInfo keyInfo = storageInfoP( storagePosixNewP(FSLASH_STR), keyFile, .ignoreMissing = true, .followLink = true); if (keyInfo.exists) { if (keyInfo.userId != userId() && keyInfo.userId != 0) // {vm_covered} { THROW_FMT( // {vm_covered} FileReadError, "key file '%s' must be owned by the '%s' user or root", strZ(keyFile), strZ(userName())); } if ((keyInfo.userId == userId() && keyInfo.mode & (S_IRWXG | S_IRWXO)) || // {vm_covered} (keyInfo.userId == 0 && keyInfo.mode & (S_IWGRP | S_IXGRP | S_IRWXO))) // {vm_covered} { THROW_FMT( FileReadError, "key file '%s' has group or other permissions\n" "HINT: file must have permissions u=rw (0600) or less if owned by the '%s' user\n" "HINT: file must have permissions u=rw, g=r (0640) or less if owned by root\n", strZ(keyFile), strZ(userName())); } } // Load key and verify that the key and cert go together cryptoError( SSL_CTX_use_PrivateKey_file(context, strZ(keyFile), SSL_FILETYPE_PEM) != 1, zNewFmt("unable to load key file '%s'", strZ(keyFile))); // Verify again that the cert and key go together. It is not clear why this is needed since the key has already been // verified in SSL_CTX_use_PrivateKey_file(), but it may be that older versions of OpenSSL need it. cryptoError( SSL_CTX_check_private_key(context) != 1, zNewFmt("cert '%s' and key '%s' do not match", strZ(certFile), strZ(keyFile))); } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void tlsInit(const String *const tlsCipher12, const String *const tlsCipher13) // {vm_covered} { FUNCTION_TEST_BEGIN(); // {vm_covered} FUNCTION_TEST_PARAM(STRING, tlsCipher12); // {vm_covered} FUNCTION_TEST_PARAM(STRING, tlsCipher13); // {vm_covered} FUNCTION_TEST_END(); // {vm_covered} // Free old mem context (if any) if (tlsCommonLocal.memContext != NULL) // {vm_covered} memContextFree(tlsCommonLocal.memContext); // {vm_covered} // Initialize mem context MEM_CONTEXT_BEGIN(memContextTop()) // {vm_covered} { MEM_CONTEXT_NEW_BEGIN(TlsCommon, .childQty = MEM_CONTEXT_QTY_MAX) // {vm_covered} { tlsCommonLocal.memContext = MEM_CONTEXT_NEW(); // {vm_covered} tlsCommonLocal.tlsCipher12 = strDup(tlsCipher12); // {vm_covered} tlsCommonLocal.tlsCipher13 = strDup(tlsCipher13); // {vm_covered} } MEM_CONTEXT_NEW_END(); // {vm_covered} } MEM_CONTEXT_END(); // {vm_covered} FUNCTION_TEST_RETURN_VOID(); // {vm_covered} } /**********************************************************************************************************************************/ FN_EXTERN SSL_CTX * tlsContext(const bool verifyPeer) { FUNCTION_TEST_VOID(); cryptoInit(); // Select the TLS method to use. This negotiates the highest version available to both sides, but SSL versions will be excluded // in SSL_CTX_set_options(). const SSL_METHOD *const method = TLS_method(); cryptoError(method == NULL, "unable to load TLS method"); // Create TLS context SSL_CTX *const result = SSL_CTX_new(method); cryptoError(result == NULL, "unable to create TLS context"); // Disable compression SSL_CTX_set_options(result, SSL_OP_NO_COMPRESSION); // Disable auto-retry to prevent SSL_read() from hanging SSL_CTX_clear_mode(result, SSL_MODE_AUTO_RETRY); // Enhance security when verifyPeer is enabled if (verifyPeer) { // Set minimum TLS version to 1.2 cryptoError(SSL_CTX_set_min_proto_version(result, TLS1_2_VERSION) != 1, "unable to set minumum TLS version to 1.2"); // Set allowed ciphers for TLSv1.2 when configured if (tlsCommonLocal.tlsCipher12 != NULL) // {vm_covered} { cryptoError( // {vm_covered} SSL_CTX_set_cipher_list(result, strZ(tlsCommonLocal.tlsCipher12)) != 1, // {vm_covered} "unable to set TLSv1.2 ciphers"); } // Set allowed ciphers for TLSv1.3 when configured if (tlsCommonLocal.tlsCipher13 != NULL) // {vm_covered} { cryptoError( // {vm_covered} SSL_CTX_set_ciphersuites(result, strZ(tlsCommonLocal.tlsCipher13)) != 1, // {vm_covered} "unable to set TLSv1.3 ciphers"); } } FUNCTION_TEST_RETURN_TYPE_P(SSL_CTX, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/tls/common.h0000664000175100017510000000276215265101745020342 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Common ***********************************************************************************************************************************/ #ifndef COMMON_IO_TLS_COMMON_H #define COMMON_IO_TLS_COMMON_H #include #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize TLS with allowed ciphers FN_EXTERN void tlsInit(const String *tlsCipher12, const String *tlsCipher13); // Convert an ASN1 string used in certificates to a Buffer FN_EXTERN Buffer *tlsAsn1ToBuf(const ASN1_STRING *nameAsn1); // Convert an ASN1 string used in certificates to a String FN_EXTERN String *tlsAsn1ToStr(const ASN1_STRING *nameAsn1); // Get common name from a certificate FN_EXTERN String *tlsCertCommonName(X509 *certificate); // Reject embedded nulls in certificate common or alternative name to prevent attacks like CVE-2009-4034 FN_EXTERN void tlsCertNameVerify(const String *name); // Load certificate and key and check that they match FN_EXTERN void tlsCertKeyLoad(SSL_CTX *context, const String *certFile, const String *keyFile); // Create TLS context FN_EXTERN SSL_CTX *tlsContext(bool verifyPeer); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/tls/server.c0000664000175100017510000003062515265101745020352 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Server ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "common/crypto/common.h" #include "common/debug.h" #include "common/io/server.h" #include "common/io/tls/common.h" #include "common/io/tls/server.h" #include "common/io/tls/session.h" #include "common/log.h" #include "common/stat.h" #include "common/type/object.h" /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_EXTERN(TLS_STAT_SERVER_STR, TLS_STAT_SERVER); /*********************************************************************************************************************************** Curve used to generate ephemeral Elliptic Curve DH keys ***********************************************************************************************************************************/ #define ECDH_CURVE "prime256v1" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct TlsServer { String *host; // Host SSL_CTX *context; // TLS context TimeMSec timeout; // Timeout for any i/o operation (connect, read, etc.) bool verifyPeer; // Will the client cert be verified? } TlsServer; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void tlsServerToLog(const THIS_VOID, StringStatic *const debugLog) { THIS(const TlsServer); strStcFmt(debugLog, "{host: %s, timeout: %" PRIu64 "}", strZ(this->host), this->timeout); } #define FUNCTION_LOG_TLS_SERVER_TYPE \ TlsServer * #define FUNCTION_LOG_TLS_SERVER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, tlsServerToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free context ***********************************************************************************************************************************/ static void tlsServerFreeResource(THIS_VOID) { THIS(TlsServer); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SERVER, this); FUNCTION_LOG_END(); ASSERT(this != NULL); SSL_CTX_free(this->context); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Authenticate client Adapted from PostgreSQL be_tls_open_server() in src/backend/libpq/be-secure-openssl.c. ***********************************************************************************************************************************/ static void tlsServerAuth(const TlsServer *const this, IoSession *const ioSession, SSL *const tlsSession) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SERVER, this); FUNCTION_LOG_PARAM(IO_SESSION, ioSession); FUNCTION_LOG_PARAM_P(VOID, tlsSession); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // If peer verification requested if (this->verifyPeer) // {vm_covered} { // If the client cert was presented then the session is authenticated. An error will be thrown automatically if the // client cert is not valid. X509 *const clientCert = SSL_get_peer_certificate(tlsSession); // {vm_covered} ioSessionAuthenticatedSet(ioSession, clientCert != NULL); // {vm_covered} // Set the peer name to the client cert common name if (clientCert != NULL) // {vm_covered} ioSessionPeerNameSet(ioSession, tlsCertCommonName(clientCert)); // {vm_covered} // Free the cert X509_free(clientCert); // {vm_covered} } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static IoSession * tlsServerAccept(THIS_VOID, IoSession *const ioSession) { THIS(TlsServer); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SERVER, this); FUNCTION_LOG_PARAM(IO_SESSION, ioSession); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(ioSession != NULL); IoSession *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Open TLS session SSL *tlsSession = SSL_new(this->context); result = tlsSessionNew(tlsSession, ioSession, this->timeout); // Authenticate TLS session tlsServerAuth(this, result, tlsSession); // Move session ioSessionMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); statInc(TLS_STAT_SESSION_STR); FUNCTION_LOG_RETURN(IO_SESSION, result); } /**********************************************************************************************************************************/ static const String * tlsServerName(THIS_VOID) // {vm_covered} { THIS(TlsServer); // {vm_covered} FUNCTION_TEST_BEGIN(); // {vm_covered} FUNCTION_TEST_PARAM(TLS_SERVER, this); // {vm_covered} FUNCTION_TEST_END(); // {vm_covered} ASSERT(this != NULL); // {vm_covered} FUNCTION_TEST_RETURN_CONST(STRING, this->host); // {vm_covered} } /*********************************************************************************************************************************** Initialize TLS context with all required security features Adapted from PostgreSQL be_tls_init() in src/backend/libpq/be-secure-openssl.c. ***********************************************************************************************************************************/ static const IoServerInterface tlsServerInterface = { .type = IO_SERVER_TLS_TYPE, .name = tlsServerName, .accept = tlsServerAccept, .toLog = tlsServerToLog, }; FN_EXTERN IoServer * tlsServerNew( const String *const host, const String *const caFile, const String *const keyFile, const String *const certFile, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(STRING, caFile); FUNCTION_LOG_PARAM(STRING, keyFile); FUNCTION_LOG_PARAM(STRING, certFile); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(host != NULL); ASSERT(keyFile != NULL); ASSERT(certFile != NULL); OBJ_NEW_BEGIN(TlsServer, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (TlsServer) { .host = strDup(host), .context = tlsContext(true), .timeout = timeout, }; // Set callback to free context memContextCallbackSet(objMemContext(this), tlsServerFreeResource, this); // Set options SSL_CTX_set_options( this->context, // Let server set cipher order SSL_OP_CIPHER_SERVER_PREFERENCE | #ifdef SSL_OP_NO_RENEGOTIATION // Disable renegotiation, available since 1.1.0h. This affects only TLSv1.2 and older protocol versions as TLSv1.3 has // no support for renegotiation. SSL_OP_NO_RENEGOTIATION | #endif // Disable session tickets SSL_OP_NO_TICKET); // Disable session caching SSL_CTX_set_session_cache_mode(this->context, SSL_SESS_CACHE_OFF); // Set parameters for generating ephemeral DH keys. Auto selects parameters that match the strength of the certificate key. // Also see https://weakdh.org and https://en.wikipedia.org/wiki/Logjam_(computer_security). cryptoError(SSL_CTX_set_dh_auto(this->context, 1) != 1, "unable to set auto dh parameters"); // Set the curve used to generate ephemeral Elliptic Curve DH keys. The OpenSSL macro casts away const on the curve list. #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wcast-qual" cryptoError(SSL_CTX_set1_groups_list(this->context, ECDH_CURVE) != 1, "unable to set ecdh curve " ECDH_CURVE); #pragma GCC diagnostic pop // Load certificate and key tlsCertKeyLoad(this->context, certFile, keyFile); // If a CA store is specified then client certificates will be verified // ------------------------------------------------------------------------------------------------------------------------- if (caFile != NULL) // {vm_covered} { // Load CA store cryptoError( // {vm_covered} SSL_CTX_load_verify_locations(this->context, strZ(caFile), NULL) != 1, // {vm_covered} zNewFmt("unable to load CA file '%s'", strZ(caFile))); // {vm_covered} // Tell OpenSSL to send the list of root certs we trust to clients in CertificateRequests. This lets a client with a // keystore select the appropriate client certificate to send to us. Also, this ensures that the SSL context will own // the rootCertList and free it when no longer needed. STACK_OF(X509_NAME) *rootCertList = SSL_load_client_CA_file(strZ(caFile)); // {vm_covered} cryptoError(rootCertList == NULL, zNewFmt("unable to generate CA list from '%s'", strZ(caFile))); // {vm_covered} SSL_CTX_set_client_CA_list(this->context, rootCertList); // {vm_covered} // Always ask for SSL client cert, but don't fail when not presented. In this case the server will disconnect after // sending a data end message to the client. The client can use this to verify that the server is running without the // need to authenticate. SSL_CTX_set_verify(this->context, SSL_VERIFY_PEER | SSL_VERIFY_CLIENT_ONCE, NULL); // {vm_covered} // Set a flag so the client cert will be checked later this->verifyPeer = true; // {vm_covered} } } OBJ_NEW_END(); statInc(TLS_STAT_SERVER_STR); FUNCTION_LOG_RETURN(IO_SERVER, ioServerNew(this, &tlsServerInterface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/tls/server.h0000664000175100017510000000333115265101745020351 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Server A simple TLS server intended to expose services by accepting TLS over an IoSession. ***********************************************************************************************************************************/ #ifndef COMMON_IO_TLS_SERVER_H #define COMMON_IO_TLS_SERVER_H #include "common/io/server.h" #include "common/io/tls/client.h" #include "common/time.h" /*********************************************************************************************************************************** Io server type ***********************************************************************************************************************************/ #define IO_SERVER_TLS_TYPE IO_CLIENT_TLS_TYPE /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ #define TLS_STAT_SERVER "tls.server" // Servers created STRING_DECLARE(TLS_STAT_SERVER_STR); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoServer *tlsServerNew( const String *host, const String *caFile, const String *keyFile, const String *certFile, TimeMSec timeout); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/tls/session.c0000664000175100017510000003320615265101745020525 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Session ***********************************************************************************************************************************/ #include #include #include "common/crypto/common.h" #include "common/debug.h" #include "common/io/io.h" #include "common/io/read.h" #include "common/io/session.h" #include "common/io/tls/client.h" #include "common/io/tls/session.h" #include "common/io/write.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct TlsSession { IoSession *ioSession; // Io session SSL *session; // TLS session on the file descriptor TimeMSec timeout; // Timeout for any i/o operation (read, write, etc.) bool shutdownOnClose; // Shutdown the TLS connection when closing the session bool ignoreUnexpectedEof; // Ignore unexpected EOF when requested IoRead *read; // Read interface IoWrite *write; // Write interface } TlsSession; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void tlsSessionToLog(const THIS_VOID, StringStatic *const debugLog) { THIS(const TlsSession); strStcCat(debugLog, "{ioSession: "); ioSessionToLog(this->ioSession, debugLog); strStcFmt(debugLog, ", timeout: %" PRIu64 ", shutdownOnClose: %s}", this->timeout, cvtBoolToConstZ(this->shutdownOnClose)); } #define FUNCTION_LOG_TLS_SESSION_TYPE \ TlsSession * #define FUNCTION_LOG_TLS_SESSION_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, tlsSessionToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free connection ***********************************************************************************************************************************/ static void tlsSessionFreeResource(THIS_VOID) { THIS(TlsSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); SSL_free(this->session); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static void tlsSessionClose(THIS_VOID) { THIS(TlsSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // If not already closed if (this->session != NULL) { // Shutdown on request if (this->shutdownOnClose) SSL_shutdown(this->session); // Close the io session ioSessionClose(this->ioSession); // Free the TLS session memContextCallbackClear(objMemContext(this)); tlsSessionFreeResource(this); this->session = NULL; } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Process result from SSL_read(), SSL_write(), SSL_connect(), and SSL_accept(). Returns: 0 if the function should be tried again with the same parameters -1 if the connection was closed gracefully > 0 with the read/write size if SSL_read()/SSL_write() was called ***********************************************************************************************************************************/ // Helper to process error conditions static int tlsSessionResultProcess( TlsSession *const this, const int errorTls, const long unsigned int errorTlsDetail, const bool closeOk) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_PARAM(INT, errorTls); FUNCTION_LOG_PARAM(UINT64, errorTlsDetail); FUNCTION_LOG_PARAM(BOOL, closeOk); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->session != NULL); int result = -1; switch (errorTls) { // Try again after waiting for read ready case SSL_ERROR_WANT_READ: ioReadReadyP(ioSessionIoReadP(this->ioSession), .error = true); result = 0; break; // Try again after waiting for write ready case SSL_ERROR_WANT_WRITE: ioWriteReadyP(ioSessionIoWrite(this->ioSession), .error = true); result = 0; break; // Handle graceful termination by the server or unexpected EOF/error default: { // Close connection on graceful termination by the server or unexpected EOF/error when allowed if (errorTls == SSL_ERROR_ZERO_RETURN || this->ignoreUnexpectedEof) { if (!closeOk) THROW(ProtocolError, "unexpected TLS eof"); this->shutdownOnClose = false; tlsSessionClose(this); } // Else error else { // Get detailed error message when available const char *const errorTlsDetailMessage = ERR_reason_error_string(errorTlsDetail); THROW_FMT( ServiceError, "TLS error [%d:%lu] %s", errorTls, errorTlsDetail, errorTlsDetailMessage == NULL ? "no details available" : errorTlsDetailMessage); } break; } } FUNCTION_LOG_RETURN(INT, result); } static int tlsSessionResult(TlsSession *const this, int result, const bool closeOk) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_PARAM(INT, result); FUNCTION_LOG_PARAM(BOOL, closeOk); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->session != NULL); // Process errors if (result <= 0) { // Get TLS error const int errorTls = SSL_get_error(this->session, result); const long unsigned int errorTlsDetail = ERR_get_error(); result = tlsSessionResultProcess(this, errorTls, errorTlsDetail, closeOk); } FUNCTION_LOG_RETURN(INT, result); } /*********************************************************************************************************************************** Read from the TLS session ***********************************************************************************************************************************/ static size_t tlsSessionRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(TlsSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->session != NULL); ASSERT(buffer != NULL); ASSERT(!bufFull(buffer)); int result = 0; // If blocking read keep reading until buffer is full do { // If no TLS data pending then check the io to reduce blocking if (!SSL_pending(this->session)) ioReadReadyP(ioSessionIoReadP(this->ioSession), .error = true); // Read and handle errors. The error queue must be cleared before this operation. ERR_clear_error(); result = tlsSessionResult(this, SSL_read(this->session, bufRemainsPtr(buffer), (int)bufRemains(buffer)), true); // Update amount of buffer used if (result > 0) { bufUsedInc(buffer, (size_t)result); } // If the connection was closed then we are at eof. It is up to the layer above TLS to decide if this is an error. else if (result == -1) break; } while (block && bufRemains(buffer) > 0); FUNCTION_LOG_RETURN(SIZE, (size_t)result); } /*********************************************************************************************************************************** Write to the TLS session ***********************************************************************************************************************************/ static void tlsSessionWrite(THIS_VOID, const Buffer *const buffer) { THIS(TlsSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->session != NULL); ASSERT(buffer != NULL); int result = 0; while (result == 0) { // Write and handle errors. The error queue must be cleared before this operation. ERR_clear_error(); result = tlsSessionResult(this, SSL_write(this->session, bufPtrConst(buffer), (int)bufUsed(buffer)), false); CHECK(ServiceError, result == 0 || (size_t)result == bufUsed(buffer), "expected retry or complete write"); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Has session been closed by the server? ***********************************************************************************************************************************/ static bool tlsSessionEof(THIS_VOID) { THIS(TlsSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(BOOL, this->session == NULL); } /**********************************************************************************************************************************/ static IoRead * tlsSessionIoRead(THIS_VOID, const bool ignoreUnexpectedEof) { THIS(TlsSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TLS_SESSION, this); FUNCTION_TEST_PARAM(BOOL, ignoreUnexpectedEof); FUNCTION_TEST_END(); ASSERT(this != NULL); this->ignoreUnexpectedEof = ignoreUnexpectedEof; FUNCTION_TEST_RETURN(IO_READ, this->read); } /**********************************************************************************************************************************/ static IoWrite * tlsSessionIoWrite(THIS_VOID) { THIS(TlsSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TLS_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(IO_WRITE, this->write); } /**********************************************************************************************************************************/ static IoSessionRole tlsSessionRole(const THIS_VOID) { THIS(const TlsSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TLS_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN_TYPE(IoSessionRole, ioSessionRole(this->ioSession)); } /**********************************************************************************************************************************/ static const IoSessionInterface tlsSessionInterface = { .type = IO_CLIENT_TLS_TYPE, .close = tlsSessionClose, .ioRead = tlsSessionIoRead, .ioWrite = tlsSessionIoWrite, .role = tlsSessionRole, .toLog = tlsSessionToLog, }; FN_EXTERN IoSession * tlsSessionNew(SSL *const session, IoSession *const ioSession, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM_P(VOID, session); FUNCTION_LOG_PARAM(IO_SESSION, ioSession); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(session != NULL); ASSERT(ioSession != NULL); OBJ_NEW_BEGIN(TlsSession, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (TlsSession) { .session = session, .ioSession = ioSessionMove(ioSession, objMemContext(this)), .timeout = timeout, .shutdownOnClose = true, }; // Ensure session is freed memContextCallbackSet(objMemContext(this), tlsSessionFreeResource, this); // Assign file descriptor to TLS session cryptoError(SSL_set_fd(this->session, ioSessionFd(this->ioSession)) != 1, "unable to add fd to TLS session"); // Negotiate TLS session. The error queue must be cleared before this operation. int result = 0; while (result == 0) { ERR_clear_error(); if (ioSessionRole(this->ioSession) == ioSessionRoleClient) result = tlsSessionResult(this, SSL_connect(this->session), false); else result = tlsSessionResult(this, SSL_accept(this->session), false); } // Create read and write interfaces this->write = ioWriteNewP(this, .write = tlsSessionWrite); ioWriteOpen(this->write); this->read = ioReadNewP(this, .block = true, .eof = tlsSessionEof, .read = tlsSessionRead); ioReadOpen(this->read); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_SESSION, ioSessionNew(this, &tlsSessionInterface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/tls/session.h0000664000175100017510000000222215265101745020524 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Session TLS sessions are created by calling ioClientOpen(). TLS sessions are generally reused so the user must be prepared to retry their transaction on a read/write error if the server closes the session before it is reused. If this behavior is not desirable then ioSessionFree()/ioClientOpen() can be called to get a new session. ***********************************************************************************************************************************/ #ifndef COMMON_IO_TLS_SESSION_H #define COMMON_IO_TLS_SESSION_H #include #include "common/io/session.h" #include "common/time.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ // Only called by TLS client/server code FN_EXTERN IoSession *tlsSessionNew(SSL *session, IoSession *ioSession, TimeMSec timeout); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/write.c0000664000175100017510000002344615265101745017377 0ustar00runnerrunner/*********************************************************************************************************************************** IO Write Interface ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/io.h" #include "common/io/write.h" #include "common/log.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoWrite { IoWritePub pub; // Publicly accessible variables Buffer *output; // Output buffer uint64_t position; // Current write position #ifdef DEBUG bool filterGroupSet; // Were filters set? bool opened; // Has the io been opened? bool closed; // Has the io been closed? #endif }; /**********************************************************************************************************************************/ FN_EXTERN IoWrite * ioWriteNew(void *const driver, const IoWriteInterface interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(IO_WRITE_INTERFACE, interface); FUNCTION_LOG_END(); ASSERT(driver != NULL); ASSERT(interface.write != NULL); OBJ_NEW_BEGIN(IoWrite, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoWrite) { .pub = { .driver = objMoveToInterface(driver, this, memContextPrior()), .interface = interface, .filterGroup = ioFilterGroupNew(), }, .output = bufNew(ioBufferSize()), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteOpen(IoWrite *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->opened && !this->closed); if (ioWriteInterface(this)->open != NULL) ioWriteInterface(this)->open(ioWriteDriver(this)); // Track whether filters were added to prevent seek()/flush() from being called later since they won't work with most filters #ifdef DEBUG this->filterGroupSet = ioFilterGroupSize(this->pub.filterGroup) > 0; #endif // Open the filter group ioFilterGroupOpen(this->pub.filterGroup); #ifdef DEBUG this->opened = true; #endif FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWrite(IoWrite *const this, const Buffer *const buffer) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->opened && !this->closed); // Only write if there is data to write if (buffer != NULL && !bufEmpty(buffer)) { do { ioFilterGroupProcess(this->pub.filterGroup, buffer, this->output); // Write data if the buffer is full if (bufRemains(this->output) == 0) { ioWriteInterface(this)->write(ioWriteDriver(this), this->output); this->position += bufUsed(this->output); bufUsedZero(this->output); } } while (ioFilterGroupInputSame(this->pub.filterGroup)); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteLine(IoWrite *const this, const Buffer *const buffer) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ioWrite(this, buffer); ioWrite(this, LF_BUF); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool ioWriteReady(IoWrite *const this, const IoWriteReadyParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(BOOL, param.error); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = true; if (ioWriteInterface(this)->ready != NULL) result = ioWriteInterface(this)->ready(ioWriteDriver(this), param.error); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteStr(IoWrite *const this, const String *const string) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(STRING, string); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(string != NULL); ioWrite(this, BUFSTR(string)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteStrLine(IoWrite *const this, const String *const string) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(STRING, string); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(string != NULL); ioWrite(this, BUFSTR(string)); ioWrite(this, LF_BUF); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteVarIntU64(IoWrite *const this, const uint64_t value) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(UINT64, value); FUNCTION_LOG_END(); ASSERT(this != NULL); uint8_t buffer[CVT_VARINT128_BUFFER_SIZE]; size_t bufferPos = 0; cvtUInt64ToVarInt128(value, buffer, &bufferPos, sizeof(buffer)); ioWrite(this, BUF(buffer, bufferPos)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteFlush(IoWrite *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->opened && !this->closed); ASSERT(!this->filterGroupSet); if (!bufEmpty(this->output)) { ioWriteInterface(this)->write(ioWriteDriver(this), this->output); bufUsedZero(this->output); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteSeek(IoWrite *const this, const uint64_t position) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(UINT64, position); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->opened && !this->closed); ASSERT(!this->filterGroupSet); ASSERT(ioWriteInterface(this)->write != NULL); // If the requested position is not the current position then flush writes and seek. Writes must be flushed first so they will // not later end up in the wrong location. if (position != this->position) { ioWriteFlush(this); ioWriteInterface(this)->seek(ioWriteDriver(this), position); // Update current position this->position = position; } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteClose(IoWrite *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->opened && !this->closed); // Flush remaining data do { ioFilterGroupProcess(this->pub.filterGroup, NULL, this->output); // Write data if the buffer is full or if this is the last buffer to be written if (bufRemains(this->output) == 0 || (ioFilterGroupDone(this->pub.filterGroup) && !bufEmpty(this->output))) { ioWriteInterface(this)->write(ioWriteDriver(this), this->output); bufUsedZero(this->output); } } while (!ioFilterGroupDone(this->pub.filterGroup)); // Close the filter group and gather results ioFilterGroupClose(this->pub.filterGroup); // Close the driver if there is a close function if (ioWriteInterface(this)->close != NULL) ioWriteInterface(this)->close(ioWriteDriver(this)); #ifdef DEBUG this->closed = true; #endif FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN int ioWriteFd(const IoWrite *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(INT, ioWriteInterface(this)->fd == NULL ? -1 : ioWriteInterface(this)->fd(ioWriteDriver(this))); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/write.h0000664000175100017510000001073315265101745017377 0ustar00runnerrunner/*********************************************************************************************************************************** IO Write Interface Objects that write to some IO destination (file, socket, etc.) are implemented using this interface. All objects are required to implement IoWriteProcess and can optionally implement IoWriteOpen or IoWriteClose. IoWriteOpen and IoWriteClose can be used to allocate/open or deallocate/free resources. An example of an IoWrite object is IoBufferWrite. ***********************************************************************************************************************************/ #ifndef COMMON_IO_WRITE_H #define COMMON_IO_WRITE_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoWrite IoWrite; #include "common/io/filter/group.h" #include "common/io/write.intern.h" #include "common/type/buffer.h" #include "common/type/object.h" /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Filter group. Filters must be set before open and cannot be reset FN_INLINE_ALWAYS IoFilterGroup * ioWriteFilterGroup(IoWrite *const this) { return THIS_PUB(IoWrite)->filterGroup; } // File descriptor for the write object. Not all write objects have a file descriptor and -1 will be returned in that case. FN_EXTERN int ioWriteFd(const IoWrite *this); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Open the IO FN_EXTERN void ioWriteOpen(IoWrite *this); // Write data to IO and process filters FN_EXTERN void ioWrite(IoWrite *this, const Buffer *buffer); // Write linefeed-terminated buffer FN_EXTERN void ioWriteLine(IoWrite *this, const Buffer *buffer); // Can bytes be written immediately? There are no guarantees on how much data can be written but it must be at least one byte. typedef struct IoWriteReadyParam { VAR_PARAM_HEADER; bool error; // Error when write not ready } IoWriteReadyParam; #define ioWriteReadyP(this, ...) \ ioWriteReady(this, (IoWriteReadyParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool ioWriteReady(IoWrite *this, IoWriteReadyParam param); // Write string FN_EXTERN void ioWriteStr(IoWrite *this, const String *string); // Write linefeed-terminated string FN_EXTERN void ioWriteStrLine(IoWrite *this, const String *string); // Write varint-128 encoding FN_EXTERN void ioWriteVarIntU64(IoWrite *this, uint64_t value); // Flush any data in the output buffer. This does not end writing and will not work if filters are present. FN_EXTERN void ioWriteFlush(IoWrite *this); // Seek to specified position relative to beginning of write FN_EXTERN void ioWriteSeek(IoWrite *this, uint64_t position); // Close the IO and write any additional data that has not been written yet FN_EXTERN void ioWriteClose(IoWrite *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioWriteFree(IoWrite *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_WRITE_TYPE \ IoWrite * #define FUNCTION_LOG_IO_WRITE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoWrite", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/io/write.intern.h0000664000175100017510000000576415265101745020705 0ustar00runnerrunner/*********************************************************************************************************************************** IO Write Interface Internal ***********************************************************************************************************************************/ #ifndef COMMON_IO_WRITE_INTERN_H #define COMMON_IO_WRITE_INTERN_H #include "common/io/write.h" #include "common/type/buffer.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct IoWriteInterface { void (*close)(void *driver); int (*fd)(const void *driver); void (*open)(void *driver); // Can bytes be written immediately? There are no guarantees on how much data can be written but it must be at least one byte. // Optionally error when write is not ready. bool (*ready)(void *driver, bool error); // Seek to specified position relative to beginning of write void (*seek)(void *driver, uint64_t position); void (*write)(void *driver, const Buffer *buffer); } IoWriteInterface; #define ioWriteNewP(driver, ...) \ ioWriteNew(driver, (IoWriteInterface){__VA_ARGS__}) FN_EXTERN IoWrite *ioWriteNew(void *driver, IoWriteInterface interface); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoWritePub { void *driver; // Driver object IoWriteInterface interface; // Driver interface IoFilterGroup *filterGroup; // IO filters } IoWritePub; // Driver for the write object FN_INLINE_ALWAYS void * ioWriteDriver(const IoWrite *const this) { return THIS_PUB(IoWrite)->driver; } // Interface for the write object FN_INLINE_ALWAYS const IoWriteInterface * ioWriteInterface(const IoWrite *const this) { return &THIS_PUB(IoWrite)->interface; } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_WRITE_INTERFACE_TYPE \ IoWriteInterface #define FUNCTION_LOG_IO_WRITE_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "IoWriteInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/lock.c0000664000175100017510000004111115265101745016553 0ustar00runnerrunner/*********************************************************************************************************************************** Lock Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/bufferWrite.h" #include "common/io/fdRead.h" #include "common/io/fdWrite.h" #include "common/io/io.h" #include "common/lock.h" #include "common/log.h" #include "common/memContext.h" #include "common/type/json.h" #include "common/user.h" #include "common/wait.h" #include "storage/posix/storage.h" #include "storage/storage.intern.h" #include "version.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ // Indicates a lock that was made by matching exec-id rather than holding an actual lock. This distinguishes it from -1, which is a // general system error. #define LOCK_ON_EXEC_ID -2 #define LOCK_KEY_EXEC_ID STRID6("execId", 0x12e0c56051) #define LOCK_KEY_PERCENT_COMPLETE STRID6("pctCplt", 0x14310a140d01) #define LOCK_KEY_PROCESS_ID STRID5("pid", 0x11300) #define LOCK_KEY_SIZE_COMPLETE STRID6("szCplt", 0x50c4286931) #define LOCK_KEY_SIZE STRID5("sz", 0x3530) /*********************************************************************************************************************************** Mem context and local variables ***********************************************************************************************************************************/ typedef struct LockFile { String *name; // Name of lock file int fd; // File descriptor for lock file bool written; // Has the lock file been written? } LockFile; static struct LockLocal { MemContext *memContext; // Mem context for locks const String *path; // Lock path const String *execId; // Process exec id List *lockList; // List of locks held const Storage *storage; // Storage object for lock path } lockLocal; /**********************************************************************************************************************************/ FN_EXTERN void lockInit(const String *const path, const String *const execId) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(STRING, execId); FUNCTION_LOG_END(); ASSERT(lockLocal.memContext == NULL); // Allocate a mem context to hold lock info MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(Lock, .childQty = MEM_CONTEXT_QTY_MAX) { lockLocal.memContext = MEM_CONTEXT_NEW(); lockLocal.path = strDup(path); lockLocal.execId = strDup(execId); lockLocal.lockList = lstNewP(sizeof(LockFile), .comparator = lstComparatorStr); lockLocal.storage = storagePosixNewP(lockLocal.path, .write = true); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN const String * lockExecId(void) { FUNCTION_TEST_VOID(); ASSERT(lockLocal.memContext != NULL); FUNCTION_TEST_RETURN_CONST(STRING, lockLocal.execId); } /*********************************************************************************************************************************** Read contents of lock file ***********************************************************************************************************************************/ // Size of initial buffer used to load lock file #define LOCK_BUFFER_SIZE 128 // Helper to read data static LockData lockReadData(const String *const lockFileName, const int fd) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, lockFileName); FUNCTION_LOG_PARAM(INT, fd); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(lockFileName != NULL); ASSERT(fd != -1); LockData result = {0}; TRY_BEGIN() { MEM_CONTEXT_TEMP_BEGIN() { // Read contents of file Buffer *const buffer = bufNew(LOCK_BUFFER_SIZE); IoWrite *const write = ioBufferWriteNewOpen(buffer); ioCopyP(ioFdReadNewOpen(lockFileName, fd, 0), write); ioWriteClose(write); JsonRead *const json = jsonReadNew(strNewBuf(buffer)); jsonReadObjectBegin(json); MEM_CONTEXT_PRIOR_BEGIN() { result.execId = jsonReadStr(jsonReadKeyRequireStrId(json, LOCK_KEY_EXEC_ID)); if (jsonReadKeyExpectStrId(json, LOCK_KEY_PERCENT_COMPLETE)) result.percentComplete = varNewUInt(jsonReadUInt(json)); result.processId = jsonReadInt(jsonReadKeyRequireStrId(json, LOCK_KEY_PROCESS_ID)); if (jsonReadKeyExpectStrId(json, LOCK_KEY_SIZE)) result.size = varNewUInt64(jsonReadUInt64(json)); if (jsonReadKeyExpectStrId(json, LOCK_KEY_SIZE_COMPLETE)) result.sizeComplete = varNewUInt64(jsonReadUInt64(json)); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); } CATCH_ANY() { THROWP_FMT(errorType(), "unable to read lock file '%s': %s", strZ(lockFileName), errorMessage()); } TRY_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN bool lockAcquire(const String *const lockFileName, const LockAcquireParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, lockFileName); FUNCTION_LOG_PARAM(BOOL, param.returnOnNoLock); FUNCTION_LOG_END(); ASSERT(lockFileName != NULL); ASSERT(lockLocal.memContext != NULL); bool result = false; if (lstFind(lockLocal.lockList, &lockFileName) != NULL) THROW_FMT(AssertError, "lock on file '%s' already held", strZ(lockFileName)); MEM_CONTEXT_TEMP_BEGIN() { const String *const lockFilePath = storagePathP(lockLocal.storage, lockFileName); bool retry; int errNo = 0; int fd = -1; do { // Assume there will be no retry retry = false; // Attempt to open the file if ((fd = open(strZ(lockFilePath), O_RDWR | O_CREAT, STORAGE_MODE_FILE_DEFAULT)) == -1) { // Save the error for reporting outside the loop errNo = errno; // If the path does not exist then create it if (errNo == ENOENT) { storagePathCreateP(storagePosixNewP(FSLASH_STR, .write = true), strPath(lockFilePath)); retry = true; } } else { // Attempt to lock the file if (flock(fd, LOCK_EX | LOCK_NB) == -1) { // Save the error for reporting outside the loop errNo = errno; // Get execId from lock file and close it const String *execId = NULL; TRY_BEGIN() { execId = lockReadData(lockFileName, fd).execId; } CATCH_ANY() { // Any errors will be reported as unable to acquire a lock. If a process is trying to get a lock but is not // synchronized with the process holding the actual lock, it is possible that it could see a short read or // have some other problem reading. Reporting the error will likely be misleading when the actual problem is // that another process owns the lock file. } FINALLY() { close(fd); } TRY_END(); // Even though we were unable to lock the file, it may be that it is already locked by another process with the // same exec-id, i.e. spawned by the same original main process. If so, report the lock as successful. if (strEq(execId, lockLocal.execId)) fd = LOCK_ON_EXEC_ID; else fd = -1; } } } while (fd == -1 && retry); // If the lock was not successful if (fd == -1) { // Error when requested if (!param.returnOnNoLock || errNo != EWOULDBLOCK) { const String *errorHint = NULL; if (errNo == EWOULDBLOCK) errorHint = strNewZ("\nHINT: is another " PROJECT_NAME " process running?"); else if (errNo == EACCES) { // Get information for the current user userInit(); errorHint = strNewFmt( "\nHINT: does '%s:%s' running " PROJECT_NAME " have permissions on the '%s' file?", strZ(userName()), strZ(groupName()), strZ(lockFilePath)); } THROW_FMT( LockAcquireError, "unable to acquire lock on file '%s': %s%s", strZ(lockFilePath), strerror(errNo), errorHint == NULL ? "" : strZ(errorHint)); } } // Else add lock to list else { MEM_CONTEXT_OBJ_BEGIN(lockLocal.lockList) { lstAdd(lockLocal.lockList, &(LockFile){.name = strDup(lockFileName), .fd = fd}); } MEM_CONTEXT_OBJ_END(); // Write lock data unless lock was acquired by matching execId if (fd != LOCK_ON_EXEC_ID) lockWriteP(lockFileName); // Success result = true; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN LockReadResult lockRead(const String *const lockFileName, const LockReadParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, lockFileName); FUNCTION_LOG_PARAM(BOOL, param.remove); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(lockLocal.memContext != NULL); ASSERT(lockFileName != NULL); LockReadResult result = {.status = lockReadStatusValid}; MEM_CONTEXT_TEMP_BEGIN() { const String *const lockFilePath = storagePathP(lockLocal.storage, lockFileName); // If we cannot open the lock file for any reason then warn int fd = -1; if ((fd = open(strZ(lockFilePath), O_RDONLY, 0)) == -1) { result.status = lockReadStatusMissing; } else { // Attempt a lock on the file - if a lock can be acquired that means the original process died without removing the lock if (flock(fd, LOCK_EX | LOCK_NB) == 0) { result.status = lockReadStatusUnlocked; } // Else attempt to read the file else { TRY_BEGIN() { MEM_CONTEXT_PRIOR_BEGIN() { result.data = lockReadData(lockFilePath, fd); } MEM_CONTEXT_PRIOR_END(); } CATCH_ANY() { result.status = lockReadStatusInvalid; } TRY_END(); } // Remove lock file if requested but do not report failures if (param.remove) unlink(strZ(lockFilePath)); // Close after unlinking to prevent a race condition where another process creates the file as we remove it close(fd); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN void lockWrite(const String *const lockFileName, const LockWriteParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, lockFileName); FUNCTION_LOG_PARAM(VARIANT, param.percentComplete); FUNCTION_LOG_PARAM(VARIANT, param.sizeComplete); FUNCTION_LOG_PARAM(VARIANT, param.size); FUNCTION_LOG_END(); ASSERT(lockFileName != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Build the json object JsonWrite *const json = jsonWriteNewP(); jsonWriteObjectBegin(json); jsonWriteStr(jsonWriteKeyStrId(json, LOCK_KEY_EXEC_ID), lockLocal.execId); if (param.percentComplete != NULL) jsonWriteUInt(jsonWriteKeyStrId(json, LOCK_KEY_PERCENT_COMPLETE), varUInt(param.percentComplete)); jsonWriteInt(jsonWriteKeyStrId(json, LOCK_KEY_PROCESS_ID), getpid()); if (param.size != NULL) jsonWriteUInt64(jsonWriteKeyStrId(json, LOCK_KEY_SIZE), varUInt64(param.size)); if (param.sizeComplete != NULL) jsonWriteUInt64(jsonWriteKeyStrId(json, LOCK_KEY_SIZE_COMPLETE), varUInt64(param.sizeComplete)); jsonWriteObjectEnd(json); // Write to lock file LockFile *const lockFile = lstFind(lockLocal.lockList, &lockFileName); if (lockFile == NULL) THROW_FMT(AssertError, "lock file '%s' not found", strZ(lockFileName)); const String *const lockFilePath = storagePathP(lockLocal.storage, lockFileName); if (lockFile->written) { // Seek to beginning of lock file THROW_ON_SYS_ERROR_FMT( lseek(lockFile->fd, 0, SEEK_SET) == -1, FileOpenError, STORAGE_ERROR_READ_SEEK, (uint64_t)0, strZ(lockFilePath)); // In case the current write is shorter than before THROW_ON_SYS_ERROR_FMT(ftruncate(lockFile->fd, 0) == -1, FileWriteError, "unable to truncate '%s'", strZ(lockFilePath)); } // Write lock file data IoWrite *const write = ioFdWriteNewOpen(lockFilePath, lockFile->fd, 0); ioCopyP(ioBufferReadNewOpen(BUFSTR(jsonWriteResult(json))), write); ioWriteClose(write); lockFile->written = true; } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool lockRelease(const LockReleaseParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BOOL, param.returnOnNoLock); FUNCTION_LOG_END(); ASSERT(lockLocal.memContext != NULL); bool result = false; // Nothing to do if lock list is empty if (lstEmpty(lockLocal.lockList)) { // Fail if requested if (!param.returnOnNoLock) THROW(AssertError, "no lock is held by this process"); } // Else release all locks else { // Release until list is empty while (!lstEmpty(lockLocal.lockList)) { LockFile *const lockFile = lstGet(lockLocal.lockList, 0); // Remove lock file if this lock was not acquired by matching execId if (lockFile->fd != LOCK_ON_EXEC_ID) { storageRemoveP(lockLocal.storage, lockFile->name); close(lockFile->fd); } strFree(lockFile->name); lstRemoveIdx(lockLocal.lockList, 0); } // Success result = true; } FUNCTION_LOG_RETURN(BOOL, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/lock.h0000664000175100017510000001164615265101745016572 0ustar00runnerrunner/*********************************************************************************************************************************** Lock Handler ***********************************************************************************************************************************/ #ifndef COMMON_LOCK_H #define COMMON_LOCK_H #include /*********************************************************************************************************************************** Lock data ***********************************************************************************************************************************/ #include "common/type/variant.h" // Lock data typedef struct LockData { pid_t processId; // Process holding the lock const String *execId; // Exec id of process holding the lock const Variant *percentComplete; // Percentage of backup complete * 100 (when not NULL) const Variant *sizeComplete; // Completed size of the backup in bytes const Variant *size; // Total size of the backup in bytes } LockData; #include "common/time.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define LOCK_FILE_EXT ".lock" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize lock module FN_EXTERN void lockInit(const String *path, const String *execId); // Exec id for this process, as set by lockInit() FN_EXTERN const String *lockExecId(void); // Acquire a lock typedef struct LockAcquireParam { VAR_PARAM_HEADER; bool returnOnNoLock; // Return when no lock acquired (rather than throw an error) } LockAcquireParam; #define lockAcquireP(lockFileName, ...) \ lockAcquire(lockFileName, (LockAcquireParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool lockAcquire(const String *lockFileName, LockAcquireParam param); // Write data to a lock file typedef struct LockWriteParam { VAR_PARAM_HEADER; const Variant *percentComplete; // Percentage of backup complete * 100 (when not NULL) const Variant *sizeComplete; // Completed size of the backup in bytes const Variant *size; // Total size of the backup in bytes } LockWriteParam; #define lockWriteP(lockFileName, ...) \ lockWrite(lockFileName, (LockWriteParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void lockWrite(const String *lockFileName, LockWriteParam param); // Read a lock file held by another process to get information about what the process is doing typedef enum { lockReadStatusMissing, // File is missing lockReadStatusUnlocked, // File is not locked lockReadStatusInvalid, // File contents are invalid lockReadStatusValid, // File is locked and contexts are valid } LockReadStatus; typedef struct LockReadResult { LockReadStatus status; // Status of file read LockData data; // Lock data } LockReadResult; typedef struct LockReadParam { VAR_PARAM_HEADER; bool remove; // Remove the lock file after locking/reading } LockReadParam; #define lockReadP(lockFileName, ...) \ lockRead(lockFileName, (LockReadParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN LockReadResult lockRead(const String *lockFileName, LockReadParam param); // Release lock(s) typedef struct LockReleaseParam { VAR_PARAM_HEADER; bool returnOnNoLock; // Return when no lock is held (rather than throw an error) } LockReleaseParam; #define lockReleaseP(...) \ lockRelease((LockReleaseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool lockRelease(LockReleaseParam param); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/log.c0000664000175100017510000005140215265101745016410 0ustar00runnerrunner/*********************************************************************************************************************************** Log Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include #include #include "common/debug.h" #include "common/log.h" #include "common/macro.h" #include "common/time.h" #include "common/type/convert.h" /*********************************************************************************************************************************** Module variables ***********************************************************************************************************************************/ // Log levels static LogLevel logLevelStdOut = logLevelError; static LogLevel logLevelStdErr = logLevelError; static LogLevel logLevelFile = logLevelOff; static LogLevel logLevelAny = logLevelError; // Log file descriptors static int logFdStdOut = STDOUT_FILENO; static int logFdStdErr = STDERR_FILENO; static int logFdFile = -1; // Has the log file banner been written yet? static bool logFileBanner = false; // Is the timestamp printed in the log? static bool logTimestamp = false; // Default process id if none is specified static unsigned int logProcessId = 0; // Size of the process id field static int logProcessSize = 2; // Prefix DRY-RUN to log messages static bool logDryRun = false; /*********************************************************************************************************************************** Dry run prefix ***********************************************************************************************************************************/ #define DRY_RUN_PREFIX "[DRY-RUN] " /*********************************************************************************************************************************** Test Asserts ***********************************************************************************************************************************/ #define ASSERT_LOG_LEVEL(logLevel) \ ASSERT(logLevel >= LOG_LEVEL_MIN && logLevel <= LOG_LEVEL_MAX) /*********************************************************************************************************************************** Log buffer -- used to format log header and message ***********************************************************************************************************************************/ static char logBuffer[LOG_BUFFER_SIZE]; /**********************************************************************************************************************************/ #define LOG_LEVEL_TOTAL (LOG_LEVEL_MAX + 1) static const char *const logLevelList[LOG_LEVEL_TOTAL] = { "OFF", "ASSERT", "ERROR", "WARN", "INFO", "DETAIL", "DEBUG", "TRACE", }; FN_EXTERN LogLevel logLevelEnum(unsigned int logLevelSeq) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, logLevelSeq); FUNCTION_TEST_END(); ASSERT(logLevelSeq < LOG_LEVEL_MAX); logLevelSeq += logLevelSeq > 0; FUNCTION_TEST_RETURN(ENUM, logLevelSeq); } FN_EXTERN const char * logLevelStr(const LogLevel logLevel) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_END(); ASSERT(logLevel <= LOG_LEVEL_MAX); FUNCTION_TEST_RETURN_CONST(STRINGZ, logLevelList[logLevel]); } /**********************************************************************************************************************************/ static void logAnySet(void) { FUNCTION_TEST_VOID(); logLevelAny = logLevelStdOut; if (logLevelStdErr > logLevelAny) logLevelAny = logLevelStdErr; if (logLevelFile > logLevelAny && logFdFile != -1) logLevelAny = logLevelFile; FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN bool logAny(const LogLevel logLevel) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_END(); ASSERT_LOG_LEVEL(logLevel); FUNCTION_TEST_RETURN(BOOL, logLevel <= logLevelAny); } /**********************************************************************************************************************************/ FN_EXTERN void logInit( const LogLevel logLevelStdOutParam, const LogLevel logLevelStdErrParam, const LogLevel logLevelFileParam, const bool logTimestampParam, const unsigned int processId, const unsigned int logProcessMax, const bool dryRunParam) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevelStdOutParam); FUNCTION_TEST_PARAM(ENUM, logLevelStdErrParam); FUNCTION_TEST_PARAM(ENUM, logLevelFileParam); FUNCTION_TEST_PARAM(BOOL, logTimestampParam); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_PARAM(UINT, logProcessMax); FUNCTION_TEST_PARAM(BOOL, dryRunParam); FUNCTION_TEST_END(); ASSERT(logLevelStdOutParam <= LOG_LEVEL_MAX); ASSERT(logLevelStdErrParam <= LOG_LEVEL_MAX); ASSERT(logLevelFileParam <= LOG_LEVEL_MAX); ASSERT(processId <= 999); ASSERT(logProcessMax <= 999); logLevelStdOut = logLevelStdOutParam; logLevelStdErr = logLevelStdErrParam; logLevelFile = logLevelFileParam; logTimestamp = logTimestampParam; logProcessId = processId; logProcessSize = logProcessMax > 99 ? 3 : 2; logDryRun = dryRunParam; logAnySet(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Close the log file ***********************************************************************************************************************************/ static void logFileClose(void) { FUNCTION_TEST_VOID(); // Close the file descriptor if it is open if (logFdFile != -1) { close(logFdFile); logFdFile = -1; } logAnySet(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool logFileSet(const char *const logFile) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, logFile); FUNCTION_TEST_END(); ASSERT(logFile != NULL); // Close the log file if it is already open logFileClose(); // Only open the file if there is a chance to log something bool result = true; if (logLevelFile != logLevelOff) { // Open the file and handle errors logFdFile = open(logFile, O_CREAT | O_APPEND | O_WRONLY, 0640); if (logFdFile == -1) { const int errNo = errno; LOG_WARN_FMT( "unable to open log file '%s': %s\nNOTE: process will continue without log file.", logFile, strerror(errNo)); result = false; } // Output the banner on first log message logFileBanner = false; logAnySet(); } logAnySet(); FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN void logClose(void) { FUNCTION_TEST_VOID(); // Disable all logging logInit(logLevelOff, logLevelOff, logLevelOff, false, 0, 1, false); // Close the log file if it is open logFileClose(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Determine if the log level is in the specified range ***********************************************************************************************************************************/ static bool logRange(const LogLevel logLevel, const LogLevel logRangeMin, const LogLevel logRangeMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_PARAM(ENUM, logRangeMin); FUNCTION_TEST_PARAM(ENUM, logRangeMax); FUNCTION_TEST_END(); ASSERT_LOG_LEVEL(logLevel); ASSERT_LOG_LEVEL(logRangeMin); ASSERT_LOG_LEVEL(logRangeMax); ASSERT(logRangeMin <= logRangeMax); FUNCTION_TEST_RETURN(BOOL, logLevel >= logRangeMin && logLevel <= logRangeMax); } /*********************************************************************************************************************************** Internal write function that handles errors ***********************************************************************************************************************************/ static void logWrite(const int fd, const char *const message, const size_t messageSize, const char *const errorDetail) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, fd); FUNCTION_TEST_PARAM(STRINGZ, message); FUNCTION_TEST_PARAM(SIZE, messageSize); FUNCTION_TEST_PARAM(STRINGZ, errorDetail); FUNCTION_TEST_END(); ASSERT(fd != -1); ASSERT(message != NULL); ASSERT(messageSize != 0); ASSERT(errorDetail != NULL); if ((size_t)write(fd, message, messageSize) != messageSize) THROW_SYS_ERROR_FMT(FileWriteError, "unable to write %s", errorDetail); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Write out log message and indent subsequent lines ***********************************************************************************************************************************/ static void logWriteIndent(const int fd, const char *message, const size_t indentSize, const char *const errorDetail) { // Indent buffer -- used to write out indent space without having to loop static const char indentBuffer[] = " "; FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, fd); FUNCTION_TEST_PARAM(STRINGZ, message); FUNCTION_TEST_PARAM(SIZE, indentSize); FUNCTION_TEST_PARAM(STRINGZ, errorDetail); FUNCTION_TEST_END(); ASSERT(fd != -1); ASSERT(message != NULL); ASSERT(indentSize > 0 && indentSize < sizeof(indentBuffer)); ASSERT(errorDetail != NULL); // Indent all lines after the first const char *linefeedPtr = strchr(message, '\n'); bool first = true; while (linefeedPtr != NULL) { if (!first) logWrite(fd, indentBuffer, indentSize, errorDetail); else first = false; logWrite(fd, message, (size_t)(linefeedPtr - message + 1), errorDetail); message += (size_t)(linefeedPtr - message + 1); linefeedPtr = strchr(message, '\n'); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Generate the log header and anything else that needs to happen before the message is output ***********************************************************************************************************************************/ typedef struct LogPreResult { size_t bufferPos; char *logBufferStdErr; size_t indentSize; } LogPreResult; static LogPreResult logPre( const LogLevel logLevel, const unsigned int processId, const char *const fileName, const char *const functionName, const int code) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_PARAM(STRINGZ, fileName); FUNCTION_TEST_PARAM(STRINGZ, functionName); FUNCTION_TEST_PARAM(INT, code); FUNCTION_TEST_END(); ASSERT_LOG_LEVEL(logLevel); ASSERT(fileName != NULL); ASSERT(functionName != NULL); ASSERT( (code == 0 && logLevel > logLevelError) || (logLevel == logLevelError && code != errorTypeCode(&AssertError)) || (logLevel == logLevelAssert && code == errorTypeCode(&AssertError))); // Initialize buffer position LogPreResult result = {.bufferPos = 0}; // Add time if (logTimestamp) { const TimeMSec logTimeMSec = timeMSec(); const time_t logTimeSec = (time_t)(logTimeMSec / MSEC_PER_SEC); result.bufferPos += cvtTimeToZP( "%Y-%m-%d %H:%M:%S", logTimeSec, logBuffer + result.bufferPos, sizeof(logBuffer) - result.bufferPos); result.bufferPos += (size_t)snprintf( logBuffer + result.bufferPos, sizeof(logBuffer) - result.bufferPos, ".%03d ", (int)(logTimeMSec % 1000)); } // Add process and aligned log level result.bufferPos += (size_t)snprintf( logBuffer + result.bufferPos, sizeof(logBuffer) - result.bufferPos, "P%0*u %*s: ", logProcessSize, processId == (unsigned int)-1 ? logProcessId : processId, 6, logLevelStr(logLevel)); // When writing to stderr the timestamp, process, and log level alignment will be skipped result.logBufferStdErr = logBuffer + result.bufferPos - strlen(logLevelStr(logLevel)) - 2; // Set the indent size -- this will need to be adjusted for stderr result.indentSize = result.bufferPos; // Add error code if (code != 0) result.bufferPos += (size_t)snprintf(logBuffer + result.bufferPos, sizeof(logBuffer) - result.bufferPos, "[%03d]: ", code); // Add dry-run prefix if (logDryRun) result.bufferPos += (size_t)snprintf(logBuffer + result.bufferPos, sizeof(logBuffer) - result.bufferPos, DRY_RUN_PREFIX); // Add debug info if (logLevel >= logLevelDebug) { // Adding padding for debug and trace levels. Cast to handle compilers (e.g. MSVC) that coerce to signed after subtraction. for (unsigned int paddingIdx = 0; paddingIdx < (unsigned int)((logLevel - logLevelDebug + 1) * 4); paddingIdx++) { logBuffer[result.bufferPos++] = ' '; result.indentSize++; } result.bufferPos += (size_t)snprintf( logBuffer + result.bufferPos, LOG_BUFFER_SIZE - result.bufferPos, "%.*s::%s: ", (int)strlen(fileName) - 2, fileName, functionName); } FUNCTION_TEST_RETURN_TYPE(LogPreResult, result); } /*********************************************************************************************************************************** Finalize formatting and log after the message has been added to the buffer ***********************************************************************************************************************************/ #define LOG_BANNER "-------------------PROCESS START-------------------\n" static void logPost(LogPreResult *const logData, const LogLevel logLevel, const LogLevel logRangeMin, const LogLevel logRangeMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, logData->bufferPos); FUNCTION_TEST_PARAM(STRINGZ, logData->logBufferStdErr); FUNCTION_TEST_PARAM(SIZE, logData->indentSize); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_PARAM(ENUM, logRangeMin); FUNCTION_TEST_PARAM(ENUM, logRangeMax); FUNCTION_TEST_END(); ASSERT_LOG_LEVEL(logLevel); ASSERT_LOG_LEVEL(logRangeMin); ASSERT_LOG_LEVEL(logRangeMax); ASSERT(logRangeMin <= logRangeMax); // Reserve room for the linefeed and null terminator. snprintf() returns the untruncated length, so a message longer than the // buffer can leave bufferPos past the end; limit it so the writes below stay in bounds. if (logData->bufferPos > LOG_BUFFER_SIZE - 2) logData->bufferPos = LOG_BUFFER_SIZE - 2; // Add linefeed logBuffer[logData->bufferPos++] = '\n'; logBuffer[logData->bufferPos] = 0; // Determine where to log the message based on log-level-stderr if (logLevel <= logLevelStdErr) { if ((logLevelStdErr > logLevelStdOut && logRange(logLevelStdErr, logRangeMin, logRangeMax)) || (logLevelStdErr <= logLevelStdOut && logRange(logLevelStdOut, logRangeMin, logRangeMax))) { logWriteIndent( logFdStdErr, logData->logBufferStdErr, logData->indentSize - (size_t)(logData->logBufferStdErr - logBuffer), "log to stderr"); } } else if (logLevel <= logLevelStdOut && logRange(logLevelStdOut, logRangeMin, logRangeMax)) logWriteIndent(logFdStdOut, logBuffer, logData->indentSize, "log to stdout"); // Log to file if (logLevel <= logLevelFile && logFdFile != -1 && logRange(logLevelFile, logRangeMin, logRangeMax)) { // If the banner has not been written if (!logFileBanner) { // Add a blank line if the file already has content if (lseek(logFdFile, 0, SEEK_END) > 0) logWrite(logFdFile, "\n", 1, "banner spacing to file"); // Write process start banner logWrite(logFdFile, LOG_BANNER, sizeof(LOG_BANNER) - 1, "banner to file"); // Mark banner as written logFileBanner = true; } logWriteIndent(logFdFile, logBuffer, logData->indentSize, "log to file"); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ #define LOG_SIGNAL_MESSAGE_PRE "terminated on signal " FN_EXTERN void logSignal(const LogLevel logLevel, const char *const signalName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_PARAM(STRINGZ, signalName); FUNCTION_TEST_END(); ASSERT(signalName != NULL); static_assert(LOG_BUFFER_SIZE >= sizeof(LOG_SIGNAL_MESSAGE_PRE), "invalid log buffer size"); // Initialize log buffer and data with static signal message memcpy(logBuffer, LOG_SIGNAL_MESSAGE_PRE, sizeof(LOG_SIGNAL_MESSAGE_PRE) - 1); LogPreResult logData = {.bufferPos = sizeof(LOG_SIGNAL_MESSAGE_PRE) - 1, .logBufferStdErr = logBuffer, .indentSize = 4}; // Add signal name and update buffer position (memcpy includes the null terminator). A signal name is always a short internal // constant that fits the buffer and memcpy is async-signal-safe, unlike snprintf(). const size_t signalNameSize = strlen(signalName); memcpy(logBuffer + logData.bufferPos, signalName, signalNameSize + 1); logData.bufferPos += signalNameSize; logPost(&logData, logLevel, LOG_LEVEL_MIN, LOG_LEVEL_MAX); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void logInternal( const LogLevel logLevel, const LogLevel logRangeMin, const LogLevel logRangeMax, const unsigned int processId, const char *const fileName, const char *const functionName, const int code, const char *const message) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_PARAM(ENUM, logRangeMin); FUNCTION_TEST_PARAM(ENUM, logRangeMax); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_PARAM(STRINGZ, fileName); FUNCTION_TEST_PARAM(STRINGZ, functionName); FUNCTION_TEST_PARAM(INT, code); FUNCTION_TEST_PARAM(STRINGZ, message); FUNCTION_TEST_END(); ASSERT(message != NULL); LogPreResult logData = logPre(logLevel, processId, fileName, functionName, code); // Copy message into buffer and update buffer position logData.bufferPos += (size_t)snprintf(logBuffer + logData.bufferPos, LOG_BUFFER_SIZE - logData.bufferPos, "%s", message); logPost(&logData, logLevel, logRangeMin, logRangeMax); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void logInternalFmt( const LogLevel logLevel, const LogLevel logRangeMin, const LogLevel logRangeMax, const unsigned int processId, const char *const fileName, const char *const functionName, const int code, const char *const format, ...) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_PARAM(ENUM, logRangeMin); FUNCTION_TEST_PARAM(ENUM, logRangeMax); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_PARAM(STRINGZ, fileName); FUNCTION_TEST_PARAM(STRINGZ, functionName); FUNCTION_TEST_PARAM(INT, code); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_END(); ASSERT(format != NULL); LogPreResult logData = logPre(logLevel, processId, fileName, functionName, code); // Format message into buffer and update buffer position va_list argumentList; va_start(argumentList, format); logData.bufferPos += (size_t)vsnprintf( logBuffer + logData.bufferPos, LOG_BUFFER_SIZE - logData.bufferPos, format, argumentList); va_end(argumentList); logPost(&logData, logLevel, logRangeMin, logRangeMax); FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/log.h0000664000175100017510000002723715265101745016426 0ustar00runnerrunner/*********************************************************************************************************************************** Log Handler ***********************************************************************************************************************************/ #ifndef COMMON_LOG_H #define COMMON_LOG_H #include #include #include "common/logLevel.h" #include "common/type/stringId.h" /*********************************************************************************************************************************** Max size allowed for a single log message including header ***********************************************************************************************************************************/ #ifndef LOG_BUFFER_SIZE #define LOG_BUFFER_SIZE ((size_t)(32 * 1024)) #endif /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize the log system FN_EXTERN void logInit( LogLevel logLevelStdOutParam, LogLevel logLevelStdErrParam, LogLevel logLevelFileParam, bool logTimestampParam, unsigned int processId, unsigned int logProcessMax, bool dryRun); // Close the log system FN_EXTERN void logClose(void); // Set the log file. Returns true if file logging is off or the log file was successfully opened, false if the log file could not be // opened. FN_EXTERN bool logFileSet(const char *logFile); // Check if a log level will be logged to any output. This is useful for log messages that are expensive to generate and should be // skipped if they will be discarded. FN_EXTERN bool logAny(LogLevel logLevel); // Convert log level to string and vice versa FN_EXTERN LogLevel logLevelEnum(unsigned int logLevelSeq); FN_EXTERN const char *logLevelStr(LogLevel logLevel); // Log exit on signal FN_EXTERN void logSignal(LogLevel logLevel, const char *signalName); /*********************************************************************************************************************************** Macros Macros only call logInternal()/logInternalFmt() if the message will be logged to one of the available outputs. Also simplify each call site by supplying commonly-used values. Note that it's possible that not all the macros below will appear in the code. In particular the ERROR and ASSERT macros should not be used directly. They are included for completeness and future usage. ***********************************************************************************************************************************/ // Define a macro to test logAny() that can be removed when performing coverage testing. Checking logAny() saves a function call for // logging calls that won't be output anywhere, but since the macro contains a branch it causes coverage problems. #ifdef DEBUG_COVERAGE #define IF_LOG_ANY(logLevel) #else #define IF_LOG_ANY(logLevel) \ if (logAny(logLevel)) #endif #define LOG_INTERNAL(logLevel, logRangeMin, logRangeMax, processId, code, message) \ do \ { \ IF_LOG_ANY(logLevel) \ { \ logInternal(logLevel, logRangeMin, logRangeMax, processId, __FILE__, __func__, code, message); \ } \ } while (0) #define LOG_INTERNAL_FMT(logLevel, logRangeMin, logRangeMax, processId, code, ...) \ do \ { \ IF_LOG_ANY(logLevel) \ { \ logInternalFmt(logLevel, logRangeMin, logRangeMax, processId, __FILE__, __func__, code, __VA_ARGS__); \ } \ } while (0) #define LOG_PID(logLevel, processId, code, message) \ LOG_INTERNAL(logLevel, LOG_LEVEL_MIN, LOG_LEVEL_MAX, processId, code, message) #define LOG_ASSERT_PID(processId, message) \ LOG_PID(logLevelAssert, processId, errorTypeCode(&AssertError), message) #define LOG_ERROR_PID(processId, code, message) \ LOG_PID(logLevelError, processId, code, message) #define LOG_WARN_PID(processId, message) \ LOG_PID(logLevelWarn, processId, 0, message) #define LOG_INFO_PID(processId, message) \ LOG_PID(logLevelInfo, processId, 0, message) #define LOG_DETAIL_PID(processId, message) \ LOG_PID(logLevelDetail, processId, 0, message) #define LOG_DEBUG_PID(processId, message) \ LOG_PID(logLevelDebug, processId, 0, message) #define LOG_TRACE_PID(processId, message) \ LOG_PID(logLevelTrace, processId, 0, message) #define LOG_PID_FMT(logLevel, processId, code, ...) \ LOG_INTERNAL_FMT(logLevel, LOG_LEVEL_MIN, LOG_LEVEL_MAX, processId, code, __VA_ARGS__) #define LOG_ASSERT_PID_FMT(processId, ...) \ LOG_PID_FMT(logLevelAssert, processId, errorTypeCode(&AssertError), __VA_ARGS__) #define LOG_ERROR_PID_FMT(processId, code, ...) \ LOG_PID_FMT(logLevelError, processId, code, __VA_ARGS__) #define LOG_WARN_PID_FMT(processId, ...) \ LOG_PID_FMT(logLevelWarn, processId, 0, __VA_ARGS__) #define LOG_INFO_PID_FMT(processId, ...) \ LOG_PID_FMT(logLevelInfo, processId, 0, __VA_ARGS__) #define LOG_DETAIL_PID_FMT(processId, ...) \ LOG_PID_FMT(logLevelDetail, processId, 0, __VA_ARGS__) #define LOG_DEBUG_PID_FMT(processId, ...) \ LOG_PID_FMT(logLevelDebug, processId, 0, __VA_ARGS__) #define LOG_TRACE_PID_FMT(processId, ...) \ LOG_PID_FMT(logLevelTrace, processId, 0, __VA_ARGS__) #define LOG(logLevel, code, message) \ LOG_PID(logLevel, UINT_MAX, code, message) #define LOG_ASSERT(message) \ LOG(logLevelAssert, errorTypeCode(&AssertError), message) #define LOG_ERROR(code, message) \ LOG(logLevelError, code, message) #define LOG_WARN(message) \ LOG(logLevelWarn, 0, message) #define LOG_INFO(message) \ LOG(logLevelInfo, 0, message) #define LOG_DETAIL(message) \ LOG(logLevelDetail, 0, message) #define LOG_DEBUG(message) \ LOG(logLevelDebug, 0, message) #define LOG_TRACE(message) \ LOG(logLevelTrace, 0, message) #define LOG_FMT(logLevel, code, ...) \ LOG_PID_FMT(logLevel, UINT_MAX, code, __VA_ARGS__) #define LOG_ASSERT_FMT(...) \ LOG_FMT(logLevelAssert, errorTypeCode(&AssertError), __VA_ARGS__) #define LOG_ERROR_FMT(code, ...) \ LOG_FMT(logLevelError, code, __VA_ARGS__) #define LOG_WARN_FMT(...) \ LOG_FMT(logLevelWarn, 0, __VA_ARGS__) #define LOG_INFO_FMT(...) \ LOG_FMT(logLevelInfo, 0, __VA_ARGS__) #define LOG_DETAIL_FMT(...) \ LOG_FMT(logLevelDetail, 0, __VA_ARGS__) #define LOG_DEBUG_FMT(...) \ LOG_FMT(logLevelDebug, 0, __VA_ARGS__) #define LOG_TRACE_FMT(...) \ LOG_FMT(logLevelTrace, 0, __VA_ARGS__) /*********************************************************************************************************************************** Internal Functions (in virtually all cases the macros above should be used in preference to these functions) ***********************************************************************************************************************************/ // Log function FN_EXTERN void logInternal( LogLevel logLevel, LogLevel logRangeMin, LogLevel logRangeMax, unsigned int processId, const char *fileName, const char *functionName, int code, const char *message); // Log function with formatting FN_EXTERN FN_PRINTF(8, 9) void logInternalFmt( LogLevel logLevel, LogLevel logRangeMin, LogLevel logRangeMax, unsigned int processId, const char *fileName, const char *functionName, int code, const char *format, ...); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/logLevel.h0000664000175100017510000000167515265101745017414 0ustar00runnerrunner/*********************************************************************************************************************************** Log Levels ***********************************************************************************************************************************/ #ifndef COMMON_LOGLEVEL_H #define COMMON_LOGLEVEL_H /*********************************************************************************************************************************** Log types ***********************************************************************************************************************************/ typedef enum { logLevelOff, logLevelAssert, logLevelError, logLevelWarn, logLevelInfo, logLevelDetail, logLevelDebug, logLevelTrace, } LogLevel; #define LOG_LEVEL_MIN logLevelAssert #define LOG_LEVEL_MAX logLevelTrace #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/macro.h0000664000175100017510000001473215265101745016742 0ustar00runnerrunner/*********************************************************************************************************************************** General Macros Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group Portions Copyright (c) 1994, Regents of the University of California ***********************************************************************************************************************************/ #ifndef COMMON_MACRO_H #define COMMON_MACRO_H #include #include /*********************************************************************************************************************************** Convert the parameter to a zero-terminated string Useful for converting non-string types (e.g. int) to strings for inclusion in messages. ***********************************************************************************************************************************/ #define STRINGIFY_HELPER(param) #param #define STRINGIFY(param) STRINGIFY_HELPER(param) /*********************************************************************************************************************************** If param2 > param1 then assign it to param1 Useful for ensuring coverage in cases where compared values may be always ascending or descending. ***********************************************************************************************************************************/ #define MAX_ASSIGN(param1, param2) \ do \ { \ if (param2 > param1) \ param1 = param2; \ } \ while (0) /*********************************************************************************************************************************** C11 supports static_assert but there are some syntactic placement restrictions. STATIC_ASSERT_EXP() makes it safe to use in an expression. ***********************************************************************************************************************************/ #define STATIC_ASSERT_EXPR(condition, message) \ ((void)({static_assert(condition, message); true;})) /*********************************************************************************************************************************** Allows casting const-ness away from an expression, but doesn't allow changing the type. Enforcement of the latter currently only works for gcc-like compilers. Note that it is not safe to cast const-ness away if the result will ever be modified (it would be undefined behavior). Doing so can cause compiler mis-optimizations or runtime crashes (by modifying read-only memory). It is only safe to use when the result will not be modified, but API design or language restrictions prevent you from declaring that (e.g. because a function returns both const and non-const variables). Note that this only works in function scope, not for global variables (it would be nice, but not trivial, to improve that). Adapted from PostgreSQL src/include/c.h. ***********************************************************************************************************************************/ #ifdef HAVE_BUILTIN_TYPES_COMPATIBLE_P #define UNCONSTIFY(type, expression) \ (STATIC_ASSERT_EXPR(__builtin_types_compatible_p(__typeof(expression), const type), "invalid cast"), (type)(expression)) #else #define UNCONSTIFY(type, expression) \ ((type)(expression)) #endif /*********************************************************************************************************************************** Determine the byte offset required to align a type after an arbitrary number of bytes This is useful for determining how to correctly align a type in a buffer that is being dynamically built up like a struct. ***********************************************************************************************************************************/ #define ALIGN_OFFSET(type, bytes) (alignof(type) - ((bytes) % alignof(type))) /*********************************************************************************************************************************** Determine size of a type in a struct Using sizeof() on a struct member requires additional syntax. ***********************************************************************************************************************************/ #define SIZE_OF_STRUCT_MEMBER(struct, member) sizeof(((struct){0}).member) /*********************************************************************************************************************************** Determine the length of an array that can be determined at compile time For this macro to work correctly the array must be declared like: int intList[] = {0, 1}; It will not work for an array declared like: int *intList; ***********************************************************************************************************************************/ #define LENGTH_OF(array) (sizeof(array) / sizeof((array)[0])) /*********************************************************************************************************************************** Hint to the compiler that a condition is unlikely to be true, e.g. a check for an error. Returns the boolean value of the condition. Adapted from PostgreSQL src/include/c.h. ***********************************************************************************************************************************/ #ifdef HAVE_BUILTIN_EXPECT #define UNLIKELY(condition) __builtin_expect((condition) != 0, 0) #else #define UNLIKELY(condition) ((condition) != 0) #endif #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/memContext.c0000664000175100017510000017031715265101745017761 0ustar00runnerrunner/*********************************************************************************************************************************** Memory Context Manager ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/macro.h" #include "common/memContext.h" #include "common/valgrind.h" /*********************************************************************************************************************************** Contains information about a memory allocation. This header is placed at the beginning of every memory allocation returned to the user by memNew(), etc. The advantage is that when an allocation is passed back by the user we know the location of the allocation header by doing some pointer arithmetic. This is much faster than searching through a list. ***********************************************************************************************************************************/ typedef struct MemContextAlloc { unsigned int allocIdx : 32; // Index in the allocation list unsigned int size : 32; // Allocation size (4GB max) } MemContextAlloc; // Get the allocation buffer pointer given the allocation header pointer #define MEM_CONTEXT_ALLOC_BUFFER(header) ((MemContextAlloc *)header + 1) // Get the allocation header pointer given the allocation buffer pointer #define MEM_CONTEXT_ALLOC_HEADER(buffer) ((MemContextAlloc *)buffer - 1) // Make sure the allocation is valid for the current memory context. This check only works correctly if the allocation is valid and // allocated as one of many but belongs to another context. Otherwise, there is likely to be a segfault. #define ASSERT_ALLOC_MANY_VALID(alloc) \ ASSERT( \ alloc != NULL && (uintptr_t)alloc != (uintptr_t)-sizeof(MemContextAlloc) && \ alloc->allocIdx < memContextAllocMany(memContextStack[memContextCurrentStackIdx].memContext)->listSize && \ memContextAllocMany(memContextStack[memContextCurrentStackIdx].memContext)->list[alloc->allocIdx]); /*********************************************************************************************************************************** Contains information about the memory context ***********************************************************************************************************************************/ // Quantity of child contexts, allocations, or callbacks typedef enum { memQtyNone = 0, // None for this type memQtyOne = 1, // One for this type memQtyMany = 2, // Many for this type } MemQty; // Main structure required by every mem context struct MemContext { #ifdef DEBUG const char *name; // Indicates what the context is being used for uint64_t sequenceNew; // Sequence when this context was created (used for audit) bool active : 1; // Is the context currently active? #endif MemQty childQty : 2; // How many child contexts can this context have? bool childInitialized : 1; // Has the child context list been initialized? MemQty allocQty : 2; // How many allocations can this context have? bool allocInitialized : 1; // Has the allocation list been initialized? MemQty callbackQty : 2; // How many callbacks can this context have? bool callbackInitialized : 1; // Has the callback been initialized? size_t allocExtra : 16; // Size of extra allocation (1kB max) unsigned int contextParentIdx; // Index in the parent context list MemContext *contextParent; // All contexts have a parent except top }; // Mem context with one allocation typedef struct MemContextAllocOne { MemContextAlloc *alloc; // Memory allocation created in this context } MemContextAllocOne; // Mem context with many allocations typedef struct MemContextAllocMany { MemContextAlloc **list; // List of memory allocations created in this context unsigned int listSize; // Size of alloc list (not the actual count of allocations) unsigned int freeIdx; // Index of first free space in the alloc list } MemContextAllocMany; // Mem context with one child context typedef struct MemContextChildOne { MemContext *context; // Context created in this context } MemContextChildOne; // Mem context with many child contexts typedef struct MemContextChildMany { MemContext **list; // List of contexts created in this context unsigned int listSize; // Size of child context list (not the actual count of contexts) unsigned int freeIdx; // Index of first free space in the context list } MemContextChildMany; // Mem context with one callback typedef struct MemContextCallbackOne { void (*function)(void *); // Function to call before the context is freed void *argument; // Argument to pass to callback function } MemContextCallbackOne; /*********************************************************************************************************************************** Possible sizes for the manifest based on options ***********************************************************************************************************************************/ // {uncrustify_off - formatting compressed to save space} static const uint8_t memContextSizePossible[memQtyMany + 1][memQtyMany + 1][memQtyOne + 1] = { // child none {// alloc none {/* callback none */ 0, /* callback one */ sizeof(MemContextCallbackOne)}, // alloc one {/* callback none */ sizeof(MemContextAllocOne), /* callback one */ sizeof(MemContextAllocOne) + sizeof(MemContextCallbackOne)}, // alloc many {/* callback none */ sizeof(MemContextAllocMany), /* callback one */ sizeof(MemContextAllocMany) + sizeof(MemContextCallbackOne)}}, // child one {// alloc none {/* callback none */ sizeof(MemContextChildOne), /* callback one */ sizeof(MemContextChildOne) + sizeof(MemContextCallbackOne)}, // alloc one {/* callback none */ sizeof(MemContextChildOne) + sizeof(MemContextAllocOne), /* callback one */ sizeof(MemContextChildOne) + sizeof(MemContextAllocOne) + sizeof(MemContextCallbackOne)}, // alloc many {/* callback none */ sizeof(MemContextChildOne) + sizeof(MemContextAllocMany), /* callback one */ sizeof(MemContextChildOne) + sizeof(MemContextAllocMany) + sizeof(MemContextCallbackOne)}}, // child many {// alloc none {/* callback none */ sizeof(MemContextChildMany), /* callback one */ sizeof(MemContextChildMany) + sizeof(MemContextCallbackOne)}, // alloc one {/* callback none */ sizeof(MemContextChildMany) + sizeof(MemContextAllocOne), /* callback one */ sizeof(MemContextChildMany) + sizeof(MemContextAllocOne) + sizeof(MemContextCallbackOne)}, // alloc many {/* callback none */ sizeof(MemContextChildMany) + sizeof(MemContextAllocMany), /* callback one */ sizeof(MemContextChildMany) + sizeof(MemContextAllocMany) + sizeof(MemContextCallbackOne)}}, }; // {uncrustify_on} /*********************************************************************************************************************************** Get pointers to optional parts of the manifest ***********************************************************************************************************************************/ // Get pointer to child part #define MEM_CONTEXT_CHILD_OFFSET(memContext) ((uint8_t *)(memContext + 1) + memContext->allocExtra) static MemContextChildOne * memContextChildOne(MemContext *const memContext) { return (MemContextChildOne *)MEM_CONTEXT_CHILD_OFFSET(memContext); } static MemContextChildMany * memContextChildMany(MemContext *const memContext) { return (MemContextChildMany *)MEM_CONTEXT_CHILD_OFFSET(memContext); } // Get pointer to allocation part #define MEM_CONTEXT_ALLOC_OFFSET(memContext) \ ((uint8_t *)(memContext + 1) + memContextSizePossible[memContext->childQty][0][0] + memContext->allocExtra) static MemContextAllocOne * memContextAllocOne(MemContext *const memContext) { return (MemContextAllocOne *)MEM_CONTEXT_ALLOC_OFFSET(memContext); } static MemContextAllocMany * memContextAllocMany(MemContext *const memContext) { return (MemContextAllocMany *)MEM_CONTEXT_ALLOC_OFFSET(memContext); } // Get pointer to callback part static MemContextCallbackOne * memContextCallbackOne(MemContext *const memContext) { return (MemContextCallbackOne *) ((uint8_t *)(memContext + 1) + memContextSizePossible[memContext->childQty][memContext->allocQty][0] + memContext->allocExtra); } /*********************************************************************************************************************************** Top context The top context always exists and can never be freed. All other contexts are children of the top context. The top context is generally used to allocate memory that exists for the life of the program. ***********************************************************************************************************************************/ static struct MemContextTop { MemContext memContext; MemContextChildMany memContextChildMany; MemContextAllocMany memContextAllocMany; } contextTop = { .memContext = { #ifdef DEBUG .name = "TOP", .active = true, #endif .childQty = memQtyMany, .allocQty = memQtyMany, }, }; /*********************************************************************************************************************************** Memory context stack types ***********************************************************************************************************************************/ typedef enum { memContextStackTypeSwitch = 0, // Context can be switched to allocate mem for new variables memContextStackTypeNew, // Context to be tracked for error handling - cannot switch to } MemContextStackType; /*********************************************************************************************************************************** Mem context stack used to pop mem contexts and cleanup after an error ***********************************************************************************************************************************/ #define MEM_CONTEXT_STACK_MAX 128 static struct MemContextStack { MemContext *memContext; MemContextStackType type; unsigned int tryDepth; } memContextStack[MEM_CONTEXT_STACK_MAX] = {{.memContext = (MemContext *)&contextTop}}; static unsigned int memContextCurrentStackIdx = 0; static unsigned int memContextMaxStackIdx = 0; /*********************************************************************************************************************************** ***********************************************************************************************************************************/ #ifdef DEBUG static uint64_t memContextSequence = 0; FN_EXTERN void memContextAuditBegin(MemContextAuditState *const state) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, state); FUNCTION_TEST_END(); ASSERT(state != NULL); ASSERT(state->memContext != NULL); ASSERT(state->memContext == memContextTop() || state->memContext->sequenceNew != 0); if (state->memContext->childInitialized) { ASSERT(state->memContext->childQty != memQtyNone); if (state->memContext->childQty == memQtyOne) { MemContextChildOne *const memContextChild = memContextChildOne(state->memContext); if (memContextChild->context != NULL) state->sequenceContextNew = memContextChild->context->sequenceNew; } else { ASSERT(state->memContext->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(state->memContext); for (unsigned int contextIdx = 0; contextIdx < memContextChild->listSize; contextIdx++) { if (memContextChild->list[contextIdx] != NULL && memContextChild->list[contextIdx]->sequenceNew > state->sequenceContextNew) { state->sequenceContextNew = memContextChild->list[contextIdx]->sequenceNew; } } } } FUNCTION_TEST_RETURN_VOID(); } static bool memContextAuditNameMatch(const char *const actual, const char *const expected) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, actual); FUNCTION_TEST_PARAM(STRINGZ, expected); FUNCTION_TEST_END(); ASSERT(actual != NULL); ASSERT(expected != NULL); unsigned int actualIdx = 0; while (actual[actualIdx] != '\0' && actual[actualIdx] == expected[actualIdx]) actualIdx++; FUNCTION_TEST_RETURN( BOOL, (actual[actualIdx] == '\0' || strncmp(actual + actualIdx, "::", 2) == 0) && (expected[actualIdx] == '\0' || strcmp(expected + actualIdx, " *") == 0)); } FN_EXTERN void memContextAuditEnd(const MemContextAuditState *const state, const char *const returnType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, state); FUNCTION_TEST_PARAM(STRINGZ, returnType); FUNCTION_TEST_END(); if (state->returnTypeAny) FUNCTION_TEST_RETURN_VOID(); if (state->memContext->childInitialized) { ASSERT(state->memContext->childQty != memQtyNone); const char *returnTypeInvalid = NULL; const char *returnTypeFound = NULL; if (state->memContext->childQty == memQtyOne) { MemContextChildOne *const memContextChild = memContextChildOne(state->memContext); if (memContextChild->context != NULL && memContextChild->context->sequenceNew > state->sequenceContextNew && !memContextAuditNameMatch(memContextChild->context->name, returnType)) { returnTypeInvalid = memContextChild->context->name; } } else { ASSERT(state->memContext->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(state->memContext); for (unsigned int contextIdx = 0; contextIdx < memContextChild->listSize; contextIdx++) { if (memContextChild->list[contextIdx] != NULL && memContextChild->list[contextIdx]->sequenceNew > state->sequenceContextNew) { if (memContextAuditNameMatch(memContextChild->list[contextIdx]->name, returnType)) { if (returnTypeFound != NULL) { returnTypeInvalid = memContextChild->list[contextIdx]->name; break; } returnTypeFound = memContextChild->list[contextIdx]->name; } else { returnTypeInvalid = memContextChild->list[contextIdx]->name; break; } } } } if (returnTypeInvalid != NULL) { if (returnTypeFound != NULL) { THROW_FMT( AssertError, "expected return type '%s' already found but also found '%s'", returnTypeFound, returnTypeInvalid); } else { THROW_FMT( AssertError, "expected return type '%s' but found '%s'", returnType, returnTypeInvalid); } } } FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void * memContextAuditAllocExtraName(void *const allocExtra, const char *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, allocExtra); FUNCTION_TEST_PARAM(STRINGZ, name); FUNCTION_TEST_END(); memContextFromAllocExtra(allocExtra)->name = name; FUNCTION_TEST_RETURN_P(VOID, allocExtra); } #endif /*********************************************************************************************************************************** Wrapper around malloc() with error handling ***********************************************************************************************************************************/ static void * memAllocInternal(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Allocate memory void *const buffer = malloc(size); // Error when malloc fails if (buffer == NULL) THROW_FMT(MemoryError, "unable to allocate %zu bytes", size); // Return the buffer FUNCTION_TEST_RETURN_P(VOID, buffer); } /*********************************************************************************************************************************** Allocate an array of pointers and set all entries to NULL ***********************************************************************************************************************************/ static void * memAllocPtrArrayInternal(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Allocate memory void **const buffer = memAllocInternal(size * sizeof(void *)); // Set all pointers to NULL for (size_t ptrIdx = 0; ptrIdx < size; ptrIdx++) buffer[ptrIdx] = NULL; // Return the buffer FUNCTION_TEST_RETURN_P(VOID, buffer); } /*********************************************************************************************************************************** Wrapper around realloc() with error handling ***********************************************************************************************************************************/ static void * memReAllocInternal(void *const bufferOld, const size_t sizeNew) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, bufferOld); FUNCTION_TEST_PARAM(SIZE, sizeNew); FUNCTION_TEST_END(); ASSERT(bufferOld != NULL); // Allocate memory void *const bufferNew = realloc(bufferOld, sizeNew); // Error when realloc fails if (bufferNew == NULL) THROW_FMT(MemoryError, "unable to reallocate %zu bytes", sizeNew); // Return the buffer FUNCTION_TEST_RETURN_P(VOID, bufferNew); } /*********************************************************************************************************************************** Wrapper around realloc() with error handling ***********************************************************************************************************************************/ static void * memReAllocPtrArrayInternal(void *const bufferOld, const size_t sizeOld, const size_t sizeNew) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, bufferOld); FUNCTION_TEST_PARAM(SIZE, sizeOld); FUNCTION_TEST_PARAM(SIZE, sizeNew); FUNCTION_TEST_END(); // Allocate memory void **const bufferNew = memReAllocInternal(bufferOld, sizeNew * sizeof(void *)); // Set all new pointers to NULL for (size_t ptrIdx = sizeOld; ptrIdx < sizeNew; ptrIdx++) bufferNew[ptrIdx] = NULL; // Return the buffer FUNCTION_TEST_RETURN_P(VOID, bufferNew); } /*********************************************************************************************************************************** Wrapper around free() ***********************************************************************************************************************************/ static void memFreeInternal(void *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_END(); ASSERT(buffer != NULL); free(buffer); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Free list for recycling large allocations Commands that process many files, e.g. backup, churn large uniformly-sized allocations (buffers, etc.) through short-lived contexts. Returning these to the allocator lets it hand the pages back to the kernel, which then faults them back in for the next file. Minor page faults are cheap on some hosts but expensive on others, so this churn can dominate run time and vary from run to run with heap layout. Retaining a bounded number of large freed blocks here keeps the pages resident and makes reuse deterministic, independent of the C library and heap layout. Only allocations at least MEM_FREE_LIST_THRESHOLD are recycled since smaller allocations are handled well by the allocator's own arena and do not cause churn. The buckets are keyed by exact total size, which is a perfect fit for the uniformly-sized allocations that churn. Retained memory is limited by MEM_FREE_LIST_BYTE_MAX so the cost is bounded regardless of buffer size. ***********************************************************************************************************************************/ #define MEM_FREE_LIST_THRESHOLD ((size_t)(64 * 1024)) #define MEM_FREE_LIST_SIZE_MAX 8 #define MEM_FREE_LIST_BLOCK_MAX 8 #define MEM_FREE_LIST_BYTE_MAX ((size_t)(64 * 1024 * 1024)) #define MEM_FREE_LIST_REUSE_MAX 8 typedef struct MemFreeBucket { unsigned int size; // Total block size (including header) held by this bucket unsigned int blockListSize; // Blocks currently retained unsigned int reuse; // Times reused, decremented on each eviction (see below) void *block[MEM_FREE_LIST_BLOCK_MAX]; // Retained block (header) pointers } MemFreeBucket; // The first bucket is reserved (bucketListSize starts at 1) so memFreeReuse() never assigns it so memContextFreeListReserve() can // claim it at any time. It stays unused (size 0) until claimed. static struct MemFreeList { size_t size; // Total size currently retained unsigned int bucketListSize; // Distinct sizes currently tracked (bucket 0 always reserved) MemFreeBucket bucket[MEM_FREE_LIST_SIZE_MAX]; } memFreeList = {.bucketListSize = 1}; /*********************************************************************************************************************************** Allocate, reusing a retained block of the same total size when one is available ***********************************************************************************************************************************/ static void * memAllocReuse(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Reuse a retained block of the same size when large enough to be worth recycling if (size >= MEM_FREE_LIST_THRESHOLD) { for (unsigned int bucketIdx = 0; bucketIdx < memFreeList.bucketListSize; bucketIdx++) { MemFreeBucket *const bucket = &memFreeList.bucket[bucketIdx]; if (bucket->size == size && bucket->blockListSize > 0) { void *const result = bucket->block[--bucket->blockListSize]; memFreeList.size -= size; // Count the reuse up to a limit so a bucket that stops being reused cannot resist eviction indefinitely if (bucket->reuse < MEM_FREE_LIST_REUSE_MAX) bucket->reuse++; // Restore uninitialized-read and bounds detection for the recycled block (matches malloc semantics) VALGRIND_MAKE_MEM_UNDEFINED(result, size); FUNCTION_TEST_RETURN_P(VOID, result); } } } FUNCTION_TEST_RETURN_P(VOID, memAllocInternal(size)); } /*********************************************************************************************************************************** Free, retaining large blocks for reuse instead of returning them to the allocator ***********************************************************************************************************************************/ static void memFreeReuse(MemContextAlloc *const alloc) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, alloc); FUNCTION_TEST_END(); ASSERT(alloc != NULL); // Read the size before the block is poisoned for Valgrind const size_t size = alloc->size; // Retain the block for reuse when large enough and the retained memory limit will not be exceeded if (size >= MEM_FREE_LIST_THRESHOLD && memFreeList.size + size <= MEM_FREE_LIST_BYTE_MAX) { // Find the bucket for this exact size MemFreeBucket *bucket = NULL; for (unsigned int bucketIdx = 0; bucketIdx < memFreeList.bucketListSize; bucketIdx++) { if (memFreeList.bucket[bucketIdx].size == size) { bucket = &memFreeList.bucket[bucketIdx]; break; } } // Create a bucket for this size if there is none and there is room for one. The reuse starts at one so a new bucket is not // evicted before it has had a chance to be reused, which would leave two sizes trading the same slot and never settling. if (bucket == NULL && memFreeList.bucketListSize < MEM_FREE_LIST_SIZE_MAX) { bucket = &memFreeList.bucket[memFreeList.bucketListSize++]; *bucket = (MemFreeBucket){.size = (unsigned int)size, .reuse = 1}; } // Otherwise evict the least reused bucket to claim its slot, ignoring the reserved bucket 0. The reuse of every remaining // bucket is then decremented so a bucket that was reused a lot but is no longer being reused erodes over time and is // eventually evicted. Specialized one-off sizes never get reused so they erode immediately and the buckets settle on the // sizes that recur. else if (bucket == NULL) { unsigned int evictIdx = 1; for (unsigned int bucketIdx = 2; bucketIdx < MEM_FREE_LIST_SIZE_MAX; bucketIdx++) { if (memFreeList.bucket[bucketIdx].reuse < memFreeList.bucket[evictIdx].reuse) evictIdx = bucketIdx; } for (unsigned int bucketIdx = 1; bucketIdx < MEM_FREE_LIST_SIZE_MAX; bucketIdx++) { if (memFreeList.bucket[bucketIdx].reuse > 0) memFreeList.bucket[bucketIdx].reuse--; } bucket = &memFreeList.bucket[evictIdx]; // Return the evicted bucket's retained blocks to the allocator while (bucket->blockListSize > 0) { memFreeInternal(bucket->block[--bucket->blockListSize]); memFreeList.size -= bucket->size; } *bucket = (MemFreeBucket){.size = (unsigned int)size, .reuse = 1}; } // Retain the block if there is room in the bucket if (bucket->blockListSize < MEM_FREE_LIST_BLOCK_MAX) { bucket->block[bucket->blockListSize++] = alloc; memFreeList.size += size; // Poison so use-after-free is still detected while the block is retained VALGRIND_MAKE_MEM_NOACCESS(alloc, size); FUNCTION_TEST_RETURN_VOID(); } } // Otherwise return the block to the allocator memFreeInternal(alloc); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void memContextFreeListReserve(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Reserve the first bucket for allocations of this size (plus the allocation header) so the size that churns most, e.g. // buffer size, always has a bucket and is checked first. The first bucket is permanently reserved and never assigned by // memFreeReuse(), so this can be called at any time, but the bucket must only be reserved once. ASSERT(memFreeList.bucket[0].size == 0); memFreeList.bucket[0].size = (unsigned int)(sizeof(MemContextAlloc) + size); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Find space for a new mem context ***********************************************************************************************************************************/ static unsigned int memContextNewIndex(MemContext *const memContext, MemContextChildMany *const memContextChild) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, memContext); FUNCTION_TEST_PARAM_P(VOID, memContextChild); FUNCTION_TEST_END(); ASSERT(memContext != NULL); ASSERT(memContextChild != NULL); // Initialize (free space will always be index 0) if (!memContext->childInitialized) { *memContextChild = (MemContextChildMany) { .list = memAllocPtrArrayInternal(MEM_CONTEXT_INITIAL_SIZE), .listSize = MEM_CONTEXT_INITIAL_SIZE, }; memContext->childInitialized = true; } else { // Try to find space for the new context for (; memContextChild->freeIdx < memContextChild->listSize; memContextChild->freeIdx++) { if (memContextChild->list[memContextChild->freeIdx] == NULL) break; } // If no space was found then allocate more if (memContextChild->freeIdx == memContextChild->listSize) { // Calculate new list size const unsigned int listSizeNew = memContextChild->listSize * 2; // ReAllocate memory before modifying anything else in case there is an error memContextChild->list = memReAllocPtrArrayInternal(memContextChild->list, memContextChild->listSize, listSizeNew); // Set new list size memContextChild->listSize = listSizeNew; } } FUNCTION_TEST_RETURN(UINT, memContextChild->freeIdx); } /**********************************************************************************************************************************/ FN_EXTERN MemContext * memContextNew( #ifdef DEBUG const char *const name, #endif const MemContextNewParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, name); FUNCTION_TEST_PARAM(UINT, param.childQty); FUNCTION_TEST_PARAM(UINT, param.allocQty); FUNCTION_TEST_PARAM(UINT, param.callbackQty); FUNCTION_TEST_PARAM(SIZE, param.allocExtra); FUNCTION_TEST_END(); ASSERT(name != NULL); ASSERT(param.childQty <= 1 || param.childQty == UINT8_MAX); ASSERT(param.allocQty <= 1 || param.allocQty == UINT8_MAX); ASSERT(param.callbackQty <= 1); // Check context name length ASSERT(name[0] != '\0'); CHECK(AssertError, memContextMaxStackIdx < MEM_CONTEXT_STACK_MAX - 1, "mem context stack is full"); // Pad allocExtra so that optional structures will be aligned. There may not be any optional structures but it doesn't seem // worth the cost of checking since memory allocations are aligned so the extra bytes would be wasted anyway. size_t allocExtra = param.allocExtra; if (allocExtra % alignof(void *) != 0) allocExtra += ALIGN_OFFSET(void *, allocExtra); // Create the new context MemContext *const contextCurrent = memContextStack[memContextCurrentStackIdx].memContext; ASSERT(contextCurrent->childQty != memQtyNone); const MemQty childQty = param.childQty > 1 ? memQtyMany : (MemQty)param.childQty; const MemQty allocQty = param.allocQty > 1 ? memQtyMany : (MemQty)param.allocQty; const MemQty callbackQty = (MemQty)param.callbackQty; MemContext *const this = memAllocInternal( sizeof(MemContext) + allocExtra + memContextSizePossible[childQty][allocQty][callbackQty]); *this = (MemContext) { #ifdef DEBUG // Set the context name .name = name, // Set audit sequence .sequenceNew = ++memContextSequence, // Set new context active .active = true, #endif // Set flags .childQty = childQty, .allocQty = allocQty, .callbackQty = callbackQty, // Set extra allocation .allocExtra = (uint16_t)allocExtra, // Set current context as the parent .contextParent = contextCurrent, }; // Find space for the new context if (contextCurrent->childQty == memQtyOne) { ASSERT(!contextCurrent->childInitialized || memContextChildOne(contextCurrent)->context == NULL); memContextChildOne(contextCurrent)->context = this; contextCurrent->childInitialized = true; } else { ASSERT(contextCurrent->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(contextCurrent); this->contextParentIdx = memContextNewIndex(contextCurrent, memContextChild); memContextChild->list[this->contextParentIdx] = this; // Possible free context must be in the next position memContextChild->freeIdx++; } // Add to the mem context stack so it will be automatically freed on error if memContextKeep() has not been called memContextMaxStackIdx++; memContextStack[memContextMaxStackIdx] = (struct MemContextStack) { .memContext = this, .type = memContextStackTypeNew, .tryDepth = errorTryDepth(), }; // Return context FUNCTION_TEST_RETURN(MEM_CONTEXT, this); } /**********************************************************************************************************************************/ FN_EXTERN void * memContextAllocExtra(MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->allocExtra != 0); FUNCTION_TEST_RETURN_P(VOID, this + 1); } /**********************************************************************************************************************************/ FN_EXTERN MemContext * memContextFromAllocExtra(void *const allocExtra) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, allocExtra); FUNCTION_TEST_END(); ASSERT(allocExtra != NULL); ASSERT(((MemContext *)allocExtra - 1)->allocExtra != 0); FUNCTION_TEST_RETURN(MEM_CONTEXT, (MemContext *)allocExtra - 1); } /**********************************************************************************************************************************/ FN_EXTERN void memContextCallbackSet(MemContext *const this, void (*const callbackFunction)(void *), void *const callbackArgument) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_PARAM(FUNCTIONP, callbackFunction); FUNCTION_TEST_PARAM_P(VOID, callbackArgument); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->active); ASSERT(callbackFunction != NULL); ASSERT(this->active); ASSERT(this->callbackQty != memQtyNone); #ifdef DEBUG // Error if callback has already been set - there may be valid use cases for this in the future but error until one is found if (this->callbackInitialized) THROW_FMT(AssertError, "callback is already set for context '%s'", this->name); #endif // Set callback function and argument *(memContextCallbackOne(this)) = (MemContextCallbackOne){.function = callbackFunction, .argument = callbackArgument}; this->callbackInitialized = true; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void memContextCallbackClear(MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->callbackQty != memQtyNone); ASSERT(this->active); // Clear callback function and argument *(memContextCallbackOne(this)) = (MemContextCallbackOne){0}; this->callbackInitialized = false; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Find an available slot in the memory context's allocation list and allocate memory ***********************************************************************************************************************************/ static MemContextAlloc * memContextAllocNew(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Allocate memory MemContextAlloc *const result = memAllocReuse(sizeof(MemContextAlloc) + size); // Find space for the new allocation MemContext *const contextCurrent = memContextStack[memContextCurrentStackIdx].memContext; ASSERT(contextCurrent->allocQty != memQtyNone); if (contextCurrent->allocQty == memQtyOne) { MemContextAllocOne *const contextAlloc = memContextAllocOne(contextCurrent); ASSERT(!contextCurrent->allocInitialized || contextAlloc->alloc == NULL); // Initialize allocation header *result = (MemContextAlloc){.size = (unsigned int)(sizeof(MemContextAlloc) + size)}; // Set pointer in allocation contextAlloc->alloc = result; contextCurrent->allocInitialized = true; } else { ASSERT(contextCurrent->allocQty == memQtyMany); MemContextAllocMany *const contextAlloc = memContextAllocMany(contextCurrent); // Initialize (free space will always be index 0) if (!contextCurrent->allocInitialized) { *contextAlloc = (MemContextAllocMany) { .list = memAllocPtrArrayInternal(MEM_CONTEXT_ALLOC_INITIAL_SIZE), .listSize = contextAlloc->listSize = MEM_CONTEXT_ALLOC_INITIAL_SIZE, }; contextCurrent->allocInitialized = true; } else { for (; contextAlloc->freeIdx < contextAlloc->listSize; contextAlloc->freeIdx++) if (contextAlloc->list[contextAlloc->freeIdx] == NULL) break; // If no space was found then allocate more if (contextAlloc->freeIdx == contextAlloc->listSize) { // Calculate new list size const unsigned int listSizeNew = contextAlloc->listSize * 2; // Reallocate memory before modifying anything else in case there is an error contextAlloc->list = memReAllocPtrArrayInternal(contextAlloc->list, contextAlloc->listSize, listSizeNew); // Set new size contextAlloc->listSize = listSizeNew; } } // Initialize allocation header *result = (MemContextAlloc){.allocIdx = contextAlloc->freeIdx, .size = (unsigned int)(sizeof(MemContextAlloc) + size)}; // Set pointer in allocation list contextAlloc->list[contextAlloc->freeIdx] = result; // Update free index to next location. This location may not be free but it is where the search should start next time. contextAlloc->freeIdx++; } FUNCTION_TEST_RETURN_TYPE_P(MemContextAlloc, result); } /*********************************************************************************************************************************** Resize memory that has already been allocated ***********************************************************************************************************************************/ static MemContextAlloc * memContextAllocResize(MemContextAlloc *alloc, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, alloc); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Resize the allocation alloc = memReAllocInternal(alloc, sizeof(MemContextAlloc) + size); alloc->size = (unsigned int)(sizeof(MemContextAlloc) + size); // Update pointer in allocation list in case the realloc moved the allocation MemContext *const currentContext = memContextStack[memContextCurrentStackIdx].memContext; ASSERT(currentContext->allocQty != memQtyNone); ASSERT(currentContext->allocInitialized); if (currentContext->allocQty == memQtyOne) { ASSERT(memContextAllocOne(currentContext)->alloc != NULL); memContextAllocOne(currentContext)->alloc = alloc; } else { ASSERT(currentContext->allocQty == memQtyMany); ASSERT_ALLOC_MANY_VALID(alloc); memContextAllocMany(currentContext)->list[alloc->allocIdx] = alloc; } FUNCTION_TEST_RETURN_TYPE_P(MemContextAlloc, alloc); } /**********************************************************************************************************************************/ FN_EXTERN void * memNew(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_P(VOID, MEM_CONTEXT_ALLOC_BUFFER(memContextAllocNew(size))); } /**********************************************************************************************************************************/ FN_EXTERN void * memNewPtrArray(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Allocate pointer array void **const buffer = (void **const)MEM_CONTEXT_ALLOC_BUFFER(memContextAllocNew(size * sizeof(void *))); // Set pointers to NULL for (size_t ptrIdx = 0; ptrIdx < size; ptrIdx++) buffer[ptrIdx] = NULL; FUNCTION_TEST_RETURN_P(VOID, buffer); } /**********************************************************************************************************************************/ FN_EXTERN void * memResize(void *const buffer, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_P(VOID, MEM_CONTEXT_ALLOC_BUFFER(memContextAllocResize(MEM_CONTEXT_ALLOC_HEADER(buffer), size))); } /**********************************************************************************************************************************/ FN_EXTERN void memFree(void *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_END(); // Get the allocation MemContext *const contextCurrent = memContextStack[memContextCurrentStackIdx].memContext; ASSERT(contextCurrent->allocQty != memQtyNone); ASSERT(contextCurrent->allocInitialized); MemContextAlloc *const alloc = MEM_CONTEXT_ALLOC_HEADER(buffer); // Remove allocation from the context if (contextCurrent->allocQty == memQtyOne) { ASSERT(memContextAllocOne(contextCurrent)->alloc == alloc); memContextAllocOne(contextCurrent)->alloc = NULL; } else { ASSERT(contextCurrent->allocQty == memQtyMany); ASSERT_ALLOC_MANY_VALID(alloc); // If this allocation is before the current free allocation then make it the current free allocation MemContextAllocMany *const contextAlloc = memContextAllocMany(contextCurrent); if (alloc->allocIdx < contextAlloc->freeIdx) contextAlloc->freeIdx = alloc->allocIdx; // Null the allocation contextAlloc->list[alloc->allocIdx] = NULL; } // Free the allocation, retaining it for reuse if large enough memFreeReuse(alloc); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void memContextMove(MemContext *const this, MemContext *const parentNew) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_PARAM(MEM_CONTEXT, parentNew); FUNCTION_TEST_END(); ASSERT(parentNew != NULL); // Only move if a valid mem context is provided and the old and new parents are not the same if (this != NULL && this->contextParent != parentNew) { ASSERT(this->active); ASSERT(this->contextParent->active); ASSERT(this->contextParent->childQty != memQtyNone); ASSERT(this->contextParent->childInitialized); // Null out the context in the old parent if (this->contextParent->childQty == memQtyOne) { ASSERT(memContextChildOne(this->contextParent)->context != NULL); memContextChildOne(this->contextParent)->context = NULL; } else { ASSERT(this->contextParent->childQty == memQtyMany); ASSERT(memContextChildMany(this->contextParent)->list[this->contextParentIdx] == this); memContextChildMany(this->contextParent)->list[this->contextParentIdx] = NULL; } // Find a place in the new parent context and assign it ASSERT(parentNew->active); ASSERT(parentNew->childQty != memQtyNone); if (parentNew->childQty == memQtyOne) { ASSERT(!parentNew->childInitialized || memContextChildOne(parentNew)->context == NULL); memContextChildOne(parentNew)->context = this; parentNew->childInitialized = true; } else { ASSERT(parentNew->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(parentNew); // The child list may move while finding a new index so store the index and use it with (what might be) the new pointer this->contextParentIdx = memContextNewIndex(parentNew, memContextChild); memContextChild->list[this->contextParentIdx] = this; } // Set new parent this->contextParent = parentNew; } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void memContextSwitch(MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->active); CHECK(AssertError, memContextMaxStackIdx < MEM_CONTEXT_STACK_MAX - 1, "mem context stack is full"); memContextMaxStackIdx++; memContextCurrentStackIdx = memContextMaxStackIdx; // Add memContext to the stack as a context that can be used for memory allocation memContextStack[memContextCurrentStackIdx] = (struct MemContextStack) { .memContext = this, .type = memContextStackTypeSwitch, .tryDepth = errorTryDepth(), }; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void memContextSwitchBack(void) { FUNCTION_TEST_VOID(); ASSERT(memContextCurrentStackIdx > 0); // Generate a detailed error to help with debugging #ifdef DEBUG if (memContextStack[memContextMaxStackIdx].type == memContextStackTypeNew) { THROW_FMT( AssertError, "current context expected but new context '%s' found", memContextStack[memContextMaxStackIdx].memContext->name); } #endif ASSERT(memContextCurrentStackIdx == memContextMaxStackIdx); memContextMaxStackIdx--; memContextCurrentStackIdx--; // memContext of type New cannot be the current context so keep going until we find a memContext we can switch to as the current // context while (memContextStack[memContextCurrentStackIdx].type == memContextStackTypeNew) memContextCurrentStackIdx--; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void memContextKeep(void) { FUNCTION_TEST_VOID(); // Generate a detailed error to help with debugging #ifdef DEBUG if (memContextStack[memContextMaxStackIdx].type != memContextStackTypeNew) { THROW_FMT( AssertError, "new context expected but current context '%s' found", memContextStack[memContextMaxStackIdx].memContext->name); } #endif memContextMaxStackIdx--; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void memContextDiscard(void) { FUNCTION_TEST_VOID(); // Generate a detailed error to help with debugging #ifdef DEBUG if (memContextStack[memContextMaxStackIdx].type != memContextStackTypeNew) { THROW_FMT( AssertError, "new context expected but current context '%s' found", memContextStack[memContextMaxStackIdx].memContext->name); } #endif memContextFree(memContextStack[memContextMaxStackIdx].memContext); memContextMaxStackIdx--; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN MemContext * memContextTop(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(MEM_CONTEXT, (MemContext *)&contextTop); } /**********************************************************************************************************************************/ FN_EXTERN MemContext * memContextCurrent(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(MEM_CONTEXT, memContextStack[memContextCurrentStackIdx].memContext); } /**********************************************************************************************************************************/ FN_EXTERN MemContext * memContextPrior(void) { FUNCTION_TEST_VOID(); ASSERT(memContextCurrentStackIdx > 0); unsigned int priorIdx = 1; while (memContextStack[memContextCurrentStackIdx - priorIdx].type == memContextStackTypeNew) priorIdx++; FUNCTION_TEST_RETURN(MEM_CONTEXT, memContextStack[memContextCurrentStackIdx - priorIdx].memContext); } /**********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN size_t memContextSize(const MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->active); // Size of struct and extra size_t total = 0; const uint8_t *offset = (const uint8_t *)(this + 1) + this->allocExtra; // Size of child contexts if (this->childQty == memQtyOne) { if (this->childInitialized) { const MemContextChildOne *const contextChild = (const MemContextChildOne *const)offset; if (contextChild->context != NULL) total += memContextSize(contextChild->context); } offset += sizeof(MemContextChildOne); } else if (this->childQty == memQtyMany) { if (this->childInitialized) { const MemContextChildMany *const contextChild = (const MemContextChildMany *const)offset; for (unsigned int contextIdx = 0; contextIdx < contextChild->listSize; contextIdx++) { if (contextChild->list[contextIdx] != NULL) total += memContextSize(contextChild->list[contextIdx]); } total += contextChild->listSize * sizeof(MemContextChildMany *); } offset += sizeof(MemContextChildMany); } // Size of allocations if (this->allocQty == memQtyOne) { if (this->allocInitialized) { const MemContextAllocOne *const contextAlloc = (const MemContextAllocOne *const)offset; if (contextAlloc->alloc != NULL) total += contextAlloc->alloc->size; } offset += sizeof(MemContextAllocOne); } else if (this->allocQty == memQtyMany) { if (this->allocInitialized) { const MemContextAllocMany *const contextAlloc = (const MemContextAllocMany *const)offset; for (unsigned int allocIdx = 0; allocIdx < contextAlloc->listSize; allocIdx++) { if (contextAlloc->list[allocIdx] != NULL) total += contextAlloc->list[allocIdx]->size; } total += contextAlloc->listSize * sizeof(MemContextAllocMany *); } offset += sizeof(MemContextAllocMany); } // Size of callback if (this->callbackQty != memQtyNone) offset += sizeof(MemContextCallbackOne); FUNCTION_TEST_RETURN(SIZE, (size_t)(offset - (const uint8_t *)this) + total); } #endif // DEBUG /**********************************************************************************************************************************/ FN_EXTERN void memContextClean(const unsigned int tryDepth, const bool fatal) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, tryDepth); FUNCTION_TEST_PARAM(BOOL, false); FUNCTION_TEST_END(); ASSERT(tryDepth > 0); // Iterate through everything pushed to the stack since the last try while (memContextStack[memContextMaxStackIdx].tryDepth >= tryDepth) { // Free memory contexts that were not kept. Skip this for fatal errors to avoid calling destructors that could error and // mask the original error. if (memContextStack[memContextMaxStackIdx].type == memContextStackTypeNew) { if (!fatal) memContextFree(memContextStack[memContextMaxStackIdx].memContext); } // Else find the prior context and make it the current context else { memContextCurrentStackIdx--; while (memContextStack[memContextCurrentStackIdx].type == memContextStackTypeNew) memContextCurrentStackIdx--; } memContextMaxStackIdx--; } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ // Helper to execute callbacks for the context and all its children static void memContextCallbackRecurse(MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); #ifdef DEBUG // Certain actions against the context are no longer allowed this->active = false; #endif // Callback if (this->callbackInitialized) { MemContextCallbackOne *const callback = memContextCallbackOne(this); callback->function(callback->argument); this->callbackInitialized = false; } // Child callbacks if (this->childInitialized) { if (this->childQty == memQtyOne) { MemContextChildOne *const memContextChild = memContextChildOne(this); if (memContextChild->context != NULL) memContextCallbackRecurse(memContextChild->context); } else { ASSERT(this->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(this); for (unsigned int contextIdx = 0; contextIdx < memContextChild->listSize; contextIdx++) { if (memContextChild->list[contextIdx] != NULL) memContextCallbackRecurse(memContextChild->list[contextIdx]); } } } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static void memContextFreeRecurse(MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); #ifdef DEBUG // Current context cannot be freed unless it is top (top is never really freed, just the stuff under it) if (this == memContextStack[memContextCurrentStackIdx].memContext && this != memContextTop()) THROW_FMT(AssertError, "cannot free current context '%s'", this->name); #endif // Free child contexts if (this->childInitialized) { if (this->childQty == memQtyOne) { MemContextChildOne *const memContextChild = memContextChildOne(this); if (memContextChild->context != NULL) memContextFreeRecurse(memContextChild->context); } else { ASSERT(this->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(this); for (unsigned int contextIdx = 0; contextIdx < memContextChild->listSize; contextIdx++) { if (memContextChild->list[contextIdx] != NULL) memContextFreeRecurse(memContextChild->list[contextIdx]); } // Free child context allocation list memFreeInternal(memContextChildMany(this)->list); } } // Free memory allocations and list if (this->allocInitialized) { ASSERT(this->allocQty != memQtyNone); if (this->allocQty == memQtyOne) { MemContextAllocOne *const contextAlloc = memContextAllocOne(this); if (contextAlloc->alloc != NULL) memFreeReuse(contextAlloc->alloc); } else { ASSERT(this->allocQty == memQtyMany); MemContextAllocMany *const contextAlloc = memContextAllocMany(this); for (unsigned int allocIdx = 0; allocIdx < contextAlloc->listSize; allocIdx++) if (contextAlloc->list[allocIdx] != NULL) memFreeReuse(contextAlloc->list[allocIdx]); memFreeInternal(contextAlloc->list); } } // Free the memory context so the slot can be reused (if not the top mem context) if (this != memContextTop()) { ASSERT(this->contextParent != NULL); if (this->contextParent->childQty == memQtyOne) memContextChildOne(this->contextParent)->context = NULL; else { ASSERT(this->contextParent->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(this->contextParent); // If the context index is lower than the current free index in the parent then replace it if (this->contextParentIdx < memContextChild->freeIdx) memContextChild->freeIdx = this->contextParentIdx; memContextChildMany(this->contextParent)->list[this->contextParentIdx] = NULL; } memFreeInternal(this); } // Else reset top context. In practice it is uncommon for the top mem context to be freed and then used again. else { this->childInitialized = false; this->allocInitialized = false; #ifdef DEBUG // Make the top mem context active again this->active = true; #endif } FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void memContextFree(MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->active); // Execute callbacks TRY_BEGIN() { memContextCallbackRecurse(this); } // Free context even if a callback fails FINALLY() { memContextFreeRecurse(this); } TRY_END(); FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/memContext.h0000664000175100017510000005062215265101745017762 0ustar00runnerrunner/*********************************************************************************************************************************** Memory Context Manager Memory is allocated inside contexts and all allocations (and child memory contexts) are freed when the context is freed. The goal is to make memory management both easier and more performant. Memory context management is encapsulated in macros so there is rarely any need to call the functions directly. Memory allocations are mostly performed in the constructors of objects and reallocated as needed. See the sections on memory context management and memory allocations below for more details. ***********************************************************************************************************************************/ #ifndef COMMON_MEMCONTEXT_H #define COMMON_MEMCONTEXT_H #include #include #include /*********************************************************************************************************************************** Memory context object ***********************************************************************************************************************************/ typedef struct MemContext MemContext; #include "common/type/param.h" /*********************************************************************************************************************************** Define initial number of memory contexts No space is reserved for child contexts when a new context is created because most contexts will be leaves. When a child context is requested then space will be reserved for this many child contexts initially. When more space is needed the size will be doubled. ***********************************************************************************************************************************/ #define MEM_CONTEXT_INITIAL_SIZE 4 /*********************************************************************************************************************************** Define initial number of memory allocations Space is reserved for this many allocations when a context is created. When more space is needed the size will be doubled. ***********************************************************************************************************************************/ #define MEM_CONTEXT_ALLOC_INITIAL_SIZE 4 /*********************************************************************************************************************************** Functions and macros to audit a mem context by detecting new child contexts/allocations that were created begin the begin/end but are not the expected return type. ***********************************************************************************************************************************/ #if defined(DEBUG) typedef struct MemContextAuditState { MemContext *memContext; // Mem context to audit bool returnTypeAny; // Skip auditing for this mem context uint64_t sequenceContextNew; // Max sequence for new contexts at beginning } MemContextAuditState; // Begin the audit FN_EXTERN void memContextAuditBegin(MemContextAuditState *state); // End the audit and make sure the return type is as expected FN_EXTERN void memContextAuditEnd(const MemContextAuditState *state, const char *returnTypeDefault); // Rename a mem context using the extra allocation pointer #define MEM_CONTEXT_AUDIT_ALLOC_EXTRA_NAME(this, name) memContextAuditAllocExtraName(this, #name) FN_EXTERN void *memContextAuditAllocExtraName(void *allocExtra, const char *name); #else #define MEM_CONTEXT_AUDIT_ALLOC_EXTRA_NAME(this, name) this #endif /*********************************************************************************************************************************** Memory management functions All these functions operate in the current memory context, including memResize() and memFree(). ***********************************************************************************************************************************/ // Allocate memory in the current memory context FN_EXTERN void *memNew(size_t size); // Allocate requested number of pointers and initialize them to NULL FN_EXTERN void *memNewPtrArray(size_t size); // Reallocate to the new size. Original buffer pointer is undefined on return. FN_EXTERN void *memResize(void *buffer, size_t size); // Free memory allocation FN_EXTERN void memFree(void *buffer); // Reserve the first free list bucket for allocations of the given size so the size that churns most, e.g. buffer size, always // has a bucket and is checked first. Must be called at startup before allocations of this size begin. FN_EXTERN void memContextFreeListReserve(size_t size); /*********************************************************************************************************************************** Ensure that the prior memory context is restored after the block executes (even on error) MEM_CONTEXT_BEGIN(memContext) { } MEM_CONTEXT_END(); ***********************************************************************************************************************************/ #define MEM_CONTEXT_BEGIN(memContext) \ do \ { \ memContextSwitch(memContext); /* Switch to the new memory context */ #define MEM_CONTEXT_END() \ memContextSwitchBack(); /* Switch back to the prior context */ \ } \ while (0) /*********************************************************************************************************************************** Switch to prior context and ensure that the previous prior memory context is restored after the block executes (even on error) MEM_CONTEXT_PRIOR_BEGIN(memContext) { } MEM_CONTEXT_PRIOR_END(); ***********************************************************************************************************************************/ #define MEM_CONTEXT_PRIOR_BEGIN() \ MEM_CONTEXT_BEGIN(memContextPrior()) #define MEM_CONTEXT_PRIOR_END() \ MEM_CONTEXT_END() /*********************************************************************************************************************************** Create a new context and make sure it is freed on error and prior context is restored in all cases MEM_CONTEXT_NEW_BEGIN(memContextName, ...) { } MEM_CONTEXT_NEW_END(); Note that memory context names are expected to live for the lifetime of the context -- no copy is made. ***********************************************************************************************************************************/ #define MEM_CONTEXT_NEW() \ MEM_CONTEXT_NEW_memContext #define MEM_CONTEXT_NEW_BEGIN(memContextName, ...) \ do \ { \ MemContext *MEM_CONTEXT_NEW() = memContextNewP(STRINGIFY(memContextName), __VA_ARGS__); \ memContextSwitch(MEM_CONTEXT_NEW()); #define MEM_CONTEXT_NEW_ALLOC() \ memContextAllocExtra(MEM_CONTEXT_NEW()) #define MEM_CONTEXT_NEW_END() \ memContextSwitchBack(); \ memContextKeep(); \ } \ while (0) /*********************************************************************************************************************************** Create a temporary memory context and make sure it is freed when done (even on error) MEM_CONTEXT_TEMP_BEGIN() { } MEM_CONTEXT_TEMP_END(); ***********************************************************************************************************************************/ #define MEM_CONTEXT_TEMP() \ MEM_CONTEXT_TEMP_memContext #define MEM_CONTEXT_TEMP_BEGIN() \ do \ { \ MemContext *MEM_CONTEXT_TEMP() = memContextNewP( \ "temporary", .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX); \ memContextSwitch(MEM_CONTEXT_TEMP()); #define MEM_CONTEXT_TEMP_RESET_BEGIN() \ MEM_CONTEXT_TEMP_BEGIN() \ unsigned int MEM_CONTEXT_TEMP_loopTotal = 0; #define MEM_CONTEXT_TEMP_RESET(resetTotal) \ do \ { \ MEM_CONTEXT_TEMP_loopTotal++; \ \ if (MEM_CONTEXT_TEMP_loopTotal >= resetTotal) \ { \ memContextSwitchBack(); \ memContextDiscard(); \ MEM_CONTEXT_TEMP() = memContextNewP("temporary", .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX); \ memContextSwitch(MEM_CONTEXT_TEMP()); \ MEM_CONTEXT_TEMP_loopTotal = 0; \ } \ } \ while (0) #define MEM_CONTEXT_TEMP_END() \ memContextSwitchBack(); \ memContextDiscard(); \ } \ while (0) /*********************************************************************************************************************************** Memory context management functions memContextSwitch(memContextNewP()); memContextSwitchBack(); memContextKeep()/memContextDiscard(); There is no need to implement any error handling. The mem context system will automatically clean up any mem contexts that were created but not marked as keep when an error occurs and reset the current mem context to whatever it was at the beginning of the nearest try block. Use the MEM_CONTEXT*() macros when possible rather than reimplement the boilerplate for every memory context block. ***********************************************************************************************************************************/ // Create a new mem context in the current mem context. The new context must be either kept with memContextKeep() or discarded with // memContextDiscard() before switching back from the parent context. typedef struct MemContextNewParam { VAR_PARAM_HEADER; uint8_t childQty; // How many child contexts can this context have? uint8_t allocQty; // How many allocations can this context have? uint8_t callbackQty; // How many callbacks can this context have? uint16_t allocExtra; // Extra memory to allocate with the context } MemContextNewParam; // Maximum amount of extra memory that can be allocated with the context using allocExtra #define MEM_CONTEXT_ALLOC_EXTRA_MAX UINT16_MAX // Specify maximum quantity of child contexts or allocations using childQty or allocQty #define MEM_CONTEXT_QTY_MAX UINT8_MAX #ifdef DEBUG #define memContextNewP(name, ...) \ memContextNew(name, (MemContextNewParam){VAR_PARAM_INIT, __VA_ARGS__}) #else #define memContextNewP(name, ...) \ memContextNew((MemContextNewParam){VAR_PARAM_INIT, __VA_ARGS__}) #endif FN_EXTERN MemContext *memContextNew( #ifdef DEBUG const char *name, #endif MemContextNewParam param); // Switch to a context making it the current mem context FN_EXTERN void memContextSwitch(MemContext *this); // Switch back to the context that was current before the last switch. If the last function called was memContextNewP(), then // memContextKeep()/memContextDiscard() must be called before calling memContextSwitchBack(), otherwise an error will occur in // debug builds and the behavior is undefined in production builds. FN_EXTERN void memContextSwitchBack(void); // Keep a context created by memContextNewP() so that it will not be automatically freed if an error occurs. If the context was // switched after the call to memContextNewP(), then it must be switched back before calling memContextKeep() or an error will occur // in debug builds and the behavior is undefined in production builds. FN_EXTERN void memContextKeep(void); // Discard a context created by memContextNewP(). If the context was switched after the call to memContextNewP(), then it must be // switched back before calling memContextDiscard() or an error will occur in debug builds and the behavior is undefined in // production builds. FN_EXTERN void memContextDiscard(void); // Move mem context to a new parent. This is generally used to move objects to a new context once they have been successfully // created. FN_EXTERN void memContextMove(MemContext *this, MemContext *parentNew); // Set a function that will be called when this mem context is freed FN_EXTERN void memContextCallbackSet(MemContext *this, void (*callbackFunction)(void *), void *); // Clear the callback function so it won't be called when the mem context is freed. This is usually done in the object free method // after resources have been freed but before memContextFree() is called. The goal is to prevent the object free method from being // called more than once. FN_EXTERN void memContextCallbackClear(MemContext *this); // Free a memory context FN_EXTERN void memContextFree(MemContext *this); /*********************************************************************************************************************************** Memory context getters ***********************************************************************************************************************************/ // Pointer to the extra memory allocated with the mem context FN_EXTERN void *memContextAllocExtra(MemContext *this); // Get mem context using pointer to the memory allocated with the mem context FN_EXTERN MemContext *memContextFromAllocExtra(void *allocExtra); // Current memory context FN_EXTERN MemContext *memContextCurrent(void); // Prior context, i.e. the context that was current before the last memContextSwitch() FN_EXTERN MemContext *memContextPrior(void); // "top" context. This context is created at initialization and is always present, i.e. it is never freed. The top context is a good // place to put long-lived mem contexts since they won't be automatically freed until the program exits. FN_EXTERN MemContext *memContextTop(void); // Get total size of mem context and all children #ifdef DEBUG FN_EXTERN size_t memContextSize(const MemContext *this); #endif // DEBUG /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_MEM_CONTEXT_TYPE \ MemContext * #define FUNCTION_LOG_MEM_CONTEXT_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "MemContext", buffer, bufferSize) /*********************************************************************************************************************************** Internal functions ***********************************************************************************************************************************/ // Clean up mem contexts after an error. Should only be called from error handling routines. FN_EXTERN void memContextClean(unsigned int tryDepth, bool fatal); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/regExp.c0000664000175100017510000001412315265101745017060 0ustar00runnerrunner/*********************************************************************************************************************************** Regular Expression Handler ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/regExp.h" /*********************************************************************************************************************************** Contains information about the regular expression handler ***********************************************************************************************************************************/ struct RegExp { regex_t regExp; }; /*********************************************************************************************************************************** Free regular expression ***********************************************************************************************************************************/ static void regExpFreeResource(THIS_VOID) { THIS(RegExp); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(REGEXP, this); FUNCTION_TEST_END(); ASSERT(this != NULL); regfree(&this->regExp); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Handle errors ***********************************************************************************************************************************/ static void regExpError(const int error) { char buffer[4096]; regerror(error, NULL, buffer, sizeof(buffer)); THROW(FormatError, buffer); } static void regExpErrorCheck(const int error) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, error); FUNCTION_TEST_END(); if (error != 0 && error != REG_NOMATCH) regExpError(error); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN RegExp * regExpNew(const String *const expression) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, expression); FUNCTION_TEST_END(); ASSERT(expression != NULL); OBJ_NEW_BEGIN(RegExp, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (RegExp){{0}}; // Extra braces are required for older gcc versions // Compile the regexp and process errors int result = 0; if ((result = regcomp(&this->regExp, strZ(expression), REG_EXTENDED)) != 0) regExpError(result); // Set free callback to ensure cipher context is freed memContextCallbackSet(objMemContext(this), regExpFreeResource, this); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(REGEXP, this); } /**********************************************************************************************************************************/ FN_EXTERN bool regExpMatch(RegExp *const this, const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(REGEXP, this); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(string != NULL); // Test for a match regmatch_t matchPtr; const int result = regexec(&this->regExp, strZ(string), 1, &matchPtr, 0); // Check for an error regExpErrorCheck(result); FUNCTION_TEST_RETURN(BOOL, result == 0); } /**********************************************************************************************************************************/ FN_EXTERN bool regExpMatchOne(const String *const expression, const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, expression); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); ASSERT(expression != NULL); ASSERT(string != NULL); bool result; MEM_CONTEXT_TEMP_BEGIN() { result = regExpMatch(regExpNew(expression), string); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN String * regExpPrefix(const String *const expression) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, expression); FUNCTION_TEST_END(); String *result = NULL; // Only generate prefix if expression is defined and has a beginning anchor if (expression != NULL && strZ(expression)[0] == '^') { const char *const expressionZ = strZ(expression); const size_t expressionSize = strSize(expression); unsigned int expressionIdx = 1; for (; expressionIdx < expressionSize; expressionIdx++) { char expressionChr = expressionZ[expressionIdx]; // Search for characters that will end the prefix if (expressionChr == '.' || expressionChr == '^' || expressionChr == '$' || expressionChr == '*' || expressionChr == '+' || expressionChr == '-' || expressionChr == '?' || expressionChr == '(' || expressionChr == '[' || expressionChr == '{' || expressionChr == ' ' || expressionChr == '|' || expressionChr == '\\') { break; } } // Will there be any characters in the prefix? if (expressionIdx > 1) { // Search the rest of the string for another begin anchor unsigned int anchorIdx = expressionIdx; for (; anchorIdx < expressionSize; anchorIdx++) { // [^ and \^ are not begin anchors if (expressionZ[anchorIdx] == '^' && expressionZ[anchorIdx - 1] != '[' && expressionZ[anchorIdx - 1] != '\\') break; } // If no other begin anchor was found then the prefix is usable if (anchorIdx == expressionSize) result = strSubN(expression, 1, expressionIdx - 1); } } FUNCTION_TEST_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/regExp.h0000664000175100017510000000617015265101745017070 0ustar00runnerrunner/*********************************************************************************************************************************** Regular Expression Handler ***********************************************************************************************************************************/ #ifndef COMMON_REGEXP_H #define COMMON_REGEXP_H /*********************************************************************************************************************************** RegExp object ***********************************************************************************************************************************/ typedef struct RegExp RegExp; #include "common/type/object.h" #include "common/type/string.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN RegExp *regExpNew(const String *expression); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Match on a regular expression FN_EXTERN bool regExpMatch(RegExp *this, const String *string); /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Match a regular expression in one call for brevity FN_EXTERN bool regExpMatchOne(const String *expression, const String *string); // Return the common prefix of a regular expression, if it has one. The common prefix consists of fixed characters that must always // be found at the beginning of the string to be matched. Escaped characters will not be included in the prefix. If there is no // usable prefix then NULL is returned. FN_EXTERN String *regExpPrefix(const String *expression); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void regExpFree(RegExp *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_REGEXP_TYPE \ RegExp * #define FUNCTION_LOG_REGEXP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "RegExp", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/stackTrace.c0000664000175100017510000003465015265101745017721 0ustar00runnerrunner/*********************************************************************************************************************************** Stack Trace Handler ***********************************************************************************************************************************/ #include #include #include #include #ifdef HAVE_LIBBACKTRACE #include #endif #include "common/assert.h" #include "common/macro.h" #include "common/stackTrace.h" /*********************************************************************************************************************************** Max call stack depth ***********************************************************************************************************************************/ #define STACK_TRACE_MAX 128 /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ // Stack trace function data typedef struct StackTraceData { const char *fileName; const char *functionName; unsigned int fileLine; LogLevel functionLogLevel; unsigned int tryDepth; bool paramOverflow; bool paramLog; char *param; size_t paramSize; } StackTraceData; static struct StackTraceLocal { int stackSize; // Stack size StackTraceData stack[STACK_TRACE_MAX]; // Stack data char functionParamBuffer[32 * 1024]; // Buffer to hold function parameters #ifdef HAVE_LIBBACKTRACE struct backtrace_state *backTraceState; // Backtrace state struct #endif } stackTraceLocal; /**********************************************************************************************************************************/ #ifdef DEBUG static struct StackTraceTestLocal { bool testFlag; // Don't log in parameter logging functions to avoid recursion } stackTraceTestLocal = {.testFlag = true}; FN_EXTERN void stackTraceTestStart(void) { stackTraceTestLocal.testFlag = true; } FN_EXTERN void stackTraceTestStop(void) { stackTraceTestLocal.testFlag = false; } FN_EXTERN bool stackTraceTest(void) { return stackTraceTestLocal.testFlag; } FN_EXTERN void stackTraceTestFileLineSet(unsigned int fileLine) { ASSERT(stackTraceLocal.stackSize > 0); stackTraceLocal.stack[stackTraceLocal.stackSize - 1].fileLine = fileLine; } #endif /**********************************************************************************************************************************/ FN_EXTERN LogLevel stackTracePush(const char *const fileName, const char *const functionName, const LogLevel functionLogLevel) { ASSERT(stackTraceLocal.stackSize < STACK_TRACE_MAX - 1); // Set function info StackTraceData *const data = &stackTraceLocal.stack[stackTraceLocal.stackSize]; *data = (StackTraceData) { .fileName = fileName, .functionName = functionName, .tryDepth = errorTryDepth(), }; // Set param pointer if (stackTraceLocal.stackSize == 0) { data->param = stackTraceLocal.functionParamBuffer; data->functionLogLevel = functionLogLevel; } else { StackTraceData *const dataPrior = &stackTraceLocal.stack[stackTraceLocal.stackSize - 1]; data->param = dataPrior->param + dataPrior->paramSize + 1; // Log level cannot be lower than the prior function if (functionLogLevel < dataPrior->functionLogLevel) data->functionLogLevel = dataPrior->functionLogLevel; else data->functionLogLevel = functionLogLevel; } stackTraceLocal.stackSize++; return data->functionLogLevel; } /**********************************************************************************************************************************/ static const char * stackTraceParamIdx(const int stackIdx) { ASSERT(stackTraceLocal.stackSize > 0); ASSERT(stackIdx < stackTraceLocal.stackSize); StackTraceData *const data = &stackTraceLocal.stack[stackIdx]; if (data->paramLog) { if (data->paramOverflow) return "buffer full - parameters not available"; if (data->paramSize == 0) return "void"; return data->param; } // If no parameters return the log level required to get them #define LOG_LEVEL_REQUIRED " log level required for parameters" return data->functionLogLevel == logLevelTrace ? "trace" LOG_LEVEL_REQUIRED : "debug" LOG_LEVEL_REQUIRED; } FN_EXTERN const char * stackTraceParam(void) { return stackTraceParamIdx(stackTraceLocal.stackSize - 1); } /**********************************************************************************************************************************/ FN_EXTERN char * stackTraceParamBuffer(const char *const paramName) { ASSERT(stackTraceLocal.stackSize > 0); StackTraceData *const data = &stackTraceLocal.stack[stackTraceLocal.stackSize - 1]; const size_t paramNameSize = strlen(paramName); // Make sure that adding this parameter will not overflow the buffer if ((size_t)(data->param - stackTraceLocal.functionParamBuffer) + data->paramSize + paramNameSize + 4 > sizeof(stackTraceLocal.functionParamBuffer) - (STACK_TRACE_PARAM_MAX * 2)) { // Set overflow to true data->paramOverflow = true; // There's no way to stop the parameter from being formatted so we reserve a space at the end where the format can safely // take place and not disturb the rest of the buffer. Hopefully overflows just won't happen but we need to be prepared in // case of runaway recursion or some other issue that fills the buffer because we don't want a segfault. return stackTraceLocal.functionParamBuffer + sizeof(stackTraceLocal.functionParamBuffer) - STACK_TRACE_PARAM_MAX; } // Add a comma if a parameter is already in the list if (data->paramSize != 0) { data->param[data->paramSize++] = ','; data->param[data->paramSize++] = ' '; } // Add the parameter name memcpy(data->param + data->paramSize, paramName, paramNameSize); data->paramSize += paramNameSize; // Add param/value separator data->param[data->paramSize++] = ':'; data->param[data->paramSize++] = ' '; return data->param + data->paramSize; } /**********************************************************************************************************************************/ FN_EXTERN void stackTraceParamAdd(const size_t bufferSize) { ASSERT(stackTraceLocal.stackSize > 0); StackTraceData *data = &stackTraceLocal.stack[stackTraceLocal.stackSize - 1]; if (!data->paramOverflow) data->paramSize += bufferSize; } /**********************************************************************************************************************************/ FN_EXTERN void stackTraceParamLog(void) { ASSERT(stackTraceLocal.stackSize > 0); stackTraceLocal.stack[stackTraceLocal.stackSize - 1].paramLog = true; } /**********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN void stackTracePop(const char *const fileName, const char *const functionName, const bool test) { ASSERT(stackTraceLocal.stackSize > 0); if (!test || stackTraceTest()) { stackTraceLocal.stackSize--; StackTraceData *data = &stackTraceLocal.stack[stackTraceLocal.stackSize]; if (strcmp(data->fileName, fileName) != 0 || strcmp(data->functionName, functionName) != 0) THROW_FMT(AssertError, "popping %s:%s but expected %s:%s", fileName, functionName, data->fileName, data->functionName); } } #else FN_EXTERN void stackTracePop(void) { stackTraceLocal.stackSize--; } #endif /*********************************************************************************************************************************** Stack trace format ***********************************************************************************************************************************/ static FN_PRINTF(4, 5) size_t stackTraceFmt(char *const buffer, const size_t bufferSize, const size_t bufferUsed, const char *const format, ...) { va_list argumentList; va_start(argumentList, format); const int result = vsnprintf( buffer + bufferUsed, bufferUsed < bufferSize ? bufferSize - bufferUsed : 0, format, argumentList); va_end(argumentList); return (size_t)result; } /*********************************************************************************************************************************** Helper to trim off extra path before the src path ***********************************************************************************************************************************/ static const char * stackTraceTrimSrc(const char *const fileName) { const char *const src = strstr(fileName, "src/"); return src == NULL ? fileName : src + 4; } /**********************************************************************************************************************************/ #ifdef HAVE_LIBBACKTRACE typedef struct StackTraceBackData { bool firstCall; bool firstLine; int stackIdx; size_t result; char *const buffer; const size_t bufferSize; } StackTraceBackData; // Callback to add backtrace data when available static int stackTraceBackCallback( void *const dataVoid, const uintptr_t pc, const char *fileName, const int fileLine, const char *const functionName) { (void)pc; StackTraceBackData *const data = dataVoid; // Catch any unset parameters which indicates the debug data is not available if (fileName == NULL || fileLine == 0 || functionName == NULL) { // If this is the first call then stop because the top of the backtrace must be one of our functions if (data->firstCall) return true; // Else return but do not stop data->firstCall = false; return false; } // Reset first call data->firstCall = false; // If the function name matches combine backtrace data with stack data if (data->stackIdx >= 0 && strcmp(functionName, stackTraceLocal.stack[data->stackIdx].functionName) == 0) { data->result += stackTraceFmt( data->buffer, data->bufferSize, data->result, "%s%s:%s:%d:(%s)", data->firstLine ? "" : "\n", stackTraceTrimSrc(stackTraceLocal.stack[data->stackIdx].fileName), functionName, fileLine, stackTraceParamIdx(data->stackIdx)); data->stackIdx--; } // Else just use stack data. Skip any functions in the error module since they are not useful for the user else { fileName = stackTraceTrimSrc(fileName); if (strcmp(fileName, "common/error/error.c") == 0) return false; data->result += stackTraceFmt( data->buffer, data->bufferSize, data->result, "%s%s:%s:%d:(no parameters available)", data->firstLine ? "" : "\n", fileName, functionName, fileLine); } // Reset first line data->firstLine = false; // Stop when the main function has been processed return strcmp(functionName, "main") == 0; } // Dummy error callback. If there is an error just generate the default stack trace. static void stackTraceBackErrorCallback(void *const data, const char *const msg, const int errnum) { (void)data; (void)msg; (void)errnum; } #endif FN_EXTERN size_t stackTraceToZ( char *const buffer, const size_t bufferSize, const char *fileName, const char *const functionName, const unsigned int fileLine) { #ifdef HAVE_LIBBACKTRACE // Attempt to use backtrace data StackTraceBackData data = { .firstCall = true, .firstLine = true, .stackIdx = stackTraceLocal.stackSize - 1, .result = 0, .buffer = buffer, .bufferSize = bufferSize, }; if (stackTraceLocal.backTraceState == NULL) stackTraceLocal.backTraceState = backtrace_create_state(NULL, false, NULL, NULL); backtrace_full(stackTraceLocal.backTraceState, 2, stackTraceBackCallback, stackTraceBackErrorCallback, &data); if (data.result != 0) return data.result; #endif // HAVE_LIBBACKTRACE size_t result = 0; const char *param = "test build required for parameters"; int stackIdx = stackTraceLocal.stackSize - 1; // If the current function passed in is the same as the top function on the stack then use the parameters for that function fileName = stackTraceTrimSrc(fileName); if (stackTraceLocal.stackSize > 0 && strcmp(fileName, stackTraceTrimSrc(stackTraceLocal.stack[stackIdx].fileName)) == 0 && strcmp(functionName, stackTraceLocal.stack[stackIdx].functionName) == 0) { param = stackTraceParamIdx(stackTraceLocal.stackSize - 1); stackIdx--; } // Output the current function result = stackTraceFmt(buffer, bufferSize, 0, "%s:%s:%u:(%s)", fileName, functionName, fileLine, param); // Output stack if there is anything on it if (stackIdx >= 0) { // If the function passed in was not at the top of the stack then some functions are missing if (stackIdx == stackTraceLocal.stackSize - 1) result += stackTraceFmt(buffer, bufferSize, result, "\n ... function(s) omitted ..."); // Output the rest of the stack for (; stackIdx >= 0; stackIdx--) { const StackTraceData *const data = &stackTraceLocal.stack[stackIdx]; result += stackTraceFmt(buffer, bufferSize, result, "\n%s:%s", stackTraceTrimSrc(data->fileName), data->functionName); if (data->fileLine > 0) result += stackTraceFmt(buffer, bufferSize, result, ":%u", data->fileLine); result += stackTraceFmt(buffer, bufferSize, result, ":(%s)", stackTraceParamIdx(stackIdx)); } } return result; } /**********************************************************************************************************************************/ FN_EXTERN void stackTraceClean(const unsigned int tryDepth, const bool fatal) { (void)fatal; // Cleanup is the same for fatal errors while (stackTraceLocal.stackSize > 0 && stackTraceLocal.stack[stackTraceLocal.stackSize - 1].tryDepth >= tryDepth) stackTraceLocal.stackSize--; } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/stackTrace.h0000664000175100017510000000677415265101745017734 0ustar00runnerrunner/*********************************************************************************************************************************** Stack Trace Handler ***********************************************************************************************************************************/ #ifndef COMMON_STACKTRACE_H #define COMMON_STACKTRACE_H #include #include #include "common/logLevel.h" /*********************************************************************************************************************************** Maximum size of a single parameter (including NULL terminator) ***********************************************************************************************************************************/ #define STACK_TRACE_PARAM_MAX 4096 /*********************************************************************************************************************************** Macros to access internal functions ***********************************************************************************************************************************/ #define STACK_TRACE_PUSH(logLevel) \ stackTracePush(__FILE__, __func__, logLevel) #ifdef DEBUG #define STACK_TRACE_POP(test) \ stackTracePop(__FILE__, __func__, test); #else #define STACK_TRACE_POP(test) \ stackTracePop(); #endif /*********************************************************************************************************************************** Internal Functions ***********************************************************************************************************************************/ #ifdef DEBUG // Enable/disable test function logging FN_EXTERN void stackTraceTestStart(void); FN_EXTERN void stackTraceTestStop(void); FN_EXTERN bool stackTraceTest(void); // Set line number for the current function on the stack FN_EXTERN void stackTraceTestFileLineSet(unsigned int fileLine); #else // Must always be valid since it might be needed to compile (though not used) during profiling #define stackTraceTestFileLineSet(fileLine) #endif // Push a new function onto the trace stack FN_EXTERN LogLevel stackTracePush(const char *fileName, const char *functionName, LogLevel functionLogLevel); // Pop a function from the trace stack #ifdef DEBUG FN_EXTERN void stackTracePop(const char *fileName, const char *functionName, bool test); #else FN_EXTERN void stackTracePop(void); #endif // Generate the stack trace FN_EXTERN size_t stackTraceToZ( char *buffer, size_t bufferSize, const char *fileName, const char *functionName, unsigned int fileLine); // Mark that parameters are being logged -- if none appear then the function is void FN_EXTERN void stackTraceParamLog(void); // Get parameters for the top function on the stack FN_EXTERN const char *stackTraceParam(void); // Get the next location where a parameter can be added in the param buffer FN_EXTERN char *stackTraceParamBuffer(const char *param); // Add a parameter to the function on the top of the stack FN_EXTERN void stackTraceParamAdd(size_t bufferSize); // Clean the stack at and below the try level. Called by the error to cleanup the stack when an exception occurs. FN_EXTERN void stackTraceClean(unsigned int tryDepth, bool fatal); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/stat.c0000664000175100017510000001101415265101745016575 0ustar00runnerrunner/*********************************************************************************************************************************** Statistics Collector ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/memContext.h" #include "common/stat.h" #include "common/type/json.h" #include "common/type/list.h" /*********************************************************************************************************************************** Cumulative statistics ***********************************************************************************************************************************/ typedef struct Stat { const String *key; uint64_t total; } Stat; /*********************************************************************************************************************************** Local data ***********************************************************************************************************************************/ static struct { MemContext *memContext; // Mem context to store data in this struct List *stat; // Cumulative stats } statLocalData; /**********************************************************************************************************************************/ FN_EXTERN void statInit(void) { FUNCTION_TEST_VOID(); ASSERT(statLocalData.memContext == NULL); MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(StatLocalData, .childQty = MEM_CONTEXT_QTY_MAX) { statLocalData.memContext = MEM_CONTEXT_NEW(); statLocalData.stat = lstNewP(sizeof(Stat), .sortOrder = sortOrderAsc, .comparator = lstComparatorStr); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Get the specified stat. If it doesn't already exist it will be created. ***********************************************************************************************************************************/ static Stat * statGetOrCreate(const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(key != NULL); // Attempt to find the stat Stat *stat = lstFind(statLocalData.stat, &key); // If not found then create it if (stat == NULL) { // Add the new stat MEM_CONTEXT_BEGIN(lstMemContext(statLocalData.stat)) { lstAdd(statLocalData.stat, &(Stat){.key = strDup(key)}); } MEM_CONTEXT_END(); // Sort stats so this stat will be easier to find later lstSort(statLocalData.stat, sortOrderAsc); // The stat might have moved so we'll need to find it and make sure we have the correct pointer stat = lstFind(statLocalData.stat, &key); ASSERT(stat != NULL); } FUNCTION_TEST_RETURN_TYPE_P(Stat, stat); } /**********************************************************************************************************************************/ FN_EXTERN void statInc(const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(statLocalData.memContext != NULL); ASSERT(key != NULL); statGetOrCreate(key)->total++; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * statToJson(void) { FUNCTION_TEST_VOID(); ASSERT(statLocalData.memContext != NULL); String *result = NULL; if (!lstEmpty(statLocalData.stat)) { result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP(.json = result)); for (unsigned int statIdx = 0; statIdx < lstSize(statLocalData.stat); statIdx++) { const Stat *const stat = lstGet(statLocalData.stat, statIdx); jsonWriteObjectBegin(jsonWriteKey(json, stat->key)); jsonWriteUInt64(jsonWriteKeyZ(json, "total"), stat->total); jsonWriteObjectEnd(json); } jsonWriteObjectEnd(json); } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/stat.h0000664000175100017510000000273515265101745016614 0ustar00runnerrunner/*********************************************************************************************************************************** Statistics Collector Collect simple statistics that can be output to a KeyValue for processing and logging. Each stat has a String that identifies it uniquely and will also be used in the output. Individual stats do not need to be created in advance since they will be created as needed at runtime. However, statInit() must be called before any other stat*() functions. NOTE: Statistics are held in a sorted list so there is some cost involved in each lookup. In general, statistics should be used for relatively important or high-latency operations where measurements are critical. For instance, using statistics to count the iterations of a loop would likely be a bad idea. ***********************************************************************************************************************************/ #ifndef COMMON_STAT_H #define COMMON_STAT_H #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize the stats collector FN_EXTERN void statInit(void); // Increment stat by one FN_EXTERN void statInc(const String *key); // Output stats to JSON FN_EXTERN String *statToJson(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/time.c0000664000175100017510000001602315265101745016565 0ustar00runnerrunner/*********************************************************************************************************************************** Time Management ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/time.h" /*********************************************************************************************************************************** Constants describing number of sub-units in an interval ***********************************************************************************************************************************/ #define MSEC_PER_USEC ((TimeMSec)1000) /**********************************************************************************************************************************/ FN_EXTERN TimeMSec timeMSec(void) { FUNCTION_TEST_VOID(); struct timeval currentTime; gettimeofday(¤tTime, NULL); FUNCTION_TEST_RETURN(TIME_MSEC, ((TimeMSec)currentTime.tv_sec * MSEC_PER_SEC) + (TimeMSec)currentTime.tv_usec / MSEC_PER_USEC); } /**********************************************************************************************************************************/ FN_EXTERN void sleepMSec(const TimeMSec sleepMSec) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, sleepMSec); FUNCTION_TEST_END(); if (sleepMSec > 0) { const struct timespec delay = { .tv_sec = (time_t)(sleepMSec / MSEC_PER_SEC), .tv_nsec = (long)(sleepMSec % MSEC_PER_SEC * 1000000), }; nanosleep(&delay, NULL); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void datePartsValid(const int year, const int month, const int day) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, year); FUNCTION_TEST_PARAM(INT, month); FUNCTION_TEST_PARAM(INT, day); FUNCTION_TEST_END(); static const int daysPerMonth[2][12] = { {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}, {31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}, }; if (!(year >= 1970 && month > 0 && month <= 12 && day > 0 && day <= daysPerMonth[yearIsLeap(year) ? 1 : 0][month - 1])) THROW_FMT(FormatError, "invalid date %04d-%02d-%02d", year, month, day); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void timePartsValid(const int hour, const int minute, const int second) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, hour); FUNCTION_TEST_PARAM(INT, minute); FUNCTION_TEST_PARAM(INT, second); FUNCTION_TEST_END(); if (!(hour >= 0 && hour < 24 && minute >= 0 && minute < 60 && second >= 0 && second < 60)) THROW_FMT(FormatError, "invalid time %02d:%02d:%02d", hour, minute, second); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void tzPartsValid(const int tzHour, const int tzMinute) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, tzHour); // signed hour part of timezone FUNCTION_TEST_PARAM(INT, tzMinute); // minutes part of timezone FUNCTION_TEST_END(); // Valid time zones range from GMT-12 all the way to GMT+14 (i.e. -1200 and +1400 are the min/max). // ??? This is only a sanity check for basic validity of timezone offset of 15 minute intervals until the timezone // database is implemented. if (!(((tzHour > -12 && tzHour < 14) && (tzMinute % 15 == 0)) || (tzHour == -12 && tzMinute == 0) || (tzHour == 14 && tzMinute == 0))) { THROW_FMT(FormatError, "invalid timezone %02d%02d", tzHour, tzMinute); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN int tzOffsetSeconds(int tzHour, const int tzMinute) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, tzHour); // signed hour part of timezone (e.g. -7) FUNCTION_TEST_PARAM(INT, tzMinute); // minutes part of timezone FUNCTION_TEST_END(); // Validate the timezone hour and minute tzPartsValid(tzHour, tzMinute); int sign = 1; // Preserve the sign and convert the hours to a positive number for calculating seconds if (tzHour < 0) { sign = -1; tzHour = sign * tzHour; } FUNCTION_TEST_RETURN(INT, sign * (tzHour * 3600 + tzMinute * 60)); } /**********************************************************************************************************************************/ FN_EXTERN bool yearIsLeap(const int year) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, year); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, (year % 4 == 0 && year % 100 != 0) || (year % 400 == 0)); } /**********************************************************************************************************************************/ FN_EXTERN int dayOfYear(const int year, const int month, const int day) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, year); FUNCTION_TEST_PARAM(INT, month); FUNCTION_TEST_PARAM(INT, day); FUNCTION_TEST_END(); datePartsValid(year, month, day); static const int cumulativeDaysPerMonth[2][12] = { {0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334}, {0, 31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335}, }; FUNCTION_TEST_RETURN(INT, cumulativeDaysPerMonth[yearIsLeap(year) ? 1 : 0][month - 1] + day); } /**********************************************************************************************************************************/ FN_EXTERN time_t epochFromParts( const int year, const int month, const int day, const int hour, const int minute, const int second, const int tzOffsetSecond) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, year); FUNCTION_TEST_PARAM(INT, month); FUNCTION_TEST_PARAM(INT, day); FUNCTION_TEST_PARAM(INT, hour); FUNCTION_TEST_PARAM(INT, minute); FUNCTION_TEST_PARAM(INT, second); FUNCTION_TEST_PARAM(INT, tzOffsetSecond); FUNCTION_TEST_END(); timePartsValid(hour, minute, second); // Return epoch using calculation from https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_16 FUNCTION_TEST_RETURN( TIME, -1 * tzOffsetSecond + second + minute * 60 + hour * 3600 + (dayOfYear(year, month, day) - 1) * 86400 + (year - 1900 - 70) * 31536000 + ((year - 1900 - 69) / 4) * 86400 - ((year - 1900 - 1) / 100) * 86400 + ((year - 1900 + 299) / 400) * 86400); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/time.h0000664000175100017510000000636415265101745016601 0ustar00runnerrunner/*********************************************************************************************************************************** Time Management ***********************************************************************************************************************************/ #ifndef COMMON_TIME_H #define COMMON_TIME_H #include #include #include /*********************************************************************************************************************************** Time types ***********************************************************************************************************************************/ typedef uint64_t TimeMSec; /*********************************************************************************************************************************** Constants describing number of sub-units in an interval ***********************************************************************************************************************************/ #define MSEC_PER_SEC ((TimeMSec)1000) #define SEC_PER_DAY ((time_t)86400) /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Sleep for specified milliseconds FN_EXTERN void sleepMSec(TimeMSec sleepMSec); // Epoch time in milliseconds FN_EXTERN TimeMSec timeMSec(void); // Are the date parts valid? (year >= 1970, month 1-12, day 1-31) FN_EXTERN void datePartsValid(int year, int month, int day); // Are the time parts valid? FN_EXTERN void timePartsValid(int hour, int minute, int second); // Are the timezone offset parts valid? FN_EXTERN void tzPartsValid(int tzHour, int tzMinute); // Given the hours, minutes and sign of a time zone (e.g. -0700 => -7, 0) return the signed number or seconds (e.g. -25200) FN_EXTERN int tzOffsetSeconds(int tzHour, int tzMinute); // Is the year a leap year? FN_EXTERN bool yearIsLeap(int year); // Get days since the beginning of the year (year >= 1970, month 1-12, day 1-31), returns 1-366 FN_EXTERN int dayOfYear(int year, int month, int day); // Return epoch time from date/time parts (year >= 1970, month 1-12, day 1-31, hour 0-23, minute 0-59, second 0-59, tzOffsetSecond // is the number of seconds plus or minus (+/-) the provided time - if 0, then the datetime is assumed to be UTC) FN_EXTERN time_t epochFromParts(int year, int month, int day, int hour, int minute, int second, int tzOffsetSecond); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_TIME_MSEC_TYPE \ TimeMSec #define FUNCTION_LOG_TIME_MSEC_FORMAT(value, buffer, bufferSize) \ cvtUInt64ToZ(value, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/0000775000175100017510000000000015265101745016442 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/blob.c0000664000175100017510000000511115265101745017522 0ustar00runnerrunner/*********************************************************************************************************************************** Blob Handler ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/memContext.h" #include "common/type/blob.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Blob { char *block; // Current block for writing size_t pos; // Position in current block }; /**********************************************************************************************************************************/ FN_EXTERN Blob * blbNew(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(Blob, .allocQty = MEM_CONTEXT_QTY_MAX) { *this = (Blob) { .block = memNew(BLOB_BLOCK_SIZE), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(BLOB, this); } /**********************************************************************************************************************************/ FN_EXTERN const void * blbAdd(Blob *const this, const void *const data, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BLOB, this); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); void *result = NULL; MEM_CONTEXT_OBJ_BEGIN(this) { // If data is >= block size then allocate a special block for the data if (size >= BLOB_BLOCK_SIZE) { result = memNew(size); memcpy(result, data, size); } else { // If data will fit in the current block if (BLOB_BLOCK_SIZE - this->pos >= size) { result = (uint8_t *)this->block + this->pos; memcpy(result, data, size); this->pos += size; } // Else allocate a new block else { this->block = memNew(BLOB_BLOCK_SIZE); result = this->block; memcpy(result, data, size); this->pos = size; } } } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN_P(VOID, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/blob.h0000664000175100017510000000627215265101745017540 0ustar00runnerrunner/*********************************************************************************************************************************** Blob Handler Efficiently store large numbers of small allocations by packing them onto larger blocks. Note that the allocations will not be aligned unless all of them are aligned. A usage example is to store zero-terminated strings, which do not need to be aligned, and where alignment might waste significant space. Note that this code is fairly primitive, and this should probably be a capability of memory contexts. However, for now it reduces memory requirements for large numbers of zero-terminated strings. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_BLOB_H #define COMMON_TYPE_BLOB_H /*********************************************************************************************************************************** Size of blocks allocated for blob data ***********************************************************************************************************************************/ #ifndef BLOB_BLOCK_SIZE #define BLOB_BLOCK_SIZE (64 * 1024) #endif /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Blob Blob; #include "common/type/object.h" /*********************************************************************************************************************************** Constructor ***********************************************************************************************************************************/ FN_EXTERN Blob *blbNew(void); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add data to the blob FN_EXTERN const void *blbAdd(Blob *this, const void *data, size_t size); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void strBlbFree(Blob *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_BLOB_TYPE \ Blob * #define FUNCTION_LOG_BLOB_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Blob", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/buffer.c0000664000175100017510000003015115265101745020057 0ustar00runnerrunner/*********************************************************************************************************************************** Buffer Handler ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/type/buffer.h" #include "common/valgrind.h" /*********************************************************************************************************************************** Constant buffers that are generally useful ***********************************************************************************************************************************/ BUFFER_STRDEF_EXTERN(BRACEL_BUF, "{"); BUFFER_STRDEF_EXTERN(BRACER_BUF, "}"); BUFFER_STRDEF_EXTERN(BRACKETL_BUF, "["); BUFFER_STRDEF_EXTERN(BRACKETR_BUF, "]"); BUFFER_STRDEF_EXTERN(COMMA_BUF, ","); BUFFER_STRDEF_EXTERN(EQ_BUF, "="); BUFFER_STRDEF_EXTERN(LF_BUF, "\n"); BUFFER_STRDEF_EXTERN(QUOTED_BUF, "\""); /*********************************************************************************************************************************** Contains information about the buffer ***********************************************************************************************************************************/ struct Buffer { BufferPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufNew(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(Buffer, .allocQty = 1) { *this = (Buffer) { .pub = { .sizeAlloc = size, .size = size, }, }; // Allocate buffer if (size > 0) this->pub.buffer = memNew(this->pub.sizeAlloc); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufNewC(const void *const buffer, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(buffer != NULL); // Create object and copy data Buffer *const this = bufNew(size); memcpy(this->pub.buffer, buffer, bufSize(this)); this->pub.used = bufSize(this); FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufNewDecode(const EncodingType type, const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); Buffer *const this = bufNew(decodeToBinSize(type, strZ(string))); decodeToBin(type, strZ(string), bufPtr(this)); bufUsedSet(this, bufSize(this)); FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufDup(const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(buffer != NULL); // Create object and copy data Buffer *const this = bufNew(bufUsed(buffer)); if (bufUsed(buffer) != 0) memcpy(this->pub.buffer, buffer->pub.buffer, bufSize(this)); this->pub.used = bufSize(this); FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufCat(Buffer *const this, const Buffer *const cat) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(BUFFER, cat); FUNCTION_TEST_END(); ASSERT(this != NULL); if (cat != NULL) bufCatC(this, cat->pub.buffer, 0, bufUsed(cat)); FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufCatC(Buffer *const this, const uint8_t *const cat, const size_t catOffset, const size_t catSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM_P(BYTEDATA, cat); FUNCTION_TEST_PARAM(SIZE, catOffset); FUNCTION_TEST_PARAM(SIZE, catSize); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(catSize == 0 || cat != NULL); if (catSize > 0) { if (bufUsed(this) + catSize > bufSize(this)) bufResize(this, bufUsed(this) + catSize); // Just here to silence nonnull warnings from clang static analyzer ASSERT(bufPtr(this) != NULL); memcpy(bufPtr(this) + bufUsed(this), cat + catOffset, catSize); this->pub.used += catSize; } FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufCatSub(Buffer *const this, const Buffer *const cat, const size_t catOffset, const size_t catSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(BUFFER, cat); FUNCTION_TEST_PARAM(SIZE, catOffset); FUNCTION_TEST_PARAM(SIZE, catSize); FUNCTION_TEST_END(); ASSERT(this != NULL); if (cat != NULL) { ASSERT(catOffset <= bufUsed(cat)); ASSERT(catSize <= bufUsed(cat) - catOffset); bufCatC(this, cat->pub.buffer, catOffset, catSize); } FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN bool bufEq(const Buffer *const this, const Buffer *const compare) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(BUFFER, compare); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(compare != NULL); if (bufUsed(this) == bufUsed(compare)) FUNCTION_TEST_RETURN(BOOL, memcmp(bufPtrConst(this), bufPtrConst(compare), bufUsed(compare)) == 0); FUNCTION_TEST_RETURN(BOOL, false); } /**********************************************************************************************************************************/ FN_EXTERN const uint8_t * bufFind(const Buffer *const this, const Buffer *const find, const BufFindParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(BUFFER, find); FUNCTION_TEST_PARAM_P(VOID, param.begin); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(find != NULL); ASSERT(param.begin == NULL || (param.begin >= bufPtrConst(this) && param.begin - bufPtrConst(this) <= (off_t)bufUsed(this))); const void *result = NULL; if (bufUsed(this) >= bufUsed(find)) { const uint8_t *haystack = param.begin != NULL ? param.begin : bufPtrConst(this); unsigned int findIdx = (unsigned int)(haystack - bufPtrConst(this)); for (; findIdx <= bufUsed(this) - bufUsed(find); findIdx++) { if (memcmp(haystack, bufPtrConst(find), bufSize(find)) == 0) { result = haystack; break; } haystack++; } } FUNCTION_TEST_RETURN_CONST_P(BYTEDATA, result); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufResize(Buffer *const this, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(this != NULL); // Only resize if it the new size is different if (bufSizeAlloc(this) != size) { // If new size is zero then free memory if allocated if (size == 0) { // When setting size down to 0 the buffer should always be allocated ASSERT(bufPtrConst(this) != NULL); MEM_CONTEXT_OBJ_BEGIN(this) { memFree(bufPtr(this)); } MEM_CONTEXT_OBJ_END(); this->pub.buffer = NULL; this->pub.sizeAlloc = 0; } // Else allocate or resize else { MEM_CONTEXT_OBJ_BEGIN(this) { if (bufPtrConst(this) == NULL) this->pub.buffer = memNew(size); else this->pub.buffer = memResize(bufPtr(this), size); } MEM_CONTEXT_OBJ_END(); this->pub.sizeAlloc = size; } if (bufUsed(this) > bufSizeAlloc(this)) this->pub.used = bufSizeAlloc(this); if (!bufSizeLimit(this)) this->pub.size = bufSizeAlloc(this); else if (bufSize(this) > bufSizeAlloc(this)) this->pub.size = bufSizeAlloc(this); } FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN void bufLimitClear(Buffer *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); this->pub.sizeLimit = false; this->pub.size = bufSizeAlloc(this); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void bufLimitSet(Buffer *const this, const size_t limit) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(SIZE, limit); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(limit <= bufSizeAlloc(this)); ASSERT(limit >= bufUsed(this)); this->pub.size = limit; this->pub.sizeLimit = limit != bufSizeAlloc(this); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void bufUsedInc(Buffer *const this, const size_t inc) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(SIZE, inc); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(bufUsed(this) + inc <= bufSize(this)); this->pub.used += inc; FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void bufUsedSet(Buffer *const this, const size_t used) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(SIZE, used); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(used <= bufSize(this)); // Invalidate the unused area so Valgrind will warn on a read from it VALGRIND_MAKE_MEM_UNDEFINED(bufPtr(this) + used, bufSize(this) - used); this->pub.used = used; FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void bufUsedZero(Buffer *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Invalidate the unused area so Valgrind will warn on a read from it VALGRIND_MAKE_MEM_UNDEFINED(bufPtr(this), bufSize(this)); this->pub.used = 0; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void bufToLog(const Buffer *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{used: %zu, size: %zu", bufUsed(this), bufSize(this)); if (bufSizeLimit(this)) strStcFmt(debugLog, ", sizeAlloc: %zu", bufSizeAlloc(this)); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/buffer.h0000664000175100017510000002506015265101745020067 0ustar00runnerrunner/*********************************************************************************************************************************** Buffer Handler ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_BUFFER_H #define COMMON_TYPE_BUFFER_H /*********************************************************************************************************************************** Buffer object ***********************************************************************************************************************************/ typedef struct Buffer Buffer; #include "common/memContext.h" #include "common/type/object.h" #include "common/type/string.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Buffer *bufNew(size_t size); // Create a new buffer from a C buffer FN_EXTERN Buffer *bufNewC(const void *buffer, size_t size); // Create a new buffer from a string encoded with the specified type FN_EXTERN Buffer *bufNewDecode(EncodingType type, const String *string); // Duplicate a buffer FN_EXTERN Buffer *bufDup(const Buffer *buffer); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct BufferPub { size_t sizeAlloc; // Allocated size of the buffer size_t size; // Reported size of the buffer bool sizeLimit; // Is the size limited to make the buffer appear smaller? size_t used; // Amount of buffer used uint8_t *buffer; // Buffer } BufferPub; // Amount of the buffer actually used. This will be updated automatically when possible but if the buffer is modified by using // bufPtr() then the user is responsible for updating used. FN_INLINE_ALWAYS size_t bufUsed(const Buffer *const this) { return THIS_PUB(Buffer)->used; } // Is the buffer empty? FN_INLINE_ALWAYS bool bufEmpty(const Buffer *const this) { return bufUsed(this) == 0; } // Buffer size FN_INLINE_ALWAYS size_t bufSize(const Buffer *const this) { return THIS_PUB(Buffer)->size; } // Is the buffer full? FN_INLINE_ALWAYS bool bufFull(const Buffer *const this) { return bufUsed(this) == bufSize(this); } // Buffer pointer FN_INLINE_ALWAYS uint8_t * bufPtr(Buffer *const this) { return THIS_PUB(Buffer)->buffer; } // Const buffer pointer FN_INLINE_ALWAYS const uint8_t * bufPtrConst(const Buffer *const this) { return THIS_PUB(Buffer)->buffer; } // Remaining space in the buffer FN_INLINE_ALWAYS size_t bufRemains(const Buffer *const this) { return bufSize(this) - bufUsed(this); } // Pointer to remaining buffer space (after used space) FN_INLINE_ALWAYS uint8_t * bufRemainsPtr(Buffer *const this) { return bufPtr(this) + bufUsed(this); } // Allocated buffer size. This may be different from bufSize() if a limit has been set. FN_INLINE_ALWAYS size_t bufSizeAlloc(const Buffer *const this) { return THIS_PUB(Buffer)->sizeAlloc; } // Is the size limited to make the buffer appear smaller? FN_INLINE_ALWAYS bool bufSizeLimit(const Buffer *const this) { return THIS_PUB(Buffer)->sizeLimit; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Append the contents of another buffer FN_EXTERN Buffer *bufCat(Buffer *this, const Buffer *cat); // Append a C buffer FN_EXTERN Buffer *bufCatC(Buffer *this, const uint8_t *cat, size_t catOffset, size_t catSize); // Append a subset of another buffer FN_EXTERN Buffer *bufCatSub(Buffer *this, const Buffer *cat, size_t catOffset, size_t catSize); // Are two buffers equal? FN_EXTERN bool bufEq(const Buffer *this, const Buffer *compare); // Find a buffer in another buffer typedef struct BufFindParam { VAR_PARAM_HEADER; const uint8_t *begin; // Begin find from this address } BufFindParam; #define bufFindP(this, find, ...) \ bufFind(this, find, (BufFindParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN const uint8_t *bufFind(const Buffer *this, const Buffer *find, BufFindParam param); // Move to a new parent mem context FN_INLINE_ALWAYS Buffer * bufMove(Buffer *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Resize the buffer FN_EXTERN Buffer *bufResize(Buffer *this, size_t size); // Manage buffer limits FN_EXTERN void bufLimitClear(Buffer *this); FN_EXTERN void bufLimitSet(Buffer *this, size_t limit); FN_EXTERN void bufUsedInc(Buffer *this, size_t inc); FN_EXTERN void bufUsedSet(Buffer *this, size_t used); FN_EXTERN void bufUsedZero(Buffer *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void bufFree(Buffer *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for constant buffers Frequently used constant buffers can be declared with these macros at compile time rather than dynamically at run time. Note that buffers created in this way are declared as const so can't be modified or freed by the buf*() methods. Casting to Buffer * will result in a segfault. By convention all buffer constant identifiers are appended with _BUF. ***********************************************************************************************************************************/ // This struct must be kept in sync with BufferPub (except for const qualifiers) typedef struct BufferPubConst { size_t sizeAlloc; // Allocated size of the buffer size_t size; // Reported size of the buffer bool sizeLimit; // Is the size limited to make the buffer appear smaller? size_t used; // Amount of buffer used const uint8_t *buffer; // Buffer } BufferPubConst; // Create a buffer constant inline from an uint8_t[] #define BUF(bufferParam, sizeParam) \ ((const Buffer *)&(const BufferPubConst){.size = sizeParam, .used = sizeParam, .buffer = (const uint8_t *)bufferParam}) // Create a buffer constant inline from a non-constant zero-terminated string #define BUFSTRZ(stringz) \ BUF((const uint8_t *)stringz, strlen(stringz)) // Create a buffer constant inline from a String #define BUFSTR(string) \ BUF((const uint8_t *)strZ(string), strSize(string)) // Create a buffer constant inline from a constant zero-terminated string #define BUFSTRDEF(stringdef) \ BUF((const uint8_t *)stringdef, (sizeof(stringdef) - 1)) // Used to define buffer constants that will be externed using BUFFER_DECLARE(). Must be used in a .c file. #define BUFFER_EXTERN(name, ...) \ static const uint8_t name##_RAW[] = {__VA_ARGS__}; \ VR_EXTERN_DEFINE const Buffer *const name = BUF(name##_RAW, sizeof(name##_RAW)); // Used to define String Buffer constants that will be externed using BUFFER_DECLARE(). Must be used in a .c file. #define BUFFER_STRDEF_EXTERN(name, string) \ VR_EXTERN_DEFINE const Buffer *const name = BUFSTRDEF(string) // Used to define String Buffer constants that will be local to the .c file. Must be used in a .c file. #define BUFFER_STRDEF_STATIC(name, string) \ static const Buffer *const name = BUFSTRDEF(string) // Used to declare externed Buffer constants defined with BUFFER*EXTERN(). Must be used in a .h file. #define BUFFER_DECLARE(name) \ VR_EXTERN_DECLARE const Buffer *const name /*********************************************************************************************************************************** Constant buffers that are generally useful ***********************************************************************************************************************************/ BUFFER_DECLARE(BRACEL_BUF); BUFFER_DECLARE(BRACER_BUF); BUFFER_DECLARE(BRACKETL_BUF); BUFFER_DECLARE(BRACKETR_BUF); BUFFER_DECLARE(COMMA_BUF); BUFFER_DECLARE(EQ_BUF); BUFFER_DECLARE(LF_BUF); BUFFER_DECLARE(QUOTED_BUF); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void bufToLog(const Buffer *this, StringStatic *debugLog); #define FUNCTION_LOG_BUFFER_TYPE \ Buffer * #define FUNCTION_LOG_BUFFER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, bufToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/convert.c0000664000175100017510000007034415265101745020276 0ustar00runnerrunner/*********************************************************************************************************************************** Convert C Types ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include #include "common/debug.h" #include "common/macro.h" #include "common/time.h" #include "common/type/convert.h" /*********************************************************************************************************************************** Check results of strto*() function for: * leading/trailing spaces * invalid characters * blank string * error in errno ***********************************************************************************************************************************/ static void cvtZToIntValid(const int errNo, const int base, const char *const value, const char *const endPtr, const char *const type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, errNo); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(STRINGZ, endPtr); FUNCTION_TEST_PARAM(STRINGZ, type); FUNCTION_TEST_END(); ASSERT(value != NULL); ASSERT(endPtr != NULL); if (errNo != 0 || *value == '\0' || isspace(*value) || *endPtr != '\0') THROW_FMT(FormatError, "unable to convert base %d string '%s' to %s", base, value, type); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Convert zero-terminated string to int64 and validate result ***********************************************************************************************************************************/ static int64_t cvtZToInt64Internal(const char *const value, const char *const type, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(STRINGZ, type); FUNCTION_TEST_END(); ASSERT(value != NULL); ASSERT(type != NULL); // Convert from string errno = 0; char *endPtr = NULL; const int64_t result = strtoll(value, &endPtr, base); // Validate the result cvtZToIntValid(errno, base, value, endPtr, type); FUNCTION_TEST_RETURN(INT64, result); } /*********************************************************************************************************************************** Convert zero-terminated string to uint64 and validate result ***********************************************************************************************************************************/ static uint64_t cvtZToUInt64Internal(const char *const value, const char *const type, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(STRINGZ, type); FUNCTION_TEST_END(); ASSERT(value != NULL); ASSERT(type != NULL); // Convert from string errno = 0; char *endPtr = NULL; const uint64_t result = strtoull(value, &endPtr, base); // Validate the result cvtZToIntValid(errno, base, value, endPtr, type); FUNCTION_TEST_RETURN(UINT64, result); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtBoolToZ(const bool value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%s", cvtBoolToConstZ(value)); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } FN_EXTERN const char * cvtBoolToConstZ(bool value) { return value ? TRUE_Z : FALSE_Z; } /*********************************************************************************************************************************** Round an integer contained in a string ***********************************************************************************************************************************/ static size_t cvtRound(size_t result, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, result); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(bufferSize > 0); for (int roundIdx = (int)result - 2; roundIdx >= 0; roundIdx--) { // Round when the rounding digit is >= 5 (current digit needs to be incremented) or the current digit > 9 (prior carry // overflowed and needs to be carried again) if (((roundIdx == (int)result - 2) && buffer[roundIdx + 1] >= '5') || buffer[roundIdx] > '9') { // Carry to the prior digit if (buffer[roundIdx] >= '9') { // If this is the first digit then add a new prior digit to carry to. Since it will start as zero we can just // set it to one. if (roundIdx == 0) { if (result + 1 >= bufferSize) THROW(AssertError, "buffer overflow"); memmove(buffer + 1, buffer, ++result); buffer[1] = '0'; buffer[0] = '1'; } // Else set current digit to zero and carry to prior digit. An overflow is handled on the next iteration. else { buffer[roundIdx] = '0'; buffer[roundIdx - 1]++; } } // Else increment current digit else buffer[roundIdx]++; } // Else stop rounding else break; } // Remove rightmost digit used to start rounding result--; buffer[result] = '\0'; // Return string length FUNCTION_TEST_RETURN(SIZE, result); } /*********************************************************************************************************************************** Separate the fractional part of an integer container in a string ***********************************************************************************************************************************/ static size_t cvtFraction(size_t result, const unsigned int precision, const bool trim, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, result); FUNCTION_TEST_PARAM(UINT, precision); FUNCTION_TEST_PARAM(BOOL, trim); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(bufferSize > 0); // Add decimal point if (result + 1 >= bufferSize) THROW(AssertError, "buffer overflow"); memmove(buffer + result - precision + 1, buffer + result - precision, precision + 1); buffer[result - precision] = '.'; buffer[++result] = '\0'; // Strip off any final 0s and the decimal point if there are no non-zero digits after it if (trim) { char *end = buffer + result - 1; while (*end == '0' || *end == '.') { // It should not be possible to go past the beginning because a decimal point is always written ASSERT(end > buffer); end--; if (*(end + 1) == '.') break; } // Zero terminate the string end[1] = 0; // Calculate length result = (size_t)(end - buffer + 1); } // Return string length FUNCTION_TEST_RETURN(SIZE, result); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cvtPctToUInt(const uint64_t dividend, const uint64_t divisor) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, dividend); FUNCTION_TEST_PARAM(UINT64, divisor); FUNCTION_TEST_END(); ASSERT(dividend <= divisor); // If 100% then return a fixed value to avoid any rounding throwing off the result if (dividend == divisor) FUNCTION_TEST_RETURN(UINT, 10000); // Calculate percentage char buffer[CVT_PCT_BUFFER_SIZE]; size_t size = (size_t)snprintf(buffer, sizeof(buffer), "%04" PRIu64, (uint64_t)((double)dividend / (double)divisor * 100000)); // Round cvtRound(size, buffer, sizeof(buffer)); FUNCTION_TEST_RETURN(UINT, cvtZToUInt(buffer)); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtPctToZ(const uint64_t dividend, const uint64_t divisor, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, dividend); FUNCTION_TEST_PARAM(UINT64, divisor); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(bufferSize > 0); ASSERT(dividend <= divisor); // Calculate percentage as an integer size_t result = (size_t)snprintf(buffer, bufferSize, "%03u", cvtPctToUInt(dividend, divisor)); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); // Separate fractional part result = cvtFraction(result, 2, false, buffer, bufferSize); // Add percent sign if (result + 1 >= bufferSize) THROW(AssertError, "buffer overflow"); buffer[result++] = '%'; buffer[result] = '\0'; // Return string length FUNCTION_TEST_RETURN(SIZE, result); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtDivToZ( const uint64_t dividend, const uint64_t divisor, const unsigned int precision, const bool trim, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, dividend); FUNCTION_TEST_PARAM(UINT64, divisor); FUNCTION_TEST_PARAM(UINT, precision); FUNCTION_TEST_PARAM(BOOL, trim); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(bufferSize > 0); // Determine multiplier for precision digits unsigned int multiplier = 1; switch (precision) { case 0: break; case 1: multiplier = 10; break; case 2: multiplier = 100; break; default: CHECK_FMT(AssertError, precision <= 3, "precision %u is invalid", precision); multiplier = 1000; break; } CHECK_FMT(AssertError, dividend <= UINT64_MAX / multiplier, "dividend %" PRIu64 " is too large", dividend); // If possible add a digit for rounding bool round = false; if (dividend <= UINT64_MAX / (multiplier * 10)) { round = true; multiplier *= 10; } // Convert to string. Zero pad to leave a digit before the fractional part, plus another for the rounding digit when one was // added, so that a quotient less than one is not left without an integer digit. size_t result = (size_t)snprintf( buffer, bufferSize, "%0*" PRIu64, (int)precision + (round ? 2 : 1), dividend * multiplier / divisor); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); // Round if (round) result = cvtRound(result, buffer, bufferSize); // Separate fractional part if (precision > 0) result = cvtFraction(result, precision, trim, buffer, bufferSize); // Return string length FUNCTION_TEST_RETURN(SIZE, result); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtIntToZ(const int value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%d", value); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } FN_EXTERN int cvtZToIntBase(const char *const value, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); const int64_t result = cvtZToInt64Internal(value, "int", base); if (result > INT_MAX || result < INT_MIN) THROW_FMT(FormatError, "unable to convert base %d string '%s' to int", base, value); FUNCTION_TEST_RETURN(INT, (int)result); } FN_EXTERN int cvtZToInt(const char *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); FUNCTION_TEST_RETURN(INT, cvtZToIntBase(value, 10)); } FN_EXTERN int cvtZSubNToIntBase(const char *const value, const size_t offset, const size_t size, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(SIZE, offset); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_PARAM(INT, base); FUNCTION_TEST_END(); ASSERT(value != NULL); char buffer[CVT_BASE10_BUFFER_SIZE + 1]; ASSERT(size <= CVT_BASE10_BUFFER_SIZE); memcpy(buffer, value + offset, size); buffer[size] = '\0'; FUNCTION_TEST_RETURN(INT, cvtZToIntBase(buffer, base)); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtInt64ToZ(const int64_t value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT64, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%" PRId64, value); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } FN_EXTERN int64_t cvtZToInt64Base(const char *const value, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); FUNCTION_TEST_RETURN(INT64, cvtZToInt64Internal(value, "int64", base)); } FN_EXTERN int64_t cvtZToInt64(const char *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); FUNCTION_TEST_RETURN(INT64, cvtZToInt64Base(value, 10)); } FN_EXTERN int64_t cvtZSubNToInt64Base(const char *const value, const size_t offset, const size_t size, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(SIZE, offset); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_PARAM(INT, base); FUNCTION_TEST_END(); ASSERT(value != NULL); char buffer[CVT_BASE10_BUFFER_SIZE + 1]; ASSERT(size <= CVT_BASE10_BUFFER_SIZE); memcpy(buffer, value + offset, size); buffer[size] = '\0'; FUNCTION_TEST_RETURN(INT64, cvtZToInt64Base(buffer, base)); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtModeToZ(const mode_t value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MODE, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%04o", value); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } FN_EXTERN mode_t cvtZToMode(const char *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); FUNCTION_TEST_RETURN(MODE, (mode_t)cvtZToUIntBase(value, 8)); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtSizeToZ(const size_t value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%zu", value); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtTimeToZ(const char *const format, const time_t value, char *const buffer, const size_t bufferSize, const CvtTimeToZParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_PARAM(TIME, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_PARAM(BOOL, param.utc); FUNCTION_TEST_END(); ASSERT(buffer != NULL); // Musl libc does not behave like other C libraries when formatting %s as output from gmtime_r() so forbid it entirely, see // https://www.openwall.com/lists/musl/2025/06/02/3 for details. ASSERT(!param.utc || strstr(format, "%s") == NULL); struct tm timePart; // We can ignore this warning here since the format parameter of cvtTimeToZP() is checked #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wformat-nonliteral" const size_t result = strftime( buffer, bufferSize, format, param.utc ? gmtime_r(&value, &timePart) : localtime_r(&value, &timePart)); #pragma GCC diagnostic pop if (result == 0) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } /**********************************************************************************************************************************/ // Helper to convert a time part, e.g. year static int cvtZNToTimePart(const char *const time, const char *const part, const size_t partSize) { int result = 0; int power = 1; for (size_t partIdx = partSize - 1; partIdx < partSize; partIdx--) { if (!isdigit(part[partIdx])) THROW_FMT(FormatError, "invalid date/time %s", time); result += (part[partIdx] - '0') * power; power *= 10; } return result; } FN_EXTERN time_t cvtZToTime(const char *const time) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, time); FUNCTION_TEST_END(); ASSERT(time != NULL); // Validate structure of date/time if (strlen(time) < 19 || time[4] != '-' || time[7] != '-' || time[10] != ' ' || time[13] != ':' || time[16] != ':') THROW_FMT(FormatError, "invalid date/time %s", time); // Parse date/time const int year = cvtZNToTimePart(time, time, 4); const int month = cvtZNToTimePart(time, time + 5, 2); const int day = cvtZNToTimePart(time, time + 8, 2); const int hour = cvtZNToTimePart(time, time + 11, 2); const int minute = cvtZNToTimePart(time, time + 14, 2); const int second = cvtZNToTimePart(time, time + 17, 2); // Confirm date and time parts are valid datePartsValid(year, month, day); timePartsValid(hour, minute, second); // Consume milliseconds when present (they are omitted from the result) const char *part = time + 19; if ((*part == '.' || *part == ',') && strlen(part) >= 2) { part++; while (*part != 0 && isdigit(*part)) part++; } // Add timezone offset when present if ((*part == '+' || *part == '-') && strlen(part) >= 3) { const int offsetHour = cvtZNToTimePart(time, part + 1, 2) * (*part == '-' ? -1 : 1); part += 3; // Offset separator is optional if (*part == ':') part++; // Offset minutes are optional int offsetMinute = 0; if (strlen(part) == 2) { offsetMinute = cvtZNToTimePart(time, part, 2); part += 2; } // Make sure there is nothing left over if (*part != 0) THROW_FMT(FormatError, "invalid date/time %s", time); FUNCTION_TEST_RETURN( TIME, epochFromParts(year, month, day, hour, minute, second, tzOffsetSeconds(offsetHour, offsetMinute))); } // Make sure there is nothing left over if (*part != 0) THROW_FMT(FormatError, "invalid date/time %s", time); // If no timezone was specified then use the current timezone. Set tm_isdst to -1 to force mktime to consider if DST. For // example, if system time is America/New_York then 2019-09-14 20:02:49 was a time in DST so the Epoch value should be // 1568505769 (and not 1568509369 which would be 2019-09-14 21:02:49 - an hour too late). struct tm timePart = { .tm_year = year - 1900, .tm_mon = month - 1, .tm_mday = day, .tm_hour = hour, .tm_min = minute, .tm_sec = second, .tm_isdst = -1, }; FUNCTION_TEST_RETURN(TIME, mktime(&timePart)); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtUIntToZ(const unsigned int value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%u", value); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } FN_EXTERN unsigned int cvtZToUIntBase(const char *const value, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); const uint64_t result = cvtZToUInt64Internal(value, "unsigned int", base); // Don't allow negative numbers even though strtoull() does and check max value if (*value == '-' || result > UINT_MAX) THROW_FMT(FormatError, "unable to convert base %d string '%s' to unsigned int", base, value); FUNCTION_TEST_RETURN(UINT, (unsigned int)result); } FN_EXTERN unsigned int cvtZToUInt(const char *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); FUNCTION_TEST_RETURN(UINT, cvtZToUIntBase(value, 10)); } FN_EXTERN unsigned int cvtZSubNToUIntBase(const char *const value, const size_t offset, const size_t size, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(SIZE, offset); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_PARAM(INT, base); FUNCTION_TEST_END(); ASSERT(value != NULL); char buffer[CVT_BASE10_BUFFER_SIZE + 1]; ASSERT(size <= CVT_BASE10_BUFFER_SIZE); memcpy(buffer, value + offset, size); buffer[size] = '\0'; FUNCTION_TEST_RETURN(UINT, cvtZToUIntBase(buffer, base)); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtUInt64ToZ(const uint64_t value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%" PRIu64, value); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } FN_EXTERN uint64_t cvtZToUInt64Base(const char *const value, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); const uint64_t result = cvtZToUInt64Internal(value, "uint64", base); // Don't allow negative numbers even though strtoull() does if (*value == '-') THROW_FMT(FormatError, "unable to convert base %d string '%s' to uint64", base, value); FUNCTION_TEST_RETURN(UINT64, result); } FN_EXTERN uint64_t cvtZToUInt64(const char *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); FUNCTION_TEST_RETURN(UINT64, cvtZToUInt64Base(value, 10)); } FN_EXTERN uint64_t cvtZSubNToUInt64Base(const char *const value, const size_t offset, const size_t size, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(SIZE, offset); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_PARAM(INT, base); FUNCTION_TEST_END(); ASSERT(value != NULL); char buffer[CVT_BASE10_BUFFER_SIZE + 1]; ASSERT(size <= CVT_BASE10_BUFFER_SIZE); memcpy(buffer, value + offset, size); buffer[size] = '\0'; FUNCTION_TEST_RETURN(UINT64, cvtZToUInt64Base(buffer, base)); } /**********************************************************************************************************************************/ FN_EXTERN void cvtUInt64ToVarInt128(uint64_t value, uint8_t *const buffer, size_t *const bufferPos, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_PARAM(UINT64, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(bufferPos != NULL); ASSERT(bufferSize > *bufferPos); // Keep encoding bytes while the remaining value is greater than 7 bits while (value >= 0x80) { // Encode the lower order 7 bits, adding the continuation bit to indicate there is more data buffer[*bufferPos] = (uint8_t)value | 0x80; // Shift the value to remove bits that have been encoded value >>= 7; // Keep track of size so we know how many bytes to write out (*bufferPos)++; // Make sure the buffer won't overflow if (*bufferPos >= bufferSize) THROW(AssertError, "buffer overflow"); } // Encode the last 7 bits of value buffer[*bufferPos] = (uint8_t)value; (*bufferPos)++; FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN uint64_t cvtUInt64FromVarInt128(const uint8_t *const buffer, size_t *const bufferPos, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_PARAM_P(SIZE, bufferPos); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(bufferPos != NULL); // Decode all bytes uint64_t result = 0; uint8_t byte; for (unsigned int bufferIdx = 0; bufferIdx < CVT_VARINT128_BUFFER_SIZE; bufferIdx++) { // Error if the buffer position is beyond the buffer size if (*bufferPos >= bufferSize) THROW(FormatError, "buffer position is beyond buffer size"); // Get the next encoded byte byte = buffer[*bufferPos]; // Shift the lower order 7 encoded bits into the uint64 in reverse order result |= (uint64_t)(byte & 0x7f) << (7 * bufferIdx); // Increment buffer position to indicate that the byte has been processed (*bufferPos)++; // Done if the high order bit is not set to indicate more data if (byte < 0x80) break; } // By this point all bytes should have been read so error if this is not the case. This could be due to a coding error or // corruption in the data stream. if (byte >= 0x80) THROW(FormatError, "unterminated varint-128 integer"); FUNCTION_TEST_RETURN(UINT64, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/convert.h0000664000175100017510000001600015265101745020270 0ustar00runnerrunner/*********************************************************************************************************************************** Convert C Types Contains conversions to/from native C types. Conversions of project types should be defined in their respective type modules. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_CONVERT_H #define COMMON_TYPE_CONVERT_H #include #include #include #include #include "common/assert.h" #include "common/type/param.h" /*********************************************************************************************************************************** Required buffer sizes ***********************************************************************************************************************************/ #define CVT_BOOL_BUFFER_SIZE 6 #define CVT_BASE10_BUFFER_SIZE 64 #define CVT_DIV_BUFFER_SIZE 68 #define CVT_PCT_BUFFER_SIZE 8 #define CVT_VARINT128_BUFFER_SIZE 10 /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Convert char to zero-terminated string FN_INLINE_ALWAYS size_t cvtCharToZ(const char value, char *const buffer, const size_t bufferSize) { ASSERT_INLINE(buffer != NULL); ASSERT_INLINE(bufferSize >= 2); return (size_t)snprintf(buffer, bufferSize, "%c", value); } // Convert percentage to an integer. This is useful for exporting a percentage in a way that does not incur rounding. FN_EXTERN unsigned int cvtPctToUInt(uint64_t dividend, uint64_t divisor); // Convert percentage to zero-terminated string. This is more reproducible than formatting the results of floating point division. FN_EXTERN size_t cvtPctToZ(uint64_t dividend, uint64_t divisor, char *buffer, size_t bufferSize); // Convert division to zero-terminated string. This is more reproducible than formatting the results of floating point division. FN_EXTERN size_t cvtDivToZ(uint64_t dividend, uint64_t divisor, unsigned int precision, bool trim, char *buffer, size_t bufferSize); // Convert int to zero-terminated string and vice versa FN_EXTERN size_t cvtIntToZ(const int value, char *buffer, size_t bufferSize); FN_EXTERN int cvtZToInt(const char *value); FN_EXTERN int cvtZToIntBase(const char *value, int base); FN_EXTERN int cvtZSubNToIntBase(const char *value, size_t offset, size_t size, int base); FN_INLINE_ALWAYS int cvtZSubNToInt(const char *const value, const size_t offset, const size_t size) { return cvtZSubNToIntBase(value, offset, size, 10); } // Convert int64 to zero-terminated string and vice versa FN_EXTERN size_t cvtInt64ToZ(int64_t value, char *buffer, size_t bufferSize); FN_EXTERN int64_t cvtZToInt64(const char *value); FN_EXTERN int64_t cvtZToInt64Base(const char *value, int base); FN_EXTERN int64_t cvtZSubNToInt64Base(const char *value, size_t offset, size_t size, int base); FN_INLINE_ALWAYS int64_t cvtZSubNToInt64(const char *const value, const size_t offset, const size_t size) { return cvtZSubNToInt64Base(value, offset, size, 10); } // Convert int32/64 to uint32/64 using zigzag encoding and vice versa. Zigzag encoding places the sign in the least significant bit // so that signed and unsigned values alternate, e.g. 0 = 0, -1 = 1, 1 = 2, -2 = 3, 2 = 4, -3 = 5, 3 = 6, etc. This moves as many // bits as possible into the low order bits which is good for other types of encoding, e.g. base-128. See // http://neurocline.github.io/dev/2015/09/17/zig-zag-encoding.html for details. FN_INLINE_ALWAYS uint32_t cvtInt32ToZigZag(const int32_t value) { return ((uint32_t)value << 1) ^ (uint32_t)(value >> 31); } FN_INLINE_ALWAYS int32_t cvtInt32FromZigZag(const uint32_t value) { return (int32_t)((value >> 1) ^ (~(value & 1) + 1)); } FN_INLINE_ALWAYS uint64_t cvtInt64ToZigZag(const int64_t value) { return ((uint64_t)value << 1) ^ (uint64_t)(value >> 63); } FN_INLINE_ALWAYS int64_t cvtInt64FromZigZag(const uint64_t value) { return (int64_t)((value >> 1) ^ (~(value & 1) + 1)); } // Convert mode to zero-terminated string and vice versa FN_EXTERN size_t cvtModeToZ(mode_t value, char *buffer, size_t bufferSize); FN_EXTERN mode_t cvtZToMode(const char *value); // Convert size/ssize to zero-terminated string FN_EXTERN size_t cvtSizeToZ(size_t value, char *buffer, size_t bufferSize); // Convert time_t to zero-terminated string typedef struct CvtTimeToZParam { VAR_PARAM_HEADER; bool utc; // Use UTC instead of local time? } CvtTimeToZParam; #define cvtTimeToZP(format, value, buffer, bufferSize, ...) \ cvtTimeToZ(format, value, buffer, bufferSize, (CvtTimeToZParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN FN_STRFTIME(1) size_t cvtTimeToZ( const char *format, time_t value, char *buffer, size_t bufferSize, CvtTimeToZParam param); // Convert zero-terminated string with the following format to time_t: // ^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}((\,|\.)[0-9]{1,6})?((\+|\-)[0-9]{2}(:?)([0-9]{2})?)?$ FN_EXTERN time_t cvtZToTime(const char *time); // Convert uint to zero-terminated string and vice versa FN_EXTERN size_t cvtUIntToZ(unsigned int value, char *buffer, size_t bufferSize); FN_EXTERN unsigned int cvtZToUInt(const char *value); FN_EXTERN unsigned int cvtZToUIntBase(const char *value, int base); FN_EXTERN unsigned int cvtZSubNToUIntBase(const char *value, size_t offset, size_t size, int base); FN_INLINE_ALWAYS unsigned int cvtZSubNToUInt(const char *const value, const size_t offset, const size_t size) { return cvtZSubNToUIntBase(value, offset, size, 10); } // Convert uint64 to zero-terminated string and vice versa FN_EXTERN size_t cvtUInt64ToZ(uint64_t value, char *buffer, size_t bufferSize); FN_EXTERN uint64_t cvtZToUInt64(const char *value); FN_EXTERN uint64_t cvtZToUInt64Base(const char *value, int base); FN_EXTERN uint64_t cvtZSubNToUInt64Base(const char *value, size_t offset, size_t size, int base); FN_INLINE_ALWAYS uint64_t cvtZSubNToUInt64(const char *const value, const size_t offset, const size_t size) { return cvtZSubNToUInt64Base(value, offset, size, 10); } // Convert uint64 to base-128 varint and vice versa FN_EXTERN void cvtUInt64ToVarInt128(uint64_t value, uint8_t *buffer, size_t *bufferPos, size_t bufferSize); FN_EXTERN uint64_t cvtUInt64FromVarInt128(const uint8_t *buffer, size_t *bufferPos, size_t bufferSize); // Convert boolean to zero-terminated string. Use cvtBoolToConstZ() whenever possible since it is more efficient. FN_EXTERN size_t cvtBoolToZ(bool value, char *buffer, size_t bufferSize); FN_EXTERN const char *cvtBoolToConstZ(bool value); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/json.c0000664000175100017510000016443215265101745017571 0ustar00runnerrunner/*********************************************************************************************************************************** Convert JSON to/from KeyValue ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/macro.h" #include "common/type/json.h" /*********************************************************************************************************************************** Maximum depth of nested containers when reading ***********************************************************************************************************************************/ #define JSON_READ_DEPTH_MAX 32 /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ typedef struct JsonReadStack { JsonType type; // Container type bool first; // First element has been read } JsonReadStack; struct JsonRead { const char *json; // JSON string List *stack; // Stack of object/array tags bool key; // Was a key read for an object value? bool complete; // JSON read is complete }; typedef struct JsonWriteStack { JsonType type; // Container type bool first; // First element has been written char keyLast[67]; // Last key (may need to be adjusted for alignment) } JsonWriteStack; struct JsonWrite { String *json; // JSON string List *stack; // Stack of object/array tags bool key; // Is a key set for an object value? bool complete; // JSON write is complete }; /*********************************************************************************************************************************** Identify container and scalar types ***********************************************************************************************************************************/ static bool jsonTypeScalar(const JsonType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_END(); switch (type) { case jsonTypeBool: case jsonTypeNull: case jsonTypeNumber: case jsonTypeString: FUNCTION_TEST_RETURN(BOOL, true); default: FUNCTION_TEST_RETURN(BOOL, false); } } static bool jsonTypeContainer(const JsonType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, !jsonTypeScalar(type)); } /**********************************************************************************************************************************/ FN_EXTERN JsonRead * jsonReadNew(const String *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, json); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(JsonRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (JsonRead) { .json = strZ(json), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(JSON_READ, this); } /*********************************************************************************************************************************** Consume whitespace ***********************************************************************************************************************************/ static void jsonReadConsumeWhiteSpace(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); bool whitespace = true; do { switch (*this->json) { case ' ': case '\t': case '\n': case '\r': this->json++; break; default: whitespace = false; break; } } while (whitespace); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN JsonType jsonReadTypeNext(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadConsumeWhiteSpace(this); // There should be some data if (*this->json == '\0') THROW(JsonFormatError, "expected data"); // Determine data type JsonType result; switch (*this->json) { // String case '"': result = jsonTypeString; break; // Integer case '-': case '0' ... '9': result = jsonTypeNumber; break; // Boolean case 't': case 'f': result = jsonTypeBool; break; // Null case 'n': { result = jsonTypeNull; break; } // Array case '[': result = jsonTypeArrayBegin; break; case ']': result = jsonTypeArrayEnd; break; // Object case '{': result = jsonTypeObjectBegin; break; case '}': result = jsonTypeObjectEnd; break; // Invalid type default: THROW_FMT(JsonFormatError, "invalid type at: %s", this->json); break; } FUNCTION_TEST_RETURN_TYPE(JsonType, result); } /*********************************************************************************************************************************** Read the next type ignoring a single comma before the type ***********************************************************************************************************************************/ static JsonType jsonReadTypeNextIgnoreComma(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadConsumeWhiteSpace(this); if (*this->json != ',') FUNCTION_TEST_RETURN_TYPE(JsonType, jsonReadTypeNext(this)); const char *const jsonBeforeComma = this->json; this->json++; const JsonType result = jsonReadTypeNext(this); this->json = jsonBeforeComma; FUNCTION_TEST_RETURN_TYPE(JsonType, result); } /*********************************************************************************************************************************** Push/pop a type on/off the stack ***********************************************************************************************************************************/ static void jsonReadPush(JsonRead *const this, const JsonType type, const bool key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_PARAM(BOOL, key); FUNCTION_TEST_END(); ASSERT(this != NULL); // Cannot read after complete if (this->complete) THROW(FormatError, "JSON read is complete"); // Check that the requested type matches the actual type. The actual type needs to be assigned to a local variable or Coverity // will complain about parameter order when jsonReadTypeNextIgnoreComma() is called again in THROW_FMT() even though it is a // noop. const JsonType typeActual = jsonReadTypeNextIgnoreComma(this); if (typeActual != type) THROW_FMT(JsonFormatError, "expected '%s' but found '%s' at: %s", zNewStrId(type), zNewStrId(typeActual), this->json); // If the container stack has not been created yet if (this->stack == NULL) { ASSERT(!key); // No comma before a value outside a container if (*this->json == ',') THROW_FMT(JsonFormatError, "unexpected comma at: %s", this->json); // If a scalar then no need to create the stack since the read is complete if (jsonTypeScalar(type)) { this->complete = true; FUNCTION_TEST_RETURN_VOID(); } // Create the container stack else { MEM_CONTEXT_OBJ_BEGIN(this) { this->stack = lstNewP(sizeof(JsonReadStack)); } MEM_CONTEXT_OBJ_END(); } } // If there is a container on the stack ASSERT((jsonTypeContainer(type) && !key) || !lstEmpty(this->stack)); if (!lstEmpty(this->stack)) { JsonWriteStack *const item = lstGetLast(this->stack); // If this is a key check that a key has not been read since the last value if (key) { ASSERT(item->type == jsonTypeObjectBegin); if (this->key) THROW_FMT(AssertError, "key has already been read at: %s", this->json); this->key = true; } // Else make sure a key has been read else if (item->type == jsonTypeObjectBegin) { if (!this->key) THROW_FMT(AssertError, "key has not been read at: %s", this->json); this->key = false; } // Consume comma if (item->first && (item->type != jsonTypeObjectBegin || key)) { if (*this->json != ',') THROW_FMT(JsonFormatError, "missing comma at: %s", this->json); this->json++; jsonReadConsumeWhiteSpace(this); } else { // No comma before the first value in a container or before an object value if (*this->json == ',') THROW_FMT(JsonFormatError, "unexpected comma at: %s", this->json); item->first = true; } } // Add container to the stack if (jsonTypeContainer(type)) { if (lstSize(this->stack) >= JSON_READ_DEPTH_MAX) THROW_FMT(JsonFormatError, "maximum depth of " STRINGIFY(JSON_READ_DEPTH_MAX) " exceeded at: %s", this->json); lstAdd(this->stack, &(JsonReadStack){.type = type}); } FUNCTION_TEST_RETURN_VOID(); } static void jsonReadPop(JsonRead *const this, const JsonType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(jsonTypeContainer(type)); // Cannot read after complete if (this->complete) THROW(FormatError, "JSON read is complete"); ASSERT(this->stack != NULL); ASSERT(!lstEmpty(this->stack)); ASSERT_DECLARE(const JsonReadStack *const container = lstGetLast(this->stack)); ASSERT( (container->type == jsonTypeArrayBegin && type == jsonTypeArrayEnd) || (container->type == jsonTypeObjectBegin && type == jsonTypeObjectEnd)); // Check that the container has ended if (jsonReadTypeNext(this) != type) { THROW_FMT( JsonFormatError, "expected %c but found %c at: %s", type == jsonTypeArrayEnd ? ']' : '}', *this->json, this->json); } // Remove container lstRemoveLast(this->stack); // Nothing more to read if this is the last container end if (lstEmpty(this->stack)) this->complete = true; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Read JSON number ***********************************************************************************************************************************/ typedef enum { jsonNumberTypeU64, jsonNumberTypeI64, } JsonNumberType; typedef struct JsonReadNumberResult { JsonNumberType type; union { int64_t i64; uint64_t u64; }; } JsonReadNumberResult; static JsonReadNumberResult jsonReadNumber(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); // Consume the - when present const bool intSigned = *this->json == '-'; // Consume all digits size_t digits = 0; while (isdigit(this->json[digits + intSigned])) digits++; // Invalid if no digits were found if (digits == 0) THROW_FMT(JsonFormatError, "no digits found at: %s", this->json); // Copy to buffer if (digits >= CVT_BASE10_BUFFER_SIZE) THROW_FMT(JsonFormatError, "invalid number at: %s", this->json); const size_t size = digits + intSigned; this->json += size; // Return result if (intSigned) { FUNCTION_TEST_RETURN_TYPE( JsonReadNumberResult, (JsonReadNumberResult){.type = jsonNumberTypeI64, .i64 = cvtZSubNToInt64(this->json - size, 0, size)}); } FUNCTION_TEST_RETURN_TYPE( JsonReadNumberResult, (JsonReadNumberResult){.type = jsonNumberTypeU64, .u64 = cvtZSubNToUInt64(this->json - size, 0, size)}); } /**********************************************************************************************************************************/ FN_EXTERN void jsonReadArrayBegin(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeArrayBegin, false); this->json++; FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void jsonReadArrayEnd(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPop(this, jsonTypeArrayEnd); this->json++; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool jsonReadBool(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeBool, false); if (strncmp(this->json, TRUE_Z, sizeof(TRUE_Z) - 1) == 0) { this->json += sizeof(TRUE_Z) - 1; FUNCTION_TEST_RETURN(BOOL, true); } if (strncmp(this->json, FALSE_Z, sizeof(FALSE_Z) - 1) == 0) { this->json += sizeof(FALSE_Z) - 1; FUNCTION_TEST_RETURN(BOOL, false); } THROW_FMT(JsonFormatError, "expected boolean at: %s", this->json); } /**********************************************************************************************************************************/ FN_EXTERN int jsonReadInt(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeNumber, false); JsonReadNumberResult number = jsonReadNumber(this); if (number.type == jsonNumberTypeU64) { if (number.u64 > INT_MAX) THROW_FMT(JsonFormatError, "%" PRIu64 " is out of range for int", number.u64); FUNCTION_TEST_RETURN(INT, (int)number.u64); } if (number.i64 < INT_MIN) THROW_FMT(JsonFormatError, "%" PRId64 " is out of range for int", number.i64); FUNCTION_TEST_RETURN(INT, (int)number.i64); } /**********************************************************************************************************************************/ FN_EXTERN int64_t jsonReadInt64(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeNumber, false); JsonReadNumberResult number = jsonReadNumber(this); if (number.type == jsonNumberTypeI64) FUNCTION_TEST_RETURN(INT64, number.i64); if (number.u64 > INT64_MAX) THROW_FMT(JsonFormatError, "%" PRIu64 " is out of range for int64", number.u64); FUNCTION_TEST_RETURN(INT64, (int64_t)number.u64); } /**********************************************************************************************************************************/ typedef struct JsonReadKeyInternalResult { const char *buffer; size_t size; } JsonReadKeyInternalResult; static JsonReadKeyInternalResult jsonReadKeyZN(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Skip the beginning " ASSERT(*this->json == '"'); this->json++; // Read string JsonReadKeyInternalResult result = {.buffer = this->json}; while (*this->json != '"') { if (*this->json == '\\') THROW_FMT(JsonFormatError, "escape character not allowed in key at: %s", this->json); else if (*this->json == '\0') THROW(JsonFormatError, "expected '\"' but found null delimiter"); this->json++; } // Set key size result.size = (size_t)(this->json - result.buffer); // Advance the character array pointer to the next element after the string this->json++; // Consume the : after the key jsonReadConsumeWhiteSpace(this); if (*this->json != ':') THROW_FMT(JsonFormatError, "expected : after key '%.*s' at: %s", (int)result.size, result.buffer, this->json); this->json++; FUNCTION_TEST_RETURN_TYPE(JsonReadKeyInternalResult, result); } FN_EXTERN String * jsonReadKey(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeString, true); JsonReadKeyInternalResult key = jsonReadKeyZN(this); FUNCTION_TEST_RETURN(STRING, strNewZN(key.buffer, key.size)); } /**********************************************************************************************************************************/ FN_EXTERN bool jsonReadKeyExpect(JsonRead *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); bool result = false; // Check that we are currently reading an object jsonReadConsumeWhiteSpace(this); JsonWriteStack *const item = lstGetLast(this->stack); ASSERT(item->type == jsonTypeObjectBegin); // Read until object end while (jsonReadTypeNextIgnoreComma(this) != jsonTypeObjectEnd) { // Save state before reading the key const char *const jsonBeforeKey = this->json; const bool firstBeforeKey = item->first; jsonReadPush(this, jsonTypeString, true); // Get and compare next key const JsonReadKeyInternalResult keyNext = jsonReadKeyZN(this); const int compare = strncmp(strZ(key), keyNext.buffer, keyNext.size); // Return true if key matches if (compare == 0) { result = true; break; } // Return false and reset state if requested key is after the actual key else if (compare < 0) { this->json = jsonBeforeKey; this->key = false; item->first = firstBeforeKey; break; } // Skip the value and continue jsonReadSkip(this); } FUNCTION_TEST_RETURN(BOOL, result); } FN_EXTERN bool jsonReadKeyExpectStrId(JsonRead *const this, const StringId key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRING_ID, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != 0); char buffer[STRID_MAX + 1]; FUNCTION_TEST_RETURN(BOOL, jsonReadKeyExpect(this, STR_SIZE(buffer, strIdToZ(key, buffer)))); } FN_EXTERN bool jsonReadKeyExpectZ(JsonRead *const this, const char *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRINGZ, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN(BOOL, jsonReadKeyExpect(this, STR(key))); } FN_EXTERN JsonRead * jsonReadKeyRequire(JsonRead *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); if (!jsonReadKeyExpect(this, key)) THROW_FMT(JsonFormatError, "required key '%s' not found", strZ(key)); FUNCTION_TEST_RETURN(JSON_READ, this); } FN_EXTERN JsonRead * jsonReadKeyRequireStrId(JsonRead *const this, const StringId key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRING_ID, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != 0); char buffer[STRID_MAX + 1]; FUNCTION_TEST_RETURN(JSON_READ, jsonReadKeyRequire(this, STR_SIZE(buffer, strIdToZ(key, buffer)))); } FN_EXTERN JsonRead * jsonReadKeyRequireZ(JsonRead *const this, const char *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRINGZ, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN(JSON_READ, jsonReadKeyRequire(this, STR(key))); } /**********************************************************************************************************************************/ FN_EXTERN void jsonReadNull(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeNull, false); if (strncmp(this->json, NULL_Z, sizeof(NULL_Z) - 1) == 0) { this->json += sizeof(NULL_Z) - 1; FUNCTION_TEST_RETURN_VOID(); } THROW_FMT(JsonFormatError, "expected null at: %s", this->json); } /**********************************************************************************************************************************/ FN_EXTERN void jsonReadObjectBegin(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeObjectBegin, false); this->json++; FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void jsonReadObjectEnd(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPop(this, jsonTypeObjectEnd); this->json++; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ // Quickly skip over a valid JSON string static void jsonReadSkipStr(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Skip the beginning " ASSERT(*this->json == '"'); this->json++; // Read string while (*this->json != '"') { if (*this->json == '\\') { this->json++; switch (*this->json) { // Unicode character case 'u': { this->json++; // Expect four hex digits unsigned int digitIdx = 0; for (; digitIdx < 4; digitIdx++) { if (!isxdigit((unsigned char)this->json[digitIdx])) break; } if (digitIdx != 4) THROW_FMT(JsonFormatError, "unable to decode at: %s", this->json - 2); // Move to the last digit this->json += 3; break; } // End of string after the escape case '\0': THROW(JsonFormatError, "expected '\"' but found null delimiter"); // Any other escape default: break; } } else if (*this->json == '\0') THROW(JsonFormatError, "expected '\"' but found null delimiter"); this->json++; } // Advance the character array pointer to the next element after the string this->json++; FUNCTION_TEST_RETURN_VOID(); } static void jsonReadSkipRecurse(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); switch (jsonReadTypeNextIgnoreComma(this)) { case jsonTypeBool: jsonReadBool(this); break; case jsonTypeNull: jsonReadNull(this); break; case jsonTypeNumber: jsonReadPush(this, jsonTypeNumber, false); jsonReadNumber(this); break; case jsonTypeString: jsonReadPush(this, jsonTypeString, false); jsonReadSkipStr(this); break; case jsonTypeArrayBegin: { jsonReadArrayBegin(this); while (jsonReadTypeNextIgnoreComma(this) != jsonTypeArrayEnd) jsonReadSkipRecurse(this); jsonReadArrayEnd(this); break; } default: { jsonReadObjectBegin(this); while (jsonReadTypeNextIgnoreComma(this) != jsonTypeObjectEnd) { // Read key jsonReadPush(this, jsonTypeString, true); jsonReadSkipStr(this); // Consume the : after the key jsonReadConsumeWhiteSpace(this); if (*this->json != ':') THROW_FMT(JsonFormatError, "expected : after key at: %s", this->json); this->json++; // Skip value jsonReadSkipRecurse(this); } jsonReadObjectEnd(this); break; } } FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void jsonReadSkip(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadSkipRecurse(this); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * jsonReadStr(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // If NULL if (jsonReadTypeNextIgnoreComma(this) == jsonTypeNull) { jsonReadNull(this); FUNCTION_TEST_RETURN(STRING, NULL); } // Read string jsonReadPush(this, jsonTypeString, false); String *const result = strNew(); // Skip the beginning " ASSERT(*this->json == '"'); this->json++; // Track portion of string with no escapes const char *noEscape = NULL; size_t noEscapeSize = 0; while (*this->json != '"') { if (*this->json == '\\') { // Copy portion of string without escapes if (noEscapeSize > 0) { strCatZN(result, noEscape, noEscapeSize); noEscapeSize = 0; } this->json++; switch (*this->json) { case '"': strCatChr(result, '"'); break; case '\\': strCatChr(result, '\\'); break; case '/': strCatChr(result, '/'); break; case 'n': strCatChr(result, '\n'); break; case 'r': strCatChr(result, '\r'); break; case 't': strCatChr(result, '\t'); break; case 'b': strCatChr(result, '\b'); break; case 'f': strCatChr(result, '\f'); break; case 'u': { this->json++; // We don't know how to decode anything except ASCII so fail if it looks like Unicode if (strncmp(this->json, "00", 2) != 0 || !isxdigit((unsigned char)this->json[2]) || !isxdigit((unsigned char)this->json[3])) { THROW_FMT(JsonFormatError, "unable to decode at: %s", this->json - 2); } // Decode char. A null character cannot be stored in a string. this->json += 2; const char decode = (char)cvtZSubNToUIntBase(this->json, 0, 2, 16); if (decode == '\0') THROW_FMT(JsonFormatError, "unable to decode at: %s", this->json - 4); strCatChr(result, decode); this->json++; break; } default: THROW_FMT(JsonFormatError, "invalid escape character at: %s", this->json - 1); } } else { if (*this->json == '\0') THROW(JsonFormatError, "expected '\"' but found null delimiter"); // If escape string is zero size then start it if (noEscapeSize == 0) noEscape = this->json; noEscapeSize++; } this->json++; } // Copy portion of string without escapes if (noEscapeSize > 0) strCatZN(result, noEscape, noEscapeSize); // Advance the character array pointer to the next element after the string this->json++; FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN StringId jsonReadStrId(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); StringId result = 0; MEM_CONTEXT_TEMP_BEGIN() { result = strStrId(jsonReadStr(this)); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING_ID, result); } /**********************************************************************************************************************************/ FN_EXTERN StringList * jsonReadStrLst(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { jsonReadArrayBegin(this); while (jsonReadTypeNextIgnoreComma(this) != jsonTypeArrayEnd) strLstAdd(result, jsonReadStr(this)); jsonReadArrayEnd(this); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int jsonReadUInt(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeNumber, false); JsonReadNumberResult number = jsonReadNumber(this); if (number.type == jsonNumberTypeI64) THROW_FMT(JsonFormatError, "%" PRId64 " is out of range for uint", number.i64); if (number.u64 > UINT_MAX) THROW_FMT(JsonFormatError, "%" PRIu64 " is out of range for uint", number.u64); FUNCTION_TEST_RETURN(UINT, (unsigned int)number.u64); } /**********************************************************************************************************************************/ FN_EXTERN uint64_t jsonReadUInt64(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeNumber, false); JsonReadNumberResult number = jsonReadNumber(this); if (number.type == jsonNumberTypeI64) THROW_FMT(JsonFormatError, "%" PRId64 " is out of range for uint64", number.i64); FUNCTION_TEST_RETURN(UINT64, number.u64); } /**********************************************************************************************************************************/ static Variant * jsonReadVarRecurse(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); switch (jsonReadTypeNextIgnoreComma(this)) { case jsonTypeBool: FUNCTION_TEST_RETURN(VARIANT, varNewBool(jsonReadBool(this))); case jsonTypeNull: jsonReadNull(this); FUNCTION_TEST_RETURN(VARIANT, NULL); case jsonTypeNumber: { jsonReadPush(this, jsonTypeNumber, false); JsonReadNumberResult number = jsonReadNumber(this); if (number.type == jsonNumberTypeU64) FUNCTION_TEST_RETURN(VARIANT, varNewUInt64(number.u64)); FUNCTION_TEST_RETURN(VARIANT, varNewInt64(number.i64)); } case jsonTypeString: { String *const string = jsonReadStr(this); Variant *const result = varNewStr(string); strFree(string); FUNCTION_TEST_RETURN(VARIANT, result); } case jsonTypeArrayBegin: { VariantList *const list = varLstNew(); jsonReadArrayBegin(this); MEM_CONTEXT_BEGIN(lstMemContext((List *)list)) { while (jsonReadTypeNextIgnoreComma(this) != jsonTypeArrayEnd) varLstAdd(list, jsonReadVarRecurse(this)); } MEM_CONTEXT_END(); jsonReadArrayEnd(this); Variant *const result = varNewVarLst(list); varLstFree(list); FUNCTION_TEST_RETURN(VARIANT, result); } default: { KeyValue *const result = kvNew(); jsonReadObjectBegin(this); while (jsonReadTypeNextIgnoreComma(this) != jsonTypeObjectEnd) { String *const key = jsonReadKey(this); Variant *const value = jsonReadVarRecurse(this); kvPut(result, VARSTR(key), value); strFree(key); varFree(value); } jsonReadObjectEnd(this); FUNCTION_TEST_RETURN(VARIANT, varNewKv(result)); } } } FN_EXTERN Variant * jsonReadVar(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); Variant *const result = jsonReadVarRecurse(this); FUNCTION_TEST_RETURN(VARIANT, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * jsonToVar(const String *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, json); FUNCTION_TEST_END(); Variant *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { JsonRead *const read = jsonReadNew(json); MEM_CONTEXT_PRIOR_BEGIN() { result = jsonReadVar(read); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(VARIANT, result); } /**********************************************************************************************************************************/ FN_EXTERN void jsonValidate(const String *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, json); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { JsonRead *const read = jsonReadNew(json); MEM_CONTEXT_PRIOR_BEGIN() { jsonReadSkip(read); jsonReadConsumeWhiteSpace(read); if (*read->json != '\0') THROW_FMT(JsonFormatError, "characters after JSON at: %s", read->json); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN void jsonReadToLog(const JsonRead *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{json: %s}", this->json); } #endif // DEBUG /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteNew(const JsonWriteNewParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, param.json); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(JsonWrite, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (JsonWrite) { .json = param.json == NULL ? strNew() : param.json, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /*********************************************************************************************************************************** Push/pop a type on/off the stack ***********************************************************************************************************************************/ static void jsonWritePush(JsonWrite *const this, const JsonType type, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); // Cannot write after complete if (this->complete) THROW(JsonFormatError, "JSON write is complete"); // If the container stack jas not been created yet if (this->stack == NULL) { ASSERT(key == NULL); // If a scalar then no need to create the stack since the write is complete if (jsonTypeScalar(type)) { this->complete = true; FUNCTION_TEST_RETURN_VOID(); } // Create the container stack else { MEM_CONTEXT_OBJ_BEGIN(this) { this->stack = lstNewP(sizeof(JsonWriteStack)); } MEM_CONTEXT_OBJ_END(); } } // If there is a container on the stack ASSERT((jsonTypeContainer(type) && key == NULL) || !lstEmpty(this->stack)); if (!lstEmpty(this->stack)) { JsonWriteStack *const item = lstGetLast(this->stack); // If this is a key check that a key has not been written since the last value if (key != NULL) { if (this->key) THROW_FMT(JsonFormatError, "key has already been written"); // Also check that the key is after the last key if (item->keyLast[0] != '\0' && strCmpZ(key, item->keyLast) <= 0) THROW_FMT(JsonFormatError, "key '%s' is not after prior key '%s'", strZ(key), item->keyLast); // Copy key to a buffer to avoid needing to allocate memory if (strSize(key) >= SIZE_OF_STRUCT_MEMBER(JsonWriteStack, keyLast)) { THROW_FMT( AssertError, "key '%s' must not be longer than %zu bytes", strZ(key), SIZE_OF_STRUCT_MEMBER(JsonWriteStack, keyLast) - 1); } memcpy(item->keyLast, strZ(key), strSize(key) + 1); this->key = true; } // Else make sure a key has been written else { if (item->type == jsonTypeObjectBegin) { if (!this->key) THROW_FMT(JsonFormatError, "key has not been written"); this->key = false; } } // Write comma if (item->first && (item->type != jsonTypeObjectBegin || key != NULL)) strCatChr(this->json, ','); else item->first = true; } // Add container to the stack if (jsonTypeContainer(type)) lstAdd(this->stack, &(JsonWriteStack){.type = type}); FUNCTION_TEST_RETURN_VOID(); } static void jsonWritePop(JsonWrite *const this ASSERT_PARAM(const JsonType type)) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->stack != NULL); ASSERT(!lstEmpty(this->stack)); ASSERT_DECLARE(const JsonWriteStack *const container = lstGetLast(this->stack)); ASSERT( (container->type == jsonTypeArrayBegin && type == jsonTypeArrayEnd) || (container->type == jsonTypeObjectBegin && type == jsonTypeObjectEnd)); ASSERT(container->type != jsonTypeObjectEnd || this->key == false); // Remove container lstRemoveLast(this->stack); // Nothing more to write if this is the last container end if (lstEmpty(this->stack)) this->complete = true; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteArrayBegin(JsonWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeArrayBegin, NULL); strCatChr(this->json, '['); FUNCTION_TEST_RETURN(JSON_WRITE, this); } FN_EXTERN JsonWrite * jsonWriteArrayEnd(JsonWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePop(this ASSERT_PARAM(jsonTypeArrayEnd)); strCatChr(this->json, ']'); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteBool(JsonWrite *const this, const bool value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(BOOL, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeBool, NULL); strCat(this->json, value ? TRUE_STR : FALSE_STR); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteInt(JsonWrite *const this, const int value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(INT, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeNumber, NULL); char working[CVT_BASE10_BUFFER_SIZE]; strCatZN(this->json, working, cvtIntToZ(value, working, sizeof(working))); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteInt64(JsonWrite *const this, const int64_t value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(INT64, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeNumber, NULL); char working[CVT_BASE10_BUFFER_SIZE]; strCatZN(this->json, working, cvtInt64ToZ(value, working, sizeof(working))); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteJson(JsonWrite *const this, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(this != NULL); if (value == NULL) jsonWriteNull(this); else { jsonWritePush(this, jsonTypeString, NULL); strCat(this->json, value); } FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteKey(JsonWrite *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); ASSERT( strchr(strZ(key), '\f') == NULL && strchr(strZ(key), '\\') == NULL && strchr(strZ(key), '\n') == NULL && strchr(strZ(key), '\r') == NULL && strchr(strZ(key), '\t') == NULL && strchr(strZ(key), '\b') == NULL && strchr(strZ(key), '"') == NULL); jsonWritePush(this, jsonTypeString, key); strCatChr(this->json, '"'); strCat(this->json, key); strCatZ(this->json, "\":"); FUNCTION_TEST_RETURN(JSON_WRITE, this); } FN_EXTERN JsonWrite * jsonWriteKeyStrId(JsonWrite *const this, const StringId key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING_ID, key); FUNCTION_TEST_END(); char buffer[STRID_MAX + 4]; size_t size = strIdToZ(key, buffer + 1); jsonWritePush(this, jsonTypeString, STR_SIZE(buffer + 1, size)); buffer[0] = '"'; buffer[++size] = '"'; buffer[++size] = ':'; strCatZN(this->json, buffer, size + 1); FUNCTION_TEST_RETURN(JSON_WRITE, this); } FN_EXTERN JsonWrite * jsonWriteKeyZ(JsonWrite *const this, const char *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRINGZ, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN(JSON_WRITE, jsonWriteKey(this, STR(key))); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteNull(JsonWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeNull, NULL); strCat(this->json, NULL_STR); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteObjectBegin(JsonWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeObjectBegin, NULL); strCatChr(this->json, '{'); FUNCTION_TEST_RETURN(JSON_WRITE, this); } FN_EXTERN JsonWrite * jsonWriteObjectEnd(JsonWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePop(this ASSERT_PARAM(jsonTypeObjectEnd)); strCatChr(this->json, '}'); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteStr(JsonWrite *const this, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(this != NULL); if (value == NULL) { jsonWriteNull(this); FUNCTION_TEST_RETURN(JSON_WRITE, this); } jsonWritePush(this, jsonTypeString, NULL); strCatChr(this->json, '"'); // Track portion of string with no escapes const char *valuePtr = strZ(value); const char *noEscape = NULL; size_t noEscapeSize = 0; for (unsigned int valueIdx = 0; valueIdx < strSize(value); valueIdx++) { switch (*valuePtr) { case '"': case '\\': case '\n': case '\r': case '\t': case '\b': case '\f': { // Copy portion of string without escapes if (noEscapeSize > 0) { strCatZN(this->json, noEscape, noEscapeSize); noEscapeSize = 0; } switch (*valuePtr) { case '"': strCatZ(this->json, "\\\""); break; case '\\': strCatZ(this->json, "\\\\"); break; case '\n': strCatZ(this->json, "\\n"); break; case '\r': strCatZ(this->json, "\\r"); break; case '\t': strCatZ(this->json, "\\t"); break; case '\b': strCatZ(this->json, "\\b"); break; case '\f': strCatZ(this->json, "\\f"); break; } break; } default: { // If escape string is zero size then start it if (noEscapeSize == 0) noEscape = valuePtr; noEscapeSize++; break; } } valuePtr++; } // Copy portion of string without escapes if (noEscapeSize > 0) strCatZN(this->json, noEscape, noEscapeSize); strCatChr(this->json, '"'); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteStrFmt(JsonWrite *const this, const char *const format, ...) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { // Determine how long the allocated string needs to be va_list argumentList; va_start(argumentList, format); const size_t size = (size_t)vsnprintf(NULL, 0, format, argumentList); va_end(argumentList); // Format string va_start(argumentList, format); char *const buffer = memNew(size + 1); vsnprintf(buffer, size + 1, format, argumentList); va_end(argumentList); jsonWriteStr(this, STR_SIZE(buffer, size)); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteStrId(JsonWrite *const this, const StringId value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING_ID, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeString, NULL); char buffer[STRID_MAX + 3]; buffer[0] = '"'; size_t size = strIdToZN(value, buffer + 1) + 1; buffer[size++] = '"'; strCatZN(this->json, buffer, size); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteStrLst(JsonWrite *const this, const StringList *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING_LIST, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(value != NULL); jsonWriteArrayBegin(this); for (unsigned int valueIdx = 0; valueIdx < strLstSize(value); valueIdx++) jsonWriteStr(this, strLstGet(value, valueIdx)); jsonWriteArrayEnd(this); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteUInt(JsonWrite *const this, const unsigned int value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(UINT, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeNumber, NULL); char working[CVT_BASE10_BUFFER_SIZE]; strCatZN(this->json, working, cvtUIntToZ(value, working, sizeof(working))); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteUInt64(JsonWrite *const this, const uint64_t value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeNumber, NULL); char working[CVT_BASE10_BUFFER_SIZE]; strCatZN(this->json, working, cvtUInt64ToZ(value, working, sizeof(working))); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ static void jsonWriteVarRecurse(JsonWrite *const this, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { // If VariantList then process each item in the array. Currently the list must be KeyValue types. if (value == NULL) jsonWriteNull(this); else { switch (varType(value)) { case varTypeBool: jsonWriteBool(this, varBool(value)); break; case varTypeInt: jsonWriteInt(this, varInt(value)); break; case varTypeInt64: jsonWriteInt64(this, varInt64(value)); break; case varTypeKeyValue: { const KeyValue *const keyValue = varKv(value); if (keyValue == NULL) jsonWriteNull(this); else { const StringList *const keyList = strLstSort(strLstNewVarLst(kvKeyList(keyValue)), sortOrderAsc); jsonWriteObjectBegin(this); for (unsigned int keyListIdx = 0; keyListIdx < strLstSize(keyList); keyListIdx++) { // Write key const String *const key = strLstGet(keyList, keyListIdx); jsonWriteKey(this, key); // Write value jsonWriteVarRecurse(this, kvGet(keyValue, VARSTR(key))); } jsonWriteObjectEnd(this); } break; } case varTypeString: { jsonWriteStr(this, varStr(value)); break; } case varTypeUInt: jsonWriteUInt(this, varUInt(value)); break; case varTypeUInt64: jsonWriteUInt64(this, varUInt64(value)); break; default: { ASSERT(varType(value) == varTypeVariantList); const VariantList *const valueList = varVarLst(value); if (valueList == NULL) jsonWriteNull(this); else { jsonWriteArrayBegin(this); for (unsigned int valueListIdx = 0; valueListIdx < varLstSize(valueList); valueListIdx++) jsonWriteVarRecurse(this, varLstGet(valueList, valueListIdx)); jsonWriteArrayEnd(this); } } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN JsonWrite * jsonWriteVar(JsonWrite *const this, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWriteVarRecurse(this, value); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteZ(JsonWrite *const this, const char *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWriteStr(this, value == NULL ? NULL : STR(value)); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN const String * jsonWriteResult(JsonWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->complete); FUNCTION_TEST_RETURN(STRING, this->json); } /**********************************************************************************************************************************/ FN_EXTERN String * jsonFromVar(const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { jsonWriteVar(jsonWriteNewP(.json = result), value); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN void jsonWriteToLog(const JsonWrite *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{size: %zu}", strSize(this->json)); } #endif // DEBUG ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/json.h0000664000175100017510000002245615265101745017575 0ustar00runnerrunner/*********************************************************************************************************************************** JSON read/write ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_JSON_H #define COMMON_TYPE_JSON_H #include "common/type/stringId.h" #include "common/type/variant.h" /*********************************************************************************************************************************** JSON types ***********************************************************************************************************************************/ typedef enum { jsonTypeBool = STRID5("bool", 0x63de20), // Boolean jsonTypeNull = STRID5("null", 0x632ae0), // Null jsonTypeNumber = STRID5("nmbr", 0x909ae0), // Number jsonTypeString = STRID5("str", 0x4a930), // String jsonTypeArrayBegin = STRID5("ary-b", 0x2de6410), // Array begin jsonTypeArrayEnd = STRID5("ary-e", 0x5de6410), // Array end jsonTypeObjectBegin = STRID5("obj-b", 0x2da84f0), // Object begin jsonTypeObjectEnd = STRID5("obj-e", 0x5da84f0), // Object end } JsonType; /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ typedef struct JsonRead JsonRead; typedef struct JsonWrite JsonWrite; /*********************************************************************************************************************************** Read Constructors ***********************************************************************************************************************************/ FN_EXTERN JsonRead *jsonReadNew(const String *string); /*********************************************************************************************************************************** Read Functions ***********************************************************************************************************************************/ // Read next JSON type. This is based on an examination of the first character so there may be an error when the type is read, but // the type will not change. FN_EXTERN JsonType jsonReadTypeNext(JsonRead *this); // Read array begin/end FN_EXTERN void jsonReadArrayBegin(JsonRead *this); FN_EXTERN void jsonReadArrayEnd(JsonRead *this); // Read boolean FN_EXTERN bool jsonReadBool(JsonRead *this); // Read integer FN_EXTERN int jsonReadInt(JsonRead *this); FN_EXTERN int64_t jsonReadInt64(JsonRead *this); FN_EXTERN unsigned int jsonReadUInt(JsonRead *this); FN_EXTERN uint64_t jsonReadUInt64(JsonRead *this); // Read key FN_EXTERN String *jsonReadKey(JsonRead *this); // Read expected key FN_EXTERN bool jsonReadKeyExpect(JsonRead *this, const String *key); FN_EXTERN bool jsonReadKeyExpectStrId(JsonRead *this, StringId key); FN_EXTERN bool jsonReadKeyExpectZ(JsonRead *this, const char *key); // Read required key FN_EXTERN JsonRead *jsonReadKeyRequire(JsonRead *this, const String *key); FN_EXTERN JsonRead *jsonReadKeyRequireStrId(JsonRead *this, StringId key); FN_EXTERN JsonRead *jsonReadKeyRequireZ(JsonRead *this, const char *key); // Read null FN_EXTERN void jsonReadNull(JsonRead *this); // Read object begin/end FN_EXTERN void jsonReadObjectBegin(JsonRead *this); FN_EXTERN void jsonReadObjectEnd(JsonRead *this); // Skip value FN_EXTERN void jsonReadSkip(JsonRead *this); // Read string FN_EXTERN String *jsonReadStr(JsonRead *this); FN_EXTERN StringId jsonReadStrId(JsonRead *this); FN_EXTERN StringList *jsonReadStrLst(JsonRead *this); // Read variant FN_EXTERN Variant *jsonReadVar(JsonRead *this); /*********************************************************************************************************************************** Read Helper Functions ***********************************************************************************************************************************/ // Convert JSON to Variant FN_EXTERN Variant *jsonToVar(const String *json); // Validate JSON FN_EXTERN void jsonValidate(const String *json); /*********************************************************************************************************************************** Read Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void jsonReadFree(JsonRead *const this) { objFree(this); } /*********************************************************************************************************************************** Write Constructors ***********************************************************************************************************************************/ typedef struct JsonWriteNewParam { VAR_PARAM_HEADER; String *json; } JsonWriteNewParam; #define jsonWriteNewP(...) \ jsonWriteNew((JsonWriteNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN JsonWrite *jsonWriteNew(JsonWriteNewParam param); /*********************************************************************************************************************************** Write Functions ***********************************************************************************************************************************/ // Write array begin/end FN_EXTERN JsonWrite *jsonWriteArrayBegin(JsonWrite *this); FN_EXTERN JsonWrite *jsonWriteArrayEnd(JsonWrite *this); // Write boolean FN_EXTERN JsonWrite *jsonWriteBool(JsonWrite *this, bool value); // Write integer FN_EXTERN JsonWrite *jsonWriteInt(JsonWrite *this, int value); FN_EXTERN JsonWrite *jsonWriteInt64(JsonWrite *this, int64_t value); FN_EXTERN JsonWrite *jsonWriteUInt(JsonWrite *this, unsigned int value); FN_EXTERN JsonWrite *jsonWriteUInt64(JsonWrite *this, uint64_t value); // Write JSON FN_EXTERN JsonWrite *jsonWriteJson(JsonWrite *this, const String *value); // Write key FN_EXTERN JsonWrite *jsonWriteKey(JsonWrite *this, const String *key); FN_EXTERN JsonWrite *jsonWriteKeyStrId(JsonWrite *this, StringId key); FN_EXTERN JsonWrite *jsonWriteKeyZ(JsonWrite *this, const char *key); // Write null FN_EXTERN JsonWrite *jsonWriteNull(JsonWrite *this); // Write object begin/end FN_EXTERN JsonWrite *jsonWriteObjectBegin(JsonWrite *this); FN_EXTERN JsonWrite *jsonWriteObjectEnd(JsonWrite *this); // Write string FN_EXTERN JsonWrite *jsonWriteStr(JsonWrite *this, const String *value); FN_EXTERN FN_PRINTF(2, 3) JsonWrite *jsonWriteStrFmt(JsonWrite *this, const char *format, ...); FN_EXTERN JsonWrite *jsonWriteStrId(JsonWrite *this, StringId value); FN_EXTERN JsonWrite *jsonWriteStrLst(JsonWrite *this, const StringList *value); FN_EXTERN JsonWrite *jsonWriteZ(JsonWrite *this, const char *value); // Write variant FN_EXTERN JsonWrite *jsonWriteVar(JsonWrite *this, const Variant *value); /*********************************************************************************************************************************** Write Getters/Setters ***********************************************************************************************************************************/ // Get JSON result FN_EXTERN const String *jsonWriteResult(JsonWrite *this); /*********************************************************************************************************************************** Write Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void jsonWriteFree(JsonWrite *const this) { objFree(this); } /*********************************************************************************************************************************** Write Helper Functions ***********************************************************************************************************************************/ // Convert Variant to JSON FN_EXTERN String *jsonFromVar(const Variant *value); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN void jsonReadToLog(const JsonRead *this, StringStatic *debugLog); #define FUNCTION_LOG_JSON_READ_TYPE \ JsonRead * #define FUNCTION_LOG_JSON_READ_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, jsonReadToLog, buffer, bufferSize) FN_EXTERN void jsonWriteToLog(const JsonWrite *this, StringStatic *debugLog); #define FUNCTION_LOG_JSON_WRITE_TYPE \ JsonWrite * #define FUNCTION_LOG_JSON_WRITE_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, jsonWriteToLog, buffer, bufferSize) #endif // DEBUG #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/keyValue.c0000664000175100017510000002653515265101745020406 0ustar00runnerrunner/*********************************************************************************************************************************** Key Value Handler ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/type/keyValue.h" #include "common/type/list.h" #include "common/type/variantList.h" /*********************************************************************************************************************************** Contains information about the key value store ***********************************************************************************************************************************/ struct KeyValue { KeyValuePub pub; // Publicly accessible variables List *list; // List of keys/values }; /*********************************************************************************************************************************** Contains information about an individual key/value pair ***********************************************************************************************************************************/ typedef struct KeyValuePair { Variant *key; // The key Variant *value; // The value (this may be NULL) } KeyValuePair; /**********************************************************************************************************************************/ FN_EXTERN KeyValue * kvNew(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(KeyValue, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (KeyValue) { .pub = { .keyList = varLstNew(), }, .list = lstNewP(sizeof(KeyValuePair)), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(KEY_VALUE, this); } /**********************************************************************************************************************************/ FN_EXTERN KeyValue * kvDup(const KeyValue *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, source); FUNCTION_TEST_END(); ASSERT(source != NULL); KeyValue *const this = kvNew(); MEM_CONTEXT_OBJ_BEGIN(this) { // Duplicate all key/values for (unsigned int listIdx = 0; listIdx < lstSize(source->list); listIdx++) { const KeyValuePair *const sourcePair = (const KeyValuePair *)lstGet(source->list, listIdx); lstAdd(this->list, &(KeyValuePair){.key = varDup(sourcePair->key), .value = varDup(sourcePair->value)}); } // Duplicate key list this->pub.keyList = varLstDup(kvKeyList(source)); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(KEY_VALUE, this); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int kvGetIdx(const KeyValue *const this, const Variant *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); // Search for the key unsigned int result = KEY_NOT_FOUND; for (unsigned int listIdx = 0; listIdx < lstSize(this->list); listIdx++) { const KeyValuePair *const pair = (const KeyValuePair *)lstGet(this->list, listIdx); // Break if the key matches if (varEq(key, pair->key)) { result = listIdx; break; } } FUNCTION_TEST_RETURN(UINT, result); } /*********************************************************************************************************************************** Internal put key/value pair Handles the common logic for the external put functions. The correct mem context should be set before calling this function. ***********************************************************************************************************************************/ static void kvPutInternal(KeyValue *const this, const Variant *const key, Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_AUDIT_HELPER(); // Find the key const unsigned int listIdx = kvGetIdx(this, key); // If the key was not found then add it if (listIdx == KEY_NOT_FOUND) { // Copy the pair const KeyValuePair pair = {.key = varDup(key), .value = value}; // Add to the list lstAdd(this->list, &pair); // Add to the key list varLstAdd(this->pub.keyList, varDup(key)); } // Else update it else { KeyValuePair *pair = (KeyValuePair *)lstGet(this->list, listIdx); if (pair->value != NULL) varFree(pair->value); pair->value = value; } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN KeyValue * kvPut(KeyValue *const this, const Variant *const key, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); MEM_CONTEXT_OBJ_BEGIN(this) { kvPutInternal(this, key, varDup(value)); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(KEY_VALUE, this); } /**********************************************************************************************************************************/ FN_EXTERN KeyValue * kvAdd(KeyValue *const this, const Variant *const key, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); MEM_CONTEXT_OBJ_BEGIN(this) { // Find the key const unsigned int listIdx = kvGetIdx(this, key); // If the key was not found then add it if (listIdx == KEY_NOT_FOUND) { kvPutInternal(this, key, varDup(value)); } // Else create or add to the variant list else { KeyValuePair *const pair = (KeyValuePair *)lstGet(this->list, listIdx); if (pair->value == NULL) pair->value = varDup(value); else if (varType(pair->value) != varTypeVariantList) { Variant *valueList = varNewVarLst(varLstNew()); varLstAdd(varVarLst(valueList), pair->value); varLstAdd(varVarLst(valueList), varDup(value)); pair->value = valueList; } else varLstAdd(varVarLst(pair->value), varDup(value)); } } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(KEY_VALUE, this); } /**********************************************************************************************************************************/ FN_EXTERN KeyValue * kvPutKv(KeyValue *const this, const Variant *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); KeyValue *const result = kvNew(); MEM_CONTEXT_OBJ_BEGIN(this) { kvPutInternal(this, key, varNewKv(result)); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(KEY_VALUE, result); } /**********************************************************************************************************************************/ FN_EXTERN const Variant * kvGet(const KeyValue *const this, const Variant *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); Variant *result = NULL; // Find the key const unsigned int listIdx = kvGetIdx(this, key); if (listIdx != KEY_NOT_FOUND) result = ((KeyValuePair *)lstGet(this->list, listIdx))->value; FUNCTION_TEST_RETURN(VARIANT, result); } /**********************************************************************************************************************************/ FN_EXTERN const Variant * kvGetDefault(const KeyValue *const this, const Variant *const key, const Variant *const defaultValue) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_PARAM(VARIANT, defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); // Find the key const unsigned int listIdx = kvGetIdx(this, key); // If key not found then return default, else return the value if (listIdx == KEY_NOT_FOUND) FUNCTION_TEST_RETURN_CONST(VARIANT, defaultValue); FUNCTION_TEST_RETURN(VARIANT, ((KeyValuePair *)lstGet(this->list, listIdx))->value); } /**********************************************************************************************************************************/ FN_EXTERN VariantList * kvGetList(const KeyValue *const this, const Variant *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); // Get the value const Variant *const value = kvGet(this, key); // Return the list if it already is one if (value != NULL && varType(value) == varTypeVariantList) FUNCTION_TEST_RETURN(VARIANT_LIST, varLstDup(varVarLst(value))); // Create a list to return VariantList *result; VariantList *const list = varLstNew(); MEM_CONTEXT_OBJ_BEGIN(list) { result = varLstAdd(list, varDup(value)); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(VARIANT_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN KeyValue * kvRemove(KeyValue *const this, const Variant *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); // Find the key const unsigned int listIdx = kvGetIdx(this, key); // If the key was found, remove it if (listIdx != KEY_NOT_FOUND) { // Free the key/value being removed and remove from the list const KeyValuePair *const pair = (KeyValuePair *)lstGet(this->list, listIdx); varFree(pair->key); varFree(pair->value); lstRemoveIdx(this->list, listIdx); // Remove from the key list (index must be the same as the key/value list) ASSERT(varEq(key, varLstGet(this->pub.keyList, listIdx))); varFree(varLstGet(this->pub.keyList, listIdx)); lstRemoveIdx((List *)this->pub.keyList, listIdx); } FUNCTION_TEST_RETURN(KEY_VALUE, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/keyValue.h0000664000175100017510000001173415265101745020406 0ustar00runnerrunner/*********************************************************************************************************************************** Key Value Handler ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_KEYVALUE_H #define COMMON_TYPE_KEYVALUE_H /*********************************************************************************************************************************** KeyValue object ***********************************************************************************************************************************/ typedef struct KeyValue KeyValue; #include "common/type/object.h" #include "common/type/variantList.h" /*********************************************************************************************************************************** Constant to indicate key not found ***********************************************************************************************************************************/ #define KEY_NOT_FOUND UINT_MAX /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN KeyValue *kvNew(void); FN_EXTERN KeyValue *kvDup(const KeyValue *source); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct KeyValuePub { VariantList *keyList; // List of keys } KeyValuePub; // List of keys FN_INLINE_ALWAYS const VariantList * kvKeyList(const KeyValue *const this) { return THIS_PUB(KeyValue)->keyList; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add a value to an existing key. If the key does not exist then this works the same as kvPut(). If the key does exist then the // value is converted to a VariantList with the existing value and the new value as list items. Subsequent calls expand the list. // // NOTE: this function is seldom required and kvPut() should be used when a key is expected to have a single value. FN_EXTERN KeyValue *kvAdd(KeyValue *this, const Variant *key, const Variant *value); // Move to a new parent mem context FN_INLINE_ALWAYS KeyValue * kvMove(KeyValue *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Put key/value pair FN_EXTERN KeyValue *kvPut(KeyValue *this, const Variant *key, const Variant *value); // Put key/value store. If this is called on an existing key it will replace the key with an empty kev/value store, even if the key // already contains a key/value store. FN_EXTERN KeyValue *kvPutKv(KeyValue *this, const Variant *key); // Get a value using the key FN_EXTERN const Variant *kvGet(const KeyValue *this, const Variant *key); // Get a value using the key and return a default if not found FN_EXTERN const Variant *kvGetDefault(const KeyValue *this, const Variant *key, const Variant *defaultValue); // Get key index if it exists FN_EXTERN unsigned int kvGetIdx(const KeyValue *this, const Variant *key); // Does the key exist (even if the value is NULL) FN_INLINE_ALWAYS bool kvKeyExists(const KeyValue *const this, const Variant *const key) { return kvGetIdx(this, key) != KEY_NOT_FOUND; } // Get a value as a list (even if there is only one value) using the key FN_EXTERN VariantList *kvGetList(const KeyValue *this, const Variant *key); // Remove a key/value pair FN_EXTERN KeyValue *kvRemove(KeyValue *this, const Variant *key); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void kvFree(KeyValue *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_KEY_VALUE_TYPE \ KeyValue * #define FUNCTION_LOG_KEY_VALUE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "KeyValue", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/list.c0000664000175100017510000003666215265101745017576 0ustar00runnerrunner/*********************************************************************************************************************************** List Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/type/list.h" #include "common/valgrind.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct List { ListPub pub; // Publicly accessible variables unsigned int listSizeMax; SortOrder sortOrder; uint8_t *listAlloc; // Pointer to memory allocated for the list ListComparator *comparator; }; /**********************************************************************************************************************************/ FN_EXTERN List * lstNew(const size_t itemSize, const ListParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, itemSize); FUNCTION_TEST_PARAM(FUNCTIONP, param.comparator); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(List, .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX) { *this = (List) { .pub = { .itemSize = itemSize, }, .sortOrder = param.sortOrder, .comparator = param.comparator, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN List * lstClear(List *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_END(); ASSERT(this != NULL); if (this->pub.list != NULL) { MEM_CONTEXT_BEGIN(lstMemContext(this)) { memFree(this->pub.list); } MEM_CONTEXT_END(); this->pub.list = NULL; this->pub.listSize = 0; this->listSizeMax = 0; } FUNCTION_TEST_RETURN(LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN int lstComparatorStr(const void *const item1, const void *const item2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, item1); FUNCTION_TEST_PARAM_P(VOID, item2); FUNCTION_TEST_END(); ASSERT(item1 != NULL); ASSERT(item2 != NULL); FUNCTION_TEST_RETURN(INT, strCmp(*(const String *const *)item1, *(const String *const *)item2)); } /**********************************************************************************************************************************/ FN_EXTERN int lstComparatorInt(const void *const intPtr1, const void *const intPtr2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, intPtr1); FUNCTION_TEST_PARAM_P(VOID, intPtr2); FUNCTION_TEST_END(); ASSERT(intPtr1 != NULL); ASSERT(intPtr2 != NULL); const int int1 = *(const int *)intPtr1; const int int2 = *(const int *)intPtr2; FUNCTION_TEST_RETURN(INT, LST_COMPARATOR_CMP(int1, int2)); } /**********************************************************************************************************************************/ FN_EXTERN int lstComparatorUInt(const void *const uintPtr1, const void *const uintPtr2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, uintPtr1); FUNCTION_TEST_PARAM_P(VOID, uintPtr2); FUNCTION_TEST_END(); ASSERT(uintPtr1 != NULL); ASSERT(uintPtr2 != NULL); const unsigned int uint1 = *(const unsigned int *)uintPtr1; const unsigned int uint2 = *(const unsigned int *)uintPtr2; FUNCTION_TEST_RETURN(INT, LST_COMPARATOR_CMP(uint1, uint2)); } /**********************************************************************************************************************************/ FN_EXTERN int lstComparatorZ(const void *const item1, const void *const item2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, item1); FUNCTION_TEST_PARAM_P(VOID, item2); FUNCTION_TEST_END(); ASSERT(item1 != NULL); ASSERT(item2 != NULL); FUNCTION_TEST_RETURN(INT, strcmp(*(const char *const *)item1, *(const char *const *)item2)); } /*********************************************************************************************************************************** General function for a descending comparator that simply switches the parameters on the main comparator (which should be asc) ***********************************************************************************************************************************/ static const List *comparatorDescList = NULL; static int lstComparatorDesc(const void *const item1, const void *const item2) { return comparatorDescList->comparator(item2, item1); } /**********************************************************************************************************************************/ FN_EXTERN void * lstGet(const List *const this, const unsigned int listIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM(UINT, listIdx); FUNCTION_TEST_END(); ASSERT(this != NULL); // Ensure list index is in range if (listIdx >= lstSize(this)) THROW_FMT(AssertError, "cannot get index %u from list with %u value(s)", listIdx, lstSize(this)); // Return pointer to list item FUNCTION_TEST_RETURN_P(VOID, this->pub.list + (listIdx * this->pub.itemSize)); } FN_EXTERN void * lstGetLast(const List *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Ensure there are items in the list if (lstSize(this) == 0) THROW(AssertError, "cannot get last from list with no values"); // Return pointer to list item FUNCTION_TEST_RETURN_P(VOID, lstGet(this, lstSize(this) - 1)); } /**********************************************************************************************************************************/ FN_EXTERN void * lstFind(const List *const this, const void *const item) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM_P(VOID, item); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->comparator != NULL); ASSERT(item != NULL); if (this->pub.list != NULL) { if (this->sortOrder == sortOrderAsc) FUNCTION_TEST_RETURN_P(VOID, bsearch(item, this->pub.list, lstSize(this), this->pub.itemSize, this->comparator)); else if (this->sortOrder == sortOrderDesc) { // Assign the list for the descending comparator to use comparatorDescList = this; FUNCTION_TEST_RETURN_P(VOID, bsearch(item, this->pub.list, lstSize(this), this->pub.itemSize, lstComparatorDesc)); } // Fall back on an iterative search for (unsigned int listIdx = 0; listIdx < lstSize(this); listIdx++) { if (this->comparator(item, lstGet(this, listIdx)) == 0) FUNCTION_TEST_RETURN_P(VOID, lstGet(this, listIdx)); } } FUNCTION_TEST_RETURN_P(VOID, NULL); } FN_EXTERN unsigned int lstFindIdx(const List *const this, const void *const item) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM_P(VOID, item); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(item != NULL); void *const result = lstFind(this, item); FUNCTION_TEST_RETURN(UINT, result == NULL ? LIST_NOT_FOUND : lstIdx(this, result)); } FN_EXTERN const void * lstFindDefault(const List *const this, const void *const item, const void *const itemDefault) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM_P(VOID, item); FUNCTION_TEST_PARAM_P(VOID, itemDefault); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(item != NULL); void *const result = lstFind(this, item); FUNCTION_TEST_RETURN_CONST_P(VOID, result == NULL ? itemDefault : result); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int lstIdx(const List *const this, const void *const item) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM_P(VOID, item); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(item != NULL); // Item pointers should always be aligned with the beginning of an item in the list ASSERT((size_t)((const uint8_t *const)item - this->pub.list) % this->pub.itemSize == 0); const size_t result = (size_t)((const uint8_t *const)item - this->pub.list) / this->pub.itemSize; // Item pointers should always be in range ASSERT(result < lstSize(this)); FUNCTION_TEST_RETURN(UINT, (unsigned int)result); } /**********************************************************************************************************************************/ FN_EXTERN void * lstInsert(List *const this, const unsigned int listIdx, const void *const item) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM_P(VOID, item); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(listIdx <= lstSize(this)); ASSERT(item != NULL); // If list size = max then allocate more space if (lstSize(this) == this->listSizeMax) { MEM_CONTEXT_BEGIN(lstMemContext(this)) { // If nothing has been allocated yet if (this->listSizeMax == 0) { this->listSizeMax = LIST_INITIAL_SIZE; this->pub.list = memNew(this->listSizeMax * this->pub.itemSize); } // Else the list needs to be extended else { this->listSizeMax *= 2; this->pub.list = memResize(this->pub.list, this->listSizeMax * this->pub.itemSize); } this->listAlloc = this->pub.list; } MEM_CONTEXT_END(); } // Else if there is space before the beginning of the list then move the list down else if ( (this->pub.list != this->listAlloc) && (lstSize(this) + ((uintptr_t)(this->pub.list - this->listAlloc) / this->pub.itemSize) == this->listSizeMax)) { memmove(this->listAlloc, this->pub.list, lstSize(this) * this->pub.itemSize); // Invalidate the area vacated by the move so Valgrind will warn on a read from it VALGRIND_MAKE_MEM_UNDEFINED( this->listAlloc + (lstSize(this) * this->pub.itemSize), (size_t)(this->pub.list - this->listAlloc)); this->pub.list = this->listAlloc; } // Calculate the position where this item will be copied void *const itemPtr = this->pub.list + (listIdx * this->pub.itemSize); // If not inserting at the end then move items down to make space if (listIdx != lstSize(this)) memmove(this->pub.list + ((listIdx + 1) * this->pub.itemSize), itemPtr, (lstSize(this) - listIdx) * this->pub.itemSize); // Copy item into the list this->sortOrder = sortOrderNone; memcpy(itemPtr, item, this->pub.itemSize); this->pub.listSize++; FUNCTION_TEST_RETURN_P(VOID, itemPtr); } /**********************************************************************************************************************************/ FN_EXTERN List * lstRemoveIdx(List *const this, const unsigned int listIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM(UINT, listIdx); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(listIdx <= lstSize(this)); // Decrement the list size this->pub.listSize--; // If this is the first item then move the list pointer up to avoid moving all the items if (listIdx == 0) { this->pub.list += this->pub.itemSize; // Invalidate the item that is no longer in the list so Valgrind will warn on a read from it VALGRIND_MAKE_MEM_UNDEFINED(this->pub.list - this->pub.itemSize, this->pub.itemSize); } // Else remove the item by moving the items after it down else { memmove( this->pub.list + (listIdx * this->pub.itemSize), this->pub.list + ((listIdx + 1) * this->pub.itemSize), (lstSize(this) - listIdx) * this->pub.itemSize); // Invalidate the item vacated at the end of the list so Valgrind will warn on a read from it VALGRIND_MAKE_MEM_UNDEFINED(this->pub.list + (lstSize(this) * this->pub.itemSize), this->pub.itemSize); } FUNCTION_TEST_RETURN(LIST, this); } FN_EXTERN bool lstRemove(List *const this, const void *const item) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM_P(VOID, item); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(item != NULL); const unsigned int listIdx = lstFindIdx(this, item); if (listIdx != LIST_NOT_FOUND) { lstRemoveIdx(this, listIdx); FUNCTION_TEST_RETURN(BOOL, true); } FUNCTION_TEST_RETURN(BOOL, false); } FN_EXTERN List * lstRemoveLast(List *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_END(); ASSERT(this != NULL); if (lstSize(this) == 0) THROW(AssertError, "cannot remove last from list with no values"); FUNCTION_TEST_RETURN(LIST, lstRemoveIdx(this, lstSize(this) - 1)); } /**********************************************************************************************************************************/ FN_EXTERN List * lstSort(List *const this, const SortOrder sortOrder) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM(ENUM, sortOrder); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->comparator != NULL); if (this->pub.list != NULL) { switch (sortOrder) { case sortOrderAsc: qsort(this->pub.list, lstSize(this), this->pub.itemSize, this->comparator); break; case sortOrderDesc: { // Assign the list that will be sorted for the comparator function to use comparatorDescList = this; qsort(this->pub.list, lstSize(this), this->pub.itemSize, lstComparatorDesc); break; } case sortOrderNone: break; } } this->sortOrder = sortOrder; FUNCTION_TEST_RETURN(LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN List * lstComparatorSet(List *const this, ListComparator *const comparator) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM(FUNCTIONP, comparator); FUNCTION_TEST_END(); ASSERT(this != NULL); this->comparator = comparator; this->sortOrder = sortOrderNone; FUNCTION_TEST_RETURN(LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN void lstToLog(const List *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{size: %u}", lstSize(this)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/list.h0000664000175100017510000002217315265101745017573 0ustar00runnerrunner/*********************************************************************************************************************************** List Handler ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_LIST_H #define COMMON_TYPE_LIST_H #include /*********************************************************************************************************************************** List object ***********************************************************************************************************************************/ typedef struct List List; #include "common/memContext.h" #include "common/type/object.h" #include "common/type/param.h" #include "common/type/string.h" /*********************************************************************************************************************************** Sort orders ***********************************************************************************************************************************/ typedef enum { sortOrderNone, sortOrderAsc, sortOrderDesc, } SortOrder; /*********************************************************************************************************************************** Define initial size of a list ***********************************************************************************************************************************/ #define LIST_INITIAL_SIZE 8 /*********************************************************************************************************************************** Item was not found in the list ***********************************************************************************************************************************/ #define LIST_NOT_FOUND UINT_MAX /*********************************************************************************************************************************** Function type for comparing items in the list The return value should be -1 when item1 < item2, 0 when item1 == item2, and 1 when item2 > item1. ***********************************************************************************************************************************/ typedef int ListComparator(const void *item1, const void *item2); // General purpose list comparator for Strings or structs with a String as the first member FN_EXTERN int lstComparatorStr(const void *item1, const void *item2); // General purpose list comparator for unsigned int or structs with an unsigned int as the first member FN_EXTERN int lstComparatorUInt(const void *item1, const void *item2); // General purpose list comparator for int or structs with an int as the first member FN_EXTERN int lstComparatorInt(const void *item1, const void *item2); // General purpose list comparator for zero-terminated strings or structs with a zero-terminated string as the first member FN_EXTERN int lstComparatorZ(const void *item1, const void *item2); // Macro to compare two values in a branchless and transitive fashion #define LST_COMPARATOR_CMP(item1, item2) \ (((item1) > (item2)) - ((item1) < (item2))) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct ListParam { VAR_PARAM_HEADER; SortOrder sortOrder; ListComparator *comparator; } ListParam; #define lstNewP(itemSize, ...) \ lstNew(itemSize, (ListParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN List *lstNew(size_t itemSize, ListParam param); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct ListPub { unsigned int listSize; // List size size_t itemSize; // Size of item stored in the list uint8_t *list; // Pointer to the current start of the list } ListPub; // Set a new comparator FN_EXTERN List *lstComparatorSet(List *this, ListComparator *comparator); // Memory context for this list FN_INLINE_ALWAYS MemContext * lstMemContext(List *const this) { return objMemContext(this); } // List size FN_INLINE_ALWAYS unsigned int lstSize(const List *const this) { return THIS_PUB(List)->listSize; } // Is the list empty? FN_INLINE_ALWAYS bool lstEmpty(const List *const this) { return lstSize(this) == 0; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Clear items from a list FN_EXTERN List *lstClear(List *this); // Get an item from the list FN_EXTERN void *lstGet(const List *this, unsigned int listIdx); FN_EXTERN void *lstGetLast(const List *this); // Find an item in the list FN_EXTERN void *lstFind(const List *this, const void *item); FN_EXTERN const void *lstFindDefault(const List *this, const void *item, const void *itemDefault); FN_EXTERN unsigned int lstFindIdx(const List *this, const void *item); // Does an item exist in the list? FN_INLINE_ALWAYS bool lstExists(const List *const this, const void *const item) { return lstFind(this, item) != NULL; } // Get the index of a list item FN_EXTERN unsigned int lstIdx(const List *this, const void *item); // Insert an item into the list FN_EXTERN void *lstInsert(List *this, unsigned int listIdx, const void *item); // Add an item to the end of the list FN_INLINE_ALWAYS void * lstAdd(List *const this, const void *const item) { return lstInsert(this, lstSize(this), item); } // Move to a new parent mem context FN_INLINE_ALWAYS List * lstMove(List *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Remove an item from the list FN_EXTERN bool lstRemove(List *this, const void *item); FN_EXTERN List *lstRemoveIdx(List *this, unsigned int listIdx); FN_EXTERN List *lstRemoveLast(List *this); // List sort FN_EXTERN List *lstSort(List *this, SortOrder sortOrder); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void lstFree(List *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for constant lists Frequently used constant lists can be declared with these macros at compile time rather than dynamically at run time. Note that lists created in this way are declared as const so can't be modified or freed by the lst*() methods. Casting to List * will result in a segfault due to modifying read-only memory. By convention all List constant identifiers are appended with _LST. ***********************************************************************************************************************************/ // This struct must be kept in sync with ListPub (except for const qualifiers) typedef struct ListPubConst { unsigned int listSize; // List size size_t itemSize; // Size of item stored in the list const uint8_t *list; // Pointer to the current start of the list } ListPubConst; // Create a List constant inline from a constant array #define LSTDEF(listParam) \ (const List *)&(ListPubConst){ \ .list = (const uint8_t *)listParam, .itemSize = sizeof((listParam)[0]), .listSize = LENGTH_OF(listParam)} /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void lstToLog(const List *this, StringStatic *debugLog); #define FUNCTION_LOG_LIST_TYPE \ List * #define FUNCTION_LOG_LIST_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, lstToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/object.c0000664000175100017510000000240615265101745020056 0ustar00runnerrunner/*********************************************************************************************************************************** Object Helper Macros and Functions ***********************************************************************************************************************************/ #include #include "common/type/object.h" /**********************************************************************************************************************************/ FN_EXTERN void * objMove(THIS_VOID, MemContext *const parentNew) { if (thisVoid != NULL) memContextMove(memContextFromAllocExtra(thisVoid), parentNew); return thisVoid; } /**********************************************************************************************************************************/ FN_EXTERN void * objMoveToInterface(THIS_VOID, void *const interfaceVoid, const MemContext *const current) { return objMemContext(thisVoid) != current ? objMove(thisVoid, objMemContext(interfaceVoid)) : thisVoid; } /**********************************************************************************************************************************/ FN_EXTERN void objFree(THIS_VOID) { if (thisVoid != NULL) memContextFree(memContextFromAllocExtra(thisVoid)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/object.h0000664000175100017510000001632115265101745020064 0ustar00runnerrunner/*********************************************************************************************************************************** Object Helper Macros and Functions These macros and functions implement common object functionality. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_OBJECT_H #define COMMON_TYPE_OBJECT_H #include "common/assert.h" #include "common/macro.h" #include "common/memContext.h" /*********************************************************************************************************************************** Create a new object This is a thin wrapper on MEM_CONTEXT_NEW_*() to do object specific initialization. The general pattern for a new object is: OBJ_NEW_BEGIN(MyObj) { *this = (MyObj) { .data = ... }; } OBJ_NEW_END(); The this variable is variable is automatically created with the specified type. The BASE and EXTRA variants are available when a custom variable must be used or different amount of memory must be allocated. ***********************************************************************************************************************************/ #define OBJ_NEW_BASE_EXTRA_BEGIN(type, extra, ...) \ MEM_CONTEXT_NEW_BEGIN(type, .allocExtra = extra, __VA_ARGS__) #define OBJ_NEW_EXTRA_BEGIN(type, extra, ...) \ memContextSwitch(memContextNewP(STRINGIFY(type), .allocExtra = extra, __VA_ARGS__)); \ type *const this = OBJ_NEW_ALLOC(); \ \ do \ { #define OBJ_NEW_BASE_BEGIN(type, ...) \ OBJ_NEW_BASE_EXTRA_BEGIN(type, sizeof(type), __VA_ARGS__) #define OBJ_NEW_BEGIN(type, ...) \ OBJ_NEW_EXTRA_BEGIN(type, sizeof(type), __VA_ARGS__) #define OBJ_NEW_ALLOC() \ memContextAllocExtra(memContextCurrent()) #define OBJ_NEW_END() \ MEM_CONTEXT_NEW_END() /*********************************************************************************************************************************** Rename an object for auditing purposes. The original name for an object is based on the type used to create the object but a different name might be needed to identify it during auditing. For example, a filter named IoSink would need to be renamed to IoFilter to identify it as a filter for auditing. This only has an effect in test builds. ***********************************************************************************************************************************/ #define OBJ_NAME(this, name) MEM_CONTEXT_AUDIT_ALLOC_EXTRA_NAME(this, name) /*********************************************************************************************************************************** Used in interface function parameter lists to discourage use of the untyped thisVoid parameter, e.g.: size_t bufferRead(THIS_VOID, Buffer *buffer) This macro should not be used unless the function is assigned to an interface or passed as a parameter. ***********************************************************************************************************************************/ #define THIS_VOID void *thisVoid /*********************************************************************************************************************************** Create a local "this" variable of the correct type from a THIS_VOID parameter ***********************************************************************************************************************************/ #define THIS(type) type *this = thisVoid /*********************************************************************************************************************************** Cast this private struct, e.g. List, to the associated public struct, e.g. ListPub. Note that the public struct must be the first member of the private struct. For example: FN_INLINE_ALWAYS unsigned int lstSize(const List *const this) { return THIS_PUB(List)->listSize; } The macro also ensures that this != NULL so there is no need to do that in the calling function. ***********************************************************************************************************************************/ #define THIS_PUB(type) ((const type##Pub *)thisNotNull(this)) FN_INLINE_ALWAYS const void * thisNotNull(const void *const this) { ASSERT_INLINE(this != NULL); return this; } /*********************************************************************************************************************************** Switch to the object memory context and ensure that the prior memory context is restored after the block executes (even on error) ***********************************************************************************************************************************/ #define MEM_CONTEXT_OBJ_BEGIN(this) MEM_CONTEXT_BEGIN(objMemContext(this)) #define MEM_CONTEXT_OBJ_END() MEM_CONTEXT_END() /*********************************************************************************************************************************** Functions To ensure proper type checking, these functions are meant to be called from inline functions created specifically for each object: FN_INLINE_ALWAYS void storageFree(Storage *this) { objFree(this); } ***********************************************************************************************************************************/ // Get the object mem context FN_INLINE_ALWAYS MemContext * objMemContext(void *const this) { return memContextFromAllocExtra(this); } // Move an object to a new context if this != NULL FN_EXTERN void *objMove(THIS_VOID, MemContext *parentNew); // Move driver object into an interface object when required. This is the general case, but sometimes an object will expose its // interfaces in a different way, e.g. methods, so the object retains ownership of the interfaces. If this function is called in the // driver object context then the driver object will retain ownership of the interface. Otherwise, the interface object will become // the owner of the driver object. FN_EXTERN void *objMoveToInterface(THIS_VOID, void *interfaceVoid, const MemContext *current); // Free the object mem context if this != NULL FN_EXTERN void objFree(THIS_VOID); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/pack.c0000664000175100017510000020457515265101745017541 0ustar00runnerrunner/*********************************************************************************************************************************** Pack Type Each pack field begins with a one byte tag. The four high order bits of the tag contain the field type (PackTypeMap). To allow more types than four bits will allow, the type bits can be set to 0xF and the rest of the type (- 0xF) will be stored by a base-128 encoded integer immediately following the tag. The four lower order bits vary by type. When the "more ID delta" indicator is set then the tag (and type, if any) will be followed by a base-128 encoded integer with the higher order ID delta bits. The ID delta represents the delta from the ID of the previous field. When the "more value indicator" then the tag (and the type and ID delta, if any) will be followed by a base-128 encoded integer with the high order value bits, i.e. the bits that were not stored directly in the tag. For integer types the value is the integer being stored but for string and binary types the value is 1 if the size is greater than 0 and 0 if the size is 0. When the size is greater than 0 the tag is immediately followed by (or after the delta ID if "more ID delta" is set) the base-128 encoded size and then by the string/binary bytes. For string and binary types the value bit indicates if there is data, not the length of the data, which is why the length is stored immediately following the tag when the value bit is set. This prevents storing an additional byte when the string/binary length is zero. The following are definitions for the pack tag field and examples of how it is interpreted. Integer types (packTypeMapData[type].valueMultiBit) when an unsigned value is <= 1 or a signed value is >= -1 and <= 0: 3 - more value indicator bit set to 0 2 - value low order bit 1 - more ID delta indicator bit 0 - ID delta low order bit Example: b704 b = unsigned int 64 type 7 = tag byte low bits: 0 1 1 1 meaning: "value low order bit" - the value of the u64 field is 1 "more ID delta indicator bit" - there exists a gap (i.e. NULLs are not stored so there is a gap between the stored IDs) "ID delta low order bit" - gaps are interpreted as the currently stored ID minus previously stored ID minus 1, therefore if the previously store ID is 1 and the ID of this u64 field is 11 then a gap of 10 exists. 10 is represented internally as 9 since there is always at least a gap of 1 which never needs to be recorded (it is a given). 9 in bit format is 1 0 0 1 - the low-order bit is 1 so the "ID delta low order bit" is set. 04 = since the low order bit of the internal ID delta was already set in bit 0 of the tag byte, then the remain bits are shifted right by one and represented in this second byte as 4. To get the ID delta for 04, shift the 4 back to the left one and then add back the "ID delta low order bit" to give a binary representation of 1 0 0 1 = 9. Add back the 1 which is never recorded and the ID gap is 10. Integer types (packTypeMapData[type].valueMultiBit) when an unsigned value is > 1 or a signed value is < -1 or > 0: 3 - more value indicator bit set to 1 2 - more ID delta indicator bit 0-1 - ID delta low order bits Example: 5e021f 5 = signed int 64 type e = tag byte low bits: 1 1 1 0 meaning: "more value indicator bit set to 1" - the actual value is < -1 or > 0 "more ID delta indicator bit" - there exists a gap (i.e. NULLs are not stored so there is a gap between the stored IDs) "ID delta low order bits" - here the bit 1 is set to 1 and bit 0 is not so the ID delta has the second low order bit set but not the first 02 = since bit 0 and bit 1 of the tag byte are accounted for then the 02 is the result of shifting the ID delta right by 2. Shifting the 2 back to the left by 2 and adding back the second low order bit as 1 and the first low order bit as 0 then the bit representation would be 1 0 1 0 which is ten (10) so the gap between the IDs is 11. 1f = signed, zigzag representation of -16 (the actual value) String, binary types, and boolean (packTypeMapData[type].valueSingleBit): 3 - value bit 2 - more ID delta indicator bit 0-1 - ID delta low order bits Note: binary type is interpreted the same way as string type Example: 8c090673616d706c65 8 = string type c = tag byte low bits: 1 1 0 0 meaning: "value bit" - there is data "more ID delta indicator bit" - there exists a gap (i.e. NULLs are not stored so there is a gap between the stored IDs) 09 = since neither "ID delta low order bits" is set in the tag, they are both 0, so shifting 9 left by 2, the 2 low order bits are now 0 so the result is 0x24 = 36 in decimal. Add back the 1 which is never recorded and the ID gap is 37. 06 = the length of the string is 6 bytes 73616d706c65 = the 6 bytes of the string value ("sample") Example: 30 3 = boolean type 0 = "value bit" 0 means the value is false Note that if the boolean had been pack written with .defaultWrite = false there would have been a gap instead of the 30. Array and object types: 3 - more ID delta indicator bit 0-2 - ID delta low order bits Note: arrays and objects are merely containers for the other pack types. Example: 1801 (container begin) 1 = array type 8 = "more ID delta indicator bit" - there exists a gap (i.e. NULLs are not stored so there is a gap between the stored IDs) 01 = since there are three "ID delta low order bits", the 01 will be shifted left by 3 with zeros, resulting in 8. Add back the 1 which is never recorded and the ID gap is 9. ... 00 = container end - the array/object container end will occur when a 0 byte (00) is encountered that is not part of a pack field within the array/object ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/bufferWrite.h" #include "common/io/io.h" #include "common/io/read.h" #include "common/io/write.h" #include "common/type/convert.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Map PackType types to the types that will be written into the pack. This hides the details of the type IDs from the user and allows the IDs used in the pack to differ from the IDs the user sees. ***********************************************************************************************************************************/ typedef enum { pckTypeMapUnknown = 0, // Used internally when the type is not known pckTypeMapArray = 1, // Maps to pckTypeArray pckTypeMapBool = 2, // Maps to pckTypeBool pckTypeMapI32 = 3, // Maps to pckTypeI32 pckTypeMapI64 = 4, // Maps to pckTypeI64 pckTypeMapObj = 5, // Maps to pckTypeObj // The empty position 6 can be used for a new type that will be encoded entirely in the tag pckTypeMapStr = 7, // Maps to pckTypeStr pckTypeMapU32 = 8, // Maps to pckTypeU32 pckTypeMapU64 = 9, // Maps to pckTypeU64 pckTypeMapStrId = 10, // Maps to pckTypeStrId // The empty positions before 15 can be used for new types that will be encoded entirely in the tag pckTypeMapTime = 15, // Maps to pckTypeTime pckTypeMapBin = 16, // Maps to pckTypeBin pckTypeMapPack = 17, // Maps to pckTypePack pckTypeMapMode = 18, // Maps to pckTypeMode } PackTypeMap; typedef struct PackTypeMapData { PackType type; // Data type bool valueSingleBit; // Can the value be stored in a single bit (e.g. bool) bool valueMultiBit; // Can the value require multiple bits (e.g. integer) bool size; // Does the type require a size (e.g. string) } PackTypeMapData; static const PackTypeMapData packTypeMapData[] = { // Unknown type map data should not be used [pckTypeMapUnknown] = {0}, // Types that can be encoded entirely in the tag [pckTypeMapArray] = { .type = pckTypeArray, }, [pckTypeMapBool] = { .type = pckTypeBool, .valueSingleBit = true, }, [pckTypeMapI32] = { .type = pckTypeI32, .valueMultiBit = true, }, [pckTypeMapI64] = { .type = pckTypeI64, .valueMultiBit = true, }, [pckTypeMapObj] = { .type = pckTypeObj, }, [pckTypeMapStr] = { .type = pckTypeStr, .valueSingleBit = true, .size = true, }, [pckTypeMapU32] = { .type = pckTypeU32, .valueMultiBit = true, }, [pckTypeMapU64] = { .type = pckTypeU64, .valueMultiBit = true, }, [pckTypeMapStrId] = { .type = pckTypeStrId, .valueMultiBit = true, }, // Types that require an extra byte to encode [pckTypeMapTime] = { .type = pckTypeTime, .valueMultiBit = true, }, [pckTypeMapBin] = { .type = pckTypeBin, .valueSingleBit = true, .size = true, }, [pckTypeMapPack] = { .type = pckTypePack, .size = true, }, [pckTypeMapMode] = { .type = pckTypeMode, .valueMultiBit = true, }, }; /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ typedef struct PackTagStackItem { PackTypeMap typeMap; // Tag type map unsigned int idLast; // Last id in the container unsigned int nullTotal; // Total nulls since last tag written } PackTagStackItem; typedef struct PackTagStack { List *stack; // Stack of object/array tags PackTagStackItem bottom; // Static bottom of the stack PackTagStackItem *top; // Top tag on the stack unsigned int depth; // Stack depth } PackTagStack; struct PackRead { IoRead *read; // Read pack from Buffer *buffer; // Buffer containing read data const uint8_t *bufferPtr; // Pointer to buffer size_t bufferPos; // Position in the buffer size_t bufferUsed; // Amount of data in the buffer unsigned int tagNextId; // Next tag id PackTypeMap tagNextTypeMap; // Next tag type map uint64_t tagNextValue; // Next tag value size_t tagNextSize; // Next tag size PackTagStack tagStack; // Stack of object/array tags }; struct PackWrite { IoWrite *write; // Write pack to Buffer *buffer; // Buffer to contain write data PackTagStack tagStack; // Stack of object/array tags }; /*********************************************************************************************************************************** Push a container onto the tag stack ***********************************************************************************************************************************/ static void pckTagStackPush(MemContext *const memContext, PackTagStack *const tagStack, const PackTypeMap typeMap) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, memContext); FUNCTION_TEST_PARAM_P(VOID, tagStack); FUNCTION_TEST_PARAM(ENUM, typeMap); FUNCTION_TEST_END(); ASSERT(tagStack != NULL); ASSERT(typeMap == pckTypeMapArray || typeMap == pckTypeMapObj); if (tagStack->stack == NULL) { MEM_CONTEXT_BEGIN(memContext) { tagStack->stack = lstNewP(sizeof(PackTagStackItem)); } MEM_CONTEXT_END(); } tagStack->top = lstAdd(tagStack->stack, &(PackTagStackItem){.typeMap = typeMap}); tagStack->depth++; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Check that a container can be popped off the stack ***********************************************************************************************************************************/ static void pckTagStackPopCheck(PackTagStack *const tagStack, const PackTypeMap typeMap) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, tagStack); FUNCTION_TEST_PARAM(ENUM, typeMap); FUNCTION_TEST_END(); ASSERT(tagStack != NULL); ASSERT(typeMap == pckTypeMapArray || typeMap == pckTypeMapObj); if (tagStack->depth == 0 || tagStack->top->typeMap != typeMap) THROW_FMT(FormatError, "not in %s", zNewStrId(packTypeMapData[typeMap].type)); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Pop a container off the tag stack ***********************************************************************************************************************************/ static void pckTagStackPop(PackTagStack *const tagStack) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, tagStack); FUNCTION_TEST_END(); ASSERT(tagStack != NULL); ASSERT(tagStack->depth != 0); ASSERT(tagStack->stack != NULL); lstRemoveLast(tagStack->stack); tagStack->depth--; if (tagStack->depth == 0) tagStack->top = &tagStack->bottom; else tagStack->top = lstGetLast(tagStack->stack); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ // Helper to create common data static PackRead * pckReadNewInternal(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(PackRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (PackRead) { .tagStack = {.bottom = {.typeMap = pckTypeMapObj}}, }; this->tagStack.top = &this->tagStack.bottom; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(PACK_READ, this); } FN_EXTERN PackRead * pckReadNewIo(IoRead *const read) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_READ, read); FUNCTION_TEST_END(); ASSERT(read != NULL); PackRead *const this = pckReadNewInternal(); MEM_CONTEXT_OBJ_BEGIN(this) { this->read = read; this->buffer = bufNew(ioBufferSize()); this->bufferPtr = bufPtr(this->buffer); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(PACK_READ, this); } FN_EXTERN PackRead * pckReadNew(const Pack *const pack) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK, pack); FUNCTION_TEST_END(); if (pack == NULL) FUNCTION_TEST_RETURN(PACK_READ, NULL); FUNCTION_TEST_RETURN(PACK_READ, pckReadNewC(bufPtrConst((const Buffer *)pack), bufUsed((const Buffer *)pack))); } FN_EXTERN PackRead * pckReadNewC(const uint8_t *const buffer, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(buffer != NULL); PackRead *const this = pckReadNewInternal(); this->bufferPtr = buffer; this->bufferUsed = size; FUNCTION_TEST_RETURN(PACK_READ, this); } /*********************************************************************************************************************************** Read bytes from the buffer IMPORTANT NOTE: To avoid having dynamically created return buffers the current buffer position (this->bufferPos) is stored in the object. Therefore this function should not be used as a parameter in other function calls since the value of this->bufferPos will change. ***********************************************************************************************************************************/ static size_t pckReadBuffer(PackRead *const this, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(this != NULL); size_t remaining = this->bufferUsed - this->bufferPos; if (remaining < size) { if (this->read != NULL) { // Nothing can be remaining since each read fetches exactly the number of bytes required ASSERT(remaining == 0); bufUsedZero(this->buffer); // Limit the buffer for the next read so we don't read past the end of the pack bufLimitSet(this->buffer, size < bufSizeAlloc(this->buffer) ? size : bufSizeAlloc(this->buffer)); // Read bytes ioReadSmall(this->read, this->buffer); this->bufferPos = 0; this->bufferUsed = bufUsed(this->buffer); remaining = this->bufferUsed; } if (remaining < 1) THROW(FormatError, "unexpected EOF"); FUNCTION_TEST_RETURN(SIZE, remaining < size ? remaining : size); } FUNCTION_TEST_RETURN(SIZE, size); } /*********************************************************************************************************************************** Consume buffer left in tagNextSize ***********************************************************************************************************************************/ static void pckReadConsumeBuffer(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); while (this->tagNextSize != 0) { const size_t size = pckReadBuffer(this, this->tagNextSize); this->bufferPos += size; this->tagNextSize -= size; } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Unpack an unsigned 64-bit integer from base-128 varint encoding ***********************************************************************************************************************************/ static uint64_t pckReadU64Internal(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); if (this->read != NULL) { // Internal buffer should be empty ASSERT(this->bufferUsed == this->bufferPos); FUNCTION_TEST_RETURN(UINT64, ioReadVarIntU64(this->read)); } FUNCTION_TEST_RETURN(UINT64, cvtUInt64FromVarInt128(this->bufferPtr, &this->bufferPos, this->bufferUsed)); } /*********************************************************************************************************************************** Read next field tag ***********************************************************************************************************************************/ static bool pckReadTagNext(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); bool result = false; // Read the tag byte pckReadBuffer(this, 1); const unsigned int tag = this->bufferPtr[this->bufferPos]; this->bufferPos++; // If the current container is complete (e.g. object) if (tag == 0) { this->tagNextId = UINT_MAX; } // Else a regular tag else { // Read field type (e.g. int64, string) this->tagNextTypeMap = tag >> 4; if (this->tagNextTypeMap == 0xF) this->tagNextTypeMap = (unsigned int)pckReadU64Internal(this) + 0xF; CHECK( FormatError, this->tagNextTypeMap < LENGTH_OF(packTypeMapData) && packTypeMapData[this->tagNextTypeMap].type != 0, "invalid tag type"); // If the value can contain multiple bits (e.g. integer) if (packTypeMapData[this->tagNextTypeMap].valueMultiBit) { // If the value is stored following the tag (value > 1 bit) if (tag & 0x8) { // Read low order bits of the field ID delta this->tagNextId = tag & 0x3; // Read high order bits of the field ID delta when specified if (tag & 0x4) this->tagNextId |= (unsigned int)pckReadU64Internal(this) << 2; // Read value this->tagNextValue = pckReadU64Internal(this); } // Else the value is stored in the tag (value == 1 bit) else { // Read low order bit of the field ID delta this->tagNextId = tag & 0x1; // Read high order bits of the field ID delta when specified if (tag & 0x2) this->tagNextId |= (unsigned int)pckReadU64Internal(this) << 1; // Read value this->tagNextValue = (tag >> 2) & 0x3; } } // Else the value is a single bit (e.g. boolean) else if (packTypeMapData[this->tagNextTypeMap].valueSingleBit) { // Read low order bits of the field ID delta this->tagNextId = tag & 0x3; // Read high order bits of the field ID delta when specified if (tag & 0x4) this->tagNextId |= (unsigned int)pckReadU64Internal(this) << 2; // Read value this->tagNextValue = (tag >> 3) & 0x1; } // Else the value is multiple tags (e.g. container) else { // Read low order bits of the field ID delta this->tagNextId = tag & 0x7; // Read high order bits of the field ID delta when specified if (tag & 0x8) this->tagNextId |= (unsigned int)pckReadU64Internal(this) << 3; // Value length is variable so is stored after the tag this->tagNextValue = 0; } // Increment the next tag id this->tagNextId += this->tagStack.top->idLast + 1; // Error when the id wrapped or is UINT_MAX (end of container) CHECK(FormatError, this->tagNextId > this->tagStack.top->idLast && this->tagNextId != UINT_MAX, "invalid field id"); // Get tag size if it exists if (packTypeMapData[this->tagNextTypeMap].size && (this->tagNextValue > 0 || !packTypeMapData[this->tagNextTypeMap].valueSingleBit)) { this->tagNextSize = (size_t)pckReadU64Internal(this); // A pack read from a buffer cannot have a field larger than the bytes remaining CHECK(FormatError, this->read != NULL || this->tagNextSize <= this->bufferUsed - this->bufferPos, "unexpected EOF"); } else this->tagNextSize = 0; // Tag was found result = true; } FUNCTION_TEST_RETURN(BOOL, result); } /*********************************************************************************************************************************** Read field tag Some tags and data may be skipped based on the value of the id parameter. ***********************************************************************************************************************************/ static uint64_t pckReadTag(PackRead *const this, unsigned int *const id, const PackTypeMap typeMap, const bool peek) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM_P(UINT, id); FUNCTION_TEST_PARAM(ENUM, typeMap); FUNCTION_TEST_PARAM(BOOL, peek); // Look at the next tag without advancing the field id FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(id != NULL); ASSERT((peek && typeMap == pckTypeMapUnknown) || (!peek && typeMap != pckTypeMapUnknown)); // Increment the id by one if no id was specified if (*id == 0) { *id = this->tagStack.top->idLast + 1; } // Else check that the id has been incremented else if (*id <= this->tagStack.top->idLast) THROW_FMT(FormatError, "field %u was already read", *id); // Search for the requested id do { // Get the next tag if it has not been read yet if (this->tagNextId == 0) pckReadTagNext(this); // Return if the id does not exist if (*id < this->tagNextId) { break; } // Else the id exists else if (*id == this->tagNextId) { // When not peeking the next tag (just to see what it is) then error if the type map is not as specified if (!peek) { if (this->tagNextTypeMap != typeMap) { THROW_FMT( FormatError, "field %u is type '%s' but expected '%s'", this->tagNextId, zNewStrId(packTypeMapData[this->tagNextTypeMap].type), zNewStrId(packTypeMapData[typeMap].type)); } this->tagStack.top->idLast = this->tagNextId; this->tagNextId = 0; } break; } // Read data for the field being skipped if this is not the field requested pckReadConsumeBuffer(this); // Increment the last id to the id just read this->tagStack.top->idLast = this->tagNextId; // Read tag on the next iteration this->tagNextId = 0; } while (1); FUNCTION_TEST_RETURN(UINT64, this->tagNextValue); } /**********************************************************************************************************************************/ FN_EXTERN bool pckReadNext(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->tagNextId == 0); FUNCTION_TEST_RETURN(BOOL, pckReadTagNext(this)); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int pckReadId(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(UINT, this->tagNextId); } /**********************************************************************************************************************************/ FN_EXTERN size_t pckReadSize(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(packTypeMapData[this->tagNextTypeMap].size); FUNCTION_TEST_RETURN(SIZE, this->tagNextSize); } /**********************************************************************************************************************************/ FN_EXTERN void pckReadConsume(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); unsigned int id = 0; pckReadTag(this, &id, pckTypeMapUnknown, true); pckReadTag(this, &id, this->tagNextTypeMap, false); pckReadConsumeBuffer(this); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN const uint8_t * pckReadBufPtr(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(packTypeMapData[this->tagNextTypeMap].size); ASSERT(this->buffer == NULL); FUNCTION_TEST_RETURN_CONST_P(BYTEDATA, this->bufferPtr + this->bufferPos); } /**********************************************************************************************************************************/ // Internal version of pckReadNull() that does not require a PackIdParam struct. Some functions already have an id variable so // assigning that to a PackIdParam struct and then copying it back is wasteful. static inline bool pckReadNullInternal(PackRead *const this, unsigned int *const id) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM_P(UINT, id); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(id != NULL); // Read tag at specified id pckReadTag(this, id, pckTypeMapUnknown, true); // If the field is NULL then set idLast (to avoid rechecking the same id on the next call) and return true if (*id < this->tagNextId) { this->tagStack.top->idLast = *id; FUNCTION_TEST_RETURN(BOOL, true); } // The field is not NULL FUNCTION_TEST_RETURN(BOOL, false); } FN_EXTERN bool pckReadNull(PackRead *const this, PackIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, pckReadNullInternal(this, ¶m.id)); } /**********************************************************************************************************************************/ FN_EXTERN PackType pckReadType(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN_TYPE(PackType, packTypeMapData[this->tagNextTypeMap].type); } /**********************************************************************************************************************************/ FN_EXTERN void pckReadArrayBegin(PackRead *const this, PackIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); // Read array begin pckReadTag(this, ¶m.id, pckTypeMapArray, false); // Add array to the tag stack so IDs can be tracked separately from the parent container pckTagStackPush(objMemContext(this), &this->tagStack, pckTypeMapArray); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void pckReadArrayEnd(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Check tag stack pop pckTagStackPopCheck(&this->tagStack, pckTypeMapArray); // Make sure we are at the end of the array unsigned int id = UINT_MAX - 1; pckReadTag(this, &id, pckTypeMapUnknown, true); // Pop array off the stack pckTagStackPop(&this->tagStack); // Reset tagNextId to keep reading this->tagNextId = 0; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * pckReadBin(PackRead *const this, PckReadBinParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(BUFFER, NULL); Buffer *result; // If buffer size > 0 if (pckReadTag(this, ¶m.id, pckTypeMapBin, false)) { // Get the buffer size result = bufNew(this->tagNextSize); // Read the buffer out in chunks while (bufUsed(result) < bufSize(result)) { size_t size = pckReadBuffer(this, bufRemains(result)); bufCatC(result, this->bufferPtr, this->bufferPos, size); this->bufferPos += size; } } // Else return a zero-sized buffer else result = bufNew(0); FUNCTION_TEST_RETURN(BUFFER, result); } /**********************************************************************************************************************************/ FN_EXTERN bool pckReadBool(PackRead *const this, PckReadBoolParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(BOOL, param.defaultValue); FUNCTION_TEST_RETURN(BOOL, pckReadTag(this, ¶m.id, pckTypeMapBool, false)); } /**********************************************************************************************************************************/ FN_EXTERN int32_t pckReadI32(PackRead *const this, PckReadI32Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(INT, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(INT, param.defaultValue); FUNCTION_TEST_RETURN(INT, cvtInt32FromZigZag((uint32_t)pckReadTag(this, ¶m.id, pckTypeMapI32, false))); } /**********************************************************************************************************************************/ FN_EXTERN int64_t pckReadI64(PackRead *const this, PckReadI64Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(INT64, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(INT64, param.defaultValue); FUNCTION_TEST_RETURN(INT64, cvtInt64FromZigZag(pckReadTag(this, ¶m.id, pckTypeMapI64, false))); } /**********************************************************************************************************************************/ FN_EXTERN mode_t pckReadMode(PackRead *const this, PckReadModeParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(MODE, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(MODE, param.defaultValue); FUNCTION_TEST_RETURN(MODE, (mode_t)pckReadTag(this, ¶m.id, pckTypeMapMode, false)); } /**********************************************************************************************************************************/ FN_EXTERN void pckReadObjBegin(PackRead *const this, PackIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); // Read object begin pckReadTag(this, ¶m.id, pckTypeMapObj, false); // Add object to the tag stack so IDs can be tracked separately from the parent container pckTagStackPush(objMemContext(this), &this->tagStack, pckTypeMapObj); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void pckReadObjEnd(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Check tag stack pop pckTagStackPopCheck(&this->tagStack, pckTypeMapObj); // Make sure we are at the end of the object unsigned id = UINT_MAX - 1; pckReadTag(this, &id, pckTypeMapUnknown, true); // Pop object off the stack pckTagStackPop(&this->tagStack); // Reset tagNextId to keep reading this->tagNextId = 0; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * pckReadPackRead(PackRead *const this, PckReadPackParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); Pack *const pack = pckReadPack(this, param); PackRead *const result = pckReadNew(pack); if (result != NULL) pckMove(pack, objMemContext(result)); FUNCTION_TEST_RETURN(PACK_READ, result); } FN_EXTERN PackRead * pckReadPackReadConst(PackRead *const this, PckReadPackParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(PACK_READ, NULL); // Read the tag pckReadTag(this, ¶m.id, pckTypeMapPack, false); // A pack has at least the end tag CHECK(FormatError, this->tagNextSize > 0, "invalid pack size"); PackRead *const result = pckReadNewC(pckReadBufPtr(this), pckReadSize(this)); pckReadConsumeBuffer(this); FUNCTION_TEST_RETURN(PACK_READ, result); } FN_EXTERN Pack * pckReadPack(PackRead *const this, PckReadPackParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(PACK, NULL); // Read the tag pckReadTag(this, ¶m.id, pckTypeMapPack, false); // A pack has at least the end tag CHECK(FormatError, this->tagNextSize > 0, "invalid pack size"); // Get the pack size Buffer *const result = OBJ_NAME(bufNew(this->tagNextSize), Pack::Buffer); // Read the pack out in chunks while (bufUsed(result) < bufSize(result)) { size_t size = pckReadBuffer(this, bufRemains(result)); bufCatC(result, this->bufferPtr, this->bufferPos, size); this->bufferPos += size; } FUNCTION_TEST_RETURN(PACK, (Pack *)result); } /**********************************************************************************************************************************/ FN_EXTERN String * pckReadStr(PackRead *const this, PckReadStrParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(STRING, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(STRING, strDup(param.defaultValue)); String *result; // If string size > 0 if (pckReadTag(this, ¶m.id, pckTypeMapStr, false)) { // Read the string size const size_t sizeExpected = (size_t)this->tagNextSize; // Read the string out in chunks result = strNew(); while (strSize(result) != sizeExpected) { const size_t sizeRead = pckReadBuffer(this, sizeExpected - strSize(result)); strCatZN(result, (const char *)this->bufferPtr + this->bufferPos, sizeRead); this->bufferPos += sizeRead; } } // Else return an empty string else result = strNew(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN StringId pckReadStrId(PackRead *const this, PckReadStrIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(UINT64, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(STRING_ID, param.defaultValue); FUNCTION_TEST_RETURN(STRING_ID, pckReadTag(this, ¶m.id, pckTypeMapStrId, false)); } /**********************************************************************************************************************************/ FN_EXTERN StringList * pckReadStrLst(PackRead *const this, PckReadStrLstParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(STRING_LIST, NULL); pckReadArrayBeginP(this, .id = param.id); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { while (!pckReadNullP(this)) strLstAdd(result, pckReadStrP(this)); } MEM_CONTEXT_TEMP_END(); pckReadArrayEndP(this); FUNCTION_TEST_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN time_t pckReadTime(PackRead *const this, PckReadTimeParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(TIME, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(TIME, param.defaultValue); FUNCTION_TEST_RETURN(TIME, (time_t)cvtInt64FromZigZag(pckReadTag(this, ¶m.id, pckTypeMapTime, false))); } /**********************************************************************************************************************************/ FN_EXTERN uint32_t pckReadU32(PackRead *const this, PckReadU32Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(UINT32, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(UINT32, param.defaultValue); FUNCTION_TEST_RETURN(UINT32, (uint32_t)pckReadTag(this, ¶m.id, pckTypeMapU32, false)); } /**********************************************************************************************************************************/ FN_EXTERN uint64_t pckReadU64(PackRead *const this, PckReadU64Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(UINT64, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(UINT64, param.defaultValue); FUNCTION_TEST_RETURN(UINT64, pckReadTag(this, ¶m.id, pckTypeMapU64, false)); } /**********************************************************************************************************************************/ FN_EXTERN void pckReadEnd(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); CHECK(FormatError, this->tagStack.depth == 0, "invalid pack end read in container"); // Make sure we are at the end of the pack unsigned int id = UINT_MAX - 1; pckReadTag(this, &id, pckTypeMapUnknown, true); this->tagStack.top = NULL; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void pckReadToLog(const PackRead *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{depth: %u, idLast: %u, tagNextId: %u, tagNextType: %u, tagNextValue %" PRIu64 "}", this->tagStack.depth, this->tagStack.top->idLast, this->tagNextId, this->tagNextTypeMap, this->tagNextValue); } /**********************************************************************************************************************************/ // Helper to create common data static PackWrite * pckWriteNewInternal(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(PackWrite, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (PackWrite) { .tagStack = {.bottom = {.typeMap = pckTypeMapObj}}, }; this->tagStack.top = &this->tagStack.bottom; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(PACK_WRITE, this); } FN_EXTERN PackWrite * pckWriteNew(const PckWriteNewParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, param.size); FUNCTION_TEST_END(); PackWrite *this = pckWriteNewInternal(); MEM_CONTEXT_OBJ_BEGIN(this) { this->buffer = bufNew(param.size == 0 ? PACK_EXTRA_MIN : param.size); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(PACK_WRITE, this); } FN_EXTERN PackWrite * pckWriteNewIo(IoWrite *const write) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_WRITE, write); FUNCTION_TEST_END(); ASSERT(write != NULL); PackWrite *const this = pckWriteNewInternal(); MEM_CONTEXT_OBJ_BEGIN(this) { this->write = write; this->buffer = bufNew(ioBufferSize()); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /*********************************************************************************************************************************** Write to io or buffer ***********************************************************************************************************************************/ static void pckWriteBuffer(PackWrite *const this, const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(this != NULL); // If writing directly to a buffer if (this->write == NULL) { // Add space in the buffer to write and add extra space so future writes won't always need to resize the buffer if (bufRemains(this->buffer) < bufUsed(buffer)) bufResize(this->buffer, (bufSizeAlloc(this->buffer) + bufUsed(buffer)) + PACK_EXTRA_MIN); // Write to the buffer bufCat(this->buffer, buffer); } // Else writing to io else { // If there's enough space to write to the internal buffer then do that if (bufRemains(this->buffer) >= bufUsed(buffer)) bufCat(this->buffer, buffer); else { // Flush the internal buffer if it has data if (!bufEmpty(this->buffer)) { ioWrite(this->write, this->buffer); bufUsedZero(this->buffer); } // If there's enough space to write to the internal buffer then do that if (bufRemains(this->buffer) >= bufUsed(buffer)) { bufCat(this->buffer, buffer); } // Else write directly to io else ioWrite(this->write, buffer); } } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Pack an unsigned 64-bit integer to base-128 varint encoding ***********************************************************************************************************************************/ static void pckWriteU64Internal(PackWrite *const this, const uint64_t value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_END(); ASSERT(this != NULL); uint8_t buffer[CVT_VARINT128_BUFFER_SIZE]; size_t size = 0; cvtUInt64ToVarInt128(value, buffer, &size, sizeof(buffer)); // Write encoded bytes to the buffer pckWriteBuffer(this, BUF(buffer, size)); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Write field tag ***********************************************************************************************************************************/ static void pckWriteTag(PackWrite *const this, const PackTypeMap typeMap, unsigned int id, uint64_t value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(ENUM, typeMap); FUNCTION_TEST_PARAM(UINT, id); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_END(); ASSERT(this != NULL); // If id is not specified then add one to previous tag (and include all NULLs) if (id == 0) { id = this->tagStack.top->idLast + this->tagStack.top->nullTotal + 1; } // Else the id must be greater than the last one else CHECK(AssertError, id > this->tagStack.top->idLast, "field id must be greater than last id"); // Clear NULLs now that field id has been calculated this->tagStack.top->nullTotal = 0; // Calculate field ID delta unsigned int tagId = id - this->tagStack.top->idLast - 1; // Write field type map (e.g. int64, string) uint64_t tag = typeMap >= 0xF ? 0xF0 : typeMap << 4; // If the value can contain multiple bits (e.g. integer) if (packTypeMapData[typeMap].valueMultiBit) { // If the value is stored in the tag (value == 1 bit) if (value < 2) { // Write low order bit of the value tag |= (value & 0x1) << 2; value >>= 1; // Write low order bit of the field ID delta tag |= tagId & 0x1; tagId >>= 1; // Set bit to indicate that high order bits of the field ID delta are be written after the tag if (tagId > 0) tag |= 0x2; } // Else the value is stored following the tag (value > 1 bit) else { // Set bit to indicate that the value is written after the tag tag |= 0x8; // Write low order bits of the field ID delta tag |= tagId & 0x3; tagId >>= 2; // Set bit to indicate that high order bits of the field ID delta are be written after the tag if (tagId > 0) tag |= 0x4; } } // Else the value is a single bit (e.g. boolean) else if (packTypeMapData[typeMap].valueSingleBit) { // Write value tag |= (value & 0x1) << 3; value >>= 1; // Write low order bits of the field ID delta tag |= tagId & 0x3; tagId >>= 2; // Set bit to indicate that high order bits of the field ID delta are be written after the tag if (tagId > 0) tag |= 0x4; } else { // No value expected ASSERT(value == 0); // Write low order bits of the field ID delta tag |= tagId & 0x7; tagId >>= 3; // Set bit to indicate that high order bits of the field ID delta must be written after the tag if (tagId > 0) tag |= 0x8; } // Write tag uint8_t tagByte = (uint8_t)tag; pckWriteBuffer(this, BUF(&tagByte, 1)); // Write remaining type map if (typeMap >= 0xF) pckWriteU64Internal(this, typeMap - 0xF); // Write low order bits of the field ID delta if (tagId > 0) pckWriteU64Internal(this, tagId); // Write low order bits of the value if (value > 0) pckWriteU64Internal(this, value); // Set last field id this->tagStack.top->idLast = id; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Write a default as NULL (missing) ***********************************************************************************************************************************/ static inline bool pckWriteDefaultNull(PackWrite *const this, const bool defaultWrite, const bool defaultEqual) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(BOOL, defaultWrite); FUNCTION_TEST_PARAM(BOOL, defaultEqual); FUNCTION_TEST_END(); ASSERT(this != NULL); // Write a NULL if not forcing the default to be written and the value passed equals the default if (!defaultWrite && defaultEqual) { this->tagStack.top->nullTotal++; FUNCTION_TEST_RETURN(BOOL, true); } // Let the caller know that it should write the value FUNCTION_TEST_RETURN(BOOL, false); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteNull(PackWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_END(); this->tagStack.top->nullTotal++; FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteArrayBegin(PackWrite *const this, const PackIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); // Write the array tag pckWriteTag(this, pckTypeMapArray, param.id, 0); // Add array to the tag stack so IDs can be tracked separately from the parent container pckTagStackPush(objMemContext(this), &this->tagStack, pckTypeMapArray); FUNCTION_TEST_RETURN(PACK_WRITE, this); } FN_EXTERN PackWrite * pckWriteArrayEnd(PackWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Check tag stack pop pckTagStackPopCheck(&this->tagStack, pckTypeMapArray); // Write end of array tag pckWriteU64Internal(this, 0); // Pop array off the stack to revert to ID tracking for the prior container pckTagStackPop(&this->tagStack); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteBin(PackWrite *const this, const Buffer *const value, const PckWriteBinParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(BUFFER, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, false, value == NULL)) { ASSERT(value != NULL); // Write buffer size if > 0 pckWriteTag(this, pckTypeMapBin, param.id, !bufEmpty(value)); // Write buffer data if size > 0 if (!bufEmpty(value)) { pckWriteU64Internal(this, bufUsed(value)); pckWriteBuffer(this, value); } } FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteBool(PackWrite *const this, const bool value, const PckWriteBoolParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(BOOL, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(BOOL, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapBool, param.id, value); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteI32(PackWrite *const this, const int32_t value, const PckWriteI32Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(INT, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(INT, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapI32, param.id, cvtInt32ToZigZag(value)); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteI64(PackWrite *const this, const int64_t value, const PckWriteI64Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(INT64, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(INT64, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapI64, param.id, cvtInt64ToZigZag(value)); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteMode(PackWrite *const this, const mode_t value, const PckWriteModeParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT32, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(MODE, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapMode, param.id, value); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteObjBegin(PackWrite *const this, const PackIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); // Write the object tag pckWriteTag(this, pckTypeMapObj, param.id, 0); // Add object to the tag stack so IDs can be tracked separately from the parent container pckTagStackPush(objMemContext(this), &this->tagStack, pckTypeMapObj); FUNCTION_TEST_RETURN(PACK_WRITE, this); } FN_EXTERN PackWrite * pckWriteObjEnd(PackWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Check tag stack pop pckTagStackPopCheck(&this->tagStack, pckTypeMapObj); // Write end of object tag pckWriteU64Internal(this, 0); // Pop object off the stack to revert to ID tracking for the prior container pckTagStackPop(&this->tagStack); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWritePack(PackWrite *const this, const Pack *const value, const PckWritePackParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(BUFFER, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, false, value == NULL)) { ASSERT(value != NULL); pckWriteTag(this, pckTypeMapPack, param.id, 0); pckWriteU64Internal(this, bufUsed((const Buffer *)value)); pckWriteBuffer(this, (const Buffer *)value); } FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteStr(PackWrite *this, const String *const value, const PckWriteStrParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(STRING, param.defaultValue); FUNCTION_TEST_END(); if (!pckWriteDefaultNull(this, param.defaultWrite, strEq(value, param.defaultValue))) { ASSERT(value != NULL); // Write string size if > 0 pckWriteTag(this, pckTypeMapStr, param.id, strSize(value) > 0); // Write string data if size > 0 if (strSize(value) > 0) { pckWriteU64Internal(this, strSize(value)); pckWriteBuffer(this, BUF(strZ(value), strSize(value))); } } FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteStrId(PackWrite *const this, const uint64_t value, const PckWriteStrIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(UINT64, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapStrId, param.id, value); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteStrLst(PackWrite *const this, const StringList *const value, const PckWriteStrLstParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(STRING_LIST, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, false, value == NULL)) { ASSERT(value != NULL); pckWriteArrayBeginP(this, .id = param.id); for (unsigned int valueIdx = 0; valueIdx < strLstSize(value); valueIdx++) pckWriteStrP(this, strLstGet(value, valueIdx)); pckWriteArrayEndP(this); } FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteTime(PackWrite *const this, const time_t value, const PckWriteTimeParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(TIME, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(TIME, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapTime, param.id, cvtInt64ToZigZag(value)); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteU32(PackWrite *const this, const uint32_t value, const PckWriteU32Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT32, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(UINT32, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapU32, param.id, value); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteU64(PackWrite *const this, const uint64_t value, const PckWriteU64Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(UINT64, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapU64, param.id, value); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteEnd(PackWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->tagStack.depth == 0); pckWriteU64Internal(this, 0); this->tagStack.top = NULL; // If writing to io flush the internal buffer if (this->write != NULL) { if (!bufEmpty(this->buffer)) ioWrite(this->write, this->buffer); } // Else resize the external buffer to trim off extra space added during processing else bufResize(this->buffer, bufUsed(this->buffer)); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN Pack * pckWriteResult(PackWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_END(); Pack *result = NULL; if (this != NULL) { ASSERT(this->tagStack.top == NULL); result = (Pack *)OBJ_NAME(this->buffer, Pack::Buffer); } FUNCTION_TEST_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN void pckWriteToLog(const PackWrite *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{depth: %u, idLast: %u}", this->tagStack.depth, this->tagStack.top == NULL ? 0 : this->tagStack.top->idLast); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/pack.h0000664000175100017510000006705315265101745017544 0ustar00runnerrunner/*********************************************************************************************************************************** Pack Type The pack type encodes binary data compactly while still maintaining structure and strict typing. The idea is based on Thrift, ProtocolBuffers, and Avro, compared here: https://medium.com/better-programming/use-binary-encoding-instead-of-json-dec745ec09b6. The pack type has been further optimized to balance between purely in-memory structures and those intended to be passed via a protocol or saved in a file. Integers are stored with base-128 varint encoding which is equivalent to network byte order, i.e., the endianness of the sending and receiving host don't matter. The overall idea is similar to JSON but IDs are used instead of names, typing is more granular, and the representation is far more compact. A pack can readily be converted to JSON but the reverse is not as precise due to loose typing in JSON. A pack is a stream format, i.e. it is intended to be read in order from beginning to end. Fields in a pack are identified by IDs. A field ID is stored as a delta from the previous ID, which is very efficient, but means that reading from the middle is generally not practical. The size of the gap between field IDs is important -- a gap of 1 never incurs extra cost, but depending on the field type larger gaps may require additional bytes to store the field ID delta. The standard default is the C default for that type (e.g. bool = false, int = 0) but can be changed with the .defaultValue parameter. For example, pckWriteBoolP(write, false, .defaultWrite = true) will write a 0 (i.e. false) with a field ID into the pack, but pckWriteBoolP(write, false) will not write to the pack, it will simply skip the ID. Note that pckWriteStrP(packWrite, NULL, .defaultWrite = true) is not valid since there is no way to explicitly write a NULL. NULLs are not stored in a pack and are therefore not typed. A NULL is essentially just a gap in the field IDs. Fields that are frequently NULL are best stored at the end of an object. When using read functions the default will always be returned when the field is NULL (i.e. missing). There are times when NULL must be explicitly passed, for example: pckWriteStrP(pack, strEmpty(checksumSha1) ? NULL : checksumSha1); In this case, NULL is declared since checksumSha1 might be empty and, following the rules for NULLs the field ID is skipped when strEmpty(checksumSha1). Upon reading, we can declare an empty string when a NULL (field ID gap) is encountered, e.g. pckReadStrP(pack, .defaultValue = EMPTY_STR)). A pack is an object by default. Objects can store fields, objects, or arrays. Objects and arrays will be referred to collectively as containers. Fields contain data to be stored, e.g. integers, strings, etc. Here is a simple example of a pack: PackWrite *write = pckWriteNewP(); pckWriteU64P(write, 77); pckWriteBoolP(write, false, .defaultWrite = true); pckWriteI32P(write, -1, .defaultValue = -1); pckWriteStringP(write, STRDEF("sample")); pckWriteEndP(); A string representation of this pack is `1:uint64:77,2:bool:false,4:str:sample`. The boolean was stored even though it was the default because a write was explicitly requested. The int32 field was not stored because the value matched the explicitly set default. Note that there is a gap in the ID stream, which represents the NULL/default value. This pack can be read with: PackRead *read = pckReadNew(pack); pckReadU64P(read); pckReadBoolP(read); pckReadI32P(read, .defaultValue = -1); pckReadStringP(read); pckReadEndP(); Note that defaults are not stored in the pack so any defaults that were applied when writing (by setting .defaultValue) must be applied again when reading by setting .defaultValue if the default value is not a standard C default. If we don't care about the NULL/default, another way to read is: PackRead *read = pckReadNew(pack); pckReadU64P(read); pckReadBoolP(read); pckReadStringP(read, .id = 4); pckReadEndP(); By default each read/write advances the field ID by one. If an ID is specified it must be unique and increasing, because it will advance the field ID beyond the value specified. An error will occur if an ID is attempted to be read/written but the field ID has advanced beyond it. An array can be read with: pckReadArrayBeginP(read); while (pckReadNext(read)) { // Read array element } pckReadArrayEndP(read); Note that any container (i.e. array or object) resets the field ID to one so there is no need for the caller to maintain a cumulative field ID. At the end of a container the numbering will continue from wherever the outer container left off. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_PACK_H #define COMMON_TYPE_PACK_H #include #include /*********************************************************************************************************************************** Minimum number of extra bytes to allocate for packs that are growing or are likely to grow ***********************************************************************************************************************************/ #ifndef PACK_EXTRA_MIN #define PACK_EXTRA_MIN 128 #endif /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ typedef struct Pack Pack; typedef struct PackRead PackRead; typedef struct PackWrite PackWrite; #include "common/io/read.h" #include "common/io/write.h" #include "common/type/object.h" #include "common/type/stringId.h" #include "common/type/stringList.h" /*********************************************************************************************************************************** Pack data type ***********************************************************************************************************************************/ typedef enum { pckTypeArray = STRID5("array", 0x190ca410), pckTypeBin = STRID5("bin", 0x39220), pckTypeBool = STRID5("bool", 0x63de20), pckTypeI32 = STRID6("i32", 0x1e7c91), pckTypeI64 = STRID6("i64", 0x208891), pckTypeObj = STRID5("obj", 0x284f0), pckTypeMode = STRID5("mode", 0x291ed0), pckTypePack = STRID5("pack", 0x58c300), pckTypeStr = STRID5("str", 0x4a930), pckTypeStrId = STRID5("strid", 0x44ca930), pckTypeTime = STRID5("time", 0x2b5340), pckTypeU32 = STRID6("u32", 0x1e7d51), pckTypeU64 = STRID6("u64", 0x208951), } PackType; /*********************************************************************************************************************************** Pack Functions ***********************************************************************************************************************************/ // Duplicate pack FN_INLINE_ALWAYS Pack * pckDup(const Pack *const this) { return (Pack *)OBJ_NAME(bufDup((const Buffer *)this), Pack::Buffer); } // Cast Buffer to Pack FN_INLINE_ALWAYS const Pack * pckFromBuf(const Buffer *const buffer) { return (const Pack *)buffer; } // Move to a new parent mem context FN_INLINE_ALWAYS Pack * pckMove(Pack *const this, MemContext *const parentNew) { return (Pack *)bufMove((Buffer *)this, parentNew); } // Cast Pack to Buffer FN_INLINE_ALWAYS const Buffer * pckToBuf(const Pack *const pack) { return (const Buffer *)pack; } /*********************************************************************************************************************************** Read Constructors ***********************************************************************************************************************************/ // Note that the pack is not moved into the PackRead mem context and must be moved explicitly if the PackRead object is moved. FN_EXTERN PackRead *pckReadNew(const Pack *pack); FN_EXTERN PackRead *pckReadNewC(const uint8_t *const buffer, size_t size); FN_EXTERN PackRead *pckReadNewIo(IoRead *read); /*********************************************************************************************************************************** Read Functions ***********************************************************************************************************************************/ typedef struct PackIdParam { VAR_PARAM_HEADER; unsigned int id; } PackIdParam; // Read next field. This is useful when the type of the next field is unknown, i.e. a completely dynamic data structure, or for // debugging. If you just need to know if the field exists or not, then use pckReadNullP(). FN_EXTERN bool pckReadNext(PackRead *this); // Current field buffer for fields that have a size, e.g. bin. Can only be used when the pack was created with pckReadNew(). Note // that this pointer is tied to the buffer the pack was created with so be careful not to free it too soon. FN_EXTERN const uint8_t *pckReadBufPtr(PackRead *this); // Consume the next field regardless of type. FN_EXTERN void pckReadConsume(PackRead *this); // Current field id. Set after a call to pckReadNext(). FN_EXTERN unsigned int pckReadId(PackRead *this); // Current field size for fields that have a size, e.g. bin FN_EXTERN size_t pckReadSize(PackRead *this); // Current field type. Set after a call to pckReadNext(). FN_EXTERN PackType pckReadType(PackRead *this); // Is the field NULL? If the field is NULL the id will be advanced so the field does not need to be read explicitly. If the field is // not NULL then the id is not advanced since a subsequent read is expected. #define pckReadNullP(this, ...) \ pckReadNull(this, (PackIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool pckReadNull(PackRead *this, PackIdParam param); // Read array begin/end #define pckReadArrayBeginP(this, ...) \ pckReadArrayBegin(this, (PackIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void pckReadArrayBegin(PackRead *this, PackIdParam param); #define pckReadArrayEndP(this) \ pckReadArrayEnd(this) FN_EXTERN void pckReadArrayEnd(PackRead *this); // Read binary typedef struct PckReadBinParam { VAR_PARAM_HEADER; unsigned int id; } PckReadBinParam; #define pckReadBinP(this, ...) \ pckReadBin(this, (PckReadBinParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN Buffer *pckReadBin(PackRead *this, PckReadBinParam param); // Read boolean typedef struct PckReadBoolParam { VAR_PARAM_HEADER; unsigned int id; uint32_t defaultValue; } PckReadBoolParam; #define pckReadBoolP(this, ...) \ pckReadBool(this, (PckReadBoolParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool pckReadBool(PackRead *this, PckReadBoolParam param); // Read 32-bit signed integer typedef struct PckReadI32Param { VAR_PARAM_HEADER; unsigned int id; int32_t defaultValue; } PckReadI32Param; #define pckReadI32P(this, ...) \ pckReadI32(this, (PckReadI32Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN int32_t pckReadI32(PackRead *this, PckReadI32Param param); // Read 64-bit signed integer typedef struct PckReadI64Param { VAR_PARAM_HEADER; unsigned int id; int64_t defaultValue; } PckReadI64Param; #define pckReadI64P(this, ...) \ pckReadI64(this, (PckReadI64Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN int64_t pckReadI64(PackRead *this, PckReadI64Param param); // Read mode typedef struct PckReadModeParam { VAR_PARAM_HEADER; unsigned int id; mode_t defaultValue; } PckReadModeParam; #define pckReadModeP(this, ...) \ pckReadMode(this, (PckReadModeParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN mode_t pckReadMode(PackRead *this, PckReadModeParam param); // Move to a new parent mem context FN_INLINE_ALWAYS PackRead * pckReadMove(PackRead *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Read object begin/end #define pckReadObjBeginP(this, ...) \ pckReadObjBegin(this, (PackIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void pckReadObjBegin(PackRead *this, PackIdParam param); #define pckReadObjEndP(this) \ pckReadObjEnd(this) FN_EXTERN void pckReadObjEnd(PackRead *this); // Read pack typedef struct PckReadPackParam { VAR_PARAM_HEADER; unsigned int id; } PckReadPackParam; #define pckReadPackReadP(this, ...) \ pckReadPackRead(this, (PckReadPackParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackRead *pckReadPackRead(PackRead *this, PckReadPackParam param); // Read pack using the same buffer passed to pckReadNew(). Note that this pointer is tied to the buffer the pack was created with so // be careful not to free it too soon. #define pckReadPackReadConstP(this, ...) \ pckReadPackReadConst(this, (PckReadPackParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackRead *pckReadPackReadConst(PackRead *this, PckReadPackParam param); // Read pack buffer #define pckReadPackP(this, ...) \ pckReadPack(this, (PckReadPackParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN Pack *pckReadPack(PackRead *this, PckReadPackParam param); // Read string typedef struct PckReadStrParam { VAR_PARAM_HEADER; unsigned int id; const String *defaultValue; } PckReadStrParam; #define pckReadStrP(this, ...) \ pckReadStr(this, (PckReadStrParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN String *pckReadStr(PackRead *this, PckReadStrParam param); // Read string id typedef struct PckReadStrIdParam { VAR_PARAM_HEADER; unsigned int id; StringId defaultValue; } PckReadStrIdParam; #define pckReadStrIdP(this, ...) \ pckReadStrId(this, (PckReadStrIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN uint64_t pckReadStrId(PackRead *this, PckReadStrIdParam param); // Read string list typedef struct PckReadStrLstParam { VAR_PARAM_HEADER; unsigned int id; } PckReadStrLstParam; #define pckReadStrLstP(this, ...) \ pckReadStrLst(this, (PckReadStrLstParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN StringList *pckReadStrLst(PackRead *const this, PckReadStrLstParam param); // Read time typedef struct PckReadTimeParam { VAR_PARAM_HEADER; unsigned int id; time_t defaultValue; } PckReadTimeParam; #define pckReadTimeP(this, ...) \ pckReadTime(this, (PckReadTimeParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN time_t pckReadTime(PackRead *this, PckReadTimeParam param); // Read 32-bit unsigned integer typedef struct PckReadU32Param { VAR_PARAM_HEADER; unsigned int id; uint32_t defaultValue; } PckReadU32Param; #define pckReadU32P(this, ...) \ pckReadU32(this, (PckReadU32Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN uint32_t pckReadU32(PackRead *this, PckReadU32Param param); // Read 64-bit unsigned integer typedef struct PckReadU64Param { VAR_PARAM_HEADER; unsigned int id; uint64_t defaultValue; } PckReadU64Param; #define pckReadU64P(this, ...) \ pckReadU64(this, (PckReadU64Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN uint64_t pckReadU64(PackRead *this, PckReadU64Param param); // Read end #define pckReadEndP(this) \ pckReadEnd(this) FN_EXTERN void pckReadEnd(PackRead *this); /*********************************************************************************************************************************** Read Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void pckReadFree(PackRead *const this) { objFree(this); } /*********************************************************************************************************************************** Write Constructors ***********************************************************************************************************************************/ typedef struct PckWriteNewParam { VAR_PARAM_HEADER; size_t size; } PckWriteNewParam; #define pckWriteNewP(...) \ pckWriteNew((PckWriteNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteNew(PckWriteNewParam param); FN_EXTERN PackWrite *pckWriteNewIo(IoWrite *write); /*********************************************************************************************************************************** Write Functions ***********************************************************************************************************************************/ // Write array begin/end #define pckWriteArrayBeginP(this, ...) \ pckWriteArrayBegin(this, (PackIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteArrayBegin(PackWrite *this, PackIdParam param); #define pckWriteArrayEndP(this) \ pckWriteArrayEnd(this) FN_EXTERN PackWrite *pckWriteArrayEnd(PackWrite *this); // Write binary typedef struct PckWriteBinParam { VAR_PARAM_HEADER; unsigned int id; } PckWriteBinParam; #define pckWriteBinP(this, value, ...) \ pckWriteBin(this, value, (PckWriteBinParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteBin(PackWrite *this, const Buffer *value, PckWriteBinParam param); // Write boolean typedef struct PckWriteBoolParam { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; uint32_t defaultValue; } PckWriteBoolParam; #define pckWriteBoolP(this, value, ...) \ pckWriteBool(this, value, (PckWriteBoolParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteBool(PackWrite *this, bool value, PckWriteBoolParam param); // Write 32-bit signed integer typedef struct PckWriteI32Param { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; int32_t defaultValue; } PckWriteI32Param; #define pckWriteI32P(this, value, ...) \ pckWriteI32(this, value, (PckWriteI32Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteI32(PackWrite *this, int32_t value, PckWriteI32Param param); // Write 64-bit signed integer typedef struct PckWriteI64Param { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; int64_t defaultValue; } PckWriteI64Param; #define pckWriteI64P(this, value, ...) \ pckWriteI64(this, value, (PckWriteI64Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteI64(PackWrite *this, int64_t value, PckWriteI64Param param); // Write mode typedef struct PckWriteModeParam { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; mode_t defaultValue; } PckWriteModeParam; #define pckWriteModeP(this, value, ...) \ pckWriteMode(this, value, (PckWriteModeParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteMode(PackWrite *this, mode_t value, PckWriteModeParam param); // Move to a new parent mem context FN_INLINE_ALWAYS PackWrite * pckWriteMove(PackWrite *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Write null #define pckWriteNullP(this) \ pckWriteNull(this) FN_EXTERN PackWrite *pckWriteNull(PackWrite *this); // Write object begin/end #define pckWriteObjBeginP(this, ...) \ pckWriteObjBegin(this, (PackIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteObjBegin(PackWrite *this, PackIdParam param); #define pckWriteObjEndP(this) \ pckWriteObjEnd(this) FN_EXTERN PackWrite *pckWriteObjEnd(PackWrite *this); // Write pack typedef struct PckWritePackParam { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; } PckWritePackParam; #define pckWritePackP(this, value, ...) \ pckWritePack(this, value, (PckWritePackParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWritePack(PackWrite *this, const Pack *value, PckWritePackParam param); // Write string typedef struct PckWriteStrParam { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; const String *defaultValue; } PckWriteStrParam; #define pckWriteStrP(this, value, ...) \ pckWriteStr(this, value, (PckWriteStrParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteStr(PackWrite *this, const String *value, PckWriteStrParam param); // Write string id typedef struct PckWriteStrIdParam { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; StringId defaultValue; } PckWriteStrIdParam; #define pckWriteStrIdP(this, value, ...) \ pckWriteStrId(this, value, (PckWriteStrIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteStrId(PackWrite *this, uint64_t value, PckWriteStrIdParam param); // Write string list typedef struct PckWriteStrLstParam { VAR_PARAM_HEADER; unsigned int id; } PckWriteStrLstParam; #define pckWriteStrLstP(this, value, ...) \ pckWriteStrLst(this, value, (PckWriteStrLstParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteStrLst(PackWrite *const this, const StringList *const value, const PckWriteStrLstParam param); // Write time typedef struct PckWriteTimeParam { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; time_t defaultValue; } PckWriteTimeParam; #define pckWriteTimeP(this, value, ...) \ pckWriteTime(this, value, (PckWriteTimeParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteTime(PackWrite *this, time_t value, PckWriteTimeParam param); // Write 32-bit unsigned integer typedef struct PckWriteU32Param { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; uint32_t defaultValue; } PckWriteU32Param; #define pckWriteU32P(this, value, ...) \ pckWriteU32(this, value, (PckWriteU32Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteU32(PackWrite *this, uint32_t value, PckWriteU32Param param); // Write 64-bit unsigned integer typedef struct PckWriteU64Param { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; uint64_t defaultValue; } PckWriteU64Param; #define pckWriteU64P(this, value, ...) \ pckWriteU64(this, value, (PckWriteU64Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteU64(PackWrite *this, uint64_t value, PckWriteU64Param param); // Write end #define pckWriteEndP(this) \ pckWriteEnd(this) FN_EXTERN PackWrite *pckWriteEnd(PackWrite *this); /*********************************************************************************************************************************** Write Getters/Setters ***********************************************************************************************************************************/ // Get Pack the PackWrite was writing to (returns NULL if pckWriteNew() was not used to construct the object). This function is only // valid after pckWriteEndP() has been called. FN_EXTERN Pack *pckWriteResult(PackWrite *this); /*********************************************************************************************************************************** Write Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void pckWriteFree(PackWrite *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_PACK_TYPE \ Pack * #define FUNCTION_LOG_PACK_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Pack", buffer, bufferSize) FN_EXTERN void pckReadToLog(const PackRead *this, StringStatic *debugLog); #define FUNCTION_LOG_PACK_READ_TYPE \ PackRead * #define FUNCTION_LOG_PACK_READ_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, pckReadToLog, buffer, bufferSize) FN_EXTERN void pckWriteToLog(const PackWrite *this, StringStatic *debugLog); #define FUNCTION_LOG_PACK_WRITE_TYPE \ PackWrite * #define FUNCTION_LOG_PACK_WRITE_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, pckWriteToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/param.h0000664000175100017510000000216415265101745017716 0ustar00runnerrunner/*********************************************************************************************************************************** Variable Parameter Helper Macros ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_PARAM_H #define COMMON_TYPE_PARAM_H /*********************************************************************************************************************************** Macros to help with implementing functions with variable parameters using structs ***********************************************************************************************************************************/ // This macro goes at the top of the parameter struct #define VAR_PARAM_HEADER \ bool dummy // This macro goes in the struct parameter list right before __VA_ARGS__ #define VAR_PARAM_INIT \ .dummy = false #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/string.c0000664000175100017510000011022115265101745020111 0ustar00runnerrunner/*********************************************************************************************************************************** String Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include "common/debug.h" #include "common/macro.h" #include "common/memContext.h" #include "common/type/string.h" #include "common/type/stringList.h" #include "common/valgrind.h" /*********************************************************************************************************************************** Constant strings that are generally useful ***********************************************************************************************************************************/ STRING_EXTERN(CR_STR, "\r"); STRING_EXTERN(CRLF_STR, "\r\n"); STRING_EXTERN(DOT_STR, "."); STRING_EXTERN(DOTDOT_STR, ".."); STRING_EXTERN(EMPTY_STR, ""); STRING_EXTERN(FALSE_STR, FALSE_Z); STRING_EXTERN(FSLASH_STR, "/"); STRING_EXTERN(LF_STR, "\n"); STRING_EXTERN(N_STR, "n"); STRING_EXTERN(NULL_STR, NULL_Z); STRING_EXTERN(TRUE_STR, TRUE_Z); STRING_EXTERN(Y_STR, "y"); STRING_EXTERN(ZERO_STR, "0"); /*********************************************************************************************************************************** Buffer macros ***********************************************************************************************************************************/ // Fixed size buffer allocated at the end of the object allocation #define STR_FIXED_BUFFER (char *)(this + 1) // Is the string using the fixed size buffer? #define STR_IS_FIXED_BUFFER() (this->pub.buffer == STR_FIXED_BUFFER) // Empty buffer #define STR_EMPTY_BUFFER (EMPTY_STR->pub.buffer) // Is the string using the empty buffer? #define STR_IS_EMPTY_BUFFER() (this->pub.buffer == STR_EMPTY_BUFFER) /*********************************************************************************************************************************** Maximum size of a string ***********************************************************************************************************************************/ #define STRING_SIZE_MAX 1073741824 #define CHECK_SIZE(size) \ do \ { \ if ((size) > STRING_SIZE_MAX) \ THROW(AssertError, "string size must be <= " STRINGIFY(STRING_SIZE_MAX) " bytes"); \ } \ while (0) /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct String { StringPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN String * strNew(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(String, .allocQty = 1) { *this = (String) { .pub = { // Set empty so nothing is allocated until needed .buffer = STR_EMPTY_BUFFER, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(STRING, this); } /*********************************************************************************************************************************** Create fixed size String ***********************************************************************************************************************************/ static String * strNewFixed(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); CHECK_SIZE(size); // If the string is larger than the extra allowed with a mem context then allocate the buffer separately const size_t allocExtra = sizeof(String) + size + 1; if (allocExtra > MEM_CONTEXT_ALLOC_EXTRA_MAX) { OBJ_NEW_BEGIN(String, .allocQty = 1) { *this = (String) { .pub = { .size = (unsigned int)size, .buffer = memNew(size + 1), }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(STRING, this); } OBJ_NEW_EXTRA_BEGIN(String, (uint16_t)(allocExtra)) { *this = (String) { .pub = { .size = (unsigned int)size, .buffer = STR_FIXED_BUFFER, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewZ(const char *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, string); FUNCTION_TEST_END(); ASSERT(string != NULL); // Create object String *const this = strNewFixed(strlen(string)); // Assign string memcpy(this->pub.buffer, string, strSize(this)); this->pub.buffer[strSize(this)] = '\0'; FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewDiv(const uint64_t dividend, const uint64_t divisor, StrNewDivParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, dividend); FUNCTION_TEST_PARAM(UINT64, divisor); FUNCTION_TEST_PARAM(UINT, param.precision); FUNCTION_TEST_PARAM(BOOL, param.trim); FUNCTION_TEST_END(); char working[CVT_DIV_BUFFER_SIZE]; size_t resultSize = cvtDivToZ(dividend, divisor, param.precision, param.trim, working, sizeof(working)); FUNCTION_TEST_RETURN(STRING, strNewZN(working, resultSize)); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewPct(const uint64_t dividend, const uint64_t divisor) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, dividend); FUNCTION_TEST_PARAM(UINT64, divisor); FUNCTION_TEST_END(); char working[CVT_PCT_BUFFER_SIZE]; size_t resultSize = cvtPctToZ(dividend, divisor, working, sizeof(working)); FUNCTION_TEST_RETURN(STRING, strNewZN(working, resultSize)); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewStrId(const StringId strId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, strId); FUNCTION_TEST_END(); char buffer[STRID_MAX + 1]; const size_t size = strIdToZN(strId, buffer); buffer[size] = '\0'; FUNCTION_TEST_RETURN(STRING, strNewZN(buffer, size)); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewTime(const char *const format, const time_t timestamp, const StrNewTimeParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_PARAM(TIME, timestamp); FUNCTION_TEST_PARAM(BOOL, param.utc); FUNCTION_TEST_END(); char buffer[64]; // We can ignore this warning here since the format parameter of strNewTimeP() is checked #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wformat-nonliteral" cvtTimeToZP(format, timestamp, buffer, sizeof(buffer), .utc = param.utc); #pragma GCC diagnostic pop FUNCTION_TEST_RETURN(STRING, strNewZ(buffer)); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewBuf(const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(buffer != NULL); // Create object String *const this = strNewFixed(bufUsed(buffer)); // Assign string if (strSize(this) != 0) memcpy(this->pub.buffer, bufPtrConst(buffer), strSize(this)); this->pub.buffer[strSize(this)] = 0; FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewEncode(const EncodingType type, const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(buffer != NULL); // Create object String *const this = strNewFixed(encodeToStrSize(type, bufUsed(buffer))); // Encode buffer if (bufUsed(buffer) > 0) { encodeToStr(type, bufPtrConst(buffer), bufUsed(buffer), this->pub.buffer); } // Else zero-terminate else this->pub.buffer[0] = '\0'; FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewFmt(const char *const format, ...) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_END(); ASSERT(format != NULL); // Determine how long the allocated string needs to be and create object va_list argumentList; va_start(argumentList, format); String *const this = strNewFixed((size_t)vsnprintf(NULL, 0, format, argumentList)); va_end(argumentList); // Format string va_start(argumentList, format); vsnprintf(this->pub.buffer, strSize(this) + 1, format, argumentList); va_end(argumentList); FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewZN(const char *const string, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(CHARDATA, string); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(string != NULL); // Create object String *const this = strNewFixed(size); // Assign string memcpy(this->pub.buffer, string, strSize(this)); this->pub.buffer[strSize(this)] = 0; FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strBase(const String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(STRING, strNewZ(strBaseZ(this))); } FN_EXTERN const char * strBaseZ(const String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); const char *end = this->pub.buffer + strSize(this); while (end > this->pub.buffer && *(end - 1) != '/') end--; FUNCTION_TEST_RETURN_CONST(STRINGZ, end); } /**********************************************************************************************************************************/ FN_EXTERN bool strBeginsWith(const String *const this, const String *const beginsWith) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRING, beginsWith); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(beginsWith != NULL); FUNCTION_TEST_RETURN(BOOL, strBeginsWithZ(this, strZ(beginsWith))); } FN_EXTERN bool strBeginsWithZ(const String *const this, const char *const beginsWith) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, beginsWith); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(beginsWith != NULL); bool result = false; const unsigned int beginsWithSize = (unsigned int)strlen(beginsWith); if (strSize(this) >= beginsWithSize) result = strncmp(strZ(this), beginsWith, beginsWithSize) == 0; FUNCTION_TEST_RETURN(BOOL, result); } /*********************************************************************************************************************************** Resize the string to allow the requested number of characters to be appended ***********************************************************************************************************************************/ static void strResize(String *const this, const size_t requested) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(SIZE, requested); FUNCTION_TEST_END(); if (requested > this->pub.extra) { // Fixed size strings may not be resized CHECK(AssertError, !STR_IS_FIXED_BUFFER(), "resize of fixed size string"); // Check size CHECK_SIZE(strSize(this) + requested); // Calculate new extra needs to satisfy request and leave extra space for new growth this->pub.extra = (unsigned int)(requested + ((strSize(this) + requested) / 2)); // Adding too little extra space usually leads to immediate resizing so enforce a minimum if (this->pub.extra < STRING_EXTRA_MIN) this->pub.extra = STRING_EXTRA_MIN; MEM_CONTEXT_OBJ_BEGIN(this) { if (STR_IS_EMPTY_BUFFER()) this->pub.buffer = memNew(strSize(this) + this->pub.extra + 1); else this->pub.buffer = memResize(this->pub.buffer, strSize(this) + this->pub.extra + 1); } MEM_CONTEXT_OBJ_END(); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * strCat(String *const this, const String *const cat) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRING, cat); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(cat != NULL); FUNCTION_TEST_RETURN(STRING, strCatZN(this, strZ(cat), strSize(cat))); } FN_EXTERN String * strCatZ(String *const this, const char *const cat) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, cat); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(cat != NULL); // Determine length of string to append const size_t sizeGrow = strlen(cat); if (sizeGrow != 0) { // Ensure there is enough space to grow the string strResize(this, sizeGrow); // Append the string memcpy(this->pub.buffer + strSize(this), cat, sizeGrow + 1); this->pub.size += (unsigned int)sizeGrow; this->pub.extra -= (unsigned int)sizeGrow; } FUNCTION_TEST_RETURN(STRING, this); } FN_EXTERN String * strCatZN(String *const this, const char *const cat, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, cat); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(this != NULL); if (size != 0) { ASSERT(cat != NULL); // Ensure there is enough space to grow the string strResize(this, size); // Append the string memcpy(this->pub.buffer + strSize(this), cat, size); this->pub.buffer[strSize(this) + size] = '\0'; // Update size/extra this->pub.size += (unsigned int)size; this->pub.extra -= (unsigned int)size; } FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strCatBuf(String *const this, const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); FUNCTION_TEST_RETURN(STRING, strCatZN(this, (const char *)bufPtrConst(buffer), bufUsed(buffer))); } /**********************************************************************************************************************************/ FN_EXTERN String * strCatChr(String *const this, const char cat) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(CHAR, cat); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(cat != 0); // Ensure there is enough space to grow the string strResize(this, 1); // Append the character this->pub.buffer[this->pub.size++] = cat; this->pub.buffer[this->pub.size] = 0; this->pub.extra--; FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strCatEncode(String *const this, const EncodingType type, const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); const size_t encodeSize = encodeToStrSize(type, bufUsed(buffer)); if (encodeSize != 0) { // Ensure there is enough space to grow the string strResize(this, encodeSize); // Append the encoded string encodeToStr(type, bufPtrConst(buffer), bufUsed(buffer), this->pub.buffer + strSize(this)); // Update size/extra this->pub.size += (unsigned int)encodeSize; this->pub.extra -= (unsigned int)encodeSize; } FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strCatTime(String *const this, const char *const format, const time_t timestamp, const StrCatTimeParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_PARAM(TIME, timestamp); FUNCTION_TEST_PARAM(BOOL, param.utc); FUNCTION_TEST_END(); char buffer[64]; // We can ignore this warning here since the format parameter of strCatTimeP() is checked #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wformat-nonliteral" cvtTimeToZP(format, timestamp, buffer, sizeof(buffer), .utc = param.utc); #pragma GCC diagnostic pop FUNCTION_TEST_RETURN(STRING, strCatZ(this, buffer)); } /**********************************************************************************************************************************/ FN_EXTERN String * strCatFmt(String *const this, const char *const format, ...) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(format != NULL); // Determine how long the allocated string needs to be va_list argumentList; va_start(argumentList, format); const size_t sizeGrow = (size_t)vsnprintf(NULL, 0, format, argumentList); va_end(argumentList); if (sizeGrow != 0) { // Ensure there is enough space to grow the string strResize(this, sizeGrow); // Append the formatted string va_start(argumentList, format); vsnprintf(this->pub.buffer + strSize(this), sizeGrow + 1, format, argumentList); va_end(argumentList); this->pub.size += (unsigned int)sizeGrow; this->pub.extra -= (unsigned int)sizeGrow; } FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN int strCmp(const String *const this, const String *const compare) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRING, compare); FUNCTION_TEST_END(); if (this != NULL && compare != NULL) FUNCTION_TEST_RETURN(INT, strcmp(strZ(this), strZ(compare))); else if (this == NULL) { if (compare == NULL) FUNCTION_TEST_RETURN(INT, 0); FUNCTION_TEST_RETURN(INT, -1); } FUNCTION_TEST_RETURN(INT, 1); } FN_EXTERN int strCmpZ(const String *const this, const char *const compare) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, compare); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(INT, strCmp(this, compare == NULL ? NULL : STR(compare))); } /**********************************************************************************************************************************/ FN_EXTERN String * strDup(const String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); String *result = NULL; if (this != NULL) result = strNewZ(strZ(this)); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN bool strEmpty(const String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, strSize(this) == 0); } /**********************************************************************************************************************************/ FN_EXTERN bool strEndsWith(const String *const this, const String *const endsWith) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRING, endsWith); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(endsWith != NULL); FUNCTION_TEST_RETURN(BOOL, strEndsWithZ(this, strZ(endsWith))); } FN_EXTERN bool strEndsWithZ(const String *const this, const char *const endsWith) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, endsWith); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(endsWith != NULL); bool result = false; const unsigned int endsWithSize = (unsigned int)strlen(endsWith); if (strSize(this) >= endsWithSize) result = strcmp(strZ(this) + (strSize(this) - endsWithSize), endsWith) == 0; FUNCTION_TEST_RETURN(BOOL, result); } /*********************************************************************************************************************************** There are two separate implementations because string objects can get the size very efficiently whereas the zero-terminated strings would need a call to strlen(). ***********************************************************************************************************************************/ FN_EXTERN bool strEq(const String *const this, const String *const compare) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRING, compare); FUNCTION_TEST_END(); bool result = false; if (this != NULL && compare != NULL) { if (strSize(this) == strSize(compare)) result = strcmp(strZ(this), strZ(compare)) == 0; } else result = this == NULL && compare == NULL; FUNCTION_TEST_RETURN(BOOL, result); } FN_EXTERN bool strEqZ(const String *const this, const char *const compare) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, compare); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(compare != NULL); FUNCTION_TEST_RETURN(BOOL, strcmp(strZ(this), compare) == 0); } /**********************************************************************************************************************************/ FN_EXTERN String * strFirstUpper(String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); if (strSize(this) > 0) this->pub.buffer[0] = (char)toupper(this->pub.buffer[0]); FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strFirstLower(String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); if (strSize(this) > 0) this->pub.buffer[0] = (char)tolower(this->pub.buffer[0]); FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strLower(String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); for (size_t idx = 0; idx < strSize(this); idx++) this->pub.buffer[idx] = (char)tolower(this->pub.buffer[idx]); FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strPath(const String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); const char *end = this->pub.buffer + strSize(this); while (end > this->pub.buffer && *(end - 1) != '/') end--; FUNCTION_TEST_RETURN( STRING, strNewZN( this->pub.buffer, end - this->pub.buffer <= 1 ? (size_t)(end - this->pub.buffer) : (size_t)(end - this->pub.buffer - 1))); } /**********************************************************************************************************************************/ FN_EXTERN String * strPathAbsolute(const String *const this, const String *const base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRING, base); FUNCTION_TEST_END(); ASSERT(this != NULL); String *result = NULL; // Path is already absolute so just return it if (strBeginsWith(this, FSLASH_STR)) { result = strDup(this); } // Else we'll need to construct the absolute path. You would hope we could use realpath() here but it is so broken in the Posix // spec that is seems best avoided. else { ASSERT(base != NULL); // Base must be absolute to start if (!strBeginsWith(base, FSLASH_STR)) THROW_FMT(AssertError, "base path '%s' is not absolute", strZ(base)); MEM_CONTEXT_TEMP_BEGIN() { StringList *const baseList = strLstNewSplit(base, FSLASH_STR); StringList *const pathList = strLstNewSplit(this, FSLASH_STR); while (!strLstEmpty(pathList)) { const String *const pathPart = strLstGet(pathList, 0); // If the last part is empty if (strSize(pathPart) == 0) { // Allow when this is the last part since it just means there was a trailing / if (strLstSize(pathList) == 1) { strLstRemoveIdx(pathList, 0); break; } THROW_FMT(AssertError, "'%s' is not a valid relative path", strZ(this)); } if (strEq(pathPart, DOTDOT_STR)) { const String *const basePart = strLstGet(baseList, strLstSize(baseList) - 1); if (strSize(basePart) == 0) THROW_FMT(AssertError, "relative path '%s' goes back too far in base path '%s'", strZ(this), strZ(base)); strLstRemoveIdx(baseList, strLstSize(baseList) - 1); } else if (!strEq(pathPart, DOT_STR)) strLstAdd(baseList, pathPart); strLstRemoveIdx(pathList, 0); } MEM_CONTEXT_PRIOR_BEGIN() { if (strLstSize(baseList) == 1) result = strDup(FSLASH_STR); else result = strLstJoin(baseList, "/"); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); } // There should not be any stray .. or // in the final result if (strstr(strZ(result), "/..") != NULL || strstr(strZ(result), "//") != NULL) THROW_FMT(AssertError, "result path '%s' is not absolute", strZ(result)); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN const char * strZNull(const String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRINGZ, this == NULL ? NULL : strZ(this)); } /**********************************************************************************************************************************/ FN_EXTERN String * strReplaceChr(String *const this, const char find, const char replace) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(CHAR, find); FUNCTION_TEST_PARAM(CHAR, replace); FUNCTION_TEST_END(); ASSERT(this != NULL); for (size_t stringIdx = 0; stringIdx < strSize(this); stringIdx++) { if (this->pub.buffer[stringIdx] == find) this->pub.buffer[stringIdx] = replace; } FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strSub(const String *const this, const size_t start) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(SIZE, start); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(start <= this->pub.size); FUNCTION_TEST_RETURN(STRING, strSubN(this, start, strSize(this) - start)); } /**********************************************************************************************************************************/ FN_EXTERN String * strSubN(const String *const this, const size_t start, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(SIZE, start); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(start <= strSize(this)); ASSERT(start + size <= strSize(this)); FUNCTION_TEST_RETURN(STRING, strNewZN(strZ(this) + start, size)); } /**********************************************************************************************************************************/ FN_EXTERN String * strTrim(String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Nothing to trim if size is zero if (strSize(this) > 0) { // Find the beginning of the string skipping all whitespace char *begin = this->pub.buffer; while (*begin != 0 && (*begin == ' ' || *begin == '\t' || *begin == '\r' || *begin == '\n')) begin++; // Find the end of the string skipping all whitespace char *end = this->pub.buffer + (strSize(this) - 1); while (end > begin && (*end == ' ' || *end == '\t' || *end == '\r' || *end == '\n')) end--; // Is there anything to trim? const size_t newSize = (size_t)(end - begin + 1); if (begin != this->pub.buffer || newSize < strSize(this)) { // Calculate new size and extra this->pub.extra = (unsigned int)(strSize(this) - newSize); this->pub.size = (unsigned int)newSize; // Move the substr to the beginning of the buffer memmove(this->pub.buffer, begin, strSize(this)); // Invalidate the area that was trimmed, including the prior terminator, so Valgrind will warn on a read from it VALGRIND_MAKE_MEM_UNDEFINED(this->pub.buffer + strSize(this), this->pub.extra + 1); this->pub.buffer[strSize(this)] = 0; } } FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN int strChr(const String *const this, const char chr) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(CHAR, chr); FUNCTION_TEST_END(); ASSERT(this != NULL); int result = -1; if (strSize(this) > 0) { const char *const ptr = strchr(this->pub.buffer, chr); if (ptr != NULL) result = (int)(ptr - this->pub.buffer); } FUNCTION_TEST_RETURN(INT, result); } /**********************************************************************************************************************************/ FN_EXTERN String * strTruncIdx(String *const this, const int idx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(idx >= 0 && (size_t)idx <= strSize(this)); if (strSize(this) > 0) { // Reset the size to end at the index this->pub.extra = (unsigned int)(strSize(this) - (size_t)idx); this->pub.size = (unsigned int)idx; // Invalidate the area that was truncated, including the prior terminator, so Valgrind will warn on a read from it VALGRIND_MAKE_MEM_UNDEFINED(this->pub.buffer + strSize(this), this->pub.extra + 1); this->pub.buffer[strSize(this)] = 0; } FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN void strToLog(const String *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{\"%s\"}", strZ(this)); } /**********************************************************************************************************************************/ FN_EXTERN String * strSizeFormat(const uint64_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, size); FUNCTION_TEST_END(); String *result; if (size < 1024) result = strNewFmt("%" PRIu64 "B", size); else { char working[CVT_DIV_BUFFER_SIZE]; uint64_t divisor = 1024 * 1024 * 1024; unsigned int precision = 1; const char *suffix = "GB"; if (size < (1024 * 1024)) { divisor = 1024; suffix = "KB"; } else if (size < (1024 * 1024 * 1024)) { divisor = 1024 * 1024; suffix = "MB"; } // Skip precision when it would cause overflow if (size > UINT64_MAX / 10) precision = 0; // Format size cvtDivToZ(size, divisor, precision, true, working, sizeof(working)); result = strNewFmt("%s%s", working, suffix); } FUNCTION_TEST_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/string.h0000664000175100017510000003675515265101745020141 0ustar00runnerrunner/*********************************************************************************************************************************** String Handler Strings are lightweight objects in that they do not have their own memory context, instead they exist in the current context in which they are instantiated. If a string is needed outside the current memory context, the memory context must be switched to the old context and then back. Below is a simplified example: String *result = NULL; <--- is created in the current memory context (referred to as "old context" below) MEM_CONTEXT_TEMP_BEGIN() <--- begins a new temporary context { String *resultStr = strNewZN("myNewStr"); <--- creates a string in the temporary memory context MEM_CONTEXT_PRIOR_BEGIN() <--- switch to the old context so the duplication of the string is in that context { result = strDup(resultStr); <--- recreates a copy of the string in the old context where "result" was created } MEM_CONTEXT_PRIOR_END(); <--- switch back to the temporary context } MEM_CONTEXT_TEMP_END(); <-- frees everything created inside this temporary memory context - i.e resultStr ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_STRING_H #define COMMON_TYPE_STRING_H #include #include "common/debug.h" /*********************************************************************************************************************************** Minimum number of extra bytes to allocate for strings that are growing or are likely to grow ***********************************************************************************************************************************/ #ifndef STRING_EXTRA_MIN #define STRING_EXTRA_MIN 64 #endif /*********************************************************************************************************************************** String object ***********************************************************************************************************************************/ typedef struct String String; #include "common/assert.h" #include "common/encode.h" #include "common/type/buffer.h" #include "common/type/stringId.h" #include "common/type/stringZ.h" /*********************************************************************************************************************************** Constructors Only strings created with strNew() can use strCat*() functions. If a string needs to be concatenated then start with, .e.g.: String *example = strCatZ(strNew(), "example"); This syntax signals that the string will be modified so the string is allocated separately from the object so it can be resized when needed. Most strings never need to be modified and can be stored more efficiently by allocating their memory with the object. ***********************************************************************************************************************************/ // Create a new empty string for concatenation FN_EXTERN String *strNew(void); // Create a new fixed length string from a zero-terminated string FN_EXTERN String *strNewZ(const char *const string); // Create a new fixed length string from a zero-terminated string with a specific length. The string may or may not be // zero-terminated but we'll use that nomenclature since we're not concerned about the end of the string. FN_EXTERN String *strNewZN(const char *string, size_t size); // Create a new fixed length string from a buffer. If the buffer has a NULL character this may not work as expected. All the data // will be copied but only the data before the NULL character will be used as a string. FN_EXTERN String *strNewBuf(const Buffer *buffer); // Create a new fixed length string by performing division typedef struct StrNewDivParam { VAR_PARAM_HEADER; unsigned int precision; // Digits of precision after the decimal bool trim; // Trim trailing zeros and decimal point? } StrNewDivParam; #define strNewDivP(dividend, divisor, ...) \ strNewDiv(dividend, divisor, (StrNewDivParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN String *strNewDiv(uint64_t dividend, uint64_t divisor, StrNewDivParam param); // Create a new fixed length string containing a percentage FN_EXTERN String *strNewPct(uint64_t dividend, uint64_t divisor); // Create string from string id FN_EXTERN String *strNewStrId(StringId strId); // Create a new fixed length string by converting a timestamp typedef struct StrNewTimeParam { VAR_PARAM_HEADER; bool utc; // Use UTC instead of local time? } StrNewTimeParam; #define strNewTimeP(format, timestamp, ...) \ strNewTime(format, timestamp, (StrNewTimeParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN FN_STRFTIME(1) String *strNewTime(const char *format, time_t timestamp, StrNewTimeParam param); // Create a new fixed length string encoded with the specified type (e.g. encodingBase64) from a buffer FN_EXTERN String *strNewEncode(EncodingType type, const Buffer *buffer); // Create a new fixed length string from a format string with parameters (i.e. sprintf) FN_EXTERN FN_PRINTF(1, 2) String *strNewFmt(const char *format, ...); // Create a new fixed length string from a string FN_EXTERN String *strDup(const String *this); /*********************************************************************************************************************************** Getters/setters ***********************************************************************************************************************************/ typedef struct StringPub { uint64_t size : 32; // Actual size of the string uint64_t extra : 32; // Extra space allocated for expansion char *buffer; // String buffer } StringPub; // String size minus null-terminator, i.e. the same value that strlen() would return FN_INLINE_ALWAYS size_t strSize(const String *this) { return THIS_PUB(String)->size; } // Pointer to zero-terminated string. strZNull() returns NULL when the String is NULL. FN_INLINE_ALWAYS const char * strZ(const String *this) { return THIS_PUB(String)->buffer; } FN_EXTERN const char *strZNull(const String *this); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Return the file part of a string (i.e. everything after the last / or the entire string if there is no /). strBaseZ() does not // make a copy of the string so it may be more appropriate for large loops where saving memory is important. FN_EXTERN String *strBase(const String *this); FN_EXTERN const char *strBaseZ(const String *this); // Does the string begin with the specified string? FN_EXTERN bool strBeginsWith(const String *this, const String *beginsWith); FN_EXTERN bool strBeginsWithZ(const String *this, const char *beginsWith); // Append a string or zero-terminated string FN_EXTERN String *strCat(String *this, const String *cat); FN_EXTERN String *strCatZ(String *this, const char *cat); // Append a buffer FN_EXTERN String *strCatBuf(String *this, const Buffer *buffer); // Append a character FN_EXTERN String *strCatChr(String *this, char cat); // Append a string encoded with the specified type (e.g. encodingBase64) from a buffer FN_EXTERN String *strCatEncode(String *this, EncodingType type, const Buffer *buffer); // Append a timestamp typedef struct StrCatTimeParam { VAR_PARAM_HEADER; bool utc; // Use UTC instead of local time? } StrCatTimeParam; #define strCatTimeP(this, format, timestamp, ...) \ strCatTime(this, format, timestamp, (StrCatTimeParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN FN_STRFTIME(2) String *strCatTime(String *this, const char *format, time_t timestamp, StrCatTimeParam param); // Append a formatted string FN_EXTERN FN_PRINTF(2, 3) String *strCatFmt(String *this, const char *format, ...); // Append N characters from a zero-terminated string. Note that the string does not actually need to be zero-terminated as long as // N is <= the end of the string being concatenated. FN_EXTERN String *strCatZN(String *this, const char *cat, size_t size); FN_INLINE_ALWAYS String * strCatN(String *this, const String *const cat, const size_t size) { ASSERT_INLINE(cat != NULL); return strCatZN(this, strZ(cat), size); } // Return the index to the location of the first occurrence of a character within a string, else -1 FN_EXTERN int strChr(const String *this, char chr); // C-style string compare FN_EXTERN int strCmp(const String *this, const String *compare); FN_EXTERN int strCmpZ(const String *this, const char *compare); // Is the string empty? FN_EXTERN bool strEmpty(const String *this); // Does the string end with the specified string? FN_EXTERN bool strEndsWith(const String *this, const String *endsWith); FN_EXTERN bool strEndsWithZ(const String *this, const char *endsWith); // Are two strings equal? FN_EXTERN bool strEq(const String *this, const String *compare); FN_EXTERN bool strEqZ(const String *this, const char *compare); // Upper-case the first letter FN_EXTERN String *strFirstUpper(String *this); // Lower-case the first letter FN_EXTERN String *strFirstLower(String *this); // Lower-case entire string FN_EXTERN String *strLower(String *this); // Return the path part of a string (i.e. everything before the last / or "" if there is no /) FN_EXTERN String *strPath(const String *this); // Combine with a base path to get an absolute path FN_EXTERN String *strPathAbsolute(const String *this, const String *base); // Replace a character with another character FN_EXTERN String *strReplaceChr(String *this, char find, char replace); // Format sizes (file, buffer, etc.) in human-readable form FN_EXTERN String *strSizeFormat(const uint64_t fileSize); // Convert string to string id FN_INLINE_ALWAYS StringId strStrId(const String *const this) { return strIdFromZN(strZ(this), strSize(this), STRING_ID_SEQ_NONE); } // Return a substring given only the start position FN_EXTERN String *strSub(const String *this, size_t start); // Return a substring given the start position and size FN_EXTERN String *strSubN(const String *this, size_t start, size_t size); // Trim whitespace from the beginning and end of a string FN_EXTERN String *strTrim(String *this); // Truncate the end of a string from the index provided to the current end (e.g. 123KB pass index of K returns 123) FN_EXTERN String *strTruncIdx(String *this, int idx); // Truncate the string to zero size FN_INLINE_ALWAYS String * strTrunc(String *const this) { return strTruncIdx(this, 0); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void strFree(String *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for constant strings Frequently used constant strings can be declared with these macros at compile time rather than dynamically at run time. Note that strings created in this way are declared as const so can't be modified or freed by the str*() methods. Casting to String * will result in a segfault due to modifying read-only memory. By convention all string constant identifiers are appended with _STR. ***********************************************************************************************************************************/ // This struct must be kept in sync with StringPub (except for const qualifiers) typedef struct StringPubConst { uint64_t size : 32; // Actual size of the string uint64_t extra : 32; // Extra space allocated for expansion const char *buffer; // String buffer } StringPubConst; #define STR_SIZE(bufferParam, sizeParam) \ ((const String *const)&(const StringPubConst){.buffer = (const char *const)(bufferParam), .size = (unsigned int)(sizeParam)}) // Create a String constant inline from any zero-terminated string #define STR(buffer) \ STR_SIZE(buffer, strlen(buffer)) // Create a String constant inline from a #define or inline string constant #define STRDEF(buffer) \ STR_SIZE(buffer, sizeof(buffer) - 1) // Used to define String constants that will be externed using STRING_DECLARE(). Must be used in a .c file. #define STRING_EXTERN(name, buffer) \ VR_EXTERN_DEFINE const String *const name = STRDEF(buffer) // Used to define String constants that will be local to the .c file. Must be used in a .c file. #define STRING_STATIC(name, buffer) \ static const String *const name = STRDEF(buffer) // Used to declare externed String constants defined with STRING_EXTERN(). Must be used in a .h file. #define STRING_DECLARE(name) \ VR_EXTERN_DECLARE const String *const name /*********************************************************************************************************************************** Constant strings that are generally useful ***********************************************************************************************************************************/ STRING_DECLARE(CR_STR); STRING_DECLARE(CRLF_STR); STRING_DECLARE(DOT_STR); STRING_DECLARE(DOTDOT_STR); STRING_DECLARE(EMPTY_STR); STRING_DECLARE(FALSE_STR); STRING_DECLARE(FSLASH_STR); STRING_DECLARE(LF_STR); STRING_DECLARE(N_STR); STRING_DECLARE(NULL_STR); STRING_DECLARE(TRUE_STR); STRING_DECLARE(Y_STR); STRING_DECLARE(ZERO_STR); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void strToLog(const String *this, StringStatic *debugLog); #define FUNCTION_LOG_STRING_TYPE \ String * #define FUNCTION_LOG_STRING_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, strToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/stringId.c0000664000175100017510000004177615265101745020410 0ustar00runnerrunner/*********************************************************************************************************************************** Represent Short Strings as Integers ***********************************************************************************************************************************/ #include #include "common/assert.h" #include "common/debug.h" #include "common/type/stringId.h" /*********************************************************************************************************************************** Number of bits to use for encoding. The number of bits affects the character set that can be encoded. ***********************************************************************************************************************************/ typedef enum { stringIdBit5 = 0, // 5-bit encoding for a-z, 2, 5, 6, and - characters stringIdBit6 = 1, // 6-bit encoding for a-z, 0-9, A-Z, and - characters } StringIdBit; /*********************************************************************************************************************************** Constants used to extract information from the header ***********************************************************************************************************************************/ #define STRING_ID_BIT 0x1 // Bit to indicate 5/6 bits #define STRING_ID_HEADER_SHIFT 4 // Shift for standard header // It is possible to store a max sequence of 69 if all bits are used in 6-bit encoding -- but this seems like overkill for now #define STRING_ID_SEQ_MAX 37 // Max sequence #define STRING_ID_SEQ_HIGH_MIN 6 // Sequence >= must be stored in high sequence bits #define STRING_ID_SEQ_HIGH_SHIFT 5 // Shift for high sequence bits #define STRING_ID_SEQ_LOW_MASK 0xE // Mask for low sequence bits #define STRING_ID_SEQ_HIGH_MASK 0x1F0 // Mask for high sequence bits /**********************************************************************************************************************************/ // Helper to do encoding for specified number of bits static StringId strIdBitFromZN(const StringIdBit bit, const char *const buffer, size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, bit); FUNCTION_TEST_PARAM(VOID, buffer); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(size > 0); // Encoding type switch (bit) { // 5-bit encoding case stringIdBit5: { // Map to convert characters to encoding // {uncrustify_off - array alignment} static const uint8_t map[256] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 27, 0, 0, 0, 0, 28, 0, 0, 29, 30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, }; // {uncrustify_on} // If size is greater than can be encoded then error if (size > STRID5_MAX) FUNCTION_TEST_RETURN(STRING_ID, 0); // Make sure the string is valid for this encoding for (size_t bufferIdx = 0; bufferIdx < size; bufferIdx++) { if (map[(uint8_t)buffer[bufferIdx]] == 0) FUNCTION_TEST_RETURN(STRING_ID, 0); } // Set encoding in header uint64_t result = stringIdBit5; // Encode based on the number of characters that need to be encoded switch (size) { case 12: result |= (uint64_t)map[(uint8_t)buffer[11]] << 59; __attribute__((fallthrough)); case 11: result |= (uint64_t)map[(uint8_t)buffer[10]] << 54; __attribute__((fallthrough)); case 10: result |= (uint64_t)map[(uint8_t)buffer[9]] << 49; __attribute__((fallthrough)); case 9: result |= (uint64_t)map[(uint8_t)buffer[8]] << 44; __attribute__((fallthrough)); case 8: result |= (uint64_t)map[(uint8_t)buffer[7]] << 39; __attribute__((fallthrough)); case 7: result |= (uint64_t)map[(uint8_t)buffer[6]] << 34; __attribute__((fallthrough)); case 6: result |= (uint64_t)map[(uint8_t)buffer[5]] << 29; __attribute__((fallthrough)); case 5: result |= (uint64_t)map[(uint8_t)buffer[4]] << 24; __attribute__((fallthrough)); case 4: result |= (uint64_t)map[(uint8_t)buffer[3]] << 19; __attribute__((fallthrough)); case 3: result |= (uint64_t)map[(uint8_t)buffer[2]] << 14; __attribute__((fallthrough)); case 2: result |= (uint64_t)map[(uint8_t)buffer[1]] << 9; __attribute__((fallthrough)); default: result |= (uint64_t)map[(uint8_t)buffer[0]] << 4; } FUNCTION_TEST_RETURN(STRING_ID, result); } // 6-bit encoding default: { ASSERT(bit == stringIdBit6); // Map to convert characters to encoding // {uncrustify_off - array alignment} static const uint8_t map[256] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 27, 0, 0, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 0, 0, 0, 0, 0, 0, 0, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, }; // {uncrustify_on} // If size is greater than can be encoded then error if (size > STRID6_MAX) FUNCTION_TEST_RETURN(STRING_ID, 0); // Make sure the string is valid for this encoding for (size_t bufferIdx = 0; bufferIdx < size; bufferIdx++) { if (map[(uint8_t)buffer[bufferIdx]] == 0) FUNCTION_TEST_RETURN(STRING_ID, 0); } // Set encoding in header uint64_t result = stringIdBit6; // Encode based on the number of characters that need to be encoded switch (size) { case 10: result |= (uint64_t)map[(uint8_t)buffer[9]] << 58; __attribute__((fallthrough)); case 9: result |= (uint64_t)map[(uint8_t)buffer[8]] << 52; __attribute__((fallthrough)); case 8: result |= (uint64_t)map[(uint8_t)buffer[7]] << 46; __attribute__((fallthrough)); case 7: result |= (uint64_t)map[(uint8_t)buffer[6]] << 40; __attribute__((fallthrough)); case 6: result |= (uint64_t)map[(uint8_t)buffer[5]] << 34; __attribute__((fallthrough)); case 5: result |= (uint64_t)map[(uint8_t)buffer[4]] << 28; __attribute__((fallthrough)); case 4: result |= (uint64_t)map[(uint8_t)buffer[3]] << 22; __attribute__((fallthrough)); case 3: result |= (uint64_t)map[(uint8_t)buffer[2]] << 16; __attribute__((fallthrough)); case 2: result |= (uint64_t)map[(uint8_t)buffer[1]] << 10; __attribute__((fallthrough)); default: result |= (uint64_t)map[(uint8_t)buffer[0]] << 4; } FUNCTION_TEST_RETURN(STRING_ID, result); } } } FN_EXTERN StringId strIdFromZN(const char *const buffer, const size_t size, const unsigned int sequence) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VOID, buffer); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_PARAM(UINT, sequence); FUNCTION_TEST_END(); StringId result = strIdBitFromZN(stringIdBit5, buffer, size); // If 5-bit encoding fails try 6-bit if (result == 0) { result = strIdBitFromZN(stringIdBit6, buffer, size); // Error when 6-bit encoding also fails if (result == 0) THROW_FMT(FormatError, "'%s' contains invalid characters", buffer); } // Add sequence when specified if (sequence != STRING_ID_SEQ_NONE) { // If sequence can fit in header if (sequence < STRING_ID_SEQ_HIGH_MIN) { result |= (sequence + 1) << STRING_ID_BIT; } // Else sequence must be stored in high bits else { const StringIdBit bit = (StringIdBit)(result & STRING_ID_BIT); ASSERT(size < (bit ? STRID6_MAX : STRID5_MAX)); ASSERT(sequence <= STRING_ID_SEQ_MAX); // Shift to remove the bit marker (will be added back later) and make space for high sequence result >>= STRING_ID_BIT; result <<= STRING_ID_SEQ_HIGH_SHIFT + STRING_ID_BIT; // Add the sequence result |= (sequence - STRING_ID_SEQ_HIGH_MIN) << STRING_ID_HEADER_SHIFT | STRING_ID_SEQ_LOW_MASK | bit; } } FUNCTION_TEST_RETURN(STRING_ID, result); } /**********************************************************************************************************************************/ FN_EXTERN size_t strIdToZN(StringId strId, char *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, strId); FUNCTION_TEST_PARAM(VOID, buffer); FUNCTION_TEST_END(); ASSERT(strId != 0); ASSERT(buffer != NULL); // Extract bits used to encode the characters StringIdBit bit = (StringIdBit)(strId & STRING_ID_BIT); // Remove header to get the encoded characters if ((strId & STRING_ID_SEQ_LOW_MASK) == STRING_ID_SEQ_LOW_MASK) strId >>= STRING_ID_SEQ_HIGH_SHIFT + STRING_ID_HEADER_SHIFT; else strId >>= STRING_ID_HEADER_SHIFT; // Decoding type switch (bit) { // 5-bit decoding case stringIdBit5: { // Map to convert encoding to characters VR_NON_STRING static const char map[32] = "!abcdefghijklmnopqrstuvwxyz-256!"; // Macro to decode all but the last character #define STR5ID_TO_ZN_IDX(idx) \ buffer[idx] = map[strId & 0x1F]; \ strId >>= 5; \ \ if (strId == 0) \ FUNCTION_TEST_RETURN(SIZE, idx + 1) // Char 1-11 STR5ID_TO_ZN_IDX(0); STR5ID_TO_ZN_IDX(1); STR5ID_TO_ZN_IDX(2); STR5ID_TO_ZN_IDX(3); STR5ID_TO_ZN_IDX(4); STR5ID_TO_ZN_IDX(5); STR5ID_TO_ZN_IDX(6); STR5ID_TO_ZN_IDX(7); STR5ID_TO_ZN_IDX(8); STR5ID_TO_ZN_IDX(9); STR5ID_TO_ZN_IDX(10); // Char 12 buffer[11] = map[strId & 0x1F]; ASSERT(strId >> 5 == 0); FUNCTION_TEST_RETURN(SIZE, 12); } // 6-bit decoding default: { ASSERT(bit == stringIdBit6); // Map to convert encoding to characters VR_NON_STRING static const char map[64] = "!abcdefghijklmnopqrstuvwxyz-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"; // Macro to decode all but the last character #define STR6ID_TO_ZN_IDX(idx) \ buffer[idx] = map[strId & 0x3F]; \ strId >>= 6; \ \ if (strId == 0) \ FUNCTION_TEST_RETURN(SIZE, idx + 1) // Char 1-9 STR6ID_TO_ZN_IDX(0); STR6ID_TO_ZN_IDX(1); STR6ID_TO_ZN_IDX(2); STR6ID_TO_ZN_IDX(3); STR6ID_TO_ZN_IDX(4); STR6ID_TO_ZN_IDX(5); STR6ID_TO_ZN_IDX(6); STR6ID_TO_ZN_IDX(7); STR6ID_TO_ZN_IDX(8); // Char 10 buffer[9] = map[strId & 0x3F]; ASSERT(strId >> 6 == 0); FUNCTION_TEST_RETURN(SIZE, 10); } } } /**********************************************************************************************************************************/ FN_EXTERN size_t strIdToZ(const StringId strId, char *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, strId); FUNCTION_TEST_PARAM(VOID, buffer); FUNCTION_TEST_END(); size_t size = strIdToZN(strId, buffer); buffer[size] = '\0'; FUNCTION_TEST_RETURN(SIZE, size); } FN_EXTERN unsigned int strIdSeq(const StringId strId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, strId); FUNCTION_TEST_END(); const unsigned int sequence = (unsigned int)strId & STRING_ID_SEQ_LOW_MASK; if (sequence != STRING_ID_SEQ_LOW_MASK) { ASSERT(sequence != 0); FUNCTION_TEST_RETURN(UINT, (sequence >> STRING_ID_BIT) - 1); } FUNCTION_TEST_RETURN( UINT, (((unsigned int)strId & STRING_ID_SEQ_HIGH_MASK) >> STRING_ID_HEADER_SHIFT) + STRING_ID_SEQ_HIGH_MIN); } /**********************************************************************************************************************************/ FN_EXTERN size_t strIdToLog(const StringId strId, char *const buffer, const size_t bufferSize) { ASSERT(bufferSize > STRID_MAX); (void)bufferSize; // Treat 0 as if it were null since this can never be a valid StringId if (strId == 0) return (size_t)snprintf(buffer, bufferSize, NULL_Z); return strIdToZ(strId, buffer); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/stringId.h0000664000175100017510000001524715265101745020407 0ustar00runnerrunner/*********************************************************************************************************************************** Represent Short Strings as Integers It is often useful to represent identifiers as strings when they cannot easily be represented as an enum/integer, e.g. because they are distributed among a number of unrelated modules or need to be passed to remote processes. Strings are also more helpful in debugging since they can be recognized without cross-referencing the source. However, strings are awkward to work with in C since they cannot be directly used in switch statements leading to less efficient if-else structures. A StringId encodes a short string into an integer so it can be used in switch statements but may also be readily converted back into a string for debugging purposes. StringIds may also be suitable for matching user input providing the strings are short enough. strIdFromZ("mytest0123") will return the StringId 0x7de75c51315464d1. Using the value, the string representation can be retrieved with strIdToZ(0x7de75c51315464d1, ...) which returns "mytest0123+" where the plus at the end signals that the original string was equal to or longer than the maximum allowed. When assigning a StringId to an enum, it will be necessary to cast the StringId to the enum type if the enum contains all 32-bit values, since some compilers will complain about the implicit conversion without a cast. The enum will be 32-bit if all values of the enum are <= 0xffffffff. See bldStrId() for information on generating StringId constants. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_STRINGID_H #define COMMON_TYPE_STRINGID_H #include #include #include #include /*********************************************************************************************************************************** Maximum number of characters in the string representation of a StringId. When the string representation is equal to or greater than the MAX, then a plus sign + will be assigned as the last (MAX) character in the string representation. This is a safe buffer size when calling strIdToZN. If the buffer needs to be zero-terminated then an extra byte should be added. ***********************************************************************************************************************************/ // Maximum for specific encodings (e.g. 5-bit, 6-bit) #define STRID5_MAX 12 #define STRID6_MAX 10 // Maximum for any encoding #define STRID_MAX STRID5_MAX /*********************************************************************************************************************************** Constants used to extract information from the header ***********************************************************************************************************************************/ #define STRING_ID_BIT_MASK 3 /*********************************************************************************************************************************** Indicator for no sequence value ***********************************************************************************************************************************/ #define STRING_ID_SEQ_NONE UINT_MAX /*********************************************************************************************************************************** StringId typedef to make them more recognizable in the code ***********************************************************************************************************************************/ typedef uint64_t StringId; /*********************************************************************************************************************************** Macros to define constant StringIds. ALWAYS use bldStrId() to create these macros. The parameters in the macros are not verified against each other so the str parameter is included only for documentation purposes. ***********************************************************************************************************************************/ #define STRID5(str, strId) strId #define STRID5S(str, seq, strId) strId #define STRID6(str, strId) strId #define STRID6S(str, seq, strId) strId /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Convert N chars to StringId and optionally add sequence FN_EXTERN StringId strIdFromZN(const char *buffer, size_t size, unsigned int sequence); // Convert zero-terminated string to StringId and add sequence FN_INLINE_ALWAYS StringId strIdSeqFromZ(const char *const str, const unsigned int sequence) { return strIdFromZN(str, strlen(str), sequence); } // Convert zero-terminated string to StringId FN_INLINE_ALWAYS StringId strIdFromZ(const char *const str) { return strIdSeqFromZ(str, STRING_ID_SEQ_NONE); } // Write StringId to characters without zero-terminating. The buffer at ptr must have enough space to write the entire StringId, // which could be eight characters. However, the caller may know the exact (or max length) in advance and act accordingly. The // actual number of bytes written is returned. FN_EXTERN size_t strIdToZN(StringId strId, char *const buffer); // Convert StringId to zero-terminated string. See strIdToZN() for buffer sizing and return value. FN_EXTERN size_t strIdToZ(StringId strId, char *const buffer); // Get the sequence for a StringId (or error if none) FN_EXTERN unsigned int strIdSeq(StringId strId); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN size_t strIdToLog(StringId strId, char *buffer, size_t bufferSize); #define FUNCTION_LOG_STRING_ID_TYPE \ StringId #define FUNCTION_LOG_STRING_ID_FORMAT(value, buffer, bufferSize) \ strIdToLog(value, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/stringList.c0000664000175100017510000003334415265101745020757 0ustar00runnerrunner/*********************************************************************************************************************************** String List Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/debug.h" #include "common/memContext.h" #include "common/type/stringList.h" /*********************************************************************************************************************************** Internal add -- the string must have been created in the list's mem context before being passed ***********************************************************************************************************************************/ FN_INLINE_ALWAYS String * strLstAddInternal(StringList *const this, String *const string) { return *(String **)lstAdd((List *)this, &string); } /*********************************************************************************************************************************** Internal insert -- the string must have been created in the list's mem context before being passed ***********************************************************************************************************************************/ FN_INLINE_ALWAYS String * strLstInsertInternal(StringList *const this, const unsigned int listIdx, String *const string) { return *(String **)lstInsert((List *)this, listIdx, &string); } /**********************************************************************************************************************************/ FN_EXTERN StringList * strLstNewSplitZ(const String *const string, const char *const delimiter) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_PARAM(STRINGZ, delimiter); FUNCTION_TEST_END(); ASSERT(string != NULL); ASSERT(delimiter != NULL); // Create the list StringList *const this = strLstNew(); // Base points to the beginning of the string that is being searched const char *stringBase = strZ(string); // Match points to the next delimiter match that has been found const char *stringMatch = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { do { // Find a delimiter match stringMatch = strstr(stringBase, delimiter); // If a match was found then add the string if (stringMatch != NULL) { strLstAddInternal(this, strNewZN(stringBase, (size_t)(stringMatch - stringBase))); stringBase = stringMatch + strlen(delimiter); } // Else make whatever is left the last string else strLstAddInternal(this, strNewZ(stringBase)); } while (stringMatch != NULL); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN StringList * strLstNewVarLst(const VariantList *const sourceList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT_LIST, sourceList); FUNCTION_TEST_END(); // Create the list StringList *this = NULL; if (sourceList != NULL) { this = strLstNew(); // Copy variants MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { for (unsigned int listIdx = 0; listIdx < varLstSize(sourceList); listIdx++) strLstAddInternal(this, strDup(varStr(varLstGet(sourceList, listIdx)))); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN(STRING_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN StringList * strLstDup(const StringList *const sourceList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, sourceList); FUNCTION_TEST_END(); StringList *this = NULL; if (sourceList != NULL) { // Create the list this = strLstNew(); // Copy strings MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { for (unsigned int listIdx = 0; listIdx < strLstSize(sourceList); listIdx++) strLstAddInternal(this, strDup(strLstGet(sourceList, listIdx))); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN(STRING_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strLstAdd(StringList *const this, const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); ASSERT(this != NULL); String *result = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { result = strLstAddInternal(this, strDup(string)); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN String * strLstAddSubN(StringList *const this, const String *const string, const size_t offset, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_PARAM(SIZE, offset); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(string != NULL); String *result = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { result = strLstAddInternal(this, strSubN(string, offset, size)); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN String * strLstAddFmt(StringList *const this, const char *const format, ...) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(format != NULL); // Determine how long the allocated string needs to be va_list argumentList; va_start(argumentList, format); const size_t size = (size_t)vsnprintf(NULL, 0, format, argumentList); va_end(argumentList); // Format string va_start(argumentList, format); char *const buffer = memNew(size + 1); vsnprintf(buffer, size + 1, format, argumentList); va_end(argumentList); String *result = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { result = strLstAddInternal(this, strNewZN(buffer, size)); } MEM_CONTEXT_END(); memFree(buffer); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN String * strLstAddIfMissing(StringList *const this, const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); ASSERT(this != NULL); String *const *const result = lstFind((List *)this, &string); if (result == NULL) FUNCTION_TEST_RETURN(STRING, strLstAdd(this, string)); FUNCTION_TEST_RETURN(STRING, *result); } FN_EXTERN String * strLstAddZ(StringList *const this, const char *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRINGZ, string); FUNCTION_TEST_END(); ASSERT(this != NULL); String *result = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { result = strLstAddInternal(this, strNewZ(string)); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN String * strLstAddZSubN(StringList *const this, const char *const string, const size_t offset, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRINGZ, string); FUNCTION_TEST_PARAM(SIZE, offset); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(string != NULL); String *result = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { result = strLstAddInternal(this, strSubN(STR(string), offset, size)); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int strLstFindIdx(const StringList *const this, const String *const string, const StrLstFindIdxParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_PARAM(BOOL, param.required); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(string != NULL); const unsigned int result = lstFindIdx((const List *)this, &string); if (result == LIST_NOT_FOUND && param.required) THROW_FMT(AssertError, "unable to find '%s' in string list", strZ(string)); FUNCTION_TEST_RETURN(UINT, result); } /**********************************************************************************************************************************/ FN_EXTERN String * strLstInsert(StringList *const this, const unsigned int listIdx, const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(UINT, listIdx); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); ASSERT(this != NULL); String *result = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { result = strLstInsertInternal(this, listIdx, strDup(string)); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * strLstJoinQuote(const StringList *const this, const char *const separator, const char *const quote) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRINGZ, separator); FUNCTION_TEST_PARAM(STRINGZ, quote); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(separator != NULL); ASSERT(quote != NULL); String *const join = strNew(); for (unsigned int listIdx = 0; listIdx < strLstSize(this); listIdx++) { if (listIdx != 0) strCatZ(join, separator); if (strLstGet(this, listIdx) == NULL) strCatZ(join, "[NULL]"); else strCatFmt(join, "%s%s%s", quote, strZ(strLstGet(this, listIdx)), quote); } FUNCTION_TEST_RETURN(STRING, join); } /**********************************************************************************************************************************/ FN_EXTERN StringList * strLstMergeAnti(const StringList *const this, const StringList *const anti) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRING_LIST, anti); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(anti != NULL); StringList *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { result = strLstNew(); unsigned int antiIdx = 0; // Check every item in this for (unsigned int thisIdx = 0; thisIdx < strLstSize(this); thisIdx++) { bool add = true; const String *const listItem = strLstGet(this, thisIdx); ASSERT(listItem != NULL); // If anything left in anti look for matches while (antiIdx < strLstSize(anti)) { ASSERT(strLstGet(anti, antiIdx) != NULL); const int compare = strCmp(listItem, strLstGet(anti, antiIdx)); // If the current item in this is less than the current item in anti then it will be added if (compare < 0) { break; } // If they are equal it will not be added else if (compare == 0) { add = false; antiIdx++; break; } // Otherwise keep searching anti for a match else antiIdx++; } // Add to the result list if no match found if (add) strLstAdd(result, listItem); } strLstMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN const char ** strLstPtr(const StringList *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_END(); ASSERT(this != NULL); const char **const list = memNew((strLstSize(this) + 1) * sizeof(char *)); for (unsigned int listIdx = 0; listIdx < strLstSize(this); listIdx++) { if (strLstGet(this, listIdx) == NULL) list[listIdx] = NULL; else list[listIdx] = strZ(strLstGet(this, listIdx)); } list[strLstSize(this)] = NULL; FUNCTION_TEST_RETURN_CONST_P(STRINGZ, list); } /**********************************************************************************************************************************/ FN_EXTERN void strLstToLog(const StringList *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{["); for (unsigned int strLstIdx = 0; strLstIdx < strLstSize(this); strLstIdx++) { const String *const value = strLstGet(this, strLstIdx); if (strLstIdx != 0) strStcCat(debugLog, ", "); if (value == NULL) strStcCat(debugLog, NULL_Z); else strStcFmt(debugLog, "\"%s\"", strZ(value)); } strStcCat(debugLog, "]}"); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/stringList.h0000664000175100017510000001636615265101745020771 0ustar00runnerrunner/*********************************************************************************************************************************** String List Handler ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_STRINGLIST_H #define COMMON_TYPE_STRINGLIST_H /*********************************************************************************************************************************** StringList object ***********************************************************************************************************************************/ typedef struct StringList StringList; #include "common/memContext.h" #include "common/type/list.h" #include "common/type/string.h" #include "common/type/variantList.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ // Create empty StringList FN_INLINE_ALWAYS StringList * strLstNew(void) { return (StringList *)OBJ_NAME(lstNewP(sizeof(String *), .comparator = lstComparatorStr), StringList::List); } // Split a string into a string list based on a delimiter FN_EXTERN StringList *strLstNewSplitZ(const String *string, const char *delimiter); FN_INLINE_ALWAYS StringList * strLstNewSplit(const String *const string, const String *const delimiter) { return strLstNewSplitZ(string, strZ(delimiter)); } // New string list from a variant list -- all variants in the list must be type string FN_EXTERN StringList *strLstNewVarLst(const VariantList *sourceList); // Duplicate a string list FN_EXTERN StringList *strLstDup(const StringList *sourceList); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Set a new comparator FN_INLINE_ALWAYS StringList * strLstComparatorSet(StringList *const this, ListComparator *const comparator) { return (StringList *)lstComparatorSet((List *)this, comparator); } // List size FN_INLINE_ALWAYS unsigned int strLstSize(const StringList *const this) { return lstSize((const List *)this); } // Is the list empty? FN_INLINE_ALWAYS bool strLstEmpty(const StringList *const this) { return strLstSize(this) == 0; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add String to the list FN_EXTERN String *strLstAdd(StringList *this, const String *string); FN_EXTERN String *strLstAddSubN(StringList *this, const String *string, size_t offset, size_t size); FN_INLINE_ALWAYS String * strLstAddSub(StringList *const this, const String *const string, const size_t size) { return strLstAddSubN(this, string, 0, size); } FN_EXTERN FN_PRINTF(2, 3) String *strLstAddFmt(StringList *this, const char *format, ...); FN_EXTERN String *strLstAddZ(StringList *this, const char *string); FN_EXTERN String *strLstAddZSubN(StringList *this, const char *string, size_t offset, size_t size); FN_INLINE_ALWAYS String * strLstAddZSub(StringList *const this, const char *const string, const size_t size) { return strLstAddZSubN(this, string, 0, size); } FN_EXTERN String *strLstAddIfMissing(StringList *this, const String *string); // Does the specified string exist in the list? FN_INLINE_ALWAYS bool strLstExists(const StringList *const this, const String *const string) { return lstExists((const List *)this, &string); } // Find string index in the list typedef struct StrLstFindIdxParam { VAR_PARAM_HEADER; bool required; } StrLstFindIdxParam; #define strLstFindIdxP(this, string, ...) \ strLstFindIdx(this, string, (StrLstFindIdxParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN unsigned int strLstFindIdx(const StringList *this, const String *string, StrLstFindIdxParam param); // Insert into the list FN_EXTERN String *strLstInsert(StringList *this, unsigned int listIdx, const String *string); // Get a string by index FN_INLINE_ALWAYS String * strLstGet(const StringList *const this, const unsigned int listIdx) { return *(String **)lstGet((const List *)this, listIdx); } // Join a list of strings into a single string using the specified separator and quote with specified quote character FN_EXTERN String *strLstJoinQuote(const StringList *this, const char *separator, const char *quote); // Join a list of strings into a single string using the specified separator FN_INLINE_ALWAYS String * strLstJoin(const StringList *const this, const char *const separator) { return strLstJoinQuote(this, separator, ""); } // Return all items in this list which are not in the anti list. The input lists must *both* be sorted ascending or the results will // be undefined. FN_EXTERN StringList *strLstMergeAnti(const StringList *this, const StringList *anti); // Move to a new parent mem context FN_INLINE_ALWAYS StringList * strLstMove(StringList *const this, MemContext *const parentNew) { return (StringList *)lstMove((List *)this, parentNew); } // Return a null-terminated array of pointers to the zero-terminated strings in this list. DO NOT override const and modify any of // the strings in this array, though it is OK to modify the array itself. FN_EXTERN const char **strLstPtr(const StringList *this); // Remove an item from the list FN_INLINE_ALWAYS bool strLstRemove(StringList *const this, const String *const item) { return lstRemove((List *)this, &item); } FN_INLINE_ALWAYS StringList * strLstRemoveIdx(StringList *const this, const unsigned int listIdx) { return (StringList *)lstRemoveIdx((List *)this, listIdx); } // List sort FN_INLINE_ALWAYS StringList * strLstSort(StringList *const this, const SortOrder sortOrder) { return (StringList *)lstSort((List *)this, sortOrder); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void strLstFree(StringList *const this) { lstFree((List *)this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void strLstToLog(const StringList *this, StringStatic *debugLog); #define FUNCTION_LOG_STRING_LIST_TYPE \ StringList * #define FUNCTION_LOG_STRING_LIST_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, strLstToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/stringStatic.c0000664000175100017510000000421615265101745021267 0ustar00runnerrunner/*********************************************************************************************************************************** Static String Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/type/stringStatic.h" /**********************************************************************************************************************************/ FN_EXTERN StringStatic * strStcFmt(StringStatic *const debugLog, const char *const format, ...) { // Proceed if there is space for at least one character if (strStcRemainsSize(debugLog) > 1) { va_list argument; va_start(argument, format); size_t result = (size_t)vsnprintf(strStcRemains(debugLog), strStcRemainsSize(debugLog), format, argument); va_end(argument); if (result >= strStcRemainsSize(debugLog)) debugLog->resultSize = debugLog->bufferSize - 1; else debugLog->resultSize += result; } return debugLog; } /**********************************************************************************************************************************/ FN_EXTERN void strStcCat(StringStatic *const debugLog, const char *const cat) { // Proceed if there is space for at least one character const size_t remainsSize = strStcRemainsSize(debugLog); if (remainsSize > 1) { const size_t catSize = strlen(cat); const size_t resultSize = catSize > remainsSize - 1 ? remainsSize - 1 : catSize; memcpy(strStcRemains(debugLog), cat, resultSize); debugLog->resultSize += resultSize; debugLog->buffer[debugLog->resultSize] = '\0'; } } /**********************************************************************************************************************************/ FN_EXTERN void strStcCatChr(StringStatic *const debugLog, const char cat) { if (strStcRemainsSize(debugLog) > 1) { debugLog->buffer[debugLog->resultSize] = cat; debugLog->buffer[++debugLog->resultSize] = '\0'; } } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/stringStatic.h0000664000175100017510000000664315265101745021302 0ustar00runnerrunner/*********************************************************************************************************************************** Static String Handler Format a statically-size char buffer with automatic truncation and size tracking. These are convenience methods on top of the built-in format and memcpy functions to make the interface simpler. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_STRINGSTATIC_H #define COMMON_TYPE_STRINGSTATIC_H #include /*********************************************************************************************************************************** Zero-terminated strings that are generally useful ***********************************************************************************************************************************/ #define FALSE_Z "false" #define NULL_Z "null" #define TRUE_Z "true" /*********************************************************************************************************************************** Type ***********************************************************************************************************************************/ typedef struct StringStatic { char *buffer; size_t bufferSize; size_t resultSize; } StringStatic; /*********************************************************************************************************************************** Constructor ***********************************************************************************************************************************/ // Init static string FN_INLINE_ALWAYS StringStatic strStcInit(char *const buffer, const size_t bufferSize) { return (StringStatic){.buffer = buffer, .bufferSize = bufferSize}; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Cat formatted string to static string FN_EXTERN FN_PRINTF(2, 3) StringStatic *strStcFmt(StringStatic *debugLog, const char *format, ...); // Cat zero-terminated string to static string FN_EXTERN void strStcCat(StringStatic *debugLog, const char *cat); // Cat character to static string FN_EXTERN void strStcCatChr(StringStatic *debugLog, char cat); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Pointer to remaining buffer FN_INLINE_ALWAYS char * strStcRemains(const StringStatic *const debugLog) { return debugLog->buffer + debugLog->resultSize; } // Remaining buffer size FN_INLINE_ALWAYS size_t strStcRemainsSize(const StringStatic *const debugLog) { return debugLog->bufferSize - debugLog->resultSize; } // Result size FN_INLINE_ALWAYS size_t strStcResultSize(const StringStatic *const debugLog) { return debugLog->resultSize; } // Increment result size FN_INLINE_ALWAYS void strStcResultSizeInc(StringStatic *const debugLog, const size_t inc) { debugLog->resultSize += inc; } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/stringZ.c0000664000175100017510000000470515265101745020254 0ustar00runnerrunner/*********************************************************************************************************************************** Zero-Terminated String Handler ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/memContext.h" #include "common/type/object.h" #include "common/type/stringId.h" #include "common/type/stringZ.h" /**********************************************************************************************************************************/ static char * zNewInternal(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Allocate the string as extra or a separate allocation based on how large it is char *result; OBJ_NEW_BASE_EXTRA_BEGIN( char *, size > MEM_CONTEXT_ALLOC_EXTRA_MAX ? 0 : (uint16_t)size, .allocQty = size > MEM_CONTEXT_ALLOC_EXTRA_MAX ? 1 : 0) { result = size > MEM_CONTEXT_ALLOC_EXTRA_MAX ? memNew(size) : OBJ_NEW_ALLOC(); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(STRINGZ, result); } /**********************************************************************************************************************************/ FN_EXTERN char * zNewFmt(const char *const format, ...) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_END(); ASSERT(format != NULL); // Determine how long the allocated string needs to be va_list argumentList; va_start(argumentList, format); const size_t size = (size_t)vsnprintf(NULL, 0, format, argumentList) + 1; va_end(argumentList); // Format string char *const result = zNewInternal(size); va_start(argumentList, format); vsnprintf(result, size, format, argumentList); va_end(argumentList); FUNCTION_TEST_RETURN(STRINGZ, result); } /**********************************************************************************************************************************/ FN_EXTERN char * zNewStrId(const StringId strId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, strId); FUNCTION_TEST_END(); ASSERT(strId != 0); ASSERT(MEM_CONTEXT_ALLOC_EXTRA_MAX >= STRID_MAX + 1); char *const result = zNewInternal(STRID_MAX + 1); result[strIdToZN(strId, result)] = '\0'; FUNCTION_TEST_RETURN(STRINGZ, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/stringZ.h0000664000175100017510000000270715265101745020261 0ustar00runnerrunner/*********************************************************************************************************************************** Zero-Terminated String Handler These functions are shortcuts for creating/modifying zero-terminated strings without needing to wrap a strNew*() call in strZ(). The benefit is simpler code with less indentation, but be aware that memory is still being allocated. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_STRINGZ_H #define COMMON_TYPE_STRINGZ_H #include "common/type/object.h" #include "common/type/stringId.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Format a zero-terminated string FN_EXTERN FN_PRINTF(1, 2) char *zNewFmt(const char *format, ...); // Convert StringId to zero-terminated string FN_EXTERN char *zNewStrId(StringId strId); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void zFree(char *const this) { objFree(this); } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/variant.c0000664000175100017510000007331415265101745020262 0ustar00runnerrunner/*********************************************************************************************************************************** Variant Data Type ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include #include "common/debug.h" #include "common/memContext.h" #include "common/type/convert.h" #include "common/type/variant.h" /*********************************************************************************************************************************** Constant variants that are generally useful ***********************************************************************************************************************************/ // Used to define Bool Variant constants that will be externed using VARIANT_DECLARE(). #define VARIANT_BOOL_EXTERN(name, dataParam) \ VR_EXTERN_DEFINE const Variant *const name = ((const Variant *)&(const VariantBoolPub){.type = varTypeBool, .data = dataParam}) VARIANT_BOOL_EXTERN(BOOL_FALSE_VAR, false); VARIANT_BOOL_EXTERN(BOOL_TRUE_VAR, true); /*********************************************************************************************************************************** Information about the variant ***********************************************************************************************************************************/ struct Variant { VariantPub pub; // Publicly accessible variables }; typedef struct VariantBool { VariantBoolPub pub; // Publicly accessible variables } VariantBool; typedef struct VariantInt { VariantIntPub pub; // Publicly accessible variables } VariantInt; typedef struct VariantInt64 { VariantInt64Pub pub; // Publicly accessible variables } VariantInt64; typedef struct VariantKeyValue { VARIANT_COMMON KeyValue *data; // KeyValue data } VariantKeyValue; typedef struct VariantString { VariantStringPub pub; // Publicly accessible variables } VariantString; typedef struct VariantUInt { VariantUIntPub pub; // Publicly accessible variables } VariantUInt; typedef struct VariantUInt64 { VariantUInt64Pub pub; // Publicly accessible variables } VariantUInt64; typedef struct VariantVariantList { VARIANT_COMMON VariantList *data; // VariantList data } VariantVariantList; /*********************************************************************************************************************************** Variant type names ***********************************************************************************************************************************/ static const char *const variantTypeName[] = { "bool", // varTypeBool "int", // varTypeInt "int64", // varTypeInt64 "KeyValue", // varTypeKeyValue "String", // varTypeString "unsigned int", // varTypeUInt "uint64", // varTypeUInt64 "VariantList", // varTypeVariantList }; /**********************************************************************************************************************************/ FN_EXTERN Variant * varDup(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); Variant *result = NULL; if (this != NULL) { switch (varType(this)) { case varTypeBool: result = varNewBool(varBool(this)); break; case varTypeInt: result = varNewInt(varInt(this)); break; case varTypeInt64: result = varNewInt64(varInt64(this)); break; case varTypeKeyValue: result = varNewKv(kvDup(varKv(this))); break; case varTypeString: result = varNewStr(varStr(this)); break; case varTypeUInt: result = varNewUInt(varUInt(this)); break; case varTypeUInt64: result = varNewUInt64(varUInt64(this)); break; case varTypeVariantList: result = varNewVarLst(varVarLst(this)); break; } } FUNCTION_TEST_RETURN(VARIANT, result); } /**********************************************************************************************************************************/ FN_EXTERN bool varEq(const Variant *const this1, const Variant *const this2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this1); FUNCTION_TEST_PARAM(VARIANT, this2); FUNCTION_TEST_END(); bool result = false; // Test if both variants are non-null if (this1 != NULL && this2 != NULL) { // Test if both variants are of the same type if (varType(this1) == varType(this2)) { switch (varType(this1)) { case varTypeBool: result = varBool(this1) == varBool(this2); break; case varTypeInt: result = varInt(this1) == varInt(this2); break; case varTypeInt64: result = varInt64(this1) == varInt64(this2); break; case varTypeString: result = strEq(varStr(this1), varStr(this2)); break; case varTypeUInt: result = varUInt(this1) == varUInt(this2); break; case varTypeUInt64: result = varUInt64(this1) == varUInt64(this2); break; default: ASSERT(varType(this1) == varTypeKeyValue || varType(this1) == varTypeVariantList); THROW_FMT(AssertError, "unable to test equality for %s", variantTypeName[varType(this1)]); } } } // Else they are equal if they are both null else result = this1 == NULL && this2 == NULL; FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewBool(const bool data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantBool) { *this = (VariantBool) { .pub = { .type = varTypeBool, .data = data, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantBool)); } /**********************************************************************************************************************************/ FN_EXTERN bool varBool(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(varType(this) == varTypeBool); FUNCTION_TEST_RETURN(BOOL, ((const VariantBool *)this)->pub.data); } FN_EXTERN bool varBoolForce(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); bool result = false; switch (varType(this)) { case varTypeBool: result = varBool(this); break; case varTypeInt: result = varInt(this) != 0; break; case varTypeInt64: result = varInt64(this) != 0; break; case varTypeString: { // List of false/true boolean string values. Note that false/true values must be equal. static const char *const boolString[] = { "n", "f", "0", "no", FALSE_Z, "off", "y", "t", "1", "yes", TRUE_Z, "on", }; // Search for the string const char *const string = strZ(varStr(this)); unsigned int boolIdx = 0; for (; boolIdx < LENGTH_OF(boolString); boolIdx++) if (strcasecmp(string, boolString[boolIdx]) == 0) break; // If string was not found then not a boolean if (boolIdx == LENGTH_OF(boolString)) THROW_FMT(FormatError, "unable to convert str '%s' to bool", string); // False if in first half of list, true if in second half result = boolIdx / (LENGTH_OF(boolString) / 2); break; } case varTypeUInt: result = varUInt(this) != 0; break; case varTypeUInt64: result = varUInt64(this) != 0; break; default: ASSERT(varType(this) == varTypeKeyValue || varType(this) == varTypeVariantList); THROW_FMT(AssertError, "unable to force %s to %s", variantTypeName[varType(this)], variantTypeName[varTypeBool]); } FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewInt(const int data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantInt) { *this = (VariantInt) { .pub = { .type = varTypeInt, .data = data, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantInt)); } /**********************************************************************************************************************************/ FN_EXTERN int varInt(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(varType(this) == varTypeInt); FUNCTION_TEST_RETURN(INT, ((const VariantInt *)this)->pub.data); } FN_EXTERN int varIntForce(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); int result = 0; switch (varType(this)) { case varTypeBool: result = varBool(this); break; case varTypeInt: result = varInt(this); break; case varTypeInt64: { const int64_t resultTest = varInt64(this); // Make sure the value fits into a normal 32-bit int range since 32-bit platforms are supported if (resultTest > INT32_MAX || resultTest < INT32_MIN) THROW_FMT( FormatError, "unable to convert %s %" PRId64 " to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeInt]); result = (int)resultTest; break; } case varTypeString: result = cvtZToInt(strZ(varStr(this))); break; case varTypeUInt: { const unsigned int resultTest = varUInt(this); // Make sure the value fits into a normal 32-bit int range if (resultTest > INT32_MAX) THROW_FMT( FormatError, "unable to convert %s %u to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeInt]); result = (int)resultTest; break; } case varTypeUInt64: { const uint64_t resultTest = varUInt64(this); // Make sure the value fits into a normal 32-bit int range if (resultTest > INT32_MAX) THROW_FMT( FormatError, "unable to convert %s %" PRIu64 " to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeInt]); result = (int)resultTest; break; } default: ASSERT(varType(this) == varTypeKeyValue || varType(this) == varTypeVariantList); THROW_FMT(AssertError, "unable to force %s to %s", variantTypeName[varType(this)], variantTypeName[varTypeInt]); } FUNCTION_TEST_RETURN(INT, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewInt64(const int64_t data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT64, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantInt64) { *this = (VariantInt64) { .pub = { .type = varTypeInt64, .data = data, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantInt64)); } /**********************************************************************************************************************************/ FN_EXTERN int64_t varInt64(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(varType(this) == varTypeInt64); FUNCTION_TEST_RETURN(INT64, ((const VariantInt64 *)this)->pub.data); } FN_EXTERN int64_t varInt64Force(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); int64_t result = 0; switch (varType(this)) { case varTypeBool: result = varBool(this); break; case varTypeInt: result = (int64_t)varInt(this); break; case varTypeInt64: result = varInt64(this); break; case varTypeString: result = cvtZToInt64(strZ(varStr(this))); break; case varTypeUInt: result = varUInt(this); break; case varTypeUInt64: { const uint64_t resultTest = varUInt64(this); // If max number of unsigned 64-bit integer is greater than max 64-bit signed integer can hold, then error if (resultTest <= INT64_MAX) result = (int64_t)resultTest; else { THROW_FMT( FormatError, "unable to convert %s %" PRIu64 " to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeInt64]); } break; } default: ASSERT(varType(this) == varTypeKeyValue || varType(this) == varTypeVariantList); THROW_FMT(AssertError, "unable to force %s to %s", variantTypeName[varType(this)], variantTypeName[varTypeInt64]); } FUNCTION_TEST_RETURN(INT64, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewUInt(const unsigned int data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantUInt) { *this = (VariantUInt) { .pub = { .type = varTypeUInt, .data = data, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantUInt)); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int varUInt(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(varType(this) == varTypeUInt); FUNCTION_TEST_RETURN(UINT, ((const VariantUInt *)this)->pub.data); } FN_EXTERN unsigned int varUIntForce(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); unsigned int result = 0; switch (varType(this)) { case varTypeBool: result = varBool(this); break; case varTypeInt: { const int resultTest = varInt(this); // If integer is a negative number, throw an error since the resulting conversion would be a different number if (resultTest >= 0) result = (unsigned int)resultTest; else { THROW_FMT( FormatError, "unable to convert %s %d to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeUInt]); } break; } case varTypeInt64: { const int64_t resultTest = varInt64(this); // If integer is a negative number or too large, throw an error since the resulting conversion would be out of bounds if (resultTest >= 0 && resultTest <= UINT_MAX) result = (unsigned int)resultTest; else { THROW_FMT( FormatError, "unable to convert %s %" PRId64 " to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeUInt]); } break; } case varTypeUInt: result = varUInt(this); break; case varTypeUInt64: { const uint64_t resultTest = varUInt64(this); // If integer is too large, throw an error since the resulting conversion would be out of bounds if (resultTest <= UINT_MAX) result = (unsigned int)resultTest; else { THROW_FMT( FormatError, "unable to convert %s %" PRIu64 " to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeUInt]); } break; } case varTypeString: result = cvtZToUInt(strZ(varStr(this))); break; default: ASSERT(varType(this) == varTypeKeyValue || varType(this) == varTypeVariantList); THROW_FMT(AssertError, "unable to force %s to %s", variantTypeName[varType(this)], variantTypeName[varTypeUInt]); } FUNCTION_TEST_RETURN(UINT, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewUInt64(const uint64_t data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantUInt64) { *this = (VariantUInt64) { .pub = { .type = varTypeUInt64, .data = data, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantUInt64)); } /**********************************************************************************************************************************/ FN_EXTERN uint64_t varUInt64(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(varType(this) == varTypeUInt64); FUNCTION_TEST_RETURN(UINT64, ((const VariantUInt64 *)this)->pub.data); } FN_EXTERN uint64_t varUInt64Force(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); uint64_t result = 0; switch (varType(this)) { case varTypeBool: result = varBool(this); break; case varTypeInt: { const int resultTest = varInt(this); // If integer is a negative number, throw an error since the resulting conversion would be a different number if (resultTest >= 0) result = (uint64_t)resultTest; else { THROW_FMT( FormatError, "unable to convert %s %d to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeUInt64]); } break; } case varTypeInt64: { const int64_t resultTest = varInt64(this); // If integer is a negative number, throw an error since the resulting conversion would be out of bounds if (resultTest >= 0) result = (uint64_t)resultTest; else { THROW_FMT( FormatError, "unable to convert %s %" PRId64 " to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeUInt64]); } break; } case varTypeString: result = cvtZToUInt64(strZ(varStr(this))); break; case varTypeUInt: result = varUInt(this); break; case varTypeUInt64: result = varUInt64(this); break; default: ASSERT(varType(this) == varTypeKeyValue || varType(this) == varTypeVariantList); THROW_FMT(AssertError, "unable to force %s to %s", variantTypeName[varType(this)], variantTypeName[varTypeUInt64]); } FUNCTION_TEST_RETURN(UINT64, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewKv(KeyValue *const data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantKeyValue, .childQty = 1) { *this = (VariantKeyValue) { .type = varTypeKeyValue, }; if (data != NULL) this->data = kvMove(data, memContextCurrent()); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantKeyValue)); } /**********************************************************************************************************************************/ FN_EXTERN KeyValue * varKv(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); KeyValue *result = NULL; if (this != NULL) { ASSERT(varType(this) == varTypeKeyValue); result = ((const VariantKeyValue *)this)->data; } FUNCTION_TEST_RETURN(KEY_VALUE, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewStr(const String *const data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, data); FUNCTION_TEST_END(); // If the variant is larger than the extra allowed with a mem context then allocate the buffer separately const size_t allocExtra = sizeof(VariantString) + (data != NULL ? sizeof(StringPub) + strSize(data) + 1 : 0); if (allocExtra > MEM_CONTEXT_ALLOC_EXTRA_MAX) { ASSERT(data != NULL); OBJ_NEW_BEGIN(VariantString, .childQty = 1) { *this = (VariantString) { .pub = { .type = varTypeString, .data = strDup(data), }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantString)); } OBJ_NEW_EXTRA_BEGIN(VariantString, (uint16_t)(allocExtra)) { *this = (VariantString) { .pub = { .type = varTypeString, }, }; if (data != NULL) { // Point the String after the VariantString struct StringPub *const pubData = (StringPub *)(this + 1); this->pub.data = (String *)pubData; // Assign the string size and buffer (after StringPub struct) *pubData = (StringPub) { .size = (unsigned int)strSize(data), .buffer = (char *)(pubData + 1), }; // Assign the string memcpy(pubData->buffer, strZ(data), strSize(data)); pubData->buffer[strSize(data)] = '\0'; } } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantString)); } FN_EXTERN Variant * varNewStrZ(const char *const data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, data); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(VARIANT, varNewStr(data == NULL ? NULL : STR(data))); } /**********************************************************************************************************************************/ FN_EXTERN const String * varStr(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); String *result = NULL; if (this != NULL) { ASSERT(varType(this) == varTypeString); result = ((const VariantString *)this)->pub.data; } FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN String * varStrForce(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); String *result = NULL; switch (varType(this)) { case varTypeBool: result = strNewZ(cvtBoolToConstZ(varBool(this))); break; case varTypeInt: { char working[CVT_BASE10_BUFFER_SIZE]; cvtIntToZ(varInt(this), working, sizeof(working)); result = strNewZ(working); break; } case varTypeInt64: { char working[CVT_BASE10_BUFFER_SIZE]; cvtInt64ToZ(varInt64(this), working, sizeof(working)); result = strNewZ(working); break; } case varTypeString: result = strDup(varStr(this)); break; case varTypeUInt: { char working[CVT_BASE10_BUFFER_SIZE]; cvtUIntToZ(varUInt(this), working, sizeof(working)); result = strNewZ(working); break; } case varTypeUInt64: { char working[CVT_BASE10_BUFFER_SIZE]; cvtUInt64ToZ(varUInt64(this), working, sizeof(working)); result = strNewZ(working); break; } default: ASSERT(varType(this) == varTypeKeyValue || varType(this) == varTypeVariantList); THROW_FMT(FormatError, "unable to force %s to %s", variantTypeName[varType(this)], variantTypeName[varTypeString]); } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewVarLst(const VariantList *const data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT_LIST, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantVariantList, .childQty = 1) { *this = (VariantVariantList) { .type = varTypeVariantList, }; if (data != NULL) this->data = varLstDup(data); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantVariantList)); } /**********************************************************************************************************************************/ FN_EXTERN VariantList * varVarLst(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); VariantList *result = NULL; if (this != NULL) { ASSERT(varType(this) == varTypeVariantList); result = ((const VariantVariantList *)this)->data; } FUNCTION_TEST_RETURN(VARIANT_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN void varToLog(const Variant *const this, StringStatic *const debugLog) { switch (varType(this)) { case varTypeString: strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT(varStr(this), strToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); break; case varTypeKeyValue: strStcCat(debugLog, "{KeyValue}"); break; case varTypeVariantList: strStcCat(debugLog, "{VariantList}"); break; case varTypeBool: strStcFmt(debugLog, "{%s}", cvtBoolToConstZ(varBool(this))); break; case varTypeInt: strStcFmt(debugLog, "{%d}", varInt(this)); break; case varTypeInt64: strStcFmt(debugLog, "{%" PRId64 "}", varInt64(this)); break; case varTypeUInt: strStcFmt(debugLog, "{%u}", varUInt(this)); break; case varTypeUInt64: strStcFmt(debugLog, "{%" PRIu64 "}", varUInt64(this)); break; } } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/variant.h0000664000175100017510000002656215265101745020272 0ustar00runnerrunner/*********************************************************************************************************************************** Variant Data Type Variants are lightweight objects in that they do not have their own memory context, instead they exist in the current context in which they are instantiated. If a variant is needed outside the current memory context, the memory context must be switched to the old context and then back. Below is a simplified example: Variant *result = NULL; <--- is created in the current memory context (referred to as "old context" below) MEM_CONTEXT_TEMP_BEGIN() <--- begins a new temporary context { String *resultStr = strNewZN("myNewStr"); <--- creates a string in the temporary memory context MEM_CONTEXT_PRIOR_BEGIN() <--- switch to old context so creation of the variant from the string is in old context { result = varNewUInt64(cvtZToUInt64(strZ(resultStr))); <--- recreates variant from the string in the old context. } MEM_CONTEXT_PRIOR_END(); <--- switch back to the temporary context } MEM_CONTEXT_TEMP_END(); <-- frees everything created inside this temporary memory context - i.e resultStr ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_VARIANT_H #define COMMON_TYPE_VARIANT_H #include /*********************************************************************************************************************************** Variant object ***********************************************************************************************************************************/ typedef struct Variant Variant; /*********************************************************************************************************************************** Variant type ***********************************************************************************************************************************/ typedef enum { varTypeBool, varTypeInt, varTypeInt64, varTypeKeyValue, varTypeString, varTypeUInt, varTypeUInt64, varTypeVariantList, } VariantType; #include "common/type/keyValue.h" #include "common/type/string.h" #include "common/type/variantList.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Variant *varNewBool(bool data); FN_EXTERN Variant *varNewInt(int data); FN_EXTERN Variant *varNewInt64(int64_t data); // Note that the KeyValue is not duplicated because it this a heavy-weight operation. It is merely moved into the same MemContext as // the Variant. FN_EXTERN Variant *varNewKv(KeyValue *data); FN_EXTERN Variant *varNewStr(const String *data); FN_EXTERN Variant *varNewStrZ(const char *data); FN_EXTERN Variant *varNewUInt(unsigned int data); FN_EXTERN Variant *varNewUInt64(uint64_t data); FN_EXTERN Variant *varNewVarLst(const VariantList *data); FN_EXTERN Variant *varDup(const Variant *this); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ #define VARIANT_COMMON \ VariantType type; /* Variant type */ typedef struct VariantPub { VARIANT_COMMON } VariantPub; typedef struct VariantBoolPub { VARIANT_COMMON bool data; // Boolean data } VariantBoolPub; typedef struct VariantIntPub { VARIANT_COMMON int data; // Signed integer data } VariantIntPub; typedef struct VariantInt64Pub { VARIANT_COMMON int64_t data; // 64-bit signed integer data } VariantInt64Pub; typedef struct VariantStringPub { VARIANT_COMMON String *data; // String data } VariantStringPub; typedef struct VariantUIntPub { VARIANT_COMMON unsigned int data; // Unsigned integer data } VariantUIntPub; typedef struct VariantUInt64Pub { VARIANT_COMMON uint64_t data; // 64-bit unsigned integer data } VariantUInt64Pub; FN_EXTERN bool varBool(const Variant *this); FN_EXTERN bool varBoolForce(const Variant *this); FN_EXTERN int varInt(const Variant *this); FN_EXTERN int varIntForce(const Variant *this); FN_EXTERN int64_t varInt64(const Variant *this); FN_EXTERN int64_t varInt64Force(const Variant *this); FN_EXTERN KeyValue *varKv(const Variant *this); FN_EXTERN const String *varStr(const Variant *this); FN_EXTERN String *varStrForce(const Variant *this); FN_EXTERN unsigned int varUInt(const Variant *this); FN_EXTERN unsigned int varUIntForce(const Variant *this); FN_EXTERN uint64_t varUInt64(const Variant *this); FN_EXTERN uint64_t varUInt64Force(const Variant *this); FN_EXTERN VariantList *varVarLst(const Variant *this); // Variant type FN_INLINE_ALWAYS VariantType varType(const Variant *const this) { return THIS_PUB(Variant)->type; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Test if Variants are equal FN_EXTERN bool varEq(const Variant *this1, const Variant *this2); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void varFree(Variant *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for constant variants Frequently used constant variants can be declared with these macros at compile time rather than dynamically at run time. Note that variants created in this way are declared as const so can't be modified or freed by the var*() methods. Casting to Variant * will generally result in a segfault. By convention all variant constant identifiers are appended with _VAR. ***********************************************************************************************************************************/ // Create a Bool Variant constant inline from a bool #define VARBOOL(dataParam) \ ((const Variant *)&(const VariantBoolPub){.type = varTypeBool, .data = dataParam}) // Create an Int Variant constant inline from an int #define VARINT(dataParam) \ ((const Variant *)&(const VariantIntPub){.type = varTypeInt, .data = dataParam}) // Create an Int64 Variant constant inline from an int64_t #define VARINT64(dataParam) \ ((const Variant *)&(const VariantInt64Pub){.type = varTypeInt64, .data = dataParam}) // Create a String Variant constant inline from any zero-terminated string. The struct must be kept in sync with VariantStringPub // (except for const qualifiers) typedef struct VariantStringPubConst { VARIANT_COMMON const String *data; // String data } VariantStringPubConst; #define VARSTRZ(dataParam) \ ((const Variant *)&(const VariantStringPubConst){.type = varTypeString, .data = (const String *)STR(dataParam)}) // Create a String Variant constant inline from a #define or inline string constant #define VARSTRDEF(dataParam) \ ((const Variant *)&(const VariantStringPubConst){.type = varTypeString, .data = (const String *)STRDEF(dataParam)}) // Create a String Variant constant inline from a String constant #define VARSTR(dataParam) \ ((const Variant *)&(const VariantStringPubConst){.type = varTypeString, .data = (const String *)(dataParam)}) // Used to define String Variant constants that will be externed using VARIANT_DECLARE(). Must be used in a .c file. #define VARIANT_STRDEF_EXTERN(name, dataParam) \ VR_EXTERN_DEFINE const Variant *const name = VARSTRDEF(dataParam) // Used to define String Variant constants that will be local to the .c file. Must be used in a .c file. #define VARIANT_STRDEF_STATIC(name, dataParam) \ static const Variant *const name = VARSTRDEF(dataParam) // Create a UInt Variant constant inline from an unsigned int #define VARUINT(dataParam) \ ((const Variant *)&(const VariantUIntPub){.type = varTypeUInt, .data = dataParam}) // Create a UInt64 Variant constant inline from a uint64_t #define VARUINT64(dataParam) \ ((const Variant *)&(const VariantUInt64Pub){.type = varTypeUInt64, .data = dataParam}) // Used to declare externed String Variant constants defined with VARIANT*EXTERN(). Must be used in a .h file. #define VARIANT_DECLARE(name) \ VR_EXTERN_DECLARE const Variant *const name /*********************************************************************************************************************************** Constant variants that are generally useful ***********************************************************************************************************************************/ VARIANT_DECLARE(BOOL_FALSE_VAR); VARIANT_DECLARE(BOOL_TRUE_VAR); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void varToLog(const Variant *this, StringStatic *debugLog); #define FUNCTION_LOG_VARIANT_TYPE \ Variant * #define FUNCTION_LOG_VARIANT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, varToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/variantList.c0000664000175100017510000000355715265101745021120 0ustar00runnerrunner/*********************************************************************************************************************************** Variant List Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/memContext.h" #include "common/type/list.h" #include "common/type/variantList.h" /**********************************************************************************************************************************/ FN_EXTERN VariantList * varLstNewStrLst(const StringList *const stringList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, stringList); FUNCTION_TEST_END(); VariantList *this = NULL; if (stringList != NULL) { this = varLstNew(); MEM_CONTEXT_OBJ_BEGIN(this) { for (unsigned int listIdx = 0; listIdx < strLstSize(stringList); listIdx++) varLstAdd(this, varNewStr(strLstGet(stringList, listIdx))); } MEM_CONTEXT_OBJ_END(); } FUNCTION_TEST_RETURN(VARIANT_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN VariantList * varLstDup(const VariantList *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT_LIST, source); FUNCTION_TEST_END(); VariantList *this = NULL; if (source != NULL) { this = varLstNew(); MEM_CONTEXT_OBJ_BEGIN(this) { for (unsigned int listIdx = 0; listIdx < varLstSize(source); listIdx++) varLstAdd(this, varDup(varLstGet(source, listIdx))); } MEM_CONTEXT_OBJ_END(); } FUNCTION_TEST_RETURN(VARIANT_LIST, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/variantList.h0000664000175100017510000000724615265101745021124 0ustar00runnerrunner/*********************************************************************************************************************************** Variant List Handler ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_VARIANTLIST_H #define COMMON_TYPE_VARIANTLIST_H /*********************************************************************************************************************************** Variant list object ***********************************************************************************************************************************/ typedef struct VariantList VariantList; #include "common/type/stringList.h" #include "common/type/variant.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ // Create empty VariantList FN_INLINE_ALWAYS VariantList * varLstNew(void) { return (VariantList *)OBJ_NAME(lstNewP(sizeof(Variant *)), VariantList::List); } // Create VariantList from StringList FN_EXTERN VariantList *varLstNewStrLst(const StringList *stringList); // Duplicate a variant list FN_EXTERN VariantList *varLstDup(const VariantList *source); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // List size FN_INLINE_ALWAYS unsigned int varLstSize(const VariantList *const this) { return lstSize((const List *)this); } // Is the list empty? FN_INLINE_ALWAYS bool varLstEmpty(const VariantList *const this) { return varLstSize(this) == 0; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add to list FN_INLINE_ALWAYS VariantList * varLstAdd(VariantList *const this, Variant *const data) { lstAdd((List *)this, &data); return this; } // Get by index FN_INLINE_ALWAYS Variant * varLstGet(const VariantList *const this, const unsigned int listIdx) { return *(Variant **)lstGet((const List *)this, listIdx); } // Move to new parent mem context FN_INLINE_ALWAYS VariantList * varLstMove(VariantList *const this, MemContext *const parentNew) { return (VariantList *)lstMove((List *)this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void varLstFree(VariantList *const this) { lstFree((List *)this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_VARIANT_LIST_TYPE \ VariantList * #define FUNCTION_LOG_VARIANT_LIST_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "VariantList", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/xml.c0000664000175100017510000003013315265101745017406 0ustar00runnerrunner/*********************************************************************************************************************************** Xml Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/type/xml.h" /*********************************************************************************************************************************** Encoding constants ***********************************************************************************************************************************/ #define XML_ENCODING_TYPE_UTF8 "UTF-8" /*********************************************************************************************************************************** Node type ***********************************************************************************************************************************/ struct XmlNode { xmlNodePtr node; }; /*********************************************************************************************************************************** Document type ***********************************************************************************************************************************/ struct XmlDocument { XmlDocumentPub pub; // Publicly accessible variables xmlDocPtr xml; }; /*********************************************************************************************************************************** Error handler For now this is a noop until more detailed error messages are needed. The function is called multiple times per error, so the messages need to be accumulated and then returned together. This empty function is required because without it libxml2 will dump errors to stdout. Really. ***********************************************************************************************************************************/ static void xmlErrorHandler(void *const ctx, const char *const format, ...) { (void)ctx; (void)format; } /*********************************************************************************************************************************** Initialize xml ***********************************************************************************************************************************/ static void xmlInit(void) { FUNCTION_TEST_VOID(); // Initialize xml if it is not already initialized static bool xmlInit = false; if (!xmlInit) { LIBXML_TEST_VERSION; // Set error handler xmlSetGenericErrorFunc(NULL, xmlErrorHandler); xmlInit = true; } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static XmlNodeList * xmlNodeLstNew(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(XML_NODE_LIST, (XmlNodeList *)OBJ_NAME(lstNewP(sizeof(XmlNode *)), XmlNodeList::List)); } /**********************************************************************************************************************************/ static XmlNode * xmlNodeNew(xmlNodePtr const node) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, node); FUNCTION_TEST_END(); ASSERT(node != NULL); OBJ_NEW_BEGIN(XmlNode) { *this = (XmlNode) { .node = node, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(XML_NODE, this); } /**********************************************************************************************************************************/ FN_EXTERN XmlNode * xmlNodeAdd(XmlNode *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); XmlNode *const result = xmlNodeNew(xmlNewNode(NULL, (const xmlChar *)strZ(name))); xmlAddChild(this->node, result->node); FUNCTION_TEST_RETURN(XML_NODE, result); } /*********************************************************************************************************************************** Add a node to a node list ***********************************************************************************************************************************/ static XmlNodeList * xmlNodeLstAdd(XmlNodeList *const this, xmlNodePtr const node) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE_LIST, this); FUNCTION_TEST_PARAM_P(VOID, node); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(node != NULL); MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { XmlNode *const item = xmlNodeNew(node); lstAdd((List *)this, &item); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(XML_NODE_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN String * xmlNodeContent(const XmlNode *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_END(); String *result = NULL; if (this != NULL) { xmlChar *const content = xmlNodeGetContent(this->node); result = strNewZ((char *)content); xmlFree(content); } FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN void xmlNodeContentSet(XmlNode *const this, const String *const content) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_PARAM(STRING, content); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(content != NULL); xmlAddChild(this->node, xmlNewText((const xmlChar *)strZ(content))); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * xmlNodeName(const XmlNode *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(STRING, strNewZ((const char *)this->node->name)); } /**********************************************************************************************************************************/ FN_EXTERN XmlNodeList * xmlNodeChildList(const XmlNode *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); XmlNodeList *const list = xmlNodeLstNew(); for (xmlNodePtr currentNode = this->node->children; currentNode != NULL; currentNode = currentNode->next) { if (currentNode->type == XML_ELEMENT_NODE && strEqZ(name, (const char *)currentNode->name)) xmlNodeLstAdd(list, currentNode); } FUNCTION_TEST_RETURN(XML_NODE_LIST, list); } /**********************************************************************************************************************************/ FN_EXTERN XmlNodeList * xmlNodeChildListMulti(const XmlNode *const this, const StringList *const nameList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_PARAM(STRING_LIST, nameList); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(nameList != NULL); XmlNodeList *const list = xmlNodeLstNew(); for (xmlNodePtr currentNode = this->node->children; currentNode != NULL; currentNode = currentNode->next) { if (currentNode->type == XML_ELEMENT_NODE && strLstExists(nameList, STR((const char *)currentNode->name))) xmlNodeLstAdd(list, currentNode); } FUNCTION_TEST_RETURN(XML_NODE_LIST, list); } /**********************************************************************************************************************************/ FN_EXTERN XmlNode * xmlNodeChildN(const XmlNode *const this, const String *const name, const unsigned int index, const bool errorOnMissing) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_PARAM(UINT, index); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); XmlNode *child = NULL; unsigned int childIdx = 0; for (xmlNodePtr currentNode = this->node->children; currentNode != NULL; currentNode = currentNode->next) { if (currentNode->type == XML_ELEMENT_NODE && strEqZ(name, (const char *)currentNode->name)) { if (childIdx == index) { child = xmlNodeNew(currentNode); break; } childIdx++; } } if (child == NULL && errorOnMissing) THROW_FMT(FormatError, "unable to find child '%s':%u in node '%s'", strZ(name), index, this->node->name); FUNCTION_TEST_RETURN(XML_NODE, child); } /*********************************************************************************************************************************** Free document ***********************************************************************************************************************************/ static void xmlDocumentFreeResource(THIS_VOID) { THIS(XmlDocument); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(XML_DOCUMENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); xmlFreeDoc(this->xml); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN XmlDocument * xmlDocumentNew(const String *const rootName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, rootName); FUNCTION_TEST_END(); ASSERT(rootName != NULL); xmlInit(); OBJ_NEW_BEGIN(XmlDocument, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (XmlDocument) { .xml = xmlNewDoc((const xmlChar *)"1.0"), }; // Set callback to ensure xml document is freed memContextCallbackSet(objMemContext(this), xmlDocumentFreeResource, this); this->pub.root = xmlNodeNew(xmlNewNode(NULL, (const xmlChar *)strZ(rootName))); xmlDocSetRootElement(this->xml, xmlDocumentRoot(this)->node); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(XML_DOCUMENT, this); } /**********************************************************************************************************************************/ FN_EXTERN XmlDocument * xmlDocumentNewBuf(const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(!bufEmpty(buffer)); xmlInit(); OBJ_NEW_BEGIN(XmlDocument, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (XmlDocument){{0}}; // Extra braces are required for older gcc versions if ((this->xml = xmlReadMemory((const char *)bufPtrConst(buffer), (int)bufUsed(buffer), "noname.xml", NULL, 0)) == NULL) THROW_FMT(FormatError, "invalid xml in %zu byte(s):\n%s", bufUsed(buffer), strZ(strNewBuf(buffer))); // Set callback to ensure xml document is freed memContextCallbackSet(objMemContext(this), xmlDocumentFreeResource, this); // Get the root node this->pub.root = xmlNodeNew(xmlDocGetRootElement(this->xml)); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(XML_DOCUMENT, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * xmlDocumentBuf(const XmlDocument *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_DOCUMENT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); xmlChar *xml; int xmlSize; xmlDocDumpMemoryEnc(this->xml, &xml, &xmlSize, XML_ENCODING_TYPE_UTF8); Buffer *result = bufNewC(xml, (size_t)xmlSize); xmlFree(xml); FUNCTION_TEST_RETURN(BUFFER, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/type/xml.h0000664000175100017510000001422415265101745017416 0ustar00runnerrunner/*********************************************************************************************************************************** Xml Handler A thin wrapper around the libxml2 library. There are many capabilities of libxml2 that are not exposed here and may need to be added to when implementing new features. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_XML_H #define COMMON_TYPE_XML_H /*********************************************************************************************************************************** Objects ***********************************************************************************************************************************/ typedef struct XmlDocument XmlDocument; typedef struct XmlNode XmlNode; typedef struct XmlNodeList XmlNodeList; #include "common/memContext.h" #include "common/type/list.h" #include "common/type/object.h" #include "common/type/stringList.h" /*********************************************************************************************************************************** Document Constructors ***********************************************************************************************************************************/ // Document with the specified root node FN_EXTERN XmlDocument *xmlDocumentNew(const String *rootNode); // Document from Buffer FN_EXTERN XmlDocument *xmlDocumentNewBuf(const Buffer *); /*********************************************************************************************************************************** Document Getters ***********************************************************************************************************************************/ typedef struct XmlDocumentPub { XmlNode *root; // Root node } XmlDocumentPub; // Dump document to a buffer FN_EXTERN Buffer *xmlDocumentBuf(const XmlDocument *this); // Root node FN_INLINE_ALWAYS XmlNode * xmlDocumentRoot(const XmlDocument *const this) { return ((const XmlDocumentPub *const)this)->root; } /*********************************************************************************************************************************** Document Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void xmlDocumentFree(XmlDocument *const this) { objFree(this); } /*********************************************************************************************************************************** Node Functions ***********************************************************************************************************************************/ // Add a node FN_EXTERN XmlNode *xmlNodeAdd(XmlNode *this, const String *name); /*********************************************************************************************************************************** Node Getters/Setters ***********************************************************************************************************************************/ // Node child (by name or index) FN_EXTERN XmlNode *xmlNodeChildN(const XmlNode *this, const String *name, unsigned int index, bool errorOnMissing); FN_INLINE_ALWAYS XmlNode * xmlNodeChild(const XmlNode *const this, const String *const name, const bool errorOnMissing) { return xmlNodeChildN(this, name, 0, errorOnMissing); } // List of child nodes FN_EXTERN XmlNodeList *xmlNodeChildList(const XmlNode *this, const String *name); // List of child nodes using multiple names FN_EXTERN XmlNodeList *xmlNodeChildListMulti(const XmlNode *this, const StringList *nameList); // Node content FN_EXTERN String *xmlNodeContent(const XmlNode *this); FN_EXTERN void xmlNodeContentSet(XmlNode *this, const String *content); // Node name FN_EXTERN String *xmlNodeName(const XmlNode *this); /*********************************************************************************************************************************** Node List Getters ***********************************************************************************************************************************/ // Get a node from the list FN_INLINE_ALWAYS XmlNode * xmlNodeLstGet(const XmlNodeList *const this, const unsigned int listIdx) { return *(XmlNode **)lstGet((const List *const)this, listIdx); } // Node list size FN_INLINE_ALWAYS unsigned int xmlNodeLstSize(const XmlNodeList *const this) { return lstSize((const List *const)this); } /*********************************************************************************************************************************** Node List Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void xmlNodeLstFree(XmlNodeList *const this) { lstFree((List *const)this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_XML_DOCUMENT_TYPE \ XmlDocument * #define FUNCTION_LOG_XML_DOCUMENT_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "XmlDocument", buffer, bufferSize) #define FUNCTION_LOG_XML_NODE_TYPE \ XmlNode * #define FUNCTION_LOG_XML_NODE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "XmlNode", buffer, bufferSize) #define FUNCTION_LOG_XML_NODE_LIST_TYPE \ XmlNodeList * #define FUNCTION_LOG_XML_NODE_LIST_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "XmlNodeList", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/user.c0000664000175100017510000002552215265101745016611 0ustar00runnerrunner/*********************************************************************************************************************************** System User/Group Management ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/memContext.h" #include "common/user.h" /*********************************************************************************************************************************** Per-process uid->name and gid->name caches. On systems where uid/gid lookups are routed to a remote name service (sssd, systemd-userdbd, LDAP via nss-pam-ldapd, etc.), every call costs a Unix-socket round-trip plus its own audit-and-syscall overhead. Profiling shows this can dominate the manifest-build phase for clusters with millions of files even though the data files almost always share a single owner (typically "postgres"). The cache is a small fixed-size array because the working set of unique ids on a PG host is tiny (usually 1-3). Linear scan is faster than a hash table at this size and avoids any allocation in the hot path. The cached String lives in memContextTop() so it survives the caller's context. Cache misses past USER_NAME_CACHE_SIZE fall through to an uncached lookup (correct but slow); in practice we never exceed the table for a backup of a normally-configured cluster. ***********************************************************************************************************************************/ #define USER_NAME_CACHE_SIZE 16 typedef struct UserNameCache { uid_t id; // The uid that was looked up const String *name; // User name returned from lookup, NULL if unresolvable } UserNameCache; typedef struct GroupNameCache { gid_t id; // The gid that was looked up const String *name; // Group name returned from lookup, NULL if unresolvable } GroupNameCache; /*********************************************************************************************************************************** User group info ***********************************************************************************************************************************/ static struct { MemContext *memContext; // Mem context to store data in this struct uid_t userId; // Real user id of the calling process from getuid() bool userRoot; // Is this the root user? const String *userName; // User name if it exists UserNameCache userNameCache[USER_NAME_CACHE_SIZE]; // User name cache unsigned int userNameCacheTotal; // User cache total gid_t groupId; // Real group id of the calling process from getgid() const String *groupName; // Group name if it exists GroupNameCache groupNameCache[USER_NAME_CACHE_SIZE]; // Group name cache unsigned int groupNameCacheTotal; // Group name cache total #ifdef HAVE_LIBSSH2 const String *userHome; // User home directory #endif // HAVE_LIBSSH2 } userLocalData; /**********************************************************************************************************************************/ static void userInitInternal(void) { FUNCTION_TEST_VOID(); MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(UserLocalData, .childQty = MEM_CONTEXT_QTY_MAX) { userLocalData.memContext = MEM_CONTEXT_NEW(); userLocalData.userId = getuid(); userLocalData.userName = userNameFromId(userLocalData.userId); #ifdef HAVE_LIBSSH2 userLocalData.userHome = userHomeFromId(userLocalData.userId); #endif // HAVE_LIBSSH2 userLocalData.userRoot = userLocalData.userId == 0; userLocalData.groupId = getgid(); userLocalData.groupName = groupNameFromId(userLocalData.groupId); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void userInit(void) { FUNCTION_TEST_VOID(); if (!userLocalData.memContext) userInitInternal(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN gid_t groupId(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_TYPE(gid_t, userLocalData.groupId); } /**********************************************************************************************************************************/ FN_EXTERN gid_t groupIdFromName(const String *const groupName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, groupName); FUNCTION_TEST_END(); if (groupName != NULL) { const struct group *const groupData = getgrnam(strZ(groupName)); if (groupData != NULL) FUNCTION_TEST_RETURN_TYPE(gid_t, groupData->gr_gid); } FUNCTION_TEST_RETURN_TYPE(gid_t, (gid_t)-1); } /**********************************************************************************************************************************/ FN_EXTERN const String * groupName(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STRING, userLocalData.groupName); } /**********************************************************************************************************************************/ FN_EXTERN String * groupNameFromId(const gid_t groupId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, groupId); FUNCTION_TEST_END(); // Cache lookup on a small fixed-size table -- linear scan is fastest at this size for (unsigned int groupIdx = 0; groupIdx < userLocalData.groupNameCacheTotal; groupIdx++) { if (userLocalData.groupNameCache[groupIdx].id == groupId) FUNCTION_TEST_RETURN(STRING, strDup(userLocalData.groupNameCache[groupIdx].name)); } // Cache miss -- do the actual lookup const struct group *const groupData = getgrgid(groupId); String *const result = groupData != NULL ? strNewZ(groupData->gr_name) : NULL; // Record the answer (including the negative case) so we never look this id up again. Past USER_NAME_CACHE_SIZE we just fall // back to per-call lookup, which is correct -- only slower for the unusual case of many distinct owners. if (userLocalData.groupNameCacheTotal < USER_NAME_CACHE_SIZE) { MEM_CONTEXT_BEGIN(memContextTop()) { userLocalData.groupNameCache[userLocalData.groupNameCacheTotal] = (GroupNameCache) { .id = groupId, .name = strDup(result), }; } MEM_CONTEXT_END(); userLocalData.groupNameCacheTotal++; } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ // Currently userHome() and userHomeFromId() are only used if we are building with libssh2 #ifdef HAVE_LIBSSH2 FN_EXTERN const String * userHome(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STRING, userLocalData.userHome); } /**********************************************************************************************************************************/ FN_EXTERN String * userHomeFromId(const uid_t userId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, userId); FUNCTION_TEST_END(); const struct passwd *const userData = getpwuid(userId); if (userData != NULL) FUNCTION_TEST_RETURN(STRING, strNewZ(userData->pw_dir)); FUNCTION_TEST_RETURN(STRING, NULL); } #endif // HAVE_LIBSSH2 /**********************************************************************************************************************************/ FN_EXTERN uid_t userId(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_TYPE(uid_t, userLocalData.userId); } /**********************************************************************************************************************************/ FN_EXTERN uid_t userIdFromName(const String *const userName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, userName); FUNCTION_TEST_END(); if (userName != NULL) { const struct passwd *const userData = getpwnam(strZ(userName)); if (userData != NULL) FUNCTION_TEST_RETURN_TYPE(uid_t, userData->pw_uid); } FUNCTION_TEST_RETURN_TYPE(uid_t, (uid_t)-1); } /**********************************************************************************************************************************/ FN_EXTERN const String * userName(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STRING, userLocalData.userName); } /**********************************************************************************************************************************/ FN_EXTERN String * userNameFromId(const uid_t userId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, userId); FUNCTION_TEST_END(); // Cache lookup on a small fixed-size table -- linear scan is fastest at this size for (unsigned int userIdx = 0; userIdx < userLocalData.userNameCacheTotal; userIdx++) { if (userLocalData.userNameCache[userIdx].id == userId) FUNCTION_TEST_RETURN(STRING, strDup(userLocalData.userNameCache[userIdx].name)); } // Cache miss -- do the actual lookup const struct passwd *const userData = getpwuid(userId); String *const result = userData != NULL ? strNewZ(userData->pw_name) : NULL; // Record the answer (including the negative case) so we never look this id up again. Past USER_NAME_CACHE_SIZE we just fall // back to per-call lookup, which is correct -- only slower for the unusual case of many distinct owners. if (userLocalData.userNameCacheTotal < USER_NAME_CACHE_SIZE) { MEM_CONTEXT_BEGIN(memContextTop()) { userLocalData.userNameCache[userLocalData.userNameCacheTotal] = (UserNameCache) { .id = userId, .name = strDup(result), }; } MEM_CONTEXT_END(); userLocalData.userNameCacheTotal++; } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN bool userRoot(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(BOOL, userLocalData.userRoot); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/user.h0000664000175100017510000000415215265101745016612 0ustar00runnerrunner/*********************************************************************************************************************************** System User/Group Management ***********************************************************************************************************************************/ #ifndef COMMON_USER_H #define COMMON_USER_H #include #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Call this initializer before using any of the functions below. Safe to call more than once. FN_EXTERN void userInit(void); // Get the primary group id of the current user FN_EXTERN gid_t groupId(void); // Get the id of the specified group. Returns (gid_t)-1 if not found. FN_EXTERN gid_t groupIdFromName(const String *groupName); // Get the primary group name of the current user. Returns NULL if there is no mapping. FN_EXTERN const String *groupName(void); // Get the group name from a group id. Returns NULL if the group id is invalid or there is no mapping. FN_EXTERN String *groupNameFromId(gid_t groupId); #ifdef HAVE_LIBSSH2 // Get the home directory of the current user. Returns NULL if there is no mapping. FN_EXTERN const String *userHome(void); // Get the user home directory from a user id. Returns NULL if the user id is invalid or there is no mapping. FN_EXTERN String *userHomeFromId(uid_t userId); #endif // HAVE_LIBSSH2 // Get the id of the current user FN_EXTERN uid_t userId(void); // Get the id of the specified user. Returns (uid_t)-1 if not found. FN_EXTERN uid_t userIdFromName(const String *userName); // Get the name of the current user. Returns NULL if there is no mapping. FN_EXTERN const String *userName(void); // Get the user name from a user id. Returns NULL if the user id is invalid or there is no mapping. FN_EXTERN String *userNameFromId(uid_t userId); // Is the current user the root user? FN_EXTERN bool userRoot(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/valgrind.h0000664000175100017510000000224215265101745017440 0ustar00runnerrunner/*********************************************************************************************************************************** Valgrind Memcheck Client Requests Memcheck tracks the state of memory using the requests made to the allocator, which is not enough when memory is retained by the allocator for reuse or when part of an allocation is invalidated without being freed, e.g. when a buffer shrinks. In these cases memory that should be inaccessible or undefined still looks valid to memcheck and errors go undetected. The client requests below update the state directly so detection remains accurate. The requests are compiled in for debug builds when the Valgrind header is available, which includes the unit tests, and are no-ops when not run under Valgrind. When the header is not available they compile to nothing. ***********************************************************************************************************************************/ #ifndef COMMON_VALGRIND_H #define COMMON_VALGRIND_H #ifdef WITH_VALGRIND #include #else #define VALGRIND_MAKE_MEM_NOACCESS(addr, size) #define VALGRIND_MAKE_MEM_UNDEFINED(addr, size) #endif #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/wait.c0000664000175100017510000001024215265101745016570 0ustar00runnerrunner/*********************************************************************************************************************************** Wait Handler ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/wait.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Wait { TimeMSec waitTime; // Total time to wait (in usec) TimeMSec sleepTime; // Next sleep time (in usec) TimeMSec sleepPrevTime; // Previous time slept (in usec) TimeMSec beginTime; // Time the wait began (in epoch usec) unsigned int retry; // Retries remaining }; /**********************************************************************************************************************************/ FN_EXTERN Wait * waitNew(const TimeMSec waitTime) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TIMEMSEC, waitTime); FUNCTION_LOG_END(); ASSERT(waitTime <= 999999000); OBJ_NEW_BEGIN(Wait, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (Wait) { .waitTime = waitTime, .retry = 2, }; // Calculate first sleep time -- start with 1/10th of a second for anything >= 1 second if (this->waitTime >= MSEC_PER_SEC) { this->sleepTime = MSEC_PER_SEC / 10; } // Unless the wait time is really small -- in that case divide wait time by 10 else this->sleepTime = this->waitTime / 10; // Get beginning time this->beginTime = timeMSec(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(WAIT, this); } /**********************************************************************************************************************************/ FN_EXTERN TimeMSec waitRemains(Wait *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(WAIT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); TimeMSec result = 0; // If sleep is 0 then the wait time has already ended if (this->sleepTime > 0) { // Get the elapsed time const TimeMSec elapsedTime = timeMSec() - this->beginTime; // Is there more time to go? if (elapsedTime < this->waitTime) { // Calculate remaining time result = this->waitTime - elapsedTime; // Calculate sleep time as a sum of current and last (a Fibonacci-type sequence) TimeMSec sleepTime = this->sleepTime + this->sleepPrevTime; // Make sure sleep time does not go beyond remaining time (this won't be negative because of the if condition above) if (sleepTime > result) sleepTime = result; // Store new sleep times this->sleepPrevTime = this->sleepTime; this->sleepTime = sleepTime; } // Else are there retries left? else if (this->retry != 0) { result = this->sleepTime; } // Else set sleep to zero else this->sleepTime = 0; // Decrement retries if (this->retry > 0) this->retry--; } FUNCTION_LOG_RETURN(TIME_MSEC, result); } /**********************************************************************************************************************************/ FN_EXTERN bool waitMore(Wait *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(WAIT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = false; // If time remains in the wait then sleep if (waitRemains(this) > 0) { sleepMSec(this->sleepTime); result = true; } FUNCTION_LOG_RETURN(BOOL, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/common/wait.h0000664000175100017510000000552715265101745016607 0ustar00runnerrunner/*********************************************************************************************************************************** Wait Handler Used for operations that may fail due to an error or some unexpected condition such as file missing. When waitMore() is called it will wait (based on a Fibonacci backoff) before returning to give the error or condition time to clear. Even when the wait time has expired before waitMore() is called, there will still be two retries to compensate for operations that use up the entire time limit. Any number of retries are allowed within the time limit. ***********************************************************************************************************************************/ #ifndef COMMON_WAIT_H #define COMMON_WAIT_H /*********************************************************************************************************************************** Wait object ***********************************************************************************************************************************/ typedef struct Wait Wait; #include "common/time.h" #include "common/type/object.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Wait *waitNew(TimeMSec waitTime); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Return the remaining time left FN_EXTERN TimeMSec waitRemains(Wait *this); // Wait and return true if the caller has more time/retries left FN_EXTERN bool waitMore(Wait *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void waitFree(Wait *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_WAIT_TYPE \ Wait * #define FUNCTION_LOG_WAIT_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Wait", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/0000775000175100017510000000000015265101745015436 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/common.c0000664000175100017510000001276415265101745017104 0ustar00runnerrunner/*********************************************************************************************************************************** Configuration Common ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/regExp.h" #include "common/time.h" #include "config/common.h" /**********************************************************************************************************************************/ // Helper to get the multiplier based on the qualifier static int64_t cfgParseSizeQualifier(const char qualifier) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CHAR, qualifier); FUNCTION_TEST_END(); int64_t result; switch (qualifier) { case 'b': result = 1; break; case 'k': result = 1024; break; case 'm': result = 1024 * 1024; break; case 'g': result = 1024 * 1024 * 1024; break; case 't': result = 1024LL * 1024LL * 1024LL * 1024LL; break; case 'p': result = 1024LL * 1024LL * 1024LL * 1024LL * 1024LL; break; default: THROW_FMT(AssertError, "'%c' is not a valid size qualifier", qualifier); } FUNCTION_TEST_RETURN(INT64, result); } FN_EXTERN int64_t cfgParseSize(const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(value != NULL); // Lowercase the value String *const valueLower = strLower(strDup(value)); // Match the value against possible values if (regExpMatchOne(STRDEF("^[0-9]+(kib|kb|k|mib|mb|m|gib|gb|g|tib|tb|t|pib|pb|p|b)*$"), valueLower)) { // Get the character array and size const char *const strArray = strZ(valueLower); const size_t size = strSize(valueLower); int chrPos = -1; // If there is a 'b' on the end, then see if the previous character is a number if (strArray[size - 1] == 'b') { // If the previous character is a number, then the letter to look at is 'b' which is the last position else it is in the // next to last position (e.g. kb - so the 'k' is the position of interest). Only need to test for <= 9 since the regex // enforces the format. Also allow an 'i' before the 'b'. if (strArray[size - 2] <= '9') chrPos = (int)(size - 1); else if (strArray[size - 2] == 'i') chrPos = (int)(size - 3); else chrPos = (int)(size - 2); } // Else if there is no 'b' at the end but the last position is not a number then it must be one of the letters, e.g. 'k' else if (strArray[size - 1] > '9') chrPos = (int)(size - 1); int64_t multiplier = 1; // If a letter was found calculate multiplier, else do nothing since assumed value is already in bytes if (chrPos != -1) { multiplier = cfgParseSizeQualifier(strArray[chrPos]); // Remove any letters strTruncIdx(valueLower, chrPos); } // Convert string to bytes const int64_t valueInt = cvtZToInt64(strZ(valueLower)); if (valueInt > INT64_MAX / multiplier) THROW_FMT(FormatError, "value '%s' is out of range", strZ(value)); strFree(valueLower); FUNCTION_TEST_RETURN(INT64, valueInt * multiplier); } THROW_FMT(FormatError, "value '%s' is not valid", strZ(value)); } /**********************************************************************************************************************************/ FN_EXTERN int64_t cfgParseTime(const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(value != NULL); // Get value ptr and size const char *const valuePtr = strZ(value); const size_t size = strSize(value); size_t qualifierSize = 0; int64_t multiplier = 1000; // Check if this is ms (the only two character qualifier) if (size > 1 && tolower(valuePtr[size - 2]) == 'm' && tolower(valuePtr[size - 1]) == 's') { qualifierSize = 2; multiplier = 1; } // Else check for single character qualifier else if (size > 0) { switch (tolower(valuePtr[size - 1])) { case 's': qualifierSize = 1; multiplier = 1000; break; case 'm': qualifierSize = 1; multiplier = 60 * 1000; break; case 'h': qualifierSize = 1; multiplier = 60 * 60 * 1000; break; case 'd': qualifierSize = 1; multiplier = 24 * 60 * 60 * 1000; break; case 'w': qualifierSize = 1; multiplier = 7 * 24 * 60 * 60 * 1000; break; } } // Only proceed if there are numbers to parse if (size - qualifierSize > 0) { // Convert string to time const int64_t valueInt = cvtZSubNToInt64Base(valuePtr, 0, size - qualifierSize, 10); if (valueInt <= INT64_MAX / multiplier) FUNCTION_TEST_RETURN(INT64, valueInt * multiplier); } THROW_FMT(FormatError, "value '%s' is not valid", strZ(value)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/common.h0000664000175100017510000000146615265101745017106 0ustar00runnerrunner/*********************************************************************************************************************************** Configuration Common ***********************************************************************************************************************************/ #ifndef CONFIG_COMMON_H #define CONFIG_COMMON_H #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Parse option size, e.g. 23m FN_EXTERN int64_t cfgParseSize(const String *value); // Parse option time, e.g. 900 FN_EXTERN int64_t cfgParseTime(const String *value); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/config.auto.h0000664000175100017510000011621415265101745020030 0ustar00runnerrunner/*********************************************************************************************************************************** Command and Option Configuration Automatically generated by 'build.py config' -- do not modify directly. ***********************************************************************************************************************************/ #ifndef CONFIG_CONFIG_AUTO_H #define CONFIG_CONFIG_AUTO_H /*********************************************************************************************************************************** Command constants ***********************************************************************************************************************************/ #define CFGCMD_ANNOTATE "annotate" #define CFGCMD_ARCHIVE_GET "archive-get" #define CFGCMD_ARCHIVE_PUSH "archive-push" #define CFGCMD_BACKUP "backup" #define CFGCMD_CHECK "check" #define CFGCMD_EXPIRE "expire" #define CFGCMD_HELP "help" #define CFGCMD_INFO "info" #define CFGCMD_MANIFEST "manifest" #define CFGCMD_REPO_GET "repo-get" #define CFGCMD_REPO_LS "repo-ls" #define CFGCMD_REPO_RM "repo-rm" #define CFGCMD_RESTORE "restore" #define CFGCMD_SERVER "server" #define CFGCMD_SERVER_PING "server-ping" #define CFGCMD_STANZA_CREATE "stanza-create" #define CFGCMD_STANZA_DELETE "stanza-delete" #define CFGCMD_STANZA_UPGRADE "stanza-upgrade" #define CFGCMD_START "start" #define CFGCMD_STOP "stop" #define CFGCMD_VERIFY "verify" #define CFGCMD_VERSION "version" #define CFG_COMMAND_TOTAL 22 /*********************************************************************************************************************************** Option group constants ***********************************************************************************************************************************/ #define CFG_OPTION_GROUP_TOTAL 2 /*********************************************************************************************************************************** Option constants ***********************************************************************************************************************************/ #define CFGOPT_ALLOW_ROOT "allow-root" #define CFGOPT_ANNOTATION "annotation" #define CFGOPT_ARCHIVE_ASYNC "archive-async" #define CFGOPT_ARCHIVE_CHECK "archive-check" #define CFGOPT_ARCHIVE_COPY "archive-copy" #define CFGOPT_ARCHIVE_EXPIRE_BEFORE "archive-expire-before" #define CFGOPT_ARCHIVE_GET_QUEUE_MAX "archive-get-queue-max" #define CFGOPT_ARCHIVE_HEADER_CHECK "archive-header-check" #define CFGOPT_ARCHIVE_MISSING_RETRY "archive-missing-retry" #define CFGOPT_ARCHIVE_MODE "archive-mode" #define CFGOPT_ARCHIVE_MODE_CHECK "archive-mode-check" #define CFGOPT_ARCHIVE_PUSH_BATCH_SIZE "archive-push-batch-size" #define CFGOPT_ARCHIVE_PUSH_QUEUE_MAX "archive-push-queue-max" #define CFGOPT_ARCHIVE_TIMEOUT "archive-timeout" #define CFGOPT_BACKUP_STANDBY "backup-standby" #define CFGOPT_BETA "beta" #define CFGOPT_BUFFER_SIZE "buffer-size" #define CFGOPT_CHECKSUM_PAGE "checksum-page" #define CFGOPT_CHECKSUM_PAGE_ERROR "checksum-page-error" #define CFGOPT_CMD "cmd" #define CFGOPT_CMD_SSH "cmd-ssh" #define CFGOPT_COMPRESS "compress" #define CFGOPT_COMPRESS_LEVEL "compress-level" #define CFGOPT_COMPRESS_LEVEL_NETWORK "compress-level-network" #define CFGOPT_COMPRESS_TYPE "compress-type" #define CFGOPT_CONFIG "config" #define CFGOPT_CONFIG_INCLUDE_PATH "config-include-path" #define CFGOPT_CONFIG_PATH "config-path" #define CFGOPT_DB_EXCLUDE "db-exclude" #define CFGOPT_DB_INCLUDE "db-include" #define CFGOPT_DB_TIMEOUT "db-timeout" #define CFGOPT_DELTA "delta" #define CFGOPT_DETAIL_LEVEL "detail-level" #define CFGOPT_DRY_RUN "dry-run" #define CFGOPT_EXCLUDE "exclude" #define CFGOPT_EXEC_ID "exec-id" #define CFGOPT_EXPIRE_AUTO "expire-auto" #define CFGOPT_FILTER "filter" #define CFGOPT_FORCE "force" #define CFGOPT_HELP "help" #define CFGOPT_IGNORE_MISSING "ignore-missing" #define CFGOPT_IO_TIMEOUT "io-timeout" #define CFGOPT_JOB_RETRY "job-retry" #define CFGOPT_JOB_RETRY_INTERVAL "job-retry-interval" #define CFGOPT_LINK_ALL "link-all" #define CFGOPT_LINK_MAP "link-map" #define CFGOPT_LOCK "lock" #define CFGOPT_LOCK_PATH "lock-path" #define CFGOPT_LOG_LEVEL_CONSOLE "log-level-console" #define CFGOPT_LOG_LEVEL_FILE "log-level-file" #define CFGOPT_LOG_LEVEL_STDERR "log-level-stderr" #define CFGOPT_LOG_PATH "log-path" #define CFGOPT_LOG_SUBPROCESS "log-subprocess" #define CFGOPT_LOG_TIMESTAMP "log-timestamp" #define CFGOPT_MANIFEST_SAVE_THRESHOLD "manifest-save-threshold" #define CFGOPT_NEUTRAL_UMASK "neutral-umask" #define CFGOPT_OLDEST "oldest" #define CFGOPT_ONLINE "online" #define CFGOPT_OUTPUT "output" #define CFGOPT_PAGE_HEADER_CHECK "page-header-check" #define CFGOPT_PG "pg" #define CFGOPT_PG_VERSION_FORCE "pg-version-force" #define CFGOPT_PRIORITY "priority" #define CFGOPT_PROCESS "process" #define CFGOPT_PROCESS_MAX "process-max" #define CFGOPT_PROTOCOL_TIMEOUT "protocol-timeout" #define CFGOPT_RAW "raw" #define CFGOPT_RECOVERY_OPTION "recovery-option" #define CFGOPT_RECURSE "recurse" #define CFGOPT_REFERENCE "reference" #define CFGOPT_REMOTE_TYPE "remote-type" #define CFGOPT_REPO "repo" #define CFGOPT_REPO_TARGET_TIME "repo-target-time" #define CFGOPT_REPORT "report" #define CFGOPT_RESUME "resume" #define CFGOPT_SCK_BLOCK "sck-block" #define CFGOPT_SCK_KEEP_ALIVE "sck-keep-alive" #define CFGOPT_SET "set" #define CFGOPT_SORT "sort" #define CFGOPT_SPOOL_PATH "spool-path" #define CFGOPT_STANZA "stanza" #define CFGOPT_START_FAST "start-fast" #define CFGOPT_STOP_AUTO "stop-auto" #define CFGOPT_TABLESPACE_MAP "tablespace-map" #define CFGOPT_TABLESPACE_MAP_ALL "tablespace-map-all" #define CFGOPT_TARGET "target" #define CFGOPT_TARGET_ACTION "target-action" #define CFGOPT_TARGET_EXCLUSIVE "target-exclusive" #define CFGOPT_TARGET_TIMELINE "target-timeline" #define CFGOPT_TCP_KEEP_ALIVE_COUNT "tcp-keep-alive-count" #define CFGOPT_TCP_KEEP_ALIVE_IDLE "tcp-keep-alive-idle" #define CFGOPT_TCP_KEEP_ALIVE_INTERVAL "tcp-keep-alive-interval" #define CFGOPT_TLS_CIPHER_12 "tls-cipher-12" #define CFGOPT_TLS_CIPHER_13 "tls-cipher-13" #define CFGOPT_TLS_SERVER_ADDRESS "tls-server-address" #define CFGOPT_TLS_SERVER_AUTH "tls-server-auth" #define CFGOPT_TLS_SERVER_CA_FILE "tls-server-ca-file" #define CFGOPT_TLS_SERVER_CERT_FILE "tls-server-cert-file" #define CFGOPT_TLS_SERVER_KEY_FILE "tls-server-key-file" #define CFGOPT_TLS_SERVER_PORT "tls-server-port" #define CFGOPT_TYPE "type" #define CFGOPT_VERBOSE "verbose" #define CFGOPT_VERSION "version" #define CFG_OPTION_TOTAL 202 /*********************************************************************************************************************************** Option value constants ***********************************************************************************************************************************/ #define CFGOPTVAL_ARCHIVE_MODE_PRESERVE 0 #define CFGOPTVAL_ARCHIVE_MODE_PRESERVE_STRID STRID5S("preserve", 0, 0x2da45996502) #define CFGOPTVAL_ARCHIVE_MODE_PRESERVE_Z "preserve" #define CFGOPTVAL_ARCHIVE_MODE_OFF 1 #define CFGOPTVAL_ARCHIVE_MODE_OFF_STRID STRID5S("off", 1, 0x18cf4) #define CFGOPTVAL_ARCHIVE_MODE_OFF_Z "off" #define CFGOPTVAL_BACKUP_STANDBY_N 0 #define CFGOPTVAL_BACKUP_STANDBY_N_STRID STRID5S("n", 0, 0xe2) #define CFGOPTVAL_BACKUP_STANDBY_N_Z "n" #define CFGOPTVAL_BACKUP_STANDBY_PREFER 1 #define CFGOPTVAL_BACKUP_STANDBY_PREFER_STRID STRID5S("prefer", 1, 0x245316504) #define CFGOPTVAL_BACKUP_STANDBY_PREFER_Z "prefer" #define CFGOPTVAL_BACKUP_STANDBY_Y 2 #define CFGOPTVAL_BACKUP_STANDBY_Y_STRID STRID5S("y", 2, 0x196) #define CFGOPTVAL_BACKUP_STANDBY_Y_Z "y" #define CFGOPTVAL_COMPRESS_TYPE_BZ2 STRID5("bz2", 0x73420) #define CFGOPTVAL_COMPRESS_TYPE_BZ2_Z "bz2" #define CFGOPTVAL_COMPRESS_TYPE_GZ STRID5("gz", 0x3470) #define CFGOPTVAL_COMPRESS_TYPE_GZ_Z "gz" #define CFGOPTVAL_COMPRESS_TYPE_LZ4 STRID6("lz4", 0x2068c1) #define CFGOPTVAL_COMPRESS_TYPE_LZ4_Z "lz4" #define CFGOPTVAL_COMPRESS_TYPE_NONE STRID5("none", 0x2b9ee0) #define CFGOPTVAL_COMPRESS_TYPE_NONE_Z "none" #define CFGOPTVAL_COMPRESS_TYPE_ZST STRID5("zst", 0x527a0) #define CFGOPTVAL_COMPRESS_TYPE_ZST_Z "zst" #define CFGOPTVAL_DETAIL_LEVEL_FULL 0 #define CFGOPTVAL_DETAIL_LEVEL_FULL_STRID STRID5S("full", 0, 0x632a62) #define CFGOPTVAL_DETAIL_LEVEL_FULL_Z "full" #define CFGOPTVAL_DETAIL_LEVEL_PROGRESS 1 #define CFGOPTVAL_DETAIL_LEVEL_PROGRESS_STRID STRID5S("progress", 1, 0x9ccb23be504) #define CFGOPTVAL_DETAIL_LEVEL_PROGRESS_Z "progress" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_OFF 0 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_OFF_STRID STRID5S("off", 0, 0x18cf2) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_OFF_Z "off" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_ERROR 1 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_ERROR_STRID STRID5S("error", 1, 0x127ca454) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_ERROR_Z "error" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_WARN 2 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_WARN_STRID STRID5S("warn", 2, 0x748376) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_WARN_Z "warn" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_INFO 3 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_INFO_STRID STRID5S("info", 3, 0x799c98) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_INFO_Z "info" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_DETAIL 4 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_DETAIL_STRID STRID5S("detail", 4, 0x1890d0a4a) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_DETAIL_Z "detail" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_DEBUG 5 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_DEBUG_STRID STRID5S("debug", 5, 0x7a88a4c) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_DEBUG_Z "debug" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_TRACE 6 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_TRACE_STRID STRID5S("trace", 6, 0xa30ca80e) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_TRACE_Z "trace" #define CFGOPTVAL_LOG_LEVEL_FILE_OFF 0 #define CFGOPTVAL_LOG_LEVEL_FILE_OFF_STRID STRID5S("off", 0, 0x18cf2) #define CFGOPTVAL_LOG_LEVEL_FILE_OFF_Z "off" #define CFGOPTVAL_LOG_LEVEL_FILE_ERROR 1 #define CFGOPTVAL_LOG_LEVEL_FILE_ERROR_STRID STRID5S("error", 1, 0x127ca454) #define CFGOPTVAL_LOG_LEVEL_FILE_ERROR_Z "error" #define CFGOPTVAL_LOG_LEVEL_FILE_WARN 2 #define CFGOPTVAL_LOG_LEVEL_FILE_WARN_STRID STRID5S("warn", 2, 0x748376) #define CFGOPTVAL_LOG_LEVEL_FILE_WARN_Z "warn" #define CFGOPTVAL_LOG_LEVEL_FILE_INFO 3 #define CFGOPTVAL_LOG_LEVEL_FILE_INFO_STRID STRID5S("info", 3, 0x799c98) #define CFGOPTVAL_LOG_LEVEL_FILE_INFO_Z "info" #define CFGOPTVAL_LOG_LEVEL_FILE_DETAIL 4 #define CFGOPTVAL_LOG_LEVEL_FILE_DETAIL_STRID STRID5S("detail", 4, 0x1890d0a4a) #define CFGOPTVAL_LOG_LEVEL_FILE_DETAIL_Z "detail" #define CFGOPTVAL_LOG_LEVEL_FILE_DEBUG 5 #define CFGOPTVAL_LOG_LEVEL_FILE_DEBUG_STRID STRID5S("debug", 5, 0x7a88a4c) #define CFGOPTVAL_LOG_LEVEL_FILE_DEBUG_Z "debug" #define CFGOPTVAL_LOG_LEVEL_FILE_TRACE 6 #define CFGOPTVAL_LOG_LEVEL_FILE_TRACE_STRID STRID5S("trace", 6, 0xa30ca80e) #define CFGOPTVAL_LOG_LEVEL_FILE_TRACE_Z "trace" #define CFGOPTVAL_LOG_LEVEL_STDERR_OFF 0 #define CFGOPTVAL_LOG_LEVEL_STDERR_OFF_STRID STRID5S("off", 0, 0x18cf2) #define CFGOPTVAL_LOG_LEVEL_STDERR_OFF_Z "off" #define CFGOPTVAL_LOG_LEVEL_STDERR_ERROR 1 #define CFGOPTVAL_LOG_LEVEL_STDERR_ERROR_STRID STRID5S("error", 1, 0x127ca454) #define CFGOPTVAL_LOG_LEVEL_STDERR_ERROR_Z "error" #define CFGOPTVAL_LOG_LEVEL_STDERR_WARN 2 #define CFGOPTVAL_LOG_LEVEL_STDERR_WARN_STRID STRID5S("warn", 2, 0x748376) #define CFGOPTVAL_LOG_LEVEL_STDERR_WARN_Z "warn" #define CFGOPTVAL_LOG_LEVEL_STDERR_INFO 3 #define CFGOPTVAL_LOG_LEVEL_STDERR_INFO_STRID STRID5S("info", 3, 0x799c98) #define CFGOPTVAL_LOG_LEVEL_STDERR_INFO_Z "info" #define CFGOPTVAL_LOG_LEVEL_STDERR_DETAIL 4 #define CFGOPTVAL_LOG_LEVEL_STDERR_DETAIL_STRID STRID5S("detail", 4, 0x1890d0a4a) #define CFGOPTVAL_LOG_LEVEL_STDERR_DETAIL_Z "detail" #define CFGOPTVAL_LOG_LEVEL_STDERR_DEBUG 5 #define CFGOPTVAL_LOG_LEVEL_STDERR_DEBUG_STRID STRID5S("debug", 5, 0x7a88a4c) #define CFGOPTVAL_LOG_LEVEL_STDERR_DEBUG_Z "debug" #define CFGOPTVAL_LOG_LEVEL_STDERR_TRACE 6 #define CFGOPTVAL_LOG_LEVEL_STDERR_TRACE_STRID STRID5S("trace", 6, 0xa30ca80e) #define CFGOPTVAL_LOG_LEVEL_STDERR_TRACE_Z "trace" #define CFGOPTVAL_OUTPUT_TEXT 0 #define CFGOPTVAL_OUTPUT_TEXT_STRID STRID5S("text", 0, 0xa60b42) #define CFGOPTVAL_OUTPUT_TEXT_Z "text" #define CFGOPTVAL_OUTPUT_JSON 1 #define CFGOPTVAL_OUTPUT_JSON_STRID STRID5S("json", 1, 0x73e6a4) #define CFGOPTVAL_OUTPUT_JSON_Z "json" #define CFGOPTVAL_VERIFY_OUTPUT_NONE 0 #define CFGOPTVAL_VERIFY_OUTPUT_NONE_STRID STRID5S("none", 0, 0x2b9ee2) #define CFGOPTVAL_VERIFY_OUTPUT_NONE_Z "none" #define CFGOPTVAL_VERIFY_OUTPUT_TEXT 1 #define CFGOPTVAL_VERIFY_OUTPUT_TEXT_STRID STRID5S("text", 1, 0xa60b44) #define CFGOPTVAL_VERIFY_OUTPUT_TEXT_Z "text" #define CFGOPTVAL_VERSION_OUTPUT_TEXT 0 #define CFGOPTVAL_VERSION_OUTPUT_TEXT_STRID STRID5S("text", 0, 0xa60b42) #define CFGOPTVAL_VERSION_OUTPUT_TEXT_Z "text" #define CFGOPTVAL_VERSION_OUTPUT_NUM 1 #define CFGOPTVAL_VERSION_OUTPUT_NUM_STRID STRID5S("num", 1, 0x36ae4) #define CFGOPTVAL_VERSION_OUTPUT_NUM_Z "num" #define CFGOPTVAL_PG_HOST_TYPE_SSH STRID5("ssh", 0x22730) #define CFGOPTVAL_PG_HOST_TYPE_SSH_Z "ssh" #define CFGOPTVAL_PG_HOST_TYPE_TLS STRID5("tls", 0x4d940) #define CFGOPTVAL_PG_HOST_TYPE_TLS_Z "tls" #define CFGOPTVAL_REMOTE_TYPE_PG STRID5("pg", 0xf00) #define CFGOPTVAL_REMOTE_TYPE_PG_Z "pg" #define CFGOPTVAL_REMOTE_TYPE_REPO STRID5("repo", 0x7c0b20) #define CFGOPTVAL_REMOTE_TYPE_REPO_Z "repo" #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_SHARED 0 #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_SHARED_STRID STRID5S("shared", 0, 0x85905132) #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_SHARED_Z "shared" #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_SAS 1 #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_SAS_STRID STRID5S("sas", 1, 0x4c334) #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_SAS_Z "sas" #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_AUTO 2 #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_AUTO_STRID STRID5S("auto", 2, 0x7d2a16) #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_AUTO_Z "auto" #define CFGOPTVAL_REPO_AZURE_URI_STYLE_HOST 0 #define CFGOPTVAL_REPO_AZURE_URI_STYLE_HOST_STRID STRID5S("host", 0, 0xa4de82) #define CFGOPTVAL_REPO_AZURE_URI_STYLE_HOST_Z "host" #define CFGOPTVAL_REPO_AZURE_URI_STYLE_PATH 1 #define CFGOPTVAL_REPO_AZURE_URI_STYLE_PATH_STRID STRID5S("path", 1, 0x450304) #define CFGOPTVAL_REPO_AZURE_URI_STYLE_PATH_Z "path" #define CFGOPTVAL_REPO_CIPHER_TYPE_AES_256_CBC STRID5("aes-256-cbc", 0xc43dfbbcdcca10) #define CFGOPTVAL_REPO_CIPHER_TYPE_AES_256_CBC_Z "aes-256-cbc" #define CFGOPTVAL_REPO_CIPHER_TYPE_NONE STRID5("none", 0x2b9ee0) #define CFGOPTVAL_REPO_CIPHER_TYPE_NONE_Z "none" #define CFGOPTVAL_REPO_GCS_KEY_TYPE_SERVICE 0 #define CFGOPTVAL_REPO_GCS_KEY_TYPE_SERVICE_STRID STRID5S("service", 0, 0x1469b48b32) #define CFGOPTVAL_REPO_GCS_KEY_TYPE_SERVICE_Z "service" #define CFGOPTVAL_REPO_GCS_KEY_TYPE_TOKEN 1 #define CFGOPTVAL_REPO_GCS_KEY_TYPE_TOKEN_STRID STRID5S("token", 1, 0xe2adf44) #define CFGOPTVAL_REPO_GCS_KEY_TYPE_TOKEN_Z "token" #define CFGOPTVAL_REPO_GCS_KEY_TYPE_AUTO 2 #define CFGOPTVAL_REPO_GCS_KEY_TYPE_AUTO_STRID STRID5S("auto", 2, 0x7d2a16) #define CFGOPTVAL_REPO_GCS_KEY_TYPE_AUTO_Z "auto" #define CFGOPTVAL_REPO_GCS_KEY_TYPE_WEB_ID 3 #define CFGOPTVAL_REPO_GCS_KEY_TYPE_WEB_ID_STRID STRID5S("web-id", 3, 0x89d88b78) #define CFGOPTVAL_REPO_GCS_KEY_TYPE_WEB_ID_Z "web-id" #define CFGOPTVAL_REPO_HOST_TYPE_SSH STRID5("ssh", 0x22730) #define CFGOPTVAL_REPO_HOST_TYPE_SSH_Z "ssh" #define CFGOPTVAL_REPO_HOST_TYPE_TLS STRID5("tls", 0x4d940) #define CFGOPTVAL_REPO_HOST_TYPE_TLS_Z "tls" #define CFGOPTVAL_REPO_RETENTION_ARCHIVE_TYPE_DIFF STRID5("diff", 0x319240) #define CFGOPTVAL_REPO_RETENTION_ARCHIVE_TYPE_DIFF_Z "diff" #define CFGOPTVAL_REPO_RETENTION_ARCHIVE_TYPE_FULL STRID5("full", 0x632a60) #define CFGOPTVAL_REPO_RETENTION_ARCHIVE_TYPE_FULL_Z "full" #define CFGOPTVAL_REPO_RETENTION_ARCHIVE_TYPE_INCR STRID5("incr", 0x90dc90) #define CFGOPTVAL_REPO_RETENTION_ARCHIVE_TYPE_INCR_Z "incr" #define CFGOPTVAL_REPO_RETENTION_FULL_TYPE_COUNT 0 #define CFGOPTVAL_REPO_RETENTION_FULL_TYPE_COUNT_STRID STRID5S("count", 0, 0x14755e32) #define CFGOPTVAL_REPO_RETENTION_FULL_TYPE_COUNT_Z "count" #define CFGOPTVAL_REPO_RETENTION_FULL_TYPE_TIME 1 #define CFGOPTVAL_REPO_RETENTION_FULL_TYPE_TIME_STRID STRID5S("time", 1, 0x2b5344) #define CFGOPTVAL_REPO_RETENTION_FULL_TYPE_TIME_Z "time" #define CFGOPTVAL_REPO_S3_KEY_TYPE_SHARED 0 #define CFGOPTVAL_REPO_S3_KEY_TYPE_SHARED_STRID STRID5S("shared", 0, 0x85905132) #define CFGOPTVAL_REPO_S3_KEY_TYPE_SHARED_Z "shared" #define CFGOPTVAL_REPO_S3_KEY_TYPE_AUTO 1 #define CFGOPTVAL_REPO_S3_KEY_TYPE_AUTO_STRID STRID5S("auto", 1, 0x7d2a14) #define CFGOPTVAL_REPO_S3_KEY_TYPE_AUTO_Z "auto" #define CFGOPTVAL_REPO_S3_KEY_TYPE_WEB_ID 2 #define CFGOPTVAL_REPO_S3_KEY_TYPE_WEB_ID_STRID STRID5S("web-id", 2, 0x89d88b76) #define CFGOPTVAL_REPO_S3_KEY_TYPE_WEB_ID_Z "web-id" #define CFGOPTVAL_REPO_S3_KEY_TYPE_POD_ID 3 #define CFGOPTVAL_REPO_S3_KEY_TYPE_POD_ID_STRID STRID5S("pod-id", 3, 0x89d91f08) #define CFGOPTVAL_REPO_S3_KEY_TYPE_POD_ID_Z "pod-id" #define CFGOPTVAL_REPO_S3_KEY_TYPE_PROCESS 4 #define CFGOPTVAL_REPO_S3_KEY_TYPE_PROCESS_STRID STRID5S("process", 4, 0x4e651be50a) #define CFGOPTVAL_REPO_S3_KEY_TYPE_PROCESS_Z "process" #define CFGOPTVAL_REPO_S3_URI_STYLE_HOST 0 #define CFGOPTVAL_REPO_S3_URI_STYLE_HOST_STRID STRID5S("host", 0, 0xa4de82) #define CFGOPTVAL_REPO_S3_URI_STYLE_HOST_Z "host" #define CFGOPTVAL_REPO_S3_URI_STYLE_PATH 1 #define CFGOPTVAL_REPO_S3_URI_STYLE_PATH_STRID STRID5S("path", 1, 0x450304) #define CFGOPTVAL_REPO_S3_URI_STYLE_PATH_Z "path" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_ACCEPT_NEW STRID5("accept-new", 0x2e576e9028c610) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_ACCEPT_NEW_Z "accept-new" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT STRID5("fingerprint", 0x51c9942453b9260) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT_Z "fingerprint" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_NONE STRID5("none", 0x2b9ee0) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_NONE_Z "none" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_STRICT STRID5("strict", 0x2834ca930) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_STRICT_Z "strict" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_HASH_TYPE_MD5 STRID5("md5", 0x748d0) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_HASH_TYPE_MD5_Z "md5" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_HASH_TYPE_SHA1 STRID6("sha1", 0x7412131) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_HASH_TYPE_SHA1_Z "sha1" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_HASH_TYPE_SHA256 STRID5("sha256", 0x3dde05130) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_HASH_TYPE_SHA256_Z "sha256" #define CFGOPTVAL_REPO_TYPE_AZURE STRID5("azure", 0x5957410) #define CFGOPTVAL_REPO_TYPE_AZURE_Z "azure" #define CFGOPTVAL_REPO_TYPE_CIFS STRID5("cifs", 0x999230) #define CFGOPTVAL_REPO_TYPE_CIFS_Z "cifs" #define CFGOPTVAL_REPO_TYPE_GCS STRID5("gcs", 0x4c670) #define CFGOPTVAL_REPO_TYPE_GCS_Z "gcs" #define CFGOPTVAL_REPO_TYPE_POSIX STRID5("posix", 0x184cdf00) #define CFGOPTVAL_REPO_TYPE_POSIX_Z "posix" #define CFGOPTVAL_REPO_TYPE_S3 STRID6("s3", 0x7d31) #define CFGOPTVAL_REPO_TYPE_S3_Z "s3" #define CFGOPTVAL_REPO_TYPE_SFTP STRID5("sftp", 0x850d30) #define CFGOPTVAL_REPO_TYPE_SFTP_Z "sftp" #define CFGOPTVAL_SORT_NONE 0 #define CFGOPTVAL_SORT_NONE_STRID STRID5S("none", 0, 0x2b9ee2) #define CFGOPTVAL_SORT_NONE_Z "none" #define CFGOPTVAL_SORT_ASC 1 #define CFGOPTVAL_SORT_ASC_STRID STRID5S("asc", 1, 0xe614) #define CFGOPTVAL_SORT_ASC_Z "asc" #define CFGOPTVAL_SORT_DESC 2 #define CFGOPTVAL_SORT_DESC_STRID STRID5S("desc", 2, 0x1cca46) #define CFGOPTVAL_SORT_DESC_Z "desc" #define CFGOPTVAL_TARGET_ACTION_PAUSE STRID5("pause", 0x59d4300) #define CFGOPTVAL_TARGET_ACTION_PAUSE_Z "pause" #define CFGOPTVAL_TARGET_ACTION_PROMOTE STRID5("promote", 0x168f6be500) #define CFGOPTVAL_TARGET_ACTION_PROMOTE_Z "promote" #define CFGOPTVAL_TARGET_ACTION_SHUTDOWN STRID5("shutdown", 0x75de4a55130) #define CFGOPTVAL_TARGET_ACTION_SHUTDOWN_Z "shutdown" #define CFGOPTVAL_TYPE_DIFF STRID5("diff", 0x319240) #define CFGOPTVAL_TYPE_DIFF_Z "diff" #define CFGOPTVAL_TYPE_FULL STRID5("full", 0x632a60) #define CFGOPTVAL_TYPE_FULL_Z "full" #define CFGOPTVAL_TYPE_INCR STRID5("incr", 0x90dc90) #define CFGOPTVAL_TYPE_INCR_Z "incr" #define CFGOPTVAL_RESTORE_TYPE_DEFAULT 0 #define CFGOPTVAL_RESTORE_TYPE_DEFAULT_STRID STRID5S("default", 0, 0x5195098a42) #define CFGOPTVAL_RESTORE_TYPE_DEFAULT_Z "default" #define CFGOPTVAL_RESTORE_TYPE_PRESERVE 1 #define CFGOPTVAL_RESTORE_TYPE_PRESERVE_STRID STRID5S("preserve", 1, 0x2da45996504) #define CFGOPTVAL_RESTORE_TYPE_PRESERVE_Z "preserve" #define CFGOPTVAL_RESTORE_TYPE_IMMEDIATE 2 #define CFGOPTVAL_RESTORE_TYPE_IMMEDIATE_STRID STRID5S("immediate", 2, 0x5a05242b5a96) #define CFGOPTVAL_RESTORE_TYPE_IMMEDIATE_Z "immediate" #define CFGOPTVAL_RESTORE_TYPE_STANDBY 3 #define CFGOPTVAL_RESTORE_TYPE_STANDBY_STRID STRID5S("standby", 3, 0x6444706938) #define CFGOPTVAL_RESTORE_TYPE_STANDBY_Z "standby" #define CFGOPTVAL_RESTORE_TYPE_NONE 4 #define CFGOPTVAL_RESTORE_TYPE_NONE_STRID STRID5S("none", 4, 0x2b9eea) #define CFGOPTVAL_RESTORE_TYPE_NONE_Z "none" #define CFGOPTVAL_RESTORE_TYPE_NAME 5 #define CFGOPTVAL_RESTORE_TYPE_NAME_STRID STRID5S("name", 5, 0x2b42ec) #define CFGOPTVAL_RESTORE_TYPE_NAME_Z "name" #define CFGOPTVAL_RESTORE_TYPE_TIME 6 #define CFGOPTVAL_RESTORE_TYPE_TIME_STRID STRID5S("time", 6, 0x56a680e) #define CFGOPTVAL_RESTORE_TYPE_TIME_Z "time" #define CFGOPTVAL_RESTORE_TYPE_LSN 7 #define CFGOPTVAL_RESTORE_TYPE_LSN_STRID STRID5S("lsn", 7, 0x74d81e) #define CFGOPTVAL_RESTORE_TYPE_LSN_Z "lsn" #define CFGOPTVAL_RESTORE_TYPE_XID 8 #define CFGOPTVAL_RESTORE_TYPE_XID_STRID STRID5S("xid", 8, 0x22702e) #define CFGOPTVAL_RESTORE_TYPE_XID_Z "xid" /*********************************************************************************************************************************** Command enum ***********************************************************************************************************************************/ typedef enum { cfgCmdAnnotate, cfgCmdArchiveGet, cfgCmdArchivePush, cfgCmdBackup, cfgCmdCheck, cfgCmdExpire, cfgCmdHelp, cfgCmdInfo, cfgCmdManifest, cfgCmdRepoGet, cfgCmdRepoLs, cfgCmdRepoRm, cfgCmdRestore, cfgCmdServer, cfgCmdServerPing, cfgCmdStanzaCreate, cfgCmdStanzaDelete, cfgCmdStanzaUpgrade, cfgCmdStart, cfgCmdStop, cfgCmdVerify, cfgCmdVersion, } ConfigCommand; /*********************************************************************************************************************************** Option group enum ***********************************************************************************************************************************/ typedef enum { cfgOptGrpPg, cfgOptGrpRepo, } ConfigOptionGroup; /*********************************************************************************************************************************** Option enum ***********************************************************************************************************************************/ typedef enum { cfgOptAllowRoot, cfgOptAnnotation, cfgOptArchiveAsync, cfgOptArchiveCheck, cfgOptArchiveCopy, cfgOptArchiveExpireBefore, cfgOptArchiveGetQueueMax, cfgOptArchiveHeaderCheck, cfgOptArchiveMissingRetry, cfgOptArchiveMode, cfgOptArchiveModeCheck, cfgOptArchivePushBatchSize, cfgOptArchivePushQueueMax, cfgOptArchiveTimeout, cfgOptBackupStandby, cfgOptBeta, cfgOptBufferSize, cfgOptChecksumPage, cfgOptChecksumPageError, cfgOptCmd, cfgOptCmdSsh, cfgOptCompress, cfgOptCompressLevel, cfgOptCompressLevelNetwork, cfgOptCompressType, cfgOptConfig, cfgOptConfigIncludePath, cfgOptConfigPath, cfgOptDbExclude, cfgOptDbInclude, cfgOptDbTimeout, cfgOptDelta, cfgOptDetailLevel, cfgOptDryRun, cfgOptExclude, cfgOptExecId, cfgOptExpireAuto, cfgOptFilter, cfgOptForce, cfgOptHelp, cfgOptIgnoreMissing, cfgOptIoTimeout, cfgOptJobRetry, cfgOptJobRetryInterval, cfgOptLinkAll, cfgOptLinkMap, cfgOptLock, cfgOptLockPath, cfgOptLogLevelConsole, cfgOptLogLevelFile, cfgOptLogLevelStderr, cfgOptLogPath, cfgOptLogSubprocess, cfgOptLogTimestamp, cfgOptManifestSaveThreshold, cfgOptNeutralUmask, cfgOptOldest, cfgOptOnline, cfgOptOutput, cfgOptPageHeaderCheck, cfgOptPg, cfgOptPgDatabase, cfgOptPgHost, cfgOptPgHostCaFile, cfgOptPgHostCaPath, cfgOptPgHostCertFile, cfgOptPgHostCmd, cfgOptPgHostConfig, cfgOptPgHostConfigIncludePath, cfgOptPgHostConfigPath, cfgOptPgHostKeyFile, cfgOptPgHostPort, cfgOptPgHostType, cfgOptPgHostUser, cfgOptPgLocal, cfgOptPgPath, cfgOptPgPort, cfgOptPgSocketPath, cfgOptPgUser, cfgOptPgVersionForce, cfgOptPriority, cfgOptProcess, cfgOptProcessMax, cfgOptProtocolTimeout, cfgOptRaw, cfgOptRecoveryOption, cfgOptRecurse, cfgOptReference, cfgOptRemoteType, cfgOptRepo, cfgOptRepoAzureAccount, cfgOptRepoAzureContainer, cfgOptRepoAzureEndpoint, cfgOptRepoAzureKey, cfgOptRepoAzureKeyType, cfgOptRepoAzureUriStyle, cfgOptRepoBlock, cfgOptRepoBlockAgeMap, cfgOptRepoBlockChecksumSizeMap, cfgOptRepoBlockSizeMap, cfgOptRepoBlockSizeSuper, cfgOptRepoBlockSizeSuperFull, cfgOptRepoBundle, cfgOptRepoBundleLimit, cfgOptRepoBundleSize, cfgOptRepoCipherPass, cfgOptRepoCipherRotate, cfgOptRepoCipherType, cfgOptRepoFormat, cfgOptRepoGcsBucket, cfgOptRepoGcsEndpoint, cfgOptRepoGcsKey, cfgOptRepoGcsKeyType, cfgOptRepoGcsUserProject, cfgOptRepoHardlink, cfgOptRepoHost, cfgOptRepoHostCaFile, cfgOptRepoHostCaPath, cfgOptRepoHostCertFile, cfgOptRepoHostCmd, cfgOptRepoHostConfig, cfgOptRepoHostConfigIncludePath, cfgOptRepoHostConfigPath, cfgOptRepoHostKeyFile, cfgOptRepoHostPort, cfgOptRepoHostType, cfgOptRepoHostUser, cfgOptRepoLocal, cfgOptRepoPath, cfgOptRepoRetentionArchive, cfgOptRepoRetentionArchiveType, cfgOptRepoRetentionDiff, cfgOptRepoRetentionFull, cfgOptRepoRetentionFullType, cfgOptRepoRetentionHistory, cfgOptRepoS3Bucket, cfgOptRepoS3Endpoint, cfgOptRepoS3Key, cfgOptRepoS3KeySecret, cfgOptRepoS3KeyType, cfgOptRepoS3KmsKeyId, cfgOptRepoS3ProcessCmd, cfgOptRepoS3Region, cfgOptRepoS3RequesterPays, cfgOptRepoS3Role, cfgOptRepoS3Service, cfgOptRepoS3SseCustomerKey, cfgOptRepoS3StsHost, cfgOptRepoS3Token, cfgOptRepoS3UriStyle, cfgOptRepoSftpHost, cfgOptRepoSftpHostFingerprint, cfgOptRepoSftpHostKeyCheckType, cfgOptRepoSftpHostKeyHashType, cfgOptRepoSftpHostPort, cfgOptRepoSftpHostUser, cfgOptRepoSftpKnownHost, cfgOptRepoSftpPrivateKeyFile, cfgOptRepoSftpPrivateKeyPassphrase, cfgOptRepoSftpPublicKeyFile, cfgOptRepoStorageCaFile, cfgOptRepoStorageCaPath, cfgOptRepoStorageHost, cfgOptRepoStoragePort, cfgOptRepoStoragePrefetch, cfgOptRepoStorageReadOver, cfgOptRepoStorageTag, cfgOptRepoStorageUploadChunkSize, cfgOptRepoStorageVerifyTls, cfgOptRepoSymlink, cfgOptRepoTargetTime, cfgOptRepoType, cfgOptReport, cfgOptResume, cfgOptSckBlock, cfgOptSckKeepAlive, cfgOptSet, cfgOptSort, cfgOptSpoolPath, cfgOptStanza, cfgOptStartFast, cfgOptStopAuto, cfgOptTablespaceMap, cfgOptTablespaceMapAll, cfgOptTarget, cfgOptTargetAction, cfgOptTargetExclusive, cfgOptTargetTimeline, cfgOptTcpKeepAliveCount, cfgOptTcpKeepAliveIdle, cfgOptTcpKeepAliveInterval, cfgOptTlsCipher12, cfgOptTlsCipher13, cfgOptTlsServerAddress, cfgOptTlsServerAuth, cfgOptTlsServerCaFile, cfgOptTlsServerCertFile, cfgOptTlsServerKeyFile, cfgOptTlsServerPort, cfgOptType, cfgOptVerbose, cfgOptVersion, } ConfigOption; #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/config.c0000664000175100017510000011617515265101745017062 0ustar00runnerrunner/*********************************************************************************************************************************** Command and Option Configuration ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/memContext.h" #include "config/config.intern.h" #include "config/parse.h" /*********************************************************************************************************************************** Data for the currently loaded configuration ***********************************************************************************************************************************/ static Config *configLocal = NULL; /**********************************************************************************************************************************/ FN_EXTERN void cfgInit(Config *const config) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, config); FUNCTION_TEST_END(); ASSERT(config != NULL); // Free the old context if (configLocal != NULL) memContextFree(configLocal->memContext); // Set config and move context to top so it persists for the life of the program configLocal = config; memContextMove(configLocal->memContext, memContextTop()); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN bool cfgInited(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(ENUM, configLocal != NULL); } /**********************************************************************************************************************************/ FN_EXTERN ConfigCommand cfgCommand(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); FUNCTION_TEST_RETURN(ENUM, configLocal->command); } FN_EXTERN ConfigCommandRole cfgCommandRole(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); FUNCTION_TEST_RETURN(ENUM, configLocal->commandRole); } FN_EXTERN void cfgCommandSet(ConfigCommand commandId, ConfigCommandRole commandRoleId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, commandId); FUNCTION_TEST_PARAM(ENUM, commandRoleId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(commandId < CFG_COMMAND_TOTAL); configLocal->command = commandId; configLocal->commandRole = commandRoleId; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgCommandHelp(void) { FUNCTION_TEST_VOID(); ASSERT(configLocal != NULL); FUNCTION_TEST_RETURN(BOOL, configLocal->help); } /**********************************************************************************************************************************/ FN_EXTERN VariantList * cfgCommandJobRetry(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); // Return NULL if no retries const unsigned int retryTotal = cfgOptionUInt(cfgOptJobRetry); if (retryTotal == 0) FUNCTION_TEST_RETURN(VARIANT_LIST, NULL); // Build retry list VariantList *const result = varLstNew(); MEM_CONTEXT_BEGIN(lstMemContext((List *)result)) { for (unsigned int retryIdx = 0; retryIdx < cfgOptionUInt(cfgOptJobRetry); retryIdx++) varLstAdd(result, varNewUInt64(retryIdx == 0 ? 0 : cfgOptionUInt64(cfgOptJobRetryInterval))); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(VARIANT_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN const char * cfgCommandName(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); ASSERT(configLocal->command < CFG_COMMAND_TOTAL); FUNCTION_TEST_RETURN_CONST(STRINGZ, cfgParseCommandName(configLocal->command)); } FN_EXTERN String * cfgCommandRoleName(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(STRING, cfgParseCommandRoleName(cfgCommand(), cfgCommandRole())); } /**********************************************************************************************************************************/ FN_EXTERN const StringList * cfgCommandParam(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); if (configLocal->paramList == NULL) { MEM_CONTEXT_BEGIN(configLocal->memContext) { configLocal->paramList = strLstNew(); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN(STRING_LIST, configLocal->paramList); } /**********************************************************************************************************************************/ FN_EXTERN const String * cfgBin(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); FUNCTION_TEST_RETURN(STRING, configLocal->bin); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgLockRequired(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); ASSERT(configLocal->command < CFG_COMMAND_TOTAL); // Local roles never take a lock and the remote role has special logic for locking FUNCTION_TEST_RETURN( BOOL, // If a lock is required for the command and the role is main (configLocal->lockRequired && cfgCommandRole() == cfgCmdRoleMain) || // Or any command when the role is async cfgCommandRole() == cfgCmdRoleAsync); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgLockRemoteRequired(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); ASSERT(configLocal->command < CFG_COMMAND_TOTAL); FUNCTION_TEST_RETURN(BOOL, configLocal->lockRemoteRequired); } /**********************************************************************************************************************************/ FN_EXTERN LockType cfgLockType(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); ASSERT(configLocal->command < CFG_COMMAND_TOTAL); FUNCTION_TEST_RETURN(ENUM, configLocal->lockType); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgLogFile(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); ASSERT(configLocal->command < CFG_COMMAND_TOTAL); FUNCTION_TEST_RETURN( BOOL, // If the command always logs to a file configLocal->logFile || // Or log-level-file was explicitly set as a param/env var (cfgOptionValid(cfgOptLogLevelFile) && cfgOptionSource(cfgOptLogLevelFile) == cfgSourceParam) || // Or the role is async cfgCommandRole() == cfgCmdRoleAsync); } /**********************************************************************************************************************************/ FN_EXTERN LogLevel cfgLogLevelDefault(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); ASSERT(configLocal->command < CFG_COMMAND_TOTAL); FUNCTION_TEST_RETURN(ENUM, configLocal->logLevelDefault); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgOptionGroup(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN(BOOL, configLocal->option[optionId].group); } /**********************************************************************************************************************************/ FN_EXTERN const char * cfgOptionGroupName(const ConfigOptionGroup groupId, const unsigned int groupIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, groupId); FUNCTION_TEST_PARAM(UINT, groupIdx); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(groupId < CFG_OPTION_GROUP_TOTAL); ASSERT(groupIdx < configLocal->optionGroup[groupId].indexTotal); // Generate names for the group the first time one is requested ConfigOptionGroupData *const group = &configLocal->optionGroup[groupId]; if (group->indexName == NULL) { MEM_CONTEXT_BEGIN(configLocal->memContext) { group->indexName = memNew(sizeof(String *) * group->indexTotal); for (unsigned int groupIdx = 0; groupIdx < group->indexTotal; groupIdx++) group->indexName[groupIdx] = strNewFmt("%s%u", group->name, group->indexMap[groupIdx] + 1); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STRINGZ, strZ(group->indexName[groupIdx])); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionGroupId(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(configLocal->option[optionId].group); FUNCTION_TEST_RETURN(UINT, configLocal->option[optionId].groupId); } /**********************************************************************************************************************************/ static bool cfgOptionSequence(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN(BOOL, configLocal->option[optionId].sequence); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionGroupIdxDefault(const ConfigOptionGroup groupId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, groupId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(groupId < CFG_OPTION_GROUP_TOTAL); ASSERT(configLocal->optionGroup[groupId].indexDefaultExists); FUNCTION_TEST_RETURN(UINT, configLocal->optionGroup[groupId].indexDefault); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionGroupIdxToKey(const ConfigOptionGroup groupId, const unsigned int groupIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, groupId); FUNCTION_TEST_PARAM(UINT, groupIdx); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(groupId < CFG_OPTION_GROUP_TOTAL); ASSERT(groupIdx < configLocal->optionGroup[groupId].indexTotal); FUNCTION_TEST_RETURN(UINT, configLocal->optionGroup[groupId].indexMap[groupIdx] + 1); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionKeyToIdx(const ConfigOption optionId, const unsigned int key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, key); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); unsigned int result = 0; // If then option is in a group then search for the key, else the index is 0 if (cfgOptionGroup(optionId)) { const unsigned int groupId = cfgOptionGroupId(optionId); // Search the group for the key for (; result < cfgOptionGroupIdxTotal(groupId); result++) { if (configLocal->optionGroup[groupId].indexMap[result] == key - 1) break; } // Error when the key is not found if (result == cfgOptionGroupIdxTotal(groupId)) THROW_FMT(AssertError, "key '%u' is not valid for '%s' option", key, configLocal->option[optionId].name); } FUNCTION_TEST_RETURN(UINT, result); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionGroupIdxTotal(const ConfigOptionGroup groupId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, groupId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(groupId < CFG_OPTION_GROUP_TOTAL); FUNCTION_TEST_RETURN(UINT, configLocal->optionGroup[groupId].indexTotal); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionIdxDefault(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT( !configLocal->option[optionId].group || configLocal->optionGroup[configLocal->option[optionId].groupId].indexDefaultExists); const ConfigOptionData *const option = &configLocal->option[optionId]; FUNCTION_TEST_RETURN(UINT, option->group ? configLocal->optionGroup[option->groupId].indexDefault : 0); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionIdxTotal(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); const ConfigOptionData *const option = &configLocal->option[optionId]; FUNCTION_TEST_RETURN(UINT, option->group ? configLocal->optionGroup[option->groupId].indexTotal : 1); } /**********************************************************************************************************************************/ FN_EXTERN const String * cfgOptionIdxDefaultValue(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); FUNCTION_TEST_RETURN_CONST(STRING, configLocal->option[optionId].index[optionIdx].defaultValue); } /**********************************************************************************************************************************/ FN_EXTERN const String * cfgOptionDisplayVar(const Variant *const value, const ConfigOptionType optionType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_PARAM(UINT, optionType); FUNCTION_TEST_END(); ASSERT(value != NULL); ASSERT(optionType != cfgOptTypeHash && optionType != cfgOptTypeList); if (varType(value) == varTypeString) { FUNCTION_TEST_RETURN_CONST(STRING, varStr(value)); } else if (optionType == cfgOptTypeBoolean) { FUNCTION_TEST_RETURN_CONST(STRING, varBool(value) ? TRUE_STR : FALSE_STR); } else if (optionType == cfgOptTypeTime) { FUNCTION_TEST_RETURN_CONST(STRING, strNewDivP(varUInt64Force(value), MSEC_PER_SEC, .precision = 3, .trim = true)); } else if (optionType == cfgOptTypeStringId) { FUNCTION_TEST_RETURN_CONST(STRING, strNewStrId(varUInt64(value))); } FUNCTION_TEST_RETURN(STRING, varStrForce(value)); } FN_EXTERN const String * cfgOptionIdxDisplay(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); // Check that the option is valid for the current command if (!cfgOptionValid(optionId)) THROW_FMT(AssertError, "option '%s' is not valid for the current command", cfgOptionIdxName(optionId, optionIdx)); // If there is already a display value set then return that ConfigOptionValue *const option = &configLocal->option[optionId].index[optionIdx]; if (option->display != NULL) FUNCTION_TEST_RETURN_CONST(STRING, option->display); // Generate the display value based on the type MEM_CONTEXT_BEGIN(configLocal->memContext) { option->display = cfgOptionDisplayVar(cfgOptionIdxVar(optionId, optionIdx), cfgParseOptionType(optionId)); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_CONST(STRING, option->display); } FN_EXTERN const String * cfgOptionDisplay(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, cfgOptionIdxDisplay(optionId, cfgOptionIdxDefault(optionId))); } /*********************************************************************************************************************************** Get option name by id ***********************************************************************************************************************************/ FN_EXTERN const char * cfgOptionName(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN_CONST(STRINGZ, cfgOptionIdxName(optionId, cfgOptionIdxDefault(optionId))); } FN_EXTERN const char * cfgOptionIdxName(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = cfgParseOptionGroup(optionId)); ASSERT_DECLARE( const unsigned int indexTotal = group ? configLocal->optionGroup[cfgParseOptionGroupId(optionId)].indexTotal : 0); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); // If an indexed option that is valid for the command. Group data is only set for valid options so the name is generated and // cached here, where the group index total is known to be stable. ConfigOptionData *const option = &configLocal->option[optionId]; if (option->group) { const ConfigOptionGroupData *const group = &configLocal->optionGroup[option->groupId]; // Generate indexed names for the option the first time one is requested if (option->indexName == NULL) { MEM_CONTEXT_BEGIN(configLocal->memContext) { option->indexName = memNew(sizeof(String *) * group->indexTotal); for (unsigned int optionIdx = 0; optionIdx < group->indexTotal; optionIdx++) { option->indexName[optionIdx] = strNewFmt( "%s%u%s", group->name, group->indexMap[optionIdx] + 1, option->name + strlen(group->name)); } } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STRINGZ, strZ(option->indexName[optionIdx])); } // Else if an indexed option that is not valid for the command. Error messages may need to name an option the command does not // accept itself, e.g. a hint that points at a repository option, so build the name from the parse rules. if (cfgParseOptionGroup(optionId)) { FUNCTION_TEST_RETURN_CONST( STRINGZ, cfgParseOptionKeyIdxName(optionId, cfgOptionGroupIdxToKey(cfgParseOptionGroupId(optionId), optionIdx) - 1)); } // Else not indexed FUNCTION_TEST_RETURN_CONST(STRINGZ, option->name); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgOptionIdxNegate(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); FUNCTION_TEST_RETURN(BOOL, configLocal->option[optionId].index[optionIdx].negate); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgOptionIdxReset(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); FUNCTION_TEST_RETURN(BOOL, configLocal->option[optionId].index[optionIdx].reset); } /**********************************************************************************************************************************/ // Helper to enforce constraints when getting options static ConfigOptionValue * cfgOptionIdxInternal( const ConfigOption optionId, const unsigned int optionIdx, const ConfigOptionDataType typeRequested, const bool nullAllowed) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_PARAM(ENUM, typeRequested); FUNCTION_TEST_PARAM(BOOL, nullAllowed); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); // Check that the option is valid for the current command if (!cfgOptionValid(optionId)) THROW_FMT(AssertError, "option '%s' is not valid for the current command", cfgOptionIdxName(optionId, optionIdx)); // If the option is not NULL then check it is the requested type const ConfigOptionData *const option = &configLocal->option[optionId]; ConfigOptionValue *const result = &option->index[optionIdx]; if (result->set) { if (option->dataType != typeRequested) { THROW_FMT( AssertError, "option '%s' is type %u but %u was requested", cfgOptionIdxName(optionId, optionIdx), option->dataType, typeRequested); } } // Else check the option is allowed to be NULL else if (!nullAllowed) THROW_FMT(AssertError, "option '%s' is null but non-null was requested", cfgOptionIdxName(optionId, optionIdx)); FUNCTION_TEST_RETURN_TYPE_P(ConfigOptionValue, result); } FN_EXTERN Variant * cfgOptionIdxVar(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); const ConfigOptionData *const option = &configLocal->option[optionId]; if (option->index[optionIdx].set) { const ConfigOptionValueType *const optionValueType = &option->index[optionIdx].value; switch (option->dataType) { case cfgOptDataTypeBoolean: FUNCTION_TEST_RETURN(VARIANT, varNewBool(optionValueType->boolean)); case cfgOptDataTypeHash: FUNCTION_TEST_RETURN(VARIANT, varNewKv(kvDup(optionValueType->keyValue))); case cfgOptDataTypeInteger: FUNCTION_TEST_RETURN(VARIANT, varNewInt64(optionValueType->integer)); case cfgOptDataTypeList: FUNCTION_TEST_RETURN(VARIANT, varNewVarLst(optionValueType->list)); case cfgOptDataTypeStringId: cfgOptionIdxStrId(optionId, optionIdx); FUNCTION_TEST_RETURN(VARIANT, varNewUInt64(optionValueType->stringId)); default: ASSERT(option->dataType == cfgOptDataTypeString); break; } FUNCTION_TEST_RETURN(VARIANT, varNewStr(optionValueType->string)); } FUNCTION_TEST_RETURN(VARIANT, NULL); } FN_EXTERN bool cfgOptionIdxBool(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN(BOOL, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeBoolean, false)->value.boolean); } FN_EXTERN int cfgOptionIdxInt(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer >= INT_MIN); ASSERT(cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer <= INT_MAX); FUNCTION_LOG_RETURN(INT, (int)cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer); } FN_EXTERN int64_t cfgOptionIdxInt64(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN(INT64, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer); } FN_EXTERN unsigned int cfgOptionIdxUInt(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer >= 0); ASSERT(cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer <= UINT_MAX); FUNCTION_LOG_RETURN(UINT, (unsigned int)cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer); } FN_EXTERN uint64_t cfgOptionIdxUInt64(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer >= 0); FUNCTION_LOG_RETURN(UINT64, (uint64_t)cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer); } FN_EXTERN const KeyValue * cfgOptionIdxKv(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN_CONST(KEY_VALUE, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeHash, false)->value.keyValue); } FN_EXTERN const KeyValue * cfgOptionIdxKvNull(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN_CONST(KEY_VALUE, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeHash, true)->value.keyValue); } FN_EXTERN const VariantList * cfgOptionIdxLst(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); ASSERT(cfgInited()); const VariantList *optionValue = cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeList, true)->value.list; if (optionValue == NULL) { MEM_CONTEXT_BEGIN(configLocal->memContext) { optionValue = varLstNew(); configLocal->option[optionId].index[optionIdx].value.list = optionValue; } MEM_CONTEXT_END(); } FUNCTION_LOG_RETURN_CONST(VARIANT_LIST, optionValue); } FN_EXTERN const String * cfgOptionIdxStr(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN_CONST(STRING, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeString, false)->value.string); } FN_EXTERN const String * cfgOptionIdxStrNull(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN_CONST(STRING, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeString, true)->value.string); } FN_EXTERN StringId cfgOptionIdxStrId(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); ConfigOptionValue *const optionValue = cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeStringId, false); ConfigOptionValueType *const optionValueType = &optionValue->value; if (optionValueType->stringId == 0) { optionValueType->stringId = strIdFromZN( strZ(optionValue->display), strSize(optionValue->display), cfgOptionSequence(optionId) ? optionValue->sequence : STRING_ID_SEQ_NONE); } FUNCTION_LOG_RETURN(STRING_ID, optionValueType->stringId); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionIdxSeq(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); ASSERT(cfgOptionSequence(optionId)); FUNCTION_LOG_RETURN(UINT, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeStringId, false)->sequence); } /**********************************************************************************************************************************/ FN_EXTERN void cfgOptionSet(const ConfigOption optionId, const ConfigSource source, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(ENUM, source); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); cfgOptionIdxSet(optionId, cfgOptionIdxDefault(optionId), source, value); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void cfgOptionIdxSet( const ConfigOption optionId, const unsigned int optionIdx, const ConfigSource source, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_PARAM(ENUM, source); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); // Set the source const ConfigOptionData *const option = &configLocal->option[optionId]; ConfigOptionValue *const optionValue = &option->index[optionIdx]; optionValue->source = source; // Only set value if it is not null ConfigOptionValueType *const optionValueType = &optionValue->value; if (value != NULL) { switch (option->dataType) { case cfgOptDataTypeBoolean: optionValueType->boolean = varBool(value); break; case cfgOptDataTypeInteger: optionValueType->integer = varInt64(value); break; case cfgOptDataTypeString: { if (varType(value) == varTypeString) { MEM_CONTEXT_BEGIN(configLocal->memContext) { optionValueType->string = strDup(varStr(value)); } MEM_CONTEXT_END(); } else { THROW_FMT( AssertError, "option '%s' must be set with String variant", cfgOptionIdxName(optionId, optionIdx)); } break; } case cfgOptDataTypeStringId: { if (varType(value) == varTypeUInt64) { optionValueType->stringId = varUInt64(value); if (option->sequence) optionValue->sequence = (uint8_t)strIdSeq(optionValueType->stringId); } else { ASSERT(!option->sequence); optionValueType->stringId = strStrId(varStr(value)); } break; } default: THROW_FMT(AssertError, "set not available for option data type %u", option->dataType); } optionValue->set = true; } else { optionValue->set = false; // Pointer values need to be set to null since they can be accessed when the option is not set, e.g. cfgOptionStrNull(). // Setting string to NULL suffices to set the other pointers in the union to NULL. optionValueType->string = NULL; } // Set the default when specified if (source == cfgSourceDefault) { if (value != NULL) { MEM_CONTEXT_BEGIN(configLocal->memContext) { optionValue->defaultValue = cfgOptionDisplayVar(value, cfgParseOptionType(optionId)); } MEM_CONTEXT_END(); } else optionValue->defaultValue = NULL; } // Clear the display value, which will be generated when needed optionValue->display = NULL; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN ConfigSource cfgOptionSource(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(ENUM, cfgOptionIdxSource(optionId, cfgOptionIdxDefault(optionId))); } FN_EXTERN ConfigSource cfgOptionIdxSource(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); FUNCTION_TEST_RETURN(ENUM, configLocal->option[optionId].index[optionIdx].source); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgOptionTest(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, cfgOptionIdxTest(optionId, cfgOptionIdxDefault(optionId))); } FN_EXTERN bool cfgOptionIdxTest(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT(!cfgOptionValid(optionId) || ((!group && optionIdx == 0) || (group && optionIdx < indexTotal))); FUNCTION_TEST_RETURN(BOOL, cfgOptionValid(optionId) && configLocal->option[optionId].index[optionIdx].set); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgOptionValid(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); FUNCTION_TEST_RETURN(BOOL, configLocal->option[optionId].valid); } FN_EXTERN void cfgOptionInvalidate(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); configLocal->option[optionId].valid = false; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN const CipherSpec * cfgCipherSpecMainIdx(const unsigned int repoIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_END(); ASSERT(repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo)); // Allocate a slot per repo. Each repo has its own cipher type and passphrase so cipher spec cannot be shared between them. if (configLocal->cipherSpecMain == NULL) { MEM_CONTEXT_BEGIN(configLocal->memContext) { configLocal->cipherSpecMain = memNewPtrArray(cfgOptionGroupIdxTotal(cfgOptGrpRepo)); } MEM_CONTEXT_END(); } // Get cipher spec if it doesn't exist if (configLocal->cipherSpecMain[repoIdx] == NULL) { MEM_CONTEXT_BEGIN(configLocal->memContext) { const CipherType cipherType = cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx); // The digest is unset because the format of the file this pass opens determines it configLocal->cipherSpecMain[repoIdx] = cipherSpecNewP( cipherType, cipherType == cipherTypeNone ? NULL : BUFSTR(cfgOptionIdxStr(cfgOptRepoCipherPass, repoIdx))); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(CIPHER_SPEC, configLocal->cipherSpecMain[repoIdx]); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/config.h0000664000175100017510000003055415265101745017063 0ustar00runnerrunner/*********************************************************************************************************************************** Command and Option Configuration This module serves as a database for the configuration options. The configuration rules reside in config/define.c and config/parse.c sets the command and options and determines which options are valid for a command. ***********************************************************************************************************************************/ #ifndef CONFIG_CONFIG_H #define CONFIG_CONFIG_H #include "common/crypto/spec.h" #include "common/log.h" #include "common/type/stringId.h" #include "common/type/stringList.h" #include "config/config.auto.h" /*********************************************************************************************************************************** Command Role Enum Commands may have multiple processes that work together to implement their functionality. These roles allow each process to know what it is supposed to do. ***********************************************************************************************************************************/ typedef enum { // Called directly by the user. This is the main process of the command that may or may not spawn other command roles. cfgCmdRoleMain = 0, // Async worker that is spawned so the main process can return a result while work continues. An async worker may spawn local or // remote workers. cfgCmdRoleAsync, // Local worker for parallelizing jobs. A local work may spawn a remote worker. cfgCmdRoleLocal, // Remote worker for accessing resources on another host cfgCmdRoleRemote, } ConfigCommandRole; // Constants for the command roles #define CONFIG_COMMAND_ROLE_ASYNC "async" #define CONFIG_COMMAND_ROLE_LOCAL "local" #define CONFIG_COMMAND_ROLE_REMOTE "remote" #define CFG_COMMAND_ROLE_TOTAL 4 /*********************************************************************************************************************************** Lock types ***********************************************************************************************************************************/ typedef enum { lockTypeArchive, lockTypeBackup, lockTypeRestore, lockTypeAll, lockTypeNone, } LockType; /*********************************************************************************************************************************** Command Functions Access the current command and command parameters. ***********************************************************************************************************************************/ // Current command FN_EXTERN ConfigCommand cfgCommand(void); // Current command role (async, local, remote) FN_EXTERN ConfigCommandRole cfgCommandRole(void); // Get command name FN_EXTERN const char *cfgCommandName(void); // Get command:role name FN_EXTERN String *cfgCommandRoleName(void); // Does this command require an immediate lock? FN_EXTERN bool cfgLockRequired(void); // Does the command require a remote lock? FN_EXTERN bool cfgLockRemoteRequired(void); // Lock type required for this command FN_EXTERN LockType cfgLockType(void); // Does this command log to a file? FN_EXTERN bool cfgLogFile(void); // Default log level -- used for log messages that are common to all commands FN_EXTERN LogLevel cfgLogLevelDefault(void); // Command parameters, if any FN_EXTERN const StringList *cfgCommandParam(void); /*********************************************************************************************************************************** Option Group Functions ***********************************************************************************************************************************/ // Format group name for display to the user. Useful for messages that do not show an option name but must use an group name that // the user will recognize. FN_EXTERN const char *cfgOptionGroupName(ConfigOptionGroup groupId, unsigned int groupIdx); // Get the default index for this group, i.e. the index that will be used if a non-indexed function like cfgOptionTest() is called. FN_EXTERN unsigned int cfgOptionGroupIdxDefault(ConfigOptionGroup groupId); // Convert the group index to a key, i.e. the key that was used in the original configuration file, command-line, etc. FN_EXTERN unsigned int cfgOptionGroupIdxToKey(ConfigOptionGroup groupId, unsigned int groupIdx); // Total indexes, 0 if the group is not valid. Will be the total configured indexes, no matter which raw indexes were used during // configuration. e.g., if pg1-path and pg8-path are configured then this function will return 2. FN_EXTERN unsigned int cfgOptionGroupIdxTotal(ConfigOptionGroup groupId); /*********************************************************************************************************************************** Option Functions Access option values, indexes, and determine if an option is valid for the current command. Most functions have a variant that accepts an index, which currently work with non-indexed options (with optionIdx 0) but they may not always do so. ***********************************************************************************************************************************/ // Get default index for a group option FN_EXTERN unsigned int cfgOptionIdxDefault(ConfigOption optionId); // Get boolean config option FN_EXTERN bool cfgOptionIdxBool(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS bool cfgOptionBool(const ConfigOption optionId) { return cfgOptionIdxBool(optionId, cfgOptionIdxDefault(optionId)); } // Get int config option FN_EXTERN int cfgOptionIdxInt(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS int cfgOptionInt(const ConfigOption optionId) { return cfgOptionIdxInt(optionId, cfgOptionIdxDefault(optionId)); } // Get int64 config option FN_EXTERN int64_t cfgOptionIdxInt64(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS int64_t cfgOptionInt64(const ConfigOption optionId) { return cfgOptionIdxInt64(optionId, cfgOptionIdxDefault(optionId)); } // Get kv config option FN_EXTERN const KeyValue *cfgOptionIdxKv(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS const KeyValue * cfgOptionKv(const ConfigOption optionId) { return cfgOptionIdxKv(optionId, cfgOptionIdxDefault(optionId)); } FN_EXTERN const KeyValue *cfgOptionIdxKvNull(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS const KeyValue * cfgOptionKvNull(const ConfigOption optionId) { return cfgOptionIdxKvNull(optionId, cfgOptionIdxDefault(optionId)); } // Get list config option FN_EXTERN const VariantList *cfgOptionIdxLst(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS const VariantList * cfgOptionLst(const ConfigOption optionId) { return cfgOptionIdxLst(optionId, cfgOptionIdxDefault(optionId)); } // Get String config option FN_EXTERN const String *cfgOptionIdxStr(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS const String * cfgOptionStr(const ConfigOption optionId) { return cfgOptionIdxStr(optionId, cfgOptionIdxDefault(optionId)); } FN_EXTERN const String *cfgOptionIdxStrNull(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS const String * cfgOptionStrNull(const ConfigOption optionId) { return cfgOptionIdxStrNull(optionId, cfgOptionIdxDefault(optionId)); } // Get StringId config option FN_EXTERN StringId cfgOptionIdxStrId(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS StringId cfgOptionStrId(const ConfigOption optionId) { return cfgOptionIdxStrId(optionId, cfgOptionIdxDefault(optionId)); } // Get config option sequence FN_EXTERN unsigned int cfgOptionIdxSeq(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS unsigned int cfgOptionSeq(const ConfigOption optionId) { return cfgOptionIdxSeq(optionId, cfgOptionIdxDefault(optionId)); } // Get uint config option FN_EXTERN unsigned int cfgOptionIdxUInt(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS unsigned int cfgOptionUInt(const ConfigOption optionId) { return cfgOptionIdxUInt(optionId, cfgOptionIdxDefault(optionId)); } // Get uint64 config option FN_EXTERN uint64_t cfgOptionIdxUInt64(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS uint64_t cfgOptionUInt64(const ConfigOption optionId) { return cfgOptionIdxUInt64(optionId, cfgOptionIdxDefault(optionId)); } // Format the configuration value for display to the user or passing on a command line. If the value was set by the user via the // command line, config, etc., then that exact value will be displayed. This makes it easier for the user to recognize the value and // saves having to format it into something reasonable, especially for time and size option types. Note that cfgOptTypeHash and // cfgOptTypeList option types are not supported by these functions, but they are generally not displayed to the user as a whole. FN_EXTERN const String *cfgOptionDisplay(const ConfigOption optionId); FN_EXTERN const String *cfgOptionIdxDisplay(const ConfigOption optionId, const unsigned int optionIdx); // Option name by id FN_EXTERN const char *cfgOptionName(ConfigOption optionId); FN_EXTERN const char *cfgOptionIdxName(ConfigOption optionId, unsigned int optionIdx); // Is the option valid for this command? FN_EXTERN bool cfgOptionValid(ConfigOption optionId); // Is the option valid for the command and also has a value? FN_EXTERN bool cfgOptionTest(ConfigOption optionId); FN_EXTERN bool cfgOptionIdxTest(ConfigOption optionId, unsigned int optionIdx); /*********************************************************************************************************************************** Cipher Functions Main cipher spec for a repository, built from the cipher options and cached per repo since the options cannot change while a command runs. This is the top of the chain. The info files are read with it and each of those contains the cipher spec for the files that depend on it, so nothing further down needs to consult the options again. ***********************************************************************************************************************************/ FN_EXTERN const CipherSpec *cfgCipherSpecMainIdx(unsigned int repoIdx); FN_INLINE_ALWAYS const CipherSpec * cfgCipherSpecMain(void) { return cfgCipherSpecMainIdx(cfgOptionIdxDefault(cfgOptRepoCipherType)); } /*********************************************************************************************************************************** Option Source Enum Defines where an option was sourced from. The source is only needed when determining what parameters should be passed to a remote process. ***********************************************************************************************************************************/ typedef enum { cfgSourceDefault, // Default value cfgSourceParam, // Passed as command-line parameter cfgSourceConfig, // From configuration file or environment variable } ConfigSource; /*********************************************************************************************************************************** Load Functions Used primarily by modules that need to manipulate the configuration. These modules include, but are not limited to, config/parse.c, config/load.c. ***********************************************************************************************************************************/ // Was help requested? FN_EXTERN bool cfgCommandHelp(void); FN_EXTERN void cfgCommandSet(ConfigCommand commandId, ConfigCommandRole commandRoleId); // Path to current pgbackrest binary FN_EXTERN const String *cfgBin(void); // Was the option negated? FN_EXTERN bool cfgOptionIdxNegate(ConfigOption optionId, unsigned int optionIdx); // Was the option reset? FN_EXTERN bool cfgOptionIdxReset(ConfigOption optionId, unsigned int optionIdx); // Set config option FN_EXTERN void cfgOptionSet(ConfigOption optionId, ConfigSource source, const Variant *value); FN_EXTERN void cfgOptionIdxSet(ConfigOption optionId, unsigned int optionIdx, ConfigSource source, const Variant *value); // How was the option set (default, param, config)? FN_EXTERN ConfigSource cfgOptionSource(ConfigOption optionId); FN_EXTERN ConfigSource cfgOptionIdxSource(ConfigOption optionId, unsigned int optionIdx); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/config.intern.h0000664000175100017510000002150515265101745020355 0ustar00runnerrunner/*********************************************************************************************************************************** Command and Option Configuration Internals These structures and functions are generally used by modules that create configurations, e.g. config/parse, or modules that manipulate the configuration as a whole, e.g. protocol/helper, in order to communicate with other processes. The general-purpose functions for querying the current configuration are found in config.h. ***********************************************************************************************************************************/ #ifndef CONFIG_CONFIG_INTERN_H #define CONFIG_CONFIG_INTERN_H #include "config/config.h" #include "config/parse.h" /*********************************************************************************************************************************** Configuration data. These structures are not directly user-created or accessible. cfgParseP() creates the structures and uses cfgInit() to load it as the current configuration. Various cfg*() functions provide access. ***********************************************************************************************************************************/ // Group options that are related to allow valid and test checks across all options in the group typedef struct ConfigOptionGroupData { const char *name; // Name bool valid; // Is option group valid for the current command? unsigned int indexTotal; // Total number of indexes with values in option group bool indexDefaultExists; // Is there a default index for non-indexed functions? unsigned int indexDefault; // Default index (usually 0) unsigned int *indexMap; // List of index to key index mappings const String **indexName; // List of index names } ConfigOptionGroupData; // Option data typedef union ConfigOptionValueType { bool boolean; // Boolean int64_t integer; // Integer const KeyValue *keyValue; // KeyValue const VariantList *list; // VariantList const String *string; // String StringId stringId; // StringId } ConfigOptionValueType; typedef struct ConfigOptionValue { bool set; // Is the option set? bool negate; // Is the option negated? bool reset; // Is the option reset? uint8_t sequence; // Sequence, if any unsigned int source; // Where the option came from, i.e. ConfigSource enum const String *display; // Current display value, if any. Used for messages, etc. const String *defaultValue; // Default value ConfigOptionValueType value; // Option value } ConfigOptionValue; typedef struct ConfigOptionData { const char *name; // Name bool valid; // Is option valid for current command? bool group; // In a group? bool sequence; // Is there a sequence? unsigned int groupId; // Id if in a group ConfigOptionDataType dataType; // Underlying data type ConfigOptionValue *index; // List of indexed values (only 1 unless the option is indexed) const String **indexName; // Index names (e.g. repo1-path, repo2-path) } ConfigOptionData; typedef struct Config { MemContext *memContext; // Mem context for config data // Generally set by the command parser but can also be set during execute to change commands, i.e. backup -> expire ConfigCommand command; // Current command ConfigCommandRole commandRole; // Current command role String *bin; // Path to current pgbackrest binary bool help; // Was help requested for the command? bool lockRequired; // Is an immediate lock required? bool lockRemoteRequired; // Is a lock required on the remote? LockType lockType; // Lock type required bool logFile; // Will the command log to a file? LogLevel logLevelDefault; // Default log level StringList *paramList; // Parameters passed to the command (if any) ConfigOptionGroupData optionGroup[CFG_OPTION_GROUP_TOTAL]; // Option group data ConfigOptionData option[CFG_OPTION_TOTAL]; // Option data CipherSpec **cipherSpecMain; // Main cipher spec for the cipher options by repo index } Config; /*********************************************************************************************************************************** Init Function ***********************************************************************************************************************************/ // Init with new configuration FN_EXTERN void cfgInit(Config *config); // Has the configuration been initialized? FN_EXTERN bool cfgInited(void); /*********************************************************************************************************************************** Command Functions ***********************************************************************************************************************************/ // List of retry intervals for local jobs. The interval for the first retry will always be 0. NULL if there are no retries. FN_EXTERN VariantList *cfgCommandJobRetry(void); /*********************************************************************************************************************************** Option Group Functions ***********************************************************************************************************************************/ // Is the option in a group? FN_EXTERN bool cfgOptionGroup(ConfigOption optionId); // Group id if the option is in a group FN_EXTERN unsigned int cfgOptionGroupId(ConfigOption optionId); /*********************************************************************************************************************************** Option Functions ***********************************************************************************************************************************/ // Option default - should only be called by the help command FN_EXTERN const String *cfgOptionIdxDefaultValue(ConfigOption optionId, unsigned int optionIdx); // Format a variant for display using the supplied option type. cfgOptionDisplay()/cfgOptionIdxDisplay() should be used whenever // possible, but sometimes the variant needs to be manipulated before being formatted. FN_EXTERN const String *cfgOptionDisplayVar(const Variant *const value, const ConfigOptionType optionType); // Convert the key used in the original configuration to a group index. This is used when an option key must be translated into the // local group index, e.g. during parsing or when getting the value of specific options from a remote. FN_EXTERN unsigned int cfgOptionKeyToIdx(ConfigOption optionId, unsigned int key); // Total indexes for the option if in a group, 1 otherwise. FN_EXTERN unsigned int cfgOptionIdxTotal(ConfigOption optionId); // Get config option as a Variant FN_EXTERN Variant *cfgOptionIdxVar(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS Variant * cfgOptionVar(const ConfigOption optionId) { return cfgOptionIdxVar(optionId, cfgOptionIdxDefault(optionId)); } // Invalidate an option so it will not be passed to other processes. This is used to manage deprecated options that have a newer // option that should be used when possible, e.g. compress and compress-type. FN_EXTERN void cfgOptionInvalidate(ConfigOption optionId); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/exec.c0000664000175100017510000001526615265101745016540 0ustar00runnerrunner/*********************************************************************************************************************************** Exec Configuration ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/log.h" #include "config/config.intern.h" #include "config/exec.h" #include "config/parse.h" /**********************************************************************************************************************************/ FN_EXTERN StringList * cfgExecParam( const ConfigCommand commandId, const ConfigCommandRole commandRoleId, const KeyValue *const optionReplace, const bool local, const bool quote) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, commandId); FUNCTION_LOG_PARAM(ENUM, commandRoleId); FUNCTION_LOG_PARAM(KEY_VALUE, optionReplace); FUNCTION_LOG_PARAM(BOOL, local); // Will the new process be running on the same host? FUNCTION_LOG_PARAM(BOOL, quote); // Do parameters with spaces need to be quoted? FUNCTION_LOG_END(); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Loop though options and add the ones that apply to the specified command for (ConfigOption optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++) { // Skip the option if it is not valid for the original/specified command or if is secure. Also skip repo1-cipher-type // because there's no point of passing it if the other process doesn't have access to repo1-cipher-pass. There is // probably a better way to do this last part... if (!cfgParseOptionValid(commandId, commandRoleId, optionId) || cfgParseOptionSecure(optionId) || optionId == cfgOptRepoCipherType) { continue; } // Loop through option indexes. The group is read from the parse rules rather than the config because an option that // is valid for the specified command may not be valid for the command being run, and in that case the config has no // group data for it. const unsigned int optionIdxTotal = cfgParseOptionGroup(optionId) ? cfgOptionGroupIdxTotal(cfgParseOptionGroupId(optionId)) : 1; for (unsigned int optionIdx = 0; optionIdx < optionIdxTotal; optionIdx++) { // First check for a replacement const Variant *const key = VARSTRZ(cfgOptionIdxName(optionId, optionIdx)); const Variant *value = NULL; bool exists = false; // If an option is requested to be replaced (usually because remote processes do not have access to the config) // then if the option exists, get the new value for replacement if (optionReplace != NULL) { exists = kvKeyExists(optionReplace, key); if (exists) value = kvGet(optionReplace, key); } // If the key exists but its value is NULL then skip this option if (exists && value == NULL) continue; // If no replacement then see if this option is not default if (value == NULL && cfgOptionValid(optionId)) { if (cfgOptionIdxNegate(optionId, optionIdx)) value = BOOL_FALSE_VAR; else if (cfgOptionIdxSource(optionId, optionIdx) != cfgSourceDefault) value = cfgOptionIdxVar(optionId, optionIdx); } // If the option was reset if (value == NULL && cfgOptionValid(optionId) && cfgOptionIdxReset(optionId, optionIdx)) { strLstAddFmt(result, "--reset-%s", cfgOptionIdxName(optionId, optionIdx)); } // Else format the value if found, even if the option is not valid for the command else if (value != NULL && (!local || exists || cfgOptionIdxSource(optionId, optionIdx) == cfgSourceParam)) { if (varType(value) == varTypeBool) strLstAddFmt(result, "--%s%s", varBool(value) ? "" : "no-", cfgOptionIdxName(optionId, optionIdx)); else { StringList *valueList = NULL; if (varType(value) == varTypeKeyValue) { valueList = strLstNew(); const KeyValue *const optionKv = varKv(value); const VariantList *const keyList = kvKeyList(optionKv); for (unsigned int keyIdx = 0; keyIdx < varLstSize(keyList); keyIdx++) { strLstAddFmt( valueList, "%s=%s", strZ(varStr(varLstGet(keyList, keyIdx))), strZ(varStrForce(kvGet(optionKv, varLstGet(keyList, keyIdx))))); } } else if (varType(value) == varTypeVariantList) { valueList = strLstNewVarLst(varVarLst(value)); } // Else only one value else { valueList = strLstNew(); strLstAdd( valueList, exists ? cfgOptionDisplayVar(value, cfgParseOptionType(optionId)) : cfgOptionIdxDisplay(optionId, optionIdx)); } // Output options and values for (unsigned int valueListIdx = 0; valueListIdx < strLstSize(valueList); valueListIdx++) { const String *value = strLstGet(valueList, valueListIdx); if (quote && strchr(strZ(value), ' ') != NULL) value = strNewFmt("\"%s\"", strZ(value)); strLstAddFmt(result, "--%s=%s", cfgOptionIdxName(optionId, optionIdx), strZ(value)); } } } } } // Add the command strLstAdd(result, cfgParseCommandRoleName(commandId, commandRoleId)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/exec.h0000664000175100017510000000240615265101745016535 0ustar00runnerrunner/*********************************************************************************************************************************** Exec Configuration ***********************************************************************************************************************************/ #ifndef CONFIG_EXEC_H #define CONFIG_EXEC_H #include "common/type/stringList.h" #include "config/config.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Generate a list of options required for execution of a new command, overriding options with the values in optionReplace when // present. If local is set then the new command must have access to the local configuration files and environment since only // options originally passed on the command-line will be added to the list. Use quote when the options will be used as a // concatenated string rather than being passed directly to exec*() as a list. FN_EXTERN StringList *cfgExecParam( ConfigCommand commandId, ConfigCommandRole commandRoleId, const KeyValue *optionReplace, bool local, bool quote); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/load.c0000664000175100017510000006666315265101745016542 0ustar00runnerrunner/*********************************************************************************************************************************** Configuration Load ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/command.h" #include "command/lock.h" #include "common/crypto/common.h" #include "common/debug.h" #include "common/io/io.h" #include "common/io/socket/common.h" #include "common/io/tls/common.h" #include "common/log.h" #include "common/memContext.h" #include "common/user.h" #include "config/config.intern.h" #include "config/load.h" #include "config/parse.h" #include "info/infoBackup.h" #include "storage/cifs/storage.h" #include "storage/helper.h" #include "storage/posix/storage.h" #include "storage/sftp/storage.h" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct ConfigLoadLocal { unsigned int argListSize; // Argument list size const char **argList; // Argument list } configLoadLocal; /*********************************************************************************************************************************** Load log settings ***********************************************************************************************************************************/ static void cfgLoadLogSetting(void) { FUNCTION_LOG_VOID(logLevelTrace); // Initialize logging LogLevel logLevelConsole = logLevelOff; LogLevel logLevelStdErr = logLevelOff; LogLevel logLevelFile = logLevelOff; bool logTimestamp = true; unsigned int logProcessMax = 1; if (cfgOptionValid(cfgOptLogLevelConsole)) logLevelConsole = logLevelEnum(cfgOptionSeq(cfgOptLogLevelConsole)); if (cfgOptionValid(cfgOptLogLevelStderr)) logLevelStdErr = logLevelEnum(cfgOptionSeq(cfgOptLogLevelStderr)); if (cfgOptionValid(cfgOptLogLevelFile)) logLevelFile = logLevelEnum(cfgOptionSeq(cfgOptLogLevelFile)); if (cfgOptionValid(cfgOptLogTimestamp)) logTimestamp = cfgOptionBool(cfgOptLogTimestamp); if (cfgOptionValid(cfgOptProcessMax)) logProcessMax = cfgOptionUInt(cfgOptProcessMax); logInit( logLevelConsole, logLevelStdErr, logLevelFile, logTimestamp, 0, logProcessMax, cfgOptionValid(cfgOptDryRun) && cfgOptionBool(cfgOptDryRun)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cfgLoadUpdateOption(void) { FUNCTION_LOG_VOID(logLevelTrace); FUNCTION_AUDIT_HELPER(); // Make sure the repo option is set for the stanza-delete command when more than one repo is configured or the first configured // repo is not key 1. if (!cfgCommandHelp() && cfgOptionValid(cfgOptRepo) && cfgCommand() == cfgCmdStanzaDelete && !cfgOptionTest(cfgOptRepo) && (cfgOptionGroupIdxTotal(cfgOptGrpRepo) > 1 || cfgOptionGroupIdxToKey(cfgOptGrpRepo, 0) != 1)) { THROW_FMT( OptionRequiredError, "%s command requires option: " CFGOPT_REPO "\n" "HINT: this command requires a specific repository to operate on", cfgCommandName()); } // If there is more than one repo configured if (cfgOptionGroupIdxTotal(cfgOptGrpRepo) > 1) { for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); optionIdx++) { // If the repo is local and either posix or cifs if (!cfgOptionIdxTest(cfgOptRepoHost, optionIdx) && (cfgOptionIdxStrId(cfgOptRepoType, optionIdx) == STORAGE_POSIX_TYPE || cfgOptionIdxStrId(cfgOptRepoType, optionIdx) == STORAGE_CIFS_TYPE)) { // Ensure a local repo does not have the same path as another local repo of the same type for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { if (optionIdx != repoIdx && !(cfgOptionIdxTest(cfgOptRepoHost, repoIdx)) && cfgOptionIdxStrId(cfgOptRepoType, optionIdx) == cfgOptionIdxStrId(cfgOptRepoType, repoIdx) && strEq(cfgOptionIdxStr(cfgOptRepoPath, optionIdx), cfgOptionIdxStr(cfgOptRepoPath, repoIdx))) { THROW_FMT( OptionInvalidValueError, "local %s and %s paths are both '%s' but must be different", cfgOptionGroupName(cfgOptGrpRepo, optionIdx), cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(cfgOptionIdxDisplay(cfgOptRepoPath, repoIdx))); } } } } } // Protocol timeout should be greater than db timeout if (cfgOptionTest(cfgOptDbTimeout) && cfgOptionTest(cfgOptProtocolTimeout) && cfgOptionInt64(cfgOptProtocolTimeout) <= cfgOptionInt64(cfgOptDbTimeout)) { // If protocol-timeout is default then increase it to be greater than db-timeout if (cfgOptionSource(cfgOptProtocolTimeout) == cfgSourceDefault) { cfgOptionSet( cfgOptProtocolTimeout, cfgSourceDefault, VARINT64(cfgOptionInt64(cfgOptDbTimeout) + (int64_t)(30 * MSEC_PER_SEC))); } else if (cfgOptionSource(cfgOptDbTimeout) == cfgSourceDefault) { const int64_t dbTimeout = cfgOptionInt64(cfgOptProtocolTimeout) - (int64_t)(30 * MSEC_PER_SEC); // Normally the protocol time will be greater than 45 seconds so db timeout can be at least 15 seconds if (dbTimeout >= (int64_t)(15 * MSEC_PER_SEC)) { cfgOptionSet(cfgOptDbTimeout, cfgSourceDefault, VARINT64(dbTimeout)); } // But in some test cases the protocol timeout will be very small so make db timeout half of protocol timeout else cfgOptionSet(cfgOptDbTimeout, cfgSourceDefault, VARINT64(cfgOptionInt64(cfgOptProtocolTimeout) / 2)); } else { THROW_FMT( OptionInvalidValueError, "'%s' is not valid for '" CFGOPT_PROTOCOL_TIMEOUT "' option\nHINT '" CFGOPT_PROTOCOL_TIMEOUT "' option (%s)" " should be greater than '" CFGOPT_DB_TIMEOUT "' option (%s).", strZ(cfgOptionDisplay(cfgOptProtocolTimeout)), strZ(cfgOptionDisplay(cfgOptProtocolTimeout)), strZ(cfgOptionDisplay(cfgOptDbTimeout))); } } // Make sure that repo and pg host settings are not both set - cannot both be remote if (cfgOptionValid(cfgOptPgHost) && cfgOptionValid(cfgOptRepoHost)) { bool pgHostFound = false; for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(cfgOptGrpPg); optionIdx++) { if (cfgOptionIdxTest(cfgOptPgHost, optionIdx)) { pgHostFound = true; break; } } // If a pg-host was found, see if a repo-host is configured if (pgHostFound == true) { for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); optionIdx++) { if (cfgOptionIdxTest(cfgOptRepoHost, optionIdx)) THROW_FMT(ConfigError, "pg and repo hosts cannot both be configured as remote"); } } } // Warn when repo-retention-full is not set on a configured repo if (!cfgCommandHelp() && cfgOptionValid(cfgOptRepoRetentionFullType)) { for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); optionIdx++) { if (!(cfgOptionIdxTest(cfgOptRepoRetentionFull, optionIdx))) { LOG_WARN_FMT( "option '%s' is not set for '%s=%s', the repository may run out of space" "\nHINT: to retain full backups indefinitely (without warning), set option '%s' to the maximum.", cfgOptionIdxName(cfgOptRepoRetentionFull, optionIdx), cfgOptionIdxName(cfgOptRepoRetentionFullType, optionIdx), strZ(cfgOptionIdxDisplay(cfgOptRepoRetentionFullType, optionIdx)), cfgOptionIdxName(cfgOptRepoRetentionFull, optionIdx)); } } } // If archive retention is valid for the command, then set archive settings if (cfgOptionValid(cfgOptRepoRetentionArchive)) { // For each possible repo, check and adjust the settings as appropriate for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); optionIdx++) { const BackupType archiveRetentionType = (BackupType)cfgOptionIdxStrId(cfgOptRepoRetentionArchiveType, optionIdx); // If the archive retention is not explicitly set then determine what it should be defaulted to if (!cfgOptionIdxTest(cfgOptRepoRetentionArchive, optionIdx)) { // If repo-retention-archive-type is default (full), then if repo-retention-full is set, set the // repo-retention-archive to this value when retention-full-type is 'count', else ignore archiving. If // retention-full-type is 'time' then the expire command will default the archive retention accordingly. const String *const msgArchiveOff = strNewFmt( "WAL segments will not be expired: option '%s=%s' but", cfgOptionIdxName(cfgOptRepoRetentionArchiveType, optionIdx), zNewStrId(archiveRetentionType)); switch (archiveRetentionType) { case backupTypeFull: { if (cfgOptionIdxSeq(cfgOptRepoRetentionFullType, optionIdx) == CFGOPTVAL_REPO_RETENTION_FULL_TYPE_COUNT && cfgOptionIdxTest(cfgOptRepoRetentionFull, optionIdx)) { cfgOptionIdxSet( cfgOptRepoRetentionArchive, optionIdx, cfgSourceDefault, VARINT64(cfgOptionIdxInt64(cfgOptRepoRetentionFull, optionIdx))); } break; } case backupTypeDiff: { // if repo-retention-diff is set then user must have set it if (cfgOptionIdxTest(cfgOptRepoRetentionDiff, optionIdx)) { cfgOptionIdxSet( cfgOptRepoRetentionArchive, optionIdx, cfgSourceDefault, VARINT64(cfgOptionIdxInt64(cfgOptRepoRetentionDiff, optionIdx))); } else { LOG_WARN_FMT( "%s neither option '%s' nor option '%s' is set", strZ(msgArchiveOff), cfgOptionIdxName(cfgOptRepoRetentionArchive, optionIdx), cfgOptionIdxName(cfgOptRepoRetentionDiff, optionIdx)); } break; } case backupTypeIncr: LOG_WARN_FMT( "%s option '%s' is not set", strZ(msgArchiveOff), cfgOptionIdxName(cfgOptRepoRetentionArchive, optionIdx)); break; } } else { // If repo-retention-archive is set then check repo-retention-archive-type and issue a warning if the // corresponding setting is UNDEF since UNDEF means backups will not be expired but they should be in the // practice of setting this value even though expiring the archive itself is OK and will be performed. if (archiveRetentionType == backupTypeDiff && !cfgOptionIdxTest(cfgOptRepoRetentionDiff, optionIdx)) { LOG_WARN_FMT( "option '%s' is not set for '%s=" CFGOPTVAL_TYPE_DIFF_Z "'\n" "HINT: to retain differential backups indefinitely (without warning), set option '%s' to the maximum.", cfgOptionIdxName(cfgOptRepoRetentionDiff, optionIdx), cfgOptionIdxName(cfgOptRepoRetentionArchiveType, optionIdx), cfgOptionIdxName(cfgOptRepoRetentionDiff, optionIdx)); } } } } // For each possible repo, error if an S3 bucket name contains dots when using host style URIs for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { if (cfgOptionIdxTest(cfgOptRepoS3Bucket, repoIdx) && cfgOptionIdxBool(cfgOptRepoStorageVerifyTls, repoIdx) && cfgOptionIdxSeq(cfgOptRepoS3UriStyle, repoIdx) == CFGOPTVAL_REPO_S3_URI_STYLE_HOST && strChr(cfgOptionIdxStr(cfgOptRepoS3Bucket, repoIdx), '.') != -1) { THROW_FMT( OptionInvalidValueError, "'%s' is not valid for option '%s'\n" "HINT: RFC-2818 forbids dots in wildcard matches.\n" "HINT: TLS/SSL verification cannot proceed with this bucket name.\n" "HINT: remove dots from the bucket name.", strZ(cfgOptionIdxDisplay(cfgOptRepoS3Bucket, repoIdx)), cfgOptionIdxName(cfgOptRepoS3Bucket, repoIdx)); } } // Check/update compress-type if compress is valid. There should be no references to the compress option outside this block. if (cfgOptionValid(cfgOptCompress)) { if (cfgOptionSource(cfgOptCompress) != cfgSourceDefault) { if (cfgOptionSource(cfgOptCompressType) != cfgSourceDefault) { LOG_WARN( "'" CFGOPT_COMPRESS "' and '" CFGOPT_COMPRESS_TYPE "' options should not both be set\n" "HINT: '" CFGOPT_COMPRESS_TYPE "' is preferred and '" CFGOPT_COMPRESS "' is deprecated."); } // Set compress-type to none. Eventually the compress option will be deprecated and removed so this reduces code churn // when that happens. if (!cfgOptionBool(cfgOptCompress) && cfgOptionSource(cfgOptCompressType) == cfgSourceDefault) { cfgOptionSet(cfgOptCompressType, cfgSourceParam, VARUINT64(CFGOPTVAL_COMPRESS_TYPE_NONE)); cfgOptionSet(cfgOptCompressLevel, cfgSourceDefault, VARINT64(0)); } } // Now invalidate compress so it can't be used and won't be passed to child processes cfgOptionInvalidate(cfgOptCompress); cfgOptionSet(cfgOptCompress, cfgSourceDefault, NULL); } // Error if repo-sftp--host-key-check-type is explicitly set to anything other than fingerprint and repo-sftp-host-fingerprint // is also specified. For backward compatibility we need to allow repo-sftp-host-fingerprint when // repo-sftp-host-key-check-type defaults to yes, but emit a warning to let the user know to change the configuration. Also // set repo-sftp-host-key-check-type=fingerprint so other code does not need to know about this exception. for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { if (cfgOptionIdxTest(cfgOptRepoSftpHostKeyCheckType, repoIdx)) { if (cfgOptionIdxTest(cfgOptRepoSftpHostFingerprint, repoIdx)) { if (cfgOptionIdxSource(cfgOptRepoSftpHostKeyCheckType, repoIdx) == cfgSourceDefault) { LOG_WARN_FMT( "option '%s' without option '%s' = '" CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT_Z "' is" " deprecated\n" "HINT: set option '%s=" CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT_Z "'", cfgOptionIdxName(cfgOptRepoSftpHostFingerprint, repoIdx), cfgOptionIdxName(cfgOptRepoSftpHostKeyCheckType, repoIdx), cfgOptionIdxName(cfgOptRepoSftpHostKeyCheckType, repoIdx)); cfgOptionIdxSet( cfgOptRepoSftpHostKeyCheckType, repoIdx, cfgSourceDefault, VARSTRZ(CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT_Z)); } else { if (cfgOptionIdxStrId(cfgOptRepoSftpHostKeyCheckType, repoIdx) != CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT) { THROW_FMT( OptionInvalidError, "option '%s' not valid without option '%s' = '" CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT_Z "'", cfgOptionIdxName(cfgOptRepoSftpHostFingerprint, repoIdx), cfgOptionIdxName(cfgOptRepoSftpHostKeyCheckType, repoIdx)); } } } else { if (cfgOptionIdxStrId(cfgOptRepoSftpHostKeyCheckType, repoIdx) == CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT) { THROW_FMT( OptionRequiredError, "%s command requires option: %s", cfgCommandName(), cfgOptionIdxName(cfgOptRepoSftpHostFingerprint, repoIdx)); } } } } // A repo must be specified when targeting time. Not all repo types support versioning so rather than try to skip repos in that // case it seems to be easier to just target a specific repo. Also, depending on the type of corruption, different repos might // require different target times. if (cfgOptionTest(cfgOptRepoTargetTime) && cfgOptionSource(cfgOptRepo) == cfgSourceDefault) THROW_FMT(OptionInvalidError, "option '" CFGOPT_REPO_TARGET_TIME "' not valid without option '" CFGOPT_REPO "'"); // The stop-auto option is deprecated if (cfgOptionValid(cfgOptStopAuto) && cfgOptionSource(cfgOptStopAuto) != cfgSourceDefault) { LOG_WARN( "option '" CFGOPT_STOP_AUTO "' is deprecated\n" "HINT: all supported versions use non-exclusive backup.\n" "HINT: stop using this option to avoid an error when it is removed."); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * cfgLoadLogFileName(const ConfigCommandRole commandRole) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, commandRole); FUNCTION_LOG_END(); // Construct log filename prefix String *const result = strCatFmt( strNew(), "%s/%s-%s", strZ(cfgOptionStr(cfgOptLogPath)), cfgOptionTest(cfgOptStanza) ? strZ(cfgOptionStr(cfgOptStanza)) : "all", cfgCommandName()); // ??? Append async for local/remote archive async commands. It would be good to find a more generic way to do this in case the // async role is added to more commands. if (commandRole == cfgCmdRoleLocal || commandRole == cfgCmdRoleRemote) { if (cfgOptionValid(cfgOptArchiveAsync) && cfgOptionBool(cfgOptArchiveAsync)) strCatZ(result, "-async"); } // Add command role if it is not main if (commandRole != cfgCmdRoleMain) strCatFmt(result, "-%s", strZ(cfgParseCommandRoleStr(commandRole))); // Add process id if local or remote role if (commandRole == cfgCmdRoleLocal || commandRole == cfgCmdRoleRemote) strCatFmt(result, "-%03u", cfgOptionUInt(cfgOptProcess)); // Add extension strCatZ(result, ".log"); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void cfgLoadLogFile(void) { if (cfgLogFile() && !cfgCommandHelp()) { MEM_CONTEXT_TEMP_BEGIN() { // Attempt to open log file if (!logFileSet(strZ(cfgLoadLogFileName(cfgCommandRole())))) cfgOptionSet(cfgOptLogLevelFile, cfgSourceParam, VARUINT64(CFGOPTVAL_LOG_LEVEL_CONSOLE_OFF_STRID)); } MEM_CONTEXT_TEMP_END(); } } /**********************************************************************************************************************************/ FN_EXTERN void cfgLoad(const unsigned int argListSize, const char *argList[]) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, argListSize); FUNCTION_LOG_PARAM(CHARPY, argList); FUNCTION_LOG_END(); ASSERT(argListSize > 0); ASSERT(argList != NULL); // Store arguments configLoadLocal.argListSize = argListSize; configLoadLocal.argList = argList; MEM_CONTEXT_TEMP_BEGIN() { // Parse config from command line and config file cfgParseP(storageLocal(), configLoadLocal.argListSize, configLoadLocal.argList); // Initialize dry-run mode for storage when valid for the current command storageHelperDryRunInit(cfgOptionValid(cfgOptDryRun) && cfgOptionBool(cfgOptDryRun)); // Load the log settings cfgLoadLogSetting(); // Error when running as the root user unless it is explicitly allowed. Running as root can create files owned by root that // the regular user is then unable to access, which is a common cause of hard-to-diagnose failures. The allow-root option // defaults to true only for the restore command, which is designed to run as root and manages file ownership carefully. The // check applies to all command roles so a non-root command cannot run as root on a remote. It is skipped for help/version // (where the option is not valid) since those may reasonably be run as root. if (!cfgCommandHelp() && cfgOptionValid(cfgOptAllowRoot) && !cfgOptionBool(cfgOptAllowRoot)) { userInit(); if (userRoot()) { THROW_FMT( OptionInvalidError, "the '%s' command must not be run as the root user\n" "HINT: running as root can create files that the regular user cannot access, causing later commands to fail.\n" "HINT: set the '" CFGOPT_ALLOW_ROOT "' option to allow running this command as root.", cfgCommandName()); } } // Neutralize the umask to make the repository file/path modes more consistent if (cfgOptionValid(cfgOptNeutralUmask) && cfgOptionBool(cfgOptNeutralUmask)) umask(0000); if (cfgOptionValid(cfgOptPriority) && cfgOptionSource(cfgOptPriority) != cfgSourceDefault) { // Sign conversion is ignored here due to type conflicts between setpriority() and getpid() on different platforms. // Linux returns _pid_t (int) from getpid() but accepts id_t (unsigned int) in setpriority(). FreeBSD (but not MacOS) // uses int for both. Presumably implicit conversion will work appropriately on each platform so just let that happen. #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wsign-conversion" THROW_ON_SYS_ERROR( setpriority(PRIO_PROCESS, getpid(), cfgOptionInt(cfgOptPriority)) == -1, KernelError, "unable to set process priority"); #pragma GCC diagnostic pop } // Initialize TCP settings if (cfgOptionValid(cfgOptSckKeepAlive)) { sckInit( cfgOptionBool(cfgOptSckBlock), cfgOptionBool(cfgOptSckKeepAlive), cfgOptionTest(cfgOptTcpKeepAliveCount) ? cfgOptionInt(cfgOptTcpKeepAliveCount) : 0, cfgOptionTest(cfgOptTcpKeepAliveIdle) ? cfgOptionInt(cfgOptTcpKeepAliveIdle) : 0, cfgOptionTest(cfgOptTcpKeepAliveInterval) ? cfgOptionInt(cfgOptTcpKeepAliveInterval) : 0); } // Initialize TLS settings if (cfgOptionValid(cfgOptTlsCipher12)) { ASSERT(cfgOptionValid(cfgOptTlsCipher13)); tlsInit(cfgOptionStrNull(cfgOptTlsCipher12), cfgOptionStrNull(cfgOptTlsCipher13)); } // Set IO buffer size (use the default for help to lower memory usage) if (cfgOptionValid(cfgOptBufferSize) && !cfgCommandHelp()) ioBufferSizeSet(cfgOptionUInt(cfgOptBufferSize)); // Set IO timeout if (cfgOptionValid(cfgOptIoTimeout)) ioTimeoutMsSet(cfgOptionUInt64(cfgOptIoTimeout)); // Open the log file if this command logs to a file cfgLoadLogFile(); // Create the exec-id used to identify all locals and remotes spawned by this process. This allows lock contention to be // easily resolved and makes it easier to associate processes from various logs. if (cfgOptionValid(cfgOptExecId) && !cfgOptionTest(cfgOptExecId)) { // Generate some random bytes uint32_t execRandom; cryptoRandomBytes((uint8_t *)&execRandom, sizeof(execRandom)); // Format a string with the pid and the random bytes to serve as the exec id cfgOptionSet(cfgOptExecId, cfgSourceParam, VARSTR(strNewFmt("%d-%08x", getpid(), execRandom))); } // Begin the command cmdBegin(); // Initialize the lock module if (cfgOptionTest(cfgOptLockPath)) lockInit(cfgOptionStr(cfgOptLockPath), cfgOptionStr(cfgOptExecId)); // Acquire a lock if this command requires a lock if (cfgLockType() != lockTypeNone && !cfgCommandHelp() && cfgLockRequired()) cmdLockAcquireP(); // Update options that have complex rules cfgLoadUpdateOption(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cfgLoadStanza(const String *const stanza) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, stanza); FUNCTION_LOG_END(); ASSERT(stanza != NULL); ASSERT(configLoadLocal.argListSize > 0); MEM_CONTEXT_TEMP_BEGIN() { // Store the exec id so it can be preserved after reload const Variant *const execId = varNewStr(cfgOptionStr(cfgOptExecId)); // Make a copy of the arguments and add the stanza (this assumes the stanza was not originally specified) StringList *const argListNew = strLstNew(); for (unsigned int argListIdx = 0; argListIdx < configLoadLocal.argListSize; argListIdx++) strLstAddZ(argListNew, configLoadLocal.argList[argListIdx]); strLstAddFmt(argListNew, "--" CFGOPT_STANZA "=%s", strZ(stanza)); // Parse config from command line and config file cfgParseP(storageLocal(), strLstSize(argListNew), strLstPtr(argListNew), .noConfigLoad = true, .noResetLogLevel = true); // Update options that have complex rules cfgLoadUpdateOption(); // Set execId to prior value cfgOptionSet(cfgOptExecId, cfgSourceParam, execId); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/load.h0000664000175100017510000000237515265101745016535 0ustar00runnerrunner/*********************************************************************************************************************************** Configuration Load ***********************************************************************************************************************************/ #ifndef CONFIG_LOAD_H #define CONFIG_LOAD_H #include #include "config/config.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Load the configuration FN_EXTERN void cfgLoad(unsigned int argListSize, const char *argList[]); // Load the configuration using the specified stanza FN_EXTERN void cfgLoadStanza(const String *stanza); // Generate log file path and name. Only the command role is configurable here because log settings may vary between commands. FN_EXTERN String *cfgLoadLogFileName(ConfigCommandRole commandRole); // Attempt to set the log file and turn file logging off if the file cannot be opened FN_EXTERN void cfgLoadLogFile(void); // Update options that have complex rules FN_EXTERN void cfgLoadUpdateOption(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/parse.auto.c.inc0000664000175100017510000616036415265101745020453 0ustar00runnerrunner/*********************************************************************************************************************************** Config Parse Rules Automatically generated by 'build.py config' -- do not modify directly. ***********************************************************************************************************************************/ /*********************************************************************************************************************************** Rule Strings ***********************************************************************************************************************************/ #define PARSE_RULE_VAL_STR(value) PARSE_RULE_U32_2(parseRuleValStr##value) #define PARSE_RULE_VAL_STRID(value) PARSE_RULE_VAL_STR(QT_##value##_QT) static const StringPubConst parseRuleValueStr[] = { PARSE_RULE_STRPUB("-1"), // val/str PARSE_RULE_STRPUB("-20"), // val/str PARSE_RULE_STRPUB("-5"), // val/str PARSE_RULE_STRPUB("-7"), // val/str PARSE_RULE_STRPUB("/tmp/pgbackrest"), // val/str PARSE_RULE_STRPUB("/var/lib/pgbackrest"), // val/str PARSE_RULE_STRPUB("/var/log/pgbackrest"), // val/str PARSE_RULE_STRPUB("/var/spool/pgbackrest"), // val/str PARSE_RULE_STRPUB("0"), // val/str PARSE_RULE_STRPUB("0B"), // val/str PARSE_RULE_STRPUB("0s"), // val/str PARSE_RULE_STRPUB("1"), // val/str PARSE_RULE_STRPUB("100ms"), // val/str PARSE_RULE_STRPUB("1024"), // val/str PARSE_RULE_STRPUB("12"), // val/str PARSE_RULE_STRPUB("128KiB"), // val/str PARSE_RULE_STRPUB("128MiB"), // val/str PARSE_RULE_STRPUB("15d"), // val/str PARSE_RULE_STRPUB("15m"), // val/str PARSE_RULE_STRPUB("15s"), // val/str PARSE_RULE_STRPUB("16GiB"), // val/str PARSE_RULE_STRPUB("16KiB"), // val/str PARSE_RULE_STRPUB("16MiB"), // val/str PARSE_RULE_STRPUB("19"), // val/str PARSE_RULE_STRPUB("1B"), // val/str PARSE_RULE_STRPUB("1GiB"), // val/str PARSE_RULE_STRPUB("1MiB"), // val/str PARSE_RULE_STRPUB("1PiB"), // val/str PARSE_RULE_STRPUB("1TiB"), // val/str PARSE_RULE_STRPUB("1d"), // val/str PARSE_RULE_STRPUB("1h"), // val/str PARSE_RULE_STRPUB("1m"), // val/str PARSE_RULE_STRPUB("2"), // val/str PARSE_RULE_STRPUB("20MiB"), // val/str PARSE_RULE_STRPUB("22"), // val/str PARSE_RULE_STRPUB("256"), // val/str PARSE_RULE_STRPUB("256KiB"), // val/str PARSE_RULE_STRPUB("2MiB"), // val/str PARSE_RULE_STRPUB("3"), // val/str PARSE_RULE_STRPUB("30m"), // val/str PARSE_RULE_STRPUB("31m"), // val/str PARSE_RULE_STRPUB("32"), // val/str PARSE_RULE_STRPUB("32KiB"), // val/str PARSE_RULE_STRPUB("360"), // val/str PARSE_RULE_STRPUB("3600"), // val/str PARSE_RULE_STRPUB("443"), // val/str PARSE_RULE_STRPUB("4MiB"), // val/str PARSE_RULE_STRPUB("4PiB"), // val/str PARSE_RULE_STRPUB("5"), // val/str PARSE_RULE_STRPUB("512KiB"), // val/str PARSE_RULE_STRPUB("5432"), // val/str PARSE_RULE_STRPUB("5MiB"), // val/str PARSE_RULE_STRPUB("6"), // val/str PARSE_RULE_STRPUB("64KiB"), // val/str PARSE_RULE_STRPUB("65535"), // val/str PARSE_RULE_STRPUB("7d"), // val/str PARSE_RULE_STRPUB("8432"), // val/str PARSE_RULE_STRPUB("8KiB"), // val/str PARSE_RULE_STRPUB("8MiB"), // val/str PARSE_RULE_STRPUB("9"), // val/str PARSE_RULE_STRPUB("900"), // val/str PARSE_RULE_STRPUB("90d"), // val/str PARSE_RULE_STRPUB("999"), // val/str PARSE_RULE_STRPUB("9999999"), // val/str PARSE_RULE_STRPUB("9999999d"), // val/str PARSE_RULE_STRPUB("accept-new"), // val/str PARSE_RULE_STRPUB("aes-256-cbc"), // val/str PARSE_RULE_STRPUB("asc"), // val/str PARSE_RULE_STRPUB("auto"), // val/str PARSE_RULE_STRPUB("azure"), // val/str PARSE_RULE_STRPUB("blob.core.windows.net"), // val/str PARSE_RULE_STRPUB("bz2"), // val/str PARSE_RULE_STRPUB("cifs"), // val/str PARSE_RULE_STRPUB("count"), // val/str PARSE_RULE_STRPUB("debug"), // val/str PARSE_RULE_STRPUB("default"), // val/str PARSE_RULE_STRPUB("desc"), // val/str PARSE_RULE_STRPUB("detail"), // val/str PARSE_RULE_STRPUB("diff"), // val/str PARSE_RULE_STRPUB("error"), // val/str PARSE_RULE_STRPUB("fingerprint"), // val/str PARSE_RULE_STRPUB("full"), // val/str PARSE_RULE_STRPUB("gcs"), // val/str PARSE_RULE_STRPUB("gz"), // val/str PARSE_RULE_STRPUB("host"), // val/str PARSE_RULE_STRPUB("immediate"), // val/str PARSE_RULE_STRPUB("incr"), // val/str PARSE_RULE_STRPUB("info"), // val/str PARSE_RULE_STRPUB("json"), // val/str PARSE_RULE_STRPUB("latest"), // val/str PARSE_RULE_STRPUB("localhost"), // val/str PARSE_RULE_STRPUB("lsn"), // val/str PARSE_RULE_STRPUB("lz4"), // val/str PARSE_RULE_STRPUB("md5"), // val/str PARSE_RULE_STRPUB("n"), // val/str PARSE_RULE_STRPUB("name"), // val/str PARSE_RULE_STRPUB("none"), // val/str PARSE_RULE_STRPUB("num"), // val/str PARSE_RULE_STRPUB("off"), // val/str PARSE_RULE_STRPUB("path"), // val/str PARSE_RULE_STRPUB("pause"), // val/str PARSE_RULE_STRPUB("pg"), // val/str PARSE_RULE_STRPUB("pgbackrest"), // val/str PARSE_RULE_STRPUB("pod-id"), // val/str PARSE_RULE_STRPUB("posix"), // val/str PARSE_RULE_STRPUB("postgres"), // val/str PARSE_RULE_STRPUB("prefer"), // val/str PARSE_RULE_STRPUB("preserve"), // val/str PARSE_RULE_STRPUB("process"), // val/str PARSE_RULE_STRPUB("progress"), // val/str PARSE_RULE_STRPUB("promote"), // val/str PARSE_RULE_STRPUB("repo"), // val/str PARSE_RULE_STRPUB("s3"), // val/str PARSE_RULE_STRPUB("sas"), // val/str PARSE_RULE_STRPUB("service"), // val/str PARSE_RULE_STRPUB("sftp"), // val/str PARSE_RULE_STRPUB("sha1"), // val/str PARSE_RULE_STRPUB("sha256"), // val/str PARSE_RULE_STRPUB("shared"), // val/str PARSE_RULE_STRPUB("shutdown"), // val/str PARSE_RULE_STRPUB("ssh"), // val/str PARSE_RULE_STRPUB("standby"), // val/str PARSE_RULE_STRPUB("storage.googleapis.com"), // val/str PARSE_RULE_STRPUB("strict"), // val/str PARSE_RULE_STRPUB("sts.amazonaws.com"), // val/str PARSE_RULE_STRPUB("text"), // val/str PARSE_RULE_STRPUB("time"), // val/str PARSE_RULE_STRPUB("tls"), // val/str PARSE_RULE_STRPUB("token"), // val/str PARSE_RULE_STRPUB("trace"), // val/str PARSE_RULE_STRPUB("warn"), // val/str PARSE_RULE_STRPUB("web-id"), // val/str PARSE_RULE_STRPUB("xid"), // val/str PARSE_RULE_STRPUB("y"), // val/str PARSE_RULE_STRPUB("zst"), // val/str PARSE_RULE_STRPUB(CFGOPTDEF_CONFIG_PATH), // val/str PARSE_RULE_STRPUB(CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE), // val/str PARSE_RULE_STRPUB(CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH), // val/str }; typedef enum { parseRuleValStrQT_DS_1_QT, // val/str/enum parseRuleValStrQT_DS_20_QT, // val/str/enum parseRuleValStrQT_DS_5_QT, // val/str/enum parseRuleValStrQT_DS_7_QT, // val/str/enum parseRuleValStrQT_FS_tmp_FS_pgbackrest_QT, // val/str/enum parseRuleValStrQT_FS_var_FS_lib_FS_pgbackrest_QT, // val/str/enum parseRuleValStrQT_FS_var_FS_log_FS_pgbackrest_QT, // val/str/enum parseRuleValStrQT_FS_var_FS_spool_FS_pgbackrest_QT, // val/str/enum parseRuleValStrQT_0_QT, // val/str/enum parseRuleValStrQT_0B_QT, // val/str/enum parseRuleValStrQT_0s_QT, // val/str/enum parseRuleValStrQT_1_QT, // val/str/enum parseRuleValStrQT_100ms_QT, // val/str/enum parseRuleValStrQT_1024_QT, // val/str/enum parseRuleValStrQT_12_QT, // val/str/enum parseRuleValStrQT_128KiB_QT, // val/str/enum parseRuleValStrQT_128MiB_QT, // val/str/enum parseRuleValStrQT_15d_QT, // val/str/enum parseRuleValStrQT_15m_QT, // val/str/enum parseRuleValStrQT_15s_QT, // val/str/enum parseRuleValStrQT_16GiB_QT, // val/str/enum parseRuleValStrQT_16KiB_QT, // val/str/enum parseRuleValStrQT_16MiB_QT, // val/str/enum parseRuleValStrQT_19_QT, // val/str/enum parseRuleValStrQT_1B_QT, // val/str/enum parseRuleValStrQT_1GiB_QT, // val/str/enum parseRuleValStrQT_1MiB_QT, // val/str/enum parseRuleValStrQT_1PiB_QT, // val/str/enum parseRuleValStrQT_1TiB_QT, // val/str/enum parseRuleValStrQT_1d_QT, // val/str/enum parseRuleValStrQT_1h_QT, // val/str/enum parseRuleValStrQT_1m_QT, // val/str/enum parseRuleValStrQT_2_QT, // val/str/enum parseRuleValStrQT_20MiB_QT, // val/str/enum parseRuleValStrQT_22_QT, // val/str/enum parseRuleValStrQT_256_QT, // val/str/enum parseRuleValStrQT_256KiB_QT, // val/str/enum parseRuleValStrQT_2MiB_QT, // val/str/enum parseRuleValStrQT_3_QT, // val/str/enum parseRuleValStrQT_30m_QT, // val/str/enum parseRuleValStrQT_31m_QT, // val/str/enum parseRuleValStrQT_32_QT, // val/str/enum parseRuleValStrQT_32KiB_QT, // val/str/enum parseRuleValStrQT_360_QT, // val/str/enum parseRuleValStrQT_3600_QT, // val/str/enum parseRuleValStrQT_443_QT, // val/str/enum parseRuleValStrQT_4MiB_QT, // val/str/enum parseRuleValStrQT_4PiB_QT, // val/str/enum parseRuleValStrQT_5_QT, // val/str/enum parseRuleValStrQT_512KiB_QT, // val/str/enum parseRuleValStrQT_5432_QT, // val/str/enum parseRuleValStrQT_5MiB_QT, // val/str/enum parseRuleValStrQT_6_QT, // val/str/enum parseRuleValStrQT_64KiB_QT, // val/str/enum parseRuleValStrQT_65535_QT, // val/str/enum parseRuleValStrQT_7d_QT, // val/str/enum parseRuleValStrQT_8432_QT, // val/str/enum parseRuleValStrQT_8KiB_QT, // val/str/enum parseRuleValStrQT_8MiB_QT, // val/str/enum parseRuleValStrQT_9_QT, // val/str/enum parseRuleValStrQT_900_QT, // val/str/enum parseRuleValStrQT_90d_QT, // val/str/enum parseRuleValStrQT_999_QT, // val/str/enum parseRuleValStrQT_9999999_QT, // val/str/enum parseRuleValStrQT_9999999d_QT, // val/str/enum parseRuleValStrQT_accept_DS_new_QT, // val/str/enum parseRuleValStrQT_aes_DS_256_DS_cbc_QT, // val/str/enum parseRuleValStrQT_asc_QT, // val/str/enum parseRuleValStrQT_auto_QT, // val/str/enum parseRuleValStrQT_azure_QT, // val/str/enum parseRuleValStrQT_blob_DT_core_DT_windows_DT_net_QT, // val/str/enum parseRuleValStrQT_bz2_QT, // val/str/enum parseRuleValStrQT_cifs_QT, // val/str/enum parseRuleValStrQT_count_QT, // val/str/enum parseRuleValStrQT_debug_QT, // val/str/enum parseRuleValStrQT_default_QT, // val/str/enum parseRuleValStrQT_desc_QT, // val/str/enum parseRuleValStrQT_detail_QT, // val/str/enum parseRuleValStrQT_diff_QT, // val/str/enum parseRuleValStrQT_error_QT, // val/str/enum parseRuleValStrQT_fingerprint_QT, // val/str/enum parseRuleValStrQT_full_QT, // val/str/enum parseRuleValStrQT_gcs_QT, // val/str/enum parseRuleValStrQT_gz_QT, // val/str/enum parseRuleValStrQT_host_QT, // val/str/enum parseRuleValStrQT_immediate_QT, // val/str/enum parseRuleValStrQT_incr_QT, // val/str/enum parseRuleValStrQT_info_QT, // val/str/enum parseRuleValStrQT_json_QT, // val/str/enum parseRuleValStrQT_latest_QT, // val/str/enum parseRuleValStrQT_localhost_QT, // val/str/enum parseRuleValStrQT_lsn_QT, // val/str/enum parseRuleValStrQT_lz4_QT, // val/str/enum parseRuleValStrQT_md5_QT, // val/str/enum parseRuleValStrQT_n_QT, // val/str/enum parseRuleValStrQT_name_QT, // val/str/enum parseRuleValStrQT_none_QT, // val/str/enum parseRuleValStrQT_num_QT, // val/str/enum parseRuleValStrQT_off_QT, // val/str/enum parseRuleValStrQT_path_QT, // val/str/enum parseRuleValStrQT_pause_QT, // val/str/enum parseRuleValStrQT_pg_QT, // val/str/enum parseRuleValStrQT_pgbackrest_QT, // val/str/enum parseRuleValStrQT_pod_DS_id_QT, // val/str/enum parseRuleValStrQT_posix_QT, // val/str/enum parseRuleValStrQT_postgres_QT, // val/str/enum parseRuleValStrQT_prefer_QT, // val/str/enum parseRuleValStrQT_preserve_QT, // val/str/enum parseRuleValStrQT_process_QT, // val/str/enum parseRuleValStrQT_progress_QT, // val/str/enum parseRuleValStrQT_promote_QT, // val/str/enum parseRuleValStrQT_repo_QT, // val/str/enum parseRuleValStrQT_s3_QT, // val/str/enum parseRuleValStrQT_sas_QT, // val/str/enum parseRuleValStrQT_service_QT, // val/str/enum parseRuleValStrQT_sftp_QT, // val/str/enum parseRuleValStrQT_sha1_QT, // val/str/enum parseRuleValStrQT_sha256_QT, // val/str/enum parseRuleValStrQT_shared_QT, // val/str/enum parseRuleValStrQT_shutdown_QT, // val/str/enum parseRuleValStrQT_ssh_QT, // val/str/enum parseRuleValStrQT_standby_QT, // val/str/enum parseRuleValStrQT_storage_DT_googleapis_DT_com_QT, // val/str/enum parseRuleValStrQT_strict_QT, // val/str/enum parseRuleValStrQT_sts_DT_amazonaws_DT_com_QT, // val/str/enum parseRuleValStrQT_text_QT, // val/str/enum parseRuleValStrQT_time_QT, // val/str/enum parseRuleValStrQT_tls_QT, // val/str/enum parseRuleValStrQT_token_QT, // val/str/enum parseRuleValStrQT_trace_QT, // val/str/enum parseRuleValStrQT_warn_QT, // val/str/enum parseRuleValStrQT_web_DS_id_QT, // val/str/enum parseRuleValStrQT_xid_QT, // val/str/enum parseRuleValStrQT_y_QT, // val/str/enum parseRuleValStrQT_zst_QT, // val/str/enum parseRuleValStrCFGOPTDEF_CONFIG_PATH, // val/str/enum parseRuleValStrCFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_FILE, // val/str/enum parseRuleValStrCFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_INCLUDE_PATH, // val/str/enum } ParseRuleValueStr; /*********************************************************************************************************************************** Rule Ints ***********************************************************************************************************************************/ #define PARSE_RULE_VAL_INT(value) PARSE_RULE_U32_1(parseRuleValInt##value) static const int parseRuleValueInt[] = { -20, // val/int -7, // val/int -5, // val/int -1, // val/int 0, // val/int 1, // val/int 2, // val/int 3, // val/int 5, // val/int 6, // val/int 9, // val/int 12, // val/int 19, // val/int 22, // val/int 32, // val/int 256, // val/int 360, // val/int 443, // val/int 900, // val/int 999, // val/int 1024, // val/int 3600, // val/int 5432, // val/int 8432, // val/int 65535, // val/int 9999999, // val/int }; static const uint8_t parseRuleValueIntStrMap[] = { parseRuleValStrQT_DS_20_QT, // val/int/strmap parseRuleValStrQT_DS_7_QT, // val/int/strmap parseRuleValStrQT_DS_5_QT, // val/int/strmap parseRuleValStrQT_DS_1_QT, // val/int/strmap parseRuleValStrQT_0_QT, // val/int/strmap parseRuleValStrQT_1_QT, // val/int/strmap parseRuleValStrQT_2_QT, // val/int/strmap parseRuleValStrQT_3_QT, // val/int/strmap parseRuleValStrQT_5_QT, // val/int/strmap parseRuleValStrQT_6_QT, // val/int/strmap parseRuleValStrQT_9_QT, // val/int/strmap parseRuleValStrQT_12_QT, // val/int/strmap parseRuleValStrQT_19_QT, // val/int/strmap parseRuleValStrQT_22_QT, // val/int/strmap parseRuleValStrQT_32_QT, // val/int/strmap parseRuleValStrQT_256_QT, // val/int/strmap parseRuleValStrQT_360_QT, // val/int/strmap parseRuleValStrQT_443_QT, // val/int/strmap parseRuleValStrQT_900_QT, // val/int/strmap parseRuleValStrQT_999_QT, // val/int/strmap parseRuleValStrQT_1024_QT, // val/int/strmap parseRuleValStrQT_3600_QT, // val/int/strmap parseRuleValStrQT_5432_QT, // val/int/strmap parseRuleValStrQT_8432_QT, // val/int/strmap parseRuleValStrQT_65535_QT, // val/int/strmap parseRuleValStrQT_9999999_QT, // val/int/strmap }; typedef enum { parseRuleValIntDS_20, // val/int/enum parseRuleValIntDS_7, // val/int/enum parseRuleValIntDS_5, // val/int/enum parseRuleValIntDS_1, // val/int/enum parseRuleValInt0, // val/int/enum parseRuleValInt1, // val/int/enum parseRuleValInt2, // val/int/enum parseRuleValInt3, // val/int/enum parseRuleValInt5, // val/int/enum parseRuleValInt6, // val/int/enum parseRuleValInt9, // val/int/enum parseRuleValInt12, // val/int/enum parseRuleValInt19, // val/int/enum parseRuleValInt22, // val/int/enum parseRuleValInt32, // val/int/enum parseRuleValInt256, // val/int/enum parseRuleValInt360, // val/int/enum parseRuleValInt443, // val/int/enum parseRuleValInt900, // val/int/enum parseRuleValInt999, // val/int/enum parseRuleValInt1024, // val/int/enum parseRuleValInt3600, // val/int/enum parseRuleValInt5432, // val/int/enum parseRuleValInt8432, // val/int/enum parseRuleValInt65535, // val/int/enum parseRuleValInt9999999, // val/int/enum } ParseRuleValueInt; /*********************************************************************************************************************************** Rule Sizes ***********************************************************************************************************************************/ #define PARSE_RULE_VAL_SIZE(value) PARSE_RULE_U32_1(parseRuleValSize##value) static const int64_t parseRuleValueSize[] = { 0, // val/size 1, // val/size 8192, // val/size 16384, // val/size 32768, // val/size 65536, // val/size 131072, // val/size 262144, // val/size 524288, // val/size 1048576, // val/size 2097152, // val/size 4194304, // val/size 5242880, // val/size 8388608, // val/size 16777216, // val/size 20971520, // val/size 134217728, // val/size 1073741824, // val/size 17179869184, // val/size 1099511627776, // val/size 1125899906842624, // val/size 4503599627370496, // val/size }; static const uint8_t parseRuleValueSizeStrMap[] = { parseRuleValStrQT_0B_QT, // val/size/strmap parseRuleValStrQT_1B_QT, // val/size/strmap parseRuleValStrQT_8KiB_QT, // val/size/strmap parseRuleValStrQT_16KiB_QT, // val/size/strmap parseRuleValStrQT_32KiB_QT, // val/size/strmap parseRuleValStrQT_64KiB_QT, // val/size/strmap parseRuleValStrQT_128KiB_QT, // val/size/strmap parseRuleValStrQT_256KiB_QT, // val/size/strmap parseRuleValStrQT_512KiB_QT, // val/size/strmap parseRuleValStrQT_1MiB_QT, // val/size/strmap parseRuleValStrQT_2MiB_QT, // val/size/strmap parseRuleValStrQT_4MiB_QT, // val/size/strmap parseRuleValStrQT_5MiB_QT, // val/size/strmap parseRuleValStrQT_8MiB_QT, // val/size/strmap parseRuleValStrQT_16MiB_QT, // val/size/strmap parseRuleValStrQT_20MiB_QT, // val/size/strmap parseRuleValStrQT_128MiB_QT, // val/size/strmap parseRuleValStrQT_1GiB_QT, // val/size/strmap parseRuleValStrQT_16GiB_QT, // val/size/strmap parseRuleValStrQT_1TiB_QT, // val/size/strmap parseRuleValStrQT_1PiB_QT, // val/size/strmap parseRuleValStrQT_4PiB_QT, // val/size/strmap }; typedef enum { parseRuleValSize0B, // val/size/enum parseRuleValSize1B, // val/size/enum parseRuleValSize8KiB, // val/size/enum parseRuleValSize16KiB, // val/size/enum parseRuleValSize32KiB, // val/size/enum parseRuleValSize64KiB, // val/size/enum parseRuleValSize128KiB, // val/size/enum parseRuleValSize256KiB, // val/size/enum parseRuleValSize512KiB, // val/size/enum parseRuleValSize1MiB, // val/size/enum parseRuleValSize2MiB, // val/size/enum parseRuleValSize4MiB, // val/size/enum parseRuleValSize5MiB, // val/size/enum parseRuleValSize8MiB, // val/size/enum parseRuleValSize16MiB, // val/size/enum parseRuleValSize20MiB, // val/size/enum parseRuleValSize128MiB, // val/size/enum parseRuleValSize1GiB, // val/size/enum parseRuleValSize16GiB, // val/size/enum parseRuleValSize1TiB, // val/size/enum parseRuleValSize1PiB, // val/size/enum parseRuleValSize4PiB, // val/size/enum } ParseRuleValueSize; /*********************************************************************************************************************************** Rule Times ***********************************************************************************************************************************/ #define PARSE_RULE_VAL_TIME(value) PARSE_RULE_U32_1(parseRuleValTime##value) static const int64_t parseRuleValueTime[] = { 0, // val/time 100, // val/time 15000, // val/time 60000, // val/time 900000, // val/time 1800000, // val/time 1860000, // val/time 3600000, // val/time 86400000, // val/time 604800000, // val/time 1296000000, // val/time 7776000000, // val/time 863999913600000, // val/time }; static const uint8_t parseRuleValueTimeStrMap[] = { parseRuleValStrQT_0s_QT, // val/time/strmap parseRuleValStrQT_100ms_QT, // val/time/strmap parseRuleValStrQT_15s_QT, // val/time/strmap parseRuleValStrQT_1m_QT, // val/time/strmap parseRuleValStrQT_15m_QT, // val/time/strmap parseRuleValStrQT_30m_QT, // val/time/strmap parseRuleValStrQT_31m_QT, // val/time/strmap parseRuleValStrQT_1h_QT, // val/time/strmap parseRuleValStrQT_1d_QT, // val/time/strmap parseRuleValStrQT_7d_QT, // val/time/strmap parseRuleValStrQT_15d_QT, // val/time/strmap parseRuleValStrQT_90d_QT, // val/time/strmap parseRuleValStrQT_9999999d_QT, // val/time/strmap }; typedef enum { parseRuleValTime0s, // val/time/enum parseRuleValTime100ms, // val/time/enum parseRuleValTime15s, // val/time/enum parseRuleValTime1m, // val/time/enum parseRuleValTime15m, // val/time/enum parseRuleValTime30m, // val/time/enum parseRuleValTime31m, // val/time/enum parseRuleValTime1h, // val/time/enum parseRuleValTime1d, // val/time/enum parseRuleValTime7d, // val/time/enum parseRuleValTime15d, // val/time/enum parseRuleValTime90d, // val/time/enum parseRuleValTime9999999d, // val/time/enum } ParseRuleValueTime; /*********************************************************************************************************************************** Dynamic default values ***********************************************************************************************************************************/ #define PARSE_RULE_DEFAULT_DYNAMIC(value) PARSE_RULE_U32_1(parseRuleDefaultDynamic##value) typedef enum { parseRuleDefaultDynamicBin, } ParseRuleDefaultDynamic; /*********************************************************************************************************************************** Command parse data ***********************************************************************************************************************************/ #define PARSE_RULE_VAL_CMD(value) PARSE_RULE_U32_1(cfgCmd##value) static const ParseRuleCommand parseRuleCommand[CFG_COMMAND_TOTAL] = { PARSE_RULE_COMMAND // cmd/annotate ( // cmd/annotate PARSE_RULE_COMMAND_NAME("annotate"), // cmd/annotate PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/annotate PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/annotate PARSE_RULE_COMMAND_LOCK_TYPE(Backup), // cmd/annotate PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/annotate // cmd/annotate PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/annotate ( // cmd/annotate PARSE_RULE_COMMAND_ROLE(Main) // cmd/annotate PARSE_RULE_COMMAND_ROLE(Remote) // cmd/annotate ), // cmd/annotate ), // cmd/annotate // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/archive-get ( // cmd/archive-get PARSE_RULE_COMMAND_NAME("archive-get"), // cmd/archive-get PARSE_RULE_COMMAND_LOCK_TYPE(Archive), // cmd/archive-get PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/archive-get PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/archive-get // cmd/archive-get PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/archive-get ( // cmd/archive-get PARSE_RULE_COMMAND_ROLE(Async) // cmd/archive-get PARSE_RULE_COMMAND_ROLE(Local) // cmd/archive-get PARSE_RULE_COMMAND_ROLE(Main) // cmd/archive-get PARSE_RULE_COMMAND_ROLE(Remote) // cmd/archive-get ), // cmd/archive-get ), // cmd/archive-get // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/archive-push ( // cmd/archive-push PARSE_RULE_COMMAND_NAME("archive-push"), // cmd/archive-push PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/archive-push PARSE_RULE_COMMAND_LOCK_TYPE(Archive), // cmd/archive-push PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/archive-push PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/archive-push // cmd/archive-push PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/archive-push ( // cmd/archive-push PARSE_RULE_COMMAND_ROLE(Async) // cmd/archive-push PARSE_RULE_COMMAND_ROLE(Local) // cmd/archive-push PARSE_RULE_COMMAND_ROLE(Main) // cmd/archive-push PARSE_RULE_COMMAND_ROLE(Remote) // cmd/archive-push ), // cmd/archive-push ), // cmd/archive-push // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/backup ( // cmd/backup PARSE_RULE_COMMAND_NAME("backup"), // cmd/backup PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/backup PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/backup PARSE_RULE_COMMAND_LOCK_TYPE(Backup), // cmd/backup PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/backup PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/backup // cmd/backup PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/backup ( // cmd/backup PARSE_RULE_COMMAND_ROLE(Local) // cmd/backup PARSE_RULE_COMMAND_ROLE(Main) // cmd/backup PARSE_RULE_COMMAND_ROLE(Remote) // cmd/backup ), // cmd/backup ), // cmd/backup // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/check ( // cmd/check PARSE_RULE_COMMAND_NAME("check"), // cmd/check PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/check PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/check // cmd/check PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/check ( // cmd/check PARSE_RULE_COMMAND_ROLE(Main) // cmd/check PARSE_RULE_COMMAND_ROLE(Remote) // cmd/check ), // cmd/check ), // cmd/check // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/expire ( // cmd/expire PARSE_RULE_COMMAND_NAME("expire"), // cmd/expire PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/expire PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/expire PARSE_RULE_COMMAND_LOCK_TYPE(Backup), // cmd/expire PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/expire PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/expire // cmd/expire PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/expire ( // cmd/expire PARSE_RULE_COMMAND_ROLE(Main) // cmd/expire PARSE_RULE_COMMAND_ROLE(Remote) // cmd/expire ), // cmd/expire ), // cmd/expire // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/help ( // cmd/help PARSE_RULE_COMMAND_NAME("help"), // cmd/help PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/help PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/help PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/help // cmd/help PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/help ( // cmd/help PARSE_RULE_COMMAND_ROLE(Main) // cmd/help ), // cmd/help ), // cmd/help // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/info ( // cmd/info PARSE_RULE_COMMAND_NAME("info"), // cmd/info PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/info PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/info // cmd/info PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/info ( // cmd/info PARSE_RULE_COMMAND_ROLE(Main) // cmd/info PARSE_RULE_COMMAND_ROLE(Remote) // cmd/info ), // cmd/info ), // cmd/info // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/manifest ( // cmd/manifest PARSE_RULE_COMMAND_NAME("manifest"), // cmd/manifest PARSE_RULE_COMMAND_INTERNAL(true) // cmd/manifest PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/manifest PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/manifest // cmd/manifest PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/manifest ( // cmd/manifest PARSE_RULE_COMMAND_ROLE(Main) // cmd/manifest PARSE_RULE_COMMAND_ROLE(Remote) // cmd/manifest ), // cmd/manifest ), // cmd/manifest // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/repo-get ( // cmd/repo-get PARSE_RULE_COMMAND_NAME("repo-get"), // cmd/repo-get PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/repo-get PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/repo-get PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/repo-get // cmd/repo-get PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/repo-get ( // cmd/repo-get PARSE_RULE_COMMAND_ROLE(Main) // cmd/repo-get PARSE_RULE_COMMAND_ROLE(Remote) // cmd/repo-get ), // cmd/repo-get ), // cmd/repo-get // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/repo-ls ( // cmd/repo-ls PARSE_RULE_COMMAND_NAME("repo-ls"), // cmd/repo-ls PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/repo-ls PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/repo-ls PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/repo-ls // cmd/repo-ls PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/repo-ls ( // cmd/repo-ls PARSE_RULE_COMMAND_ROLE(Main) // cmd/repo-ls PARSE_RULE_COMMAND_ROLE(Remote) // cmd/repo-ls ), // cmd/repo-ls ), // cmd/repo-ls // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/repo-rm ( // cmd/repo-rm PARSE_RULE_COMMAND_NAME("repo-rm"), // cmd/repo-rm PARSE_RULE_COMMAND_INTERNAL(true) // cmd/repo-rm PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/repo-rm PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/repo-rm PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/repo-rm // cmd/repo-rm PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/repo-rm ( // cmd/repo-rm PARSE_RULE_COMMAND_ROLE(Main) // cmd/repo-rm PARSE_RULE_COMMAND_ROLE(Remote) // cmd/repo-rm ), // cmd/repo-rm ), // cmd/repo-rm // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/restore ( // cmd/restore PARSE_RULE_COMMAND_NAME("restore"), // cmd/restore PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/restore PARSE_RULE_COMMAND_LOCK_TYPE(Restore), // cmd/restore PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/restore PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/restore // cmd/restore PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/restore ( // cmd/restore PARSE_RULE_COMMAND_ROLE(Local) // cmd/restore PARSE_RULE_COMMAND_ROLE(Main) // cmd/restore PARSE_RULE_COMMAND_ROLE(Remote) // cmd/restore ), // cmd/restore ), // cmd/restore // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/server ( // cmd/server PARSE_RULE_COMMAND_NAME("server"), // cmd/server PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/server PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/server PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/server // cmd/server PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/server ( // cmd/server PARSE_RULE_COMMAND_ROLE(Main) // cmd/server ), // cmd/server ), // cmd/server // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/server-ping ( // cmd/server-ping PARSE_RULE_COMMAND_NAME("server-ping"), // cmd/server-ping PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/server-ping PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/server-ping PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/server-ping // cmd/server-ping PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/server-ping ( // cmd/server-ping PARSE_RULE_COMMAND_ROLE(Main) // cmd/server-ping ), // cmd/server-ping ), // cmd/server-ping // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/stanza-create ( // cmd/stanza-create PARSE_RULE_COMMAND_NAME("stanza-create"), // cmd/stanza-create PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/stanza-create PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/stanza-create PARSE_RULE_COMMAND_LOCK_TYPE(All), // cmd/stanza-create PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/stanza-create PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/stanza-create // cmd/stanza-create PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/stanza-create ( // cmd/stanza-create PARSE_RULE_COMMAND_ROLE(Main) // cmd/stanza-create PARSE_RULE_COMMAND_ROLE(Remote) // cmd/stanza-create ), // cmd/stanza-create ), // cmd/stanza-create // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/stanza-delete ( // cmd/stanza-delete PARSE_RULE_COMMAND_NAME("stanza-delete"), // cmd/stanza-delete PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/stanza-delete PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/stanza-delete PARSE_RULE_COMMAND_LOCK_TYPE(All), // cmd/stanza-delete PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/stanza-delete PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/stanza-delete // cmd/stanza-delete PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/stanza-delete ( // cmd/stanza-delete PARSE_RULE_COMMAND_ROLE(Main) // cmd/stanza-delete PARSE_RULE_COMMAND_ROLE(Remote) // cmd/stanza-delete ), // cmd/stanza-delete ), // cmd/stanza-delete // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/stanza-upgrade ( // cmd/stanza-upgrade PARSE_RULE_COMMAND_NAME("stanza-upgrade"), // cmd/stanza-upgrade PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/stanza-upgrade PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/stanza-upgrade PARSE_RULE_COMMAND_LOCK_TYPE(All), // cmd/stanza-upgrade PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/stanza-upgrade PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/stanza-upgrade // cmd/stanza-upgrade PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/stanza-upgrade ( // cmd/stanza-upgrade PARSE_RULE_COMMAND_ROLE(Main) // cmd/stanza-upgrade PARSE_RULE_COMMAND_ROLE(Remote) // cmd/stanza-upgrade ), // cmd/stanza-upgrade ), // cmd/stanza-upgrade // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/start ( // cmd/start PARSE_RULE_COMMAND_NAME("start"), // cmd/start PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/start PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/start PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/start // cmd/start PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/start ( // cmd/start PARSE_RULE_COMMAND_ROLE(Main) // cmd/start ), // cmd/start ), // cmd/start // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/stop ( // cmd/stop PARSE_RULE_COMMAND_NAME("stop"), // cmd/stop PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/stop PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/stop PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/stop // cmd/stop PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/stop ( // cmd/stop PARSE_RULE_COMMAND_ROLE(Main) // cmd/stop ), // cmd/stop ), // cmd/stop // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/verify ( // cmd/verify PARSE_RULE_COMMAND_NAME("verify"), // cmd/verify PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/verify PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/verify PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/verify // cmd/verify PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/verify ( // cmd/verify PARSE_RULE_COMMAND_ROLE(Local) // cmd/verify PARSE_RULE_COMMAND_ROLE(Main) // cmd/verify PARSE_RULE_COMMAND_ROLE(Remote) // cmd/verify ), // cmd/verify ), // cmd/verify // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/version ( // cmd/version PARSE_RULE_COMMAND_NAME("version"), // cmd/version PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/version PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/version // cmd/version PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/version ( // cmd/version PARSE_RULE_COMMAND_ROLE(Main) // cmd/version ), // cmd/version ), // cmd/version }; /*********************************************************************************************************************************** Option group parse data ***********************************************************************************************************************************/ static const ParseRuleOptionGroup parseRuleOptionGroup[CFG_OPTION_GROUP_TOTAL] = { PARSE_RULE_OPTION_GROUP // opt-grp/pg ( // opt-grp/pg PARSE_RULE_OPTION_GROUP_NAME("pg"), // opt-grp/pg ), // opt-grp/pg // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION_GROUP // opt-grp/repo ( // opt-grp/repo PARSE_RULE_OPTION_GROUP_NAME("repo"), // opt-grp/repo ), // opt-grp/repo }; /*********************************************************************************************************************************** Option parse data ***********************************************************************************************************************************/ #define PARSE_RULE_VAL_OPT(value) PARSE_RULE_U32_2(cfgOpt##value) static const ParseRuleOption parseRuleOption[CFG_OPTION_TOTAL] = { PARSE_RULE_OPTION // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTION_NAME("allow-root"), // opt/allow-root PARSE_RULE_OPTION_TYPE(Boolean), // opt/allow-root PARSE_RULE_OPTION_NEGATE(true), // opt/allow-root PARSE_RULE_OPTION_RESET(true), // opt/allow-root PARSE_RULE_OPTION_REQUIRED(true), // opt/allow-root PARSE_RULE_OPTION_SECTION(Global), // opt/allow-root // opt/allow-root PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTION_COMMAND(Annotate) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Backup) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Check) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Expire) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Info) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Manifest) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Restore) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Server) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/allow-root PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/allow-root PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/allow-root PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Start) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Stop) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Verify) // opt/allow-root ), // opt/allow-root // opt/allow-root PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/allow-root ), // opt/allow-root // opt/allow-root PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Backup) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Restore) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Verify) // opt/allow-root ), // opt/allow-root // opt/allow-root PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTION_COMMAND(Annotate) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Backup) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Check) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Expire) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Info) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Manifest) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Restore) // opt/allow-root PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/allow-root PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/allow-root PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Verify) // opt/allow-root ), // opt/allow-root // opt/allow-root PARSE_RULE_OPTIONAL // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTIONAL_GROUP // opt/allow-root ( // opt/allow-root PARSE_RULE_FILTER_CMD // opt/allow-root ( // opt/allow-root PARSE_RULE_VAL_CMD(Restore), // opt/allow-root ), // opt/allow-root // opt/allow-root PARSE_RULE_OPTIONAL_DEFAULT // opt/allow-root ( // opt/allow-root PARSE_RULE_VAL_BOOL_TRUE, // opt/allow-root ), // opt/allow-root ), // opt/allow-root // opt/allow-root PARSE_RULE_OPTIONAL_GROUP // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTIONAL_DEFAULT // opt/allow-root ( // opt/allow-root PARSE_RULE_VAL_BOOL_FALSE, // opt/allow-root ), // opt/allow-root ), // opt/allow-root ), // opt/allow-root ), // opt/allow-root // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/annotation ( // opt/annotation PARSE_RULE_OPTION_NAME("annotation"), // opt/annotation PARSE_RULE_OPTION_TYPE(Hash), // opt/annotation PARSE_RULE_OPTION_RESET(true), // opt/annotation PARSE_RULE_OPTION_REQUIRED(false), // opt/annotation PARSE_RULE_OPTION_SECTION(Global), // opt/annotation PARSE_RULE_OPTION_MULTI(true), // opt/annotation // opt/annotation PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/annotation ( // opt/annotation PARSE_RULE_OPTION_COMMAND(Annotate) // opt/annotation PARSE_RULE_OPTION_COMMAND(Backup) // opt/annotation ), // opt/annotation // opt/annotation PARSE_RULE_OPTIONAL // opt/annotation ( // opt/annotation PARSE_RULE_OPTIONAL_GROUP // opt/annotation ( // opt/annotation PARSE_RULE_FILTER_CMD // opt/annotation ( // opt/annotation PARSE_RULE_VAL_CMD(Annotate), // opt/annotation ), // opt/annotation // opt/annotation PARSE_RULE_OPTIONAL_REQUIRED(), // opt/annotation ), // opt/annotation ), // opt/annotation ), // opt/annotation // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTION_NAME("archive-async"), // opt/archive-async PARSE_RULE_OPTION_TYPE(Boolean), // opt/archive-async PARSE_RULE_OPTION_NEGATE(true), // opt/archive-async PARSE_RULE_OPTION_RESET(true), // opt/archive-async PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-async PARSE_RULE_OPTION_SECTION(Global), // opt/archive-async // opt/archive-async PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-async ), // opt/archive-async // opt/archive-async PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-async ), // opt/archive-async // opt/archive-async PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-async ), // opt/archive-async // opt/archive-async PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-async ), // opt/archive-async // opt/archive-async PARSE_RULE_OPTIONAL // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTIONAL_GROUP // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-async ( // opt/archive-async PARSE_RULE_VAL_BOOL_FALSE, // opt/archive-async ), // opt/archive-async ), // opt/archive-async ), // opt/archive-async ), // opt/archive-async // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-check ( // opt/archive-check PARSE_RULE_OPTION_NAME("archive-check"), // opt/archive-check PARSE_RULE_OPTION_TYPE(Boolean), // opt/archive-check PARSE_RULE_OPTION_NEGATE(true), // opt/archive-check PARSE_RULE_OPTION_RESET(true), // opt/archive-check PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-check PARSE_RULE_OPTION_SECTION(Global), // opt/archive-check // opt/archive-check PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-check ( // opt/archive-check PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-check PARSE_RULE_OPTION_COMMAND(Backup) // opt/archive-check PARSE_RULE_OPTION_COMMAND(Check) // opt/archive-check ), // opt/archive-check // opt/archive-check PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-check ( // opt/archive-check PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-check ), // opt/archive-check // opt/archive-check PARSE_RULE_OPTIONAL // opt/archive-check ( // opt/archive-check PARSE_RULE_OPTIONAL_GROUP // opt/archive-check ( // opt/archive-check PARSE_RULE_FILTER_CMD // opt/archive-check ( // opt/archive-check PARSE_RULE_VAL_CMD(Backup), // opt/archive-check ), // opt/archive-check // opt/archive-check PARSE_RULE_OPTIONAL_DEPEND // opt/archive-check ( // opt/archive-check PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_BOOL_FALSE), // opt/archive-check PARSE_RULE_VAL_OPT(Online), // opt/archive-check PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-check ), // opt/archive-check // opt/archive-check PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-check ( // opt/archive-check PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-check ), // opt/archive-check ), // opt/archive-check // opt/archive-check PARSE_RULE_OPTIONAL_GROUP // opt/archive-check ( // opt/archive-check PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-check ( // opt/archive-check PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-check ), // opt/archive-check ), // opt/archive-check ), // opt/archive-check ), // opt/archive-check // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-copy ( // opt/archive-copy PARSE_RULE_OPTION_NAME("archive-copy"), // opt/archive-copy PARSE_RULE_OPTION_TYPE(Boolean), // opt/archive-copy PARSE_RULE_OPTION_NEGATE(true), // opt/archive-copy PARSE_RULE_OPTION_RESET(true), // opt/archive-copy PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-copy PARSE_RULE_OPTION_SECTION(Global), // opt/archive-copy // opt/archive-copy PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-copy ( // opt/archive-copy PARSE_RULE_OPTION_COMMAND(Backup) // opt/archive-copy ), // opt/archive-copy // opt/archive-copy PARSE_RULE_OPTIONAL // opt/archive-copy ( // opt/archive-copy PARSE_RULE_OPTIONAL_GROUP // opt/archive-copy ( // opt/archive-copy PARSE_RULE_FILTER_CMD // opt/archive-copy ( // opt/archive-copy PARSE_RULE_VAL_CMD(Backup), // opt/archive-copy ), // opt/archive-copy // opt/archive-copy PARSE_RULE_OPTIONAL_DEPEND // opt/archive-copy ( // opt/archive-copy PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_BOOL_FALSE), // opt/archive-copy PARSE_RULE_VAL_OPT(ArchiveCheck), // opt/archive-copy PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-copy ), // opt/archive-copy // opt/archive-copy PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-copy ( // opt/archive-copy PARSE_RULE_VAL_BOOL_FALSE, // opt/archive-copy ), // opt/archive-copy ), // opt/archive-copy // opt/archive-copy PARSE_RULE_OPTIONAL_GROUP // opt/archive-copy ( // opt/archive-copy PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-copy ( // opt/archive-copy PARSE_RULE_VAL_BOOL_FALSE, // opt/archive-copy ), // opt/archive-copy ), // opt/archive-copy ), // opt/archive-copy ), // opt/archive-copy // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-expire-before ( // opt/archive-expire-before PARSE_RULE_OPTION_NAME("archive-expire-before"), // opt/archive-expire-before PARSE_RULE_OPTION_TYPE(String), // opt/archive-expire-before PARSE_RULE_OPTION_REQUIRED(false), // opt/archive-expire-before PARSE_RULE_OPTION_SECTION(CommandLine), // opt/archive-expire-before // opt/archive-expire-before PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-expire-before ( // opt/archive-expire-before PARSE_RULE_OPTION_COMMAND(Expire) // opt/archive-expire-before ), // opt/archive-expire-before ), // opt/archive-expire-before // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_OPTION_NAME("archive-get-queue-max"), // opt/archive-get-queue-max PARSE_RULE_OPTION_TYPE(Size), // opt/archive-get-queue-max PARSE_RULE_OPTION_RESET(true), // opt/archive-get-queue-max PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-get-queue-max PARSE_RULE_OPTION_SECTION(Global), // opt/archive-get-queue-max // opt/archive-get-queue-max PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-get-queue-max ), // opt/archive-get-queue-max // opt/archive-get-queue-max PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-get-queue-max ), // opt/archive-get-queue-max // opt/archive-get-queue-max PARSE_RULE_OPTIONAL // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_OPTIONAL_GROUP // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_VAL_SIZE(0B), // opt/archive-get-queue-max PARSE_RULE_VAL_SIZE(4PiB), // opt/archive-get-queue-max ), // opt/archive-get-queue-max // opt/archive-get-queue-max PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_VAL_SIZE(128MiB), // opt/archive-get-queue-max ), // opt/archive-get-queue-max ), // opt/archive-get-queue-max ), // opt/archive-get-queue-max ), // opt/archive-get-queue-max // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-header-check ( // opt/archive-header-check PARSE_RULE_OPTION_NAME("archive-header-check"), // opt/archive-header-check PARSE_RULE_OPTION_TYPE(Boolean), // opt/archive-header-check PARSE_RULE_OPTION_NEGATE(true), // opt/archive-header-check PARSE_RULE_OPTION_RESET(true), // opt/archive-header-check PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-header-check PARSE_RULE_OPTION_SECTION(Global), // opt/archive-header-check // opt/archive-header-check PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-header-check ( // opt/archive-header-check PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-header-check ), // opt/archive-header-check // opt/archive-header-check PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-header-check ( // opt/archive-header-check PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-header-check ), // opt/archive-header-check // opt/archive-header-check PARSE_RULE_OPTIONAL // opt/archive-header-check ( // opt/archive-header-check PARSE_RULE_OPTIONAL_GROUP // opt/archive-header-check ( // opt/archive-header-check PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-header-check ( // opt/archive-header-check PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-header-check ), // opt/archive-header-check ), // opt/archive-header-check ), // opt/archive-header-check ), // opt/archive-header-check // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-missing-retry ( // opt/archive-missing-retry PARSE_RULE_OPTION_NAME("archive-missing-retry"), // opt/archive-missing-retry PARSE_RULE_OPTION_TYPE(Boolean), // opt/archive-missing-retry PARSE_RULE_OPTION_NEGATE(true), // opt/archive-missing-retry PARSE_RULE_OPTION_RESET(true), // opt/archive-missing-retry PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-missing-retry PARSE_RULE_OPTION_SECTION(Global), // opt/archive-missing-retry // opt/archive-missing-retry PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-missing-retry ( // opt/archive-missing-retry PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-missing-retry ), // opt/archive-missing-retry // opt/archive-missing-retry PARSE_RULE_OPTIONAL // opt/archive-missing-retry ( // opt/archive-missing-retry PARSE_RULE_OPTIONAL_GROUP // opt/archive-missing-retry ( // opt/archive-missing-retry PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-missing-retry ( // opt/archive-missing-retry PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-missing-retry ), // opt/archive-missing-retry ), // opt/archive-missing-retry ), // opt/archive-missing-retry ), // opt/archive-missing-retry // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-mode ( // opt/archive-mode PARSE_RULE_OPTION_NAME("archive-mode"), // opt/archive-mode PARSE_RULE_OPTION_TYPE(StringId), // opt/archive-mode PARSE_RULE_OPTION_SEQUENCE(true), // opt/archive-mode PARSE_RULE_OPTION_RESET(true), // opt/archive-mode PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-mode PARSE_RULE_OPTION_SECTION(Global), // opt/archive-mode // opt/archive-mode PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-mode ( // opt/archive-mode PARSE_RULE_OPTION_COMMAND(Restore) // opt/archive-mode ), // opt/archive-mode // opt/archive-mode PARSE_RULE_OPTIONAL // opt/archive-mode ( // opt/archive-mode PARSE_RULE_OPTIONAL_GROUP // opt/archive-mode ( // opt/archive-mode PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/archive-mode ( // opt/archive-mode PARSE_RULE_VAL_STRID(preserve), // opt/archive-mode PARSE_RULE_VAL_STRID(off), // opt/archive-mode ), // opt/archive-mode // opt/archive-mode PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-mode ( // opt/archive-mode PARSE_RULE_VAL_STRID(preserve), // opt/archive-mode ), // opt/archive-mode ), // opt/archive-mode ), // opt/archive-mode ), // opt/archive-mode // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_OPTION_NAME("archive-mode-check"), // opt/archive-mode-check PARSE_RULE_OPTION_TYPE(Boolean), // opt/archive-mode-check PARSE_RULE_OPTION_NEGATE(true), // opt/archive-mode-check PARSE_RULE_OPTION_RESET(true), // opt/archive-mode-check PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-mode-check PARSE_RULE_OPTION_SECTION(Global), // opt/archive-mode-check // opt/archive-mode-check PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-mode-check PARSE_RULE_OPTION_COMMAND(Backup) // opt/archive-mode-check PARSE_RULE_OPTION_COMMAND(Check) // opt/archive-mode-check ), // opt/archive-mode-check // opt/archive-mode-check PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-mode-check ), // opt/archive-mode-check // opt/archive-mode-check PARSE_RULE_OPTIONAL // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_OPTIONAL_GROUP // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_OPTIONAL_DEPEND // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_BOOL_FALSE), // opt/archive-mode-check PARSE_RULE_VAL_OPT(ArchiveCheck), // opt/archive-mode-check PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-mode-check ), // opt/archive-mode-check // opt/archive-mode-check PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-mode-check ), // opt/archive-mode-check ), // opt/archive-mode-check ), // opt/archive-mode-check ), // opt/archive-mode-check // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_OPTION_NAME("archive-push-batch-size"), // opt/archive-push-batch-size PARSE_RULE_OPTION_TYPE(Size), // opt/archive-push-batch-size PARSE_RULE_OPTION_RESET(true), // opt/archive-push-batch-size PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-push-batch-size PARSE_RULE_OPTION_SECTION(Global), // opt/archive-push-batch-size // opt/archive-push-batch-size PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-push-batch-size ), // opt/archive-push-batch-size // opt/archive-push-batch-size PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-push-batch-size ), // opt/archive-push-batch-size // opt/archive-push-batch-size PARSE_RULE_OPTIONAL // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_OPTIONAL_GROUP // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_VAL_SIZE(1MiB), // opt/archive-push-batch-size PARSE_RULE_VAL_SIZE(4PiB), // opt/archive-push-batch-size ), // opt/archive-push-batch-size // opt/archive-push-batch-size PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_VAL_SIZE(16GiB), // opt/archive-push-batch-size ), // opt/archive-push-batch-size ), // opt/archive-push-batch-size ), // opt/archive-push-batch-size ), // opt/archive-push-batch-size // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-push-queue-max ( // opt/archive-push-queue-max PARSE_RULE_OPTION_NAME("archive-push-queue-max"), // opt/archive-push-queue-max PARSE_RULE_OPTION_TYPE(Size), // opt/archive-push-queue-max PARSE_RULE_OPTION_RESET(true), // opt/archive-push-queue-max PARSE_RULE_OPTION_REQUIRED(false), // opt/archive-push-queue-max PARSE_RULE_OPTION_SECTION(Global), // opt/archive-push-queue-max // opt/archive-push-queue-max PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-push-queue-max ( // opt/archive-push-queue-max PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-push-queue-max ), // opt/archive-push-queue-max // opt/archive-push-queue-max PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-push-queue-max ( // opt/archive-push-queue-max PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-push-queue-max ), // opt/archive-push-queue-max // opt/archive-push-queue-max PARSE_RULE_OPTIONAL // opt/archive-push-queue-max ( // opt/archive-push-queue-max PARSE_RULE_OPTIONAL_GROUP // opt/archive-push-queue-max ( // opt/archive-push-queue-max PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/archive-push-queue-max ( // opt/archive-push-queue-max PARSE_RULE_VAL_SIZE(0B), // opt/archive-push-queue-max PARSE_RULE_VAL_SIZE(4PiB), // opt/archive-push-queue-max ), // opt/archive-push-queue-max ), // opt/archive-push-queue-max ), // opt/archive-push-queue-max ), // opt/archive-push-queue-max // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-timeout ( // opt/archive-timeout PARSE_RULE_OPTION_NAME("archive-timeout"), // opt/archive-timeout PARSE_RULE_OPTION_TYPE(Time), // opt/archive-timeout PARSE_RULE_OPTION_RESET(true), // opt/archive-timeout PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-timeout PARSE_RULE_OPTION_SECTION(Global), // opt/archive-timeout // opt/archive-timeout PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-timeout ( // opt/archive-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/archive-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/archive-timeout ), // opt/archive-timeout // opt/archive-timeout PARSE_RULE_OPTIONAL // opt/archive-timeout ( // opt/archive-timeout PARSE_RULE_OPTIONAL_GROUP // opt/archive-timeout ( // opt/archive-timeout PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/archive-timeout ( // opt/archive-timeout PARSE_RULE_VAL_TIME(100ms), // opt/archive-timeout PARSE_RULE_VAL_TIME(1d), // opt/archive-timeout ), // opt/archive-timeout // opt/archive-timeout PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-timeout ( // opt/archive-timeout PARSE_RULE_VAL_TIME(1m), // opt/archive-timeout ), // opt/archive-timeout ), // opt/archive-timeout ), // opt/archive-timeout ), // opt/archive-timeout // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/backup-standby ( // opt/backup-standby PARSE_RULE_OPTION_NAME("backup-standby"), // opt/backup-standby PARSE_RULE_OPTION_TYPE(StringId), // opt/backup-standby PARSE_RULE_OPTION_SEQUENCE(true), // opt/backup-standby PARSE_RULE_OPTION_BOOL_LIKE(true), // opt/backup-standby PARSE_RULE_OPTION_NEGATE(true), // opt/backup-standby PARSE_RULE_OPTION_RESET(true), // opt/backup-standby PARSE_RULE_OPTION_REQUIRED(true), // opt/backup-standby PARSE_RULE_OPTION_SECTION(Global), // opt/backup-standby // opt/backup-standby PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/backup-standby ( // opt/backup-standby PARSE_RULE_OPTION_COMMAND(Backup) // opt/backup-standby PARSE_RULE_OPTION_COMMAND(Check) // opt/backup-standby ), // opt/backup-standby // opt/backup-standby PARSE_RULE_OPTIONAL // opt/backup-standby ( // opt/backup-standby PARSE_RULE_OPTIONAL_GROUP // opt/backup-standby ( // opt/backup-standby PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/backup-standby ( // opt/backup-standby PARSE_RULE_VAL_STRID(n), // opt/backup-standby PARSE_RULE_VAL_STRID(prefer), // opt/backup-standby PARSE_RULE_VAL_STRID(y), // opt/backup-standby ), // opt/backup-standby // opt/backup-standby PARSE_RULE_OPTIONAL_DEFAULT // opt/backup-standby ( // opt/backup-standby PARSE_RULE_VAL_STRID(n), // opt/backup-standby ), // opt/backup-standby ), // opt/backup-standby ), // opt/backup-standby ), // opt/backup-standby // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/beta ( // opt/beta PARSE_RULE_OPTION_NAME("beta"), // opt/beta PARSE_RULE_OPTION_TYPE(Boolean), // opt/beta PARSE_RULE_OPTION_INTERNAL(true) // opt/beta PARSE_RULE_OPTION_BETA(true), // opt/beta PARSE_RULE_OPTION_NEGATE(true), // opt/beta PARSE_RULE_OPTION_RESET(true), // opt/beta PARSE_RULE_OPTION_REQUIRED(true), // opt/beta PARSE_RULE_OPTION_SECTION(Global), // opt/beta // opt/beta PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/beta ( // opt/beta PARSE_RULE_OPTION_COMMAND(Annotate) // opt/beta PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/beta PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/beta PARSE_RULE_OPTION_COMMAND(Backup) // opt/beta PARSE_RULE_OPTION_COMMAND(Check) // opt/beta PARSE_RULE_OPTION_COMMAND(Expire) // opt/beta PARSE_RULE_OPTION_COMMAND(Info) // opt/beta PARSE_RULE_OPTION_COMMAND(Manifest) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/beta PARSE_RULE_OPTION_COMMAND(Restore) // opt/beta PARSE_RULE_OPTION_COMMAND(Server) // opt/beta PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/beta PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/beta PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/beta PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/beta PARSE_RULE_OPTION_COMMAND(Start) // opt/beta PARSE_RULE_OPTION_COMMAND(Stop) // opt/beta PARSE_RULE_OPTION_COMMAND(Verify) // opt/beta ), // opt/beta // opt/beta PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/beta ( // opt/beta PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/beta PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/beta ), // opt/beta // opt/beta PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/beta ( // opt/beta PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/beta PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/beta PARSE_RULE_OPTION_COMMAND(Backup) // opt/beta PARSE_RULE_OPTION_COMMAND(Restore) // opt/beta PARSE_RULE_OPTION_COMMAND(Verify) // opt/beta ), // opt/beta // opt/beta PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/beta ( // opt/beta PARSE_RULE_OPTION_COMMAND(Annotate) // opt/beta PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/beta PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/beta PARSE_RULE_OPTION_COMMAND(Backup) // opt/beta PARSE_RULE_OPTION_COMMAND(Check) // opt/beta PARSE_RULE_OPTION_COMMAND(Expire) // opt/beta PARSE_RULE_OPTION_COMMAND(Info) // opt/beta PARSE_RULE_OPTION_COMMAND(Manifest) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/beta PARSE_RULE_OPTION_COMMAND(Restore) // opt/beta PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/beta PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/beta PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/beta PARSE_RULE_OPTION_COMMAND(Verify) // opt/beta ), // opt/beta // opt/beta PARSE_RULE_OPTIONAL // opt/beta ( // opt/beta PARSE_RULE_OPTIONAL_GROUP // opt/beta ( // opt/beta PARSE_RULE_OPTIONAL_DEFAULT // opt/beta ( // opt/beta PARSE_RULE_VAL_BOOL_FALSE, // opt/beta ), // opt/beta ), // opt/beta ), // opt/beta ), // opt/beta // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTION_NAME("buffer-size"), // opt/buffer-size PARSE_RULE_OPTION_TYPE(Size), // opt/buffer-size PARSE_RULE_OPTION_RESET(true), // opt/buffer-size PARSE_RULE_OPTION_REQUIRED(true), // opt/buffer-size PARSE_RULE_OPTION_SECTION(Global), // opt/buffer-size // opt/buffer-size PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Annotate) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Check) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Expire) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Info) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Manifest) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Restore) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Server) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Verify) // opt/buffer-size ), // opt/buffer-size // opt/buffer-size PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/buffer-size ), // opt/buffer-size // opt/buffer-size PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Restore) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Verify) // opt/buffer-size ), // opt/buffer-size // opt/buffer-size PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Annotate) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Check) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Expire) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Info) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Manifest) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Restore) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Verify) // opt/buffer-size ), // opt/buffer-size // opt/buffer-size PARSE_RULE_OPTIONAL // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTIONAL_GROUP // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/buffer-size ( // opt/buffer-size PARSE_RULE_VAL_SIZE(16KiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(32KiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(64KiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(128KiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(256KiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(512KiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(1MiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(2MiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(4MiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(8MiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(16MiB), // opt/buffer-size ), // opt/buffer-size // opt/buffer-size PARSE_RULE_OPTIONAL_DEFAULT // opt/buffer-size ( // opt/buffer-size PARSE_RULE_VAL_SIZE(1MiB), // opt/buffer-size ), // opt/buffer-size ), // opt/buffer-size ), // opt/buffer-size ), // opt/buffer-size // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/checksum-page ( // opt/checksum-page PARSE_RULE_OPTION_NAME("checksum-page"), // opt/checksum-page PARSE_RULE_OPTION_TYPE(Boolean), // opt/checksum-page PARSE_RULE_OPTION_NEGATE(true), // opt/checksum-page PARSE_RULE_OPTION_RESET(true), // opt/checksum-page PARSE_RULE_OPTION_REQUIRED(false), // opt/checksum-page PARSE_RULE_OPTION_SECTION(Global), // opt/checksum-page // opt/checksum-page PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/checksum-page ( // opt/checksum-page PARSE_RULE_OPTION_COMMAND(Backup) // opt/checksum-page ), // opt/checksum-page ), // opt/checksum-page // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/checksum-page-error ( // opt/checksum-page-error PARSE_RULE_OPTION_NAME("checksum-page-error"), // opt/checksum-page-error PARSE_RULE_OPTION_TYPE(Boolean), // opt/checksum-page-error PARSE_RULE_OPTION_NEGATE(true), // opt/checksum-page-error PARSE_RULE_OPTION_RESET(true), // opt/checksum-page-error PARSE_RULE_OPTION_REQUIRED(true), // opt/checksum-page-error PARSE_RULE_OPTION_SECTION(Global), // opt/checksum-page-error // opt/checksum-page-error PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/checksum-page-error ( // opt/checksum-page-error PARSE_RULE_OPTION_COMMAND(Backup) // opt/checksum-page-error PARSE_RULE_OPTION_COMMAND(Expire) // opt/checksum-page-error PARSE_RULE_OPTION_COMMAND(Restore) // opt/checksum-page-error ), // opt/checksum-page-error // opt/checksum-page-error PARSE_RULE_OPTIONAL // opt/checksum-page-error ( // opt/checksum-page-error PARSE_RULE_OPTIONAL_GROUP // opt/checksum-page-error ( // opt/checksum-page-error PARSE_RULE_FILTER_CMD // opt/checksum-page-error ( // opt/checksum-page-error PARSE_RULE_VAL_CMD(Backup), // opt/checksum-page-error ), // opt/checksum-page-error // opt/checksum-page-error PARSE_RULE_OPTIONAL_DEFAULT // opt/checksum-page-error ( // opt/checksum-page-error PARSE_RULE_VAL_BOOL_FALSE, // opt/checksum-page-error ), // opt/checksum-page-error ), // opt/checksum-page-error // opt/checksum-page-error PARSE_RULE_OPTIONAL_GROUP // opt/checksum-page-error ( // opt/checksum-page-error PARSE_RULE_FILTER_CMD // opt/checksum-page-error ( // opt/checksum-page-error PARSE_RULE_VAL_CMD(Expire), // opt/checksum-page-error PARSE_RULE_VAL_CMD(Restore), // opt/checksum-page-error ), // opt/checksum-page-error // opt/checksum-page-error PARSE_RULE_OPTIONAL_DEFAULT // opt/checksum-page-error ( // opt/checksum-page-error PARSE_RULE_VAL_BOOL_TRUE, // opt/checksum-page-error ), // opt/checksum-page-error ), // opt/checksum-page-error ), // opt/checksum-page-error ), // opt/checksum-page-error // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/cmd ( // opt/cmd PARSE_RULE_OPTION_NAME("cmd"), // opt/cmd PARSE_RULE_OPTION_TYPE(String), // opt/cmd PARSE_RULE_OPTION_DEFAULT_TYPE(Dynamic), // opt/cmd PARSE_RULE_OPTION_RESET(true), // opt/cmd PARSE_RULE_OPTION_REQUIRED(true), // opt/cmd PARSE_RULE_OPTION_SECTION(Global), // opt/cmd // opt/cmd PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/cmd ( // opt/cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/cmd ), // opt/cmd // opt/cmd PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/cmd ( // opt/cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/cmd ), // opt/cmd // opt/cmd PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/cmd ( // opt/cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/cmd ), // opt/cmd // opt/cmd PARSE_RULE_OPTIONAL // opt/cmd ( // opt/cmd PARSE_RULE_OPTIONAL_GROUP // opt/cmd ( // opt/cmd PARSE_RULE_OPTIONAL_DEFAULT // opt/cmd ( // opt/cmd PARSE_RULE_DEFAULT_DYNAMIC(Bin), // opt/cmd ), // opt/cmd ), // opt/cmd ), // opt/cmd ), // opt/cmd // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_OPTION_NAME("cmd-ssh"), // opt/cmd-ssh PARSE_RULE_OPTION_TYPE(String), // opt/cmd-ssh PARSE_RULE_OPTION_RESET(true), // opt/cmd-ssh PARSE_RULE_OPTION_REQUIRED(true), // opt/cmd-ssh PARSE_RULE_OPTION_SECTION(Global), // opt/cmd-ssh // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Annotate) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Backup) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Check) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Expire) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Info) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Manifest) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Restore) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Verify) // opt/cmd-ssh ), // opt/cmd-ssh // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/cmd-ssh ), // opt/cmd-ssh // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Backup) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Restore) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Verify) // opt/cmd-ssh ), // opt/cmd-ssh // opt/cmd-ssh PARSE_RULE_OPTIONAL // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_OPTIONAL_GROUP // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_OPTIONAL_DEFAULT // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_VAL_STR(QT_ssh_QT), // opt/cmd-ssh ), // opt/cmd-ssh ), // opt/cmd-ssh ), // opt/cmd-ssh ), // opt/cmd-ssh // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/compress ( // opt/compress PARSE_RULE_OPTION_NAME("compress"), // opt/compress PARSE_RULE_OPTION_TYPE(Boolean), // opt/compress PARSE_RULE_OPTION_NEGATE(true), // opt/compress PARSE_RULE_OPTION_RESET(true), // opt/compress PARSE_RULE_OPTION_REQUIRED(true), // opt/compress PARSE_RULE_OPTION_SECTION(Global), // opt/compress // opt/compress PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/compress ( // opt/compress PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress PARSE_RULE_OPTION_COMMAND(Backup) // opt/compress ), // opt/compress // opt/compress PARSE_RULE_OPTIONAL // opt/compress ( // opt/compress PARSE_RULE_OPTIONAL_GROUP // opt/compress ( // opt/compress PARSE_RULE_OPTIONAL_DEFAULT // opt/compress ( // opt/compress PARSE_RULE_VAL_BOOL_TRUE, // opt/compress ), // opt/compress ), // opt/compress ), // opt/compress ), // opt/compress // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTION_NAME("compress-level"), // opt/compress-level PARSE_RULE_OPTION_TYPE(Integer), // opt/compress-level PARSE_RULE_OPTION_RESET(true), // opt/compress-level PARSE_RULE_OPTION_REQUIRED(false), // opt/compress-level PARSE_RULE_OPTION_SECTION(Global), // opt/compress-level // opt/compress-level PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-level PARSE_RULE_OPTION_COMMAND(Backup) // opt/compress-level ), // opt/compress-level // opt/compress-level PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-level ), // opt/compress-level // opt/compress-level PARSE_RULE_OPTIONAL // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTIONAL_GROUP // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTIONAL_DEPEND // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_INT(0)), // opt/compress-level PARSE_RULE_VAL_OPT(CompressType), // opt/compress-level PARSE_RULE_VAL_STRID(bz2), // opt/compress-level PARSE_RULE_VAL_STRID(gz), // opt/compress-level PARSE_RULE_VAL_STRID(lz4), // opt/compress-level PARSE_RULE_VAL_STRID(zst), // opt/compress-level ), // opt/compress-level // opt/compress-level PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTIONAL_ALLOW_RANGE_MAP // opt/compress-level ( // opt/compress-level PARSE_RULE_VAL_STRID(bz2), // opt/compress-level PARSE_RULE_VAL_INT(1), // opt/compress-level PARSE_RULE_VAL_INT(9), // opt/compress-level PARSE_RULE_VAL_STRID(gz), // opt/compress-level PARSE_RULE_VAL_INT(DS_1), // opt/compress-level PARSE_RULE_VAL_INT(9), // opt/compress-level PARSE_RULE_VAL_STRID(lz4), // opt/compress-level PARSE_RULE_VAL_INT(DS_5), // opt/compress-level PARSE_RULE_VAL_INT(12), // opt/compress-level PARSE_RULE_VAL_STRID(zst), // opt/compress-level PARSE_RULE_VAL_INT(DS_7), // opt/compress-level PARSE_RULE_VAL_INT(22), // opt/compress-level ), // opt/compress-level ), // opt/compress-level // opt/compress-level PARSE_RULE_OPTIONAL_DEFAULT // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTIONAL_DEFAULT_MAP // opt/compress-level ( // opt/compress-level PARSE_RULE_VAL_STRID(bz2), // opt/compress-level PARSE_RULE_VAL_INT(9), // opt/compress-level PARSE_RULE_VAL_STRID(gz), // opt/compress-level PARSE_RULE_VAL_INT(6), // opt/compress-level PARSE_RULE_VAL_STRID(lz4), // opt/compress-level PARSE_RULE_VAL_INT(1), // opt/compress-level PARSE_RULE_VAL_STRID(zst), // opt/compress-level PARSE_RULE_VAL_INT(3), // opt/compress-level ), // opt/compress-level ), // opt/compress-level ), // opt/compress-level ), // opt/compress-level ), // opt/compress-level // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_OPTION_NAME("compress-level-network"), // opt/compress-level-network PARSE_RULE_OPTION_TYPE(Integer), // opt/compress-level-network PARSE_RULE_OPTION_RESET(true), // opt/compress-level-network PARSE_RULE_OPTION_REQUIRED(true), // opt/compress-level-network PARSE_RULE_OPTION_SECTION(Global), // opt/compress-level-network // opt/compress-level-network PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Annotate) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Backup) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Check) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Expire) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Info) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Manifest) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Restore) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Verify) // opt/compress-level-network ), // opt/compress-level-network // opt/compress-level-network PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-level-network ), // opt/compress-level-network // opt/compress-level-network PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Backup) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Restore) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Verify) // opt/compress-level-network ), // opt/compress-level-network // opt/compress-level-network PARSE_RULE_OPTIONAL // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_OPTIONAL_GROUP // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_VAL_INT(DS_5), // opt/compress-level-network PARSE_RULE_VAL_INT(12), // opt/compress-level-network ), // opt/compress-level-network // opt/compress-level-network PARSE_RULE_OPTIONAL_DEFAULT // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_VAL_INT(1), // opt/compress-level-network ), // opt/compress-level-network ), // opt/compress-level-network ), // opt/compress-level-network ), // opt/compress-level-network // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/compress-type ( // opt/compress-type PARSE_RULE_OPTION_NAME("compress-type"), // opt/compress-type PARSE_RULE_OPTION_TYPE(StringId), // opt/compress-type PARSE_RULE_OPTION_RESET(true), // opt/compress-type PARSE_RULE_OPTION_REQUIRED(true), // opt/compress-type PARSE_RULE_OPTION_SECTION(Global), // opt/compress-type // opt/compress-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/compress-type ( // opt/compress-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/compress-type ), // opt/compress-type // opt/compress-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/compress-type ( // opt/compress-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-type ), // opt/compress-type // opt/compress-type PARSE_RULE_OPTIONAL // opt/compress-type ( // opt/compress-type PARSE_RULE_OPTIONAL_GROUP // opt/compress-type ( // opt/compress-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/compress-type ( // opt/compress-type PARSE_RULE_VAL_STRID(none), // opt/compress-type PARSE_RULE_VAL_STRID(bz2), // opt/compress-type PARSE_RULE_VAL_STRID(gz), // opt/compress-type PARSE_RULE_VAL_STRID(lz4), // opt/compress-type PARSE_RULE_VAL_STRID(zst), // opt/compress-type #ifndef HAVE_LIBZST // opt/compress-type PARSE_RULE_BOOL_FALSE, // opt/compress-type #endif // opt/compress-type ), // opt/compress-type // opt/compress-type PARSE_RULE_OPTIONAL_DEFAULT // opt/compress-type ( // opt/compress-type PARSE_RULE_VAL_STRID(gz), // opt/compress-type ), // opt/compress-type ), // opt/compress-type ), // opt/compress-type ), // opt/compress-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/config ( // opt/config PARSE_RULE_OPTION_NAME("config"), // opt/config PARSE_RULE_OPTION_TYPE(String), // opt/config PARSE_RULE_OPTION_NEGATE(true), // opt/config PARSE_RULE_OPTION_REQUIRED(true), // opt/config PARSE_RULE_OPTION_SECTION(CommandLine), // opt/config // opt/config PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/config ( // opt/config PARSE_RULE_OPTION_COMMAND(Annotate) // opt/config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config PARSE_RULE_OPTION_COMMAND(Backup) // opt/config PARSE_RULE_OPTION_COMMAND(Check) // opt/config PARSE_RULE_OPTION_COMMAND(Expire) // opt/config PARSE_RULE_OPTION_COMMAND(Info) // opt/config PARSE_RULE_OPTION_COMMAND(Manifest) // opt/config PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/config PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/config PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/config PARSE_RULE_OPTION_COMMAND(Restore) // opt/config PARSE_RULE_OPTION_COMMAND(Server) // opt/config PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/config PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/config PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/config PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/config PARSE_RULE_OPTION_COMMAND(Start) // opt/config PARSE_RULE_OPTION_COMMAND(Stop) // opt/config PARSE_RULE_OPTION_COMMAND(Verify) // opt/config ), // opt/config // opt/config PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/config ( // opt/config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config ), // opt/config // opt/config PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/config ( // opt/config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config PARSE_RULE_OPTION_COMMAND(Backup) // opt/config PARSE_RULE_OPTION_COMMAND(Restore) // opt/config PARSE_RULE_OPTION_COMMAND(Verify) // opt/config ), // opt/config // opt/config PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/config ( // opt/config PARSE_RULE_OPTION_COMMAND(Annotate) // opt/config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config PARSE_RULE_OPTION_COMMAND(Backup) // opt/config PARSE_RULE_OPTION_COMMAND(Check) // opt/config PARSE_RULE_OPTION_COMMAND(Expire) // opt/config PARSE_RULE_OPTION_COMMAND(Info) // opt/config PARSE_RULE_OPTION_COMMAND(Manifest) // opt/config PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/config PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/config PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/config PARSE_RULE_OPTION_COMMAND(Restore) // opt/config PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/config PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/config PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/config PARSE_RULE_OPTION_COMMAND(Verify) // opt/config ), // opt/config // opt/config PARSE_RULE_OPTIONAL // opt/config ( // opt/config PARSE_RULE_OPTIONAL_GROUP // opt/config ( // opt/config PARSE_RULE_OPTIONAL_DEFAULT // opt/config ( // opt/config PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_FILE), // opt/config ), // opt/config ), // opt/config ), // opt/config ), // opt/config // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTION_NAME("config-include-path"), // opt/config-include-path PARSE_RULE_OPTION_TYPE(Path), // opt/config-include-path PARSE_RULE_OPTION_REQUIRED(true), // opt/config-include-path PARSE_RULE_OPTION_SECTION(CommandLine), // opt/config-include-path // opt/config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Check) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Info) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Server) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Start) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Stop) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/config-include-path ), // opt/config-include-path // opt/config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-include-path ), // opt/config-include-path // opt/config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/config-include-path ), // opt/config-include-path // opt/config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Check) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Info) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/config-include-path ), // opt/config-include-path // opt/config-include-path PARSE_RULE_OPTIONAL // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTIONAL_GROUP // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTIONAL_DEFAULT // opt/config-include-path ( // opt/config-include-path PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_INCLUDE_PATH), // opt/config-include-path ), // opt/config-include-path ), // opt/config-include-path ), // opt/config-include-path ), // opt/config-include-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/config-path ( // opt/config-path PARSE_RULE_OPTION_NAME("config-path"), // opt/config-path PARSE_RULE_OPTION_TYPE(Path), // opt/config-path PARSE_RULE_OPTION_REQUIRED(true), // opt/config-path PARSE_RULE_OPTION_SECTION(CommandLine), // opt/config-path // opt/config-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/config-path ( // opt/config-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/config-path PARSE_RULE_OPTION_COMMAND(Check) // opt/config-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/config-path PARSE_RULE_OPTION_COMMAND(Info) // opt/config-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/config-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/config-path PARSE_RULE_OPTION_COMMAND(Server) // opt/config-path PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/config-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/config-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/config-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/config-path PARSE_RULE_OPTION_COMMAND(Start) // opt/config-path PARSE_RULE_OPTION_COMMAND(Stop) // opt/config-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/config-path ), // opt/config-path // opt/config-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/config-path ( // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-path ), // opt/config-path // opt/config-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/config-path ( // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/config-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/config-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/config-path ), // opt/config-path // opt/config-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/config-path ( // opt/config-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/config-path PARSE_RULE_OPTION_COMMAND(Check) // opt/config-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/config-path PARSE_RULE_OPTION_COMMAND(Info) // opt/config-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/config-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/config-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/config-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/config-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/config-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/config-path ), // opt/config-path // opt/config-path PARSE_RULE_OPTIONAL // opt/config-path ( // opt/config-path PARSE_RULE_OPTIONAL_GROUP // opt/config-path ( // opt/config-path PARSE_RULE_OPTIONAL_DEFAULT // opt/config-path ( // opt/config-path PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH), // opt/config-path ), // opt/config-path ), // opt/config-path ), // opt/config-path ), // opt/config-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/db-exclude ( // opt/db-exclude PARSE_RULE_OPTION_NAME("db-exclude"), // opt/db-exclude PARSE_RULE_OPTION_TYPE(List), // opt/db-exclude PARSE_RULE_OPTION_RESET(true), // opt/db-exclude PARSE_RULE_OPTION_REQUIRED(false), // opt/db-exclude PARSE_RULE_OPTION_SECTION(Global), // opt/db-exclude PARSE_RULE_OPTION_MULTI(true), // opt/db-exclude // opt/db-exclude PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/db-exclude ( // opt/db-exclude PARSE_RULE_OPTION_COMMAND(Restore) // opt/db-exclude ), // opt/db-exclude ), // opt/db-exclude // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/db-include ( // opt/db-include PARSE_RULE_OPTION_NAME("db-include"), // opt/db-include PARSE_RULE_OPTION_TYPE(List), // opt/db-include PARSE_RULE_OPTION_RESET(true), // opt/db-include PARSE_RULE_OPTION_REQUIRED(false), // opt/db-include PARSE_RULE_OPTION_SECTION(Global), // opt/db-include PARSE_RULE_OPTION_MULTI(true), // opt/db-include // opt/db-include PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/db-include ( // opt/db-include PARSE_RULE_OPTION_COMMAND(Restore) // opt/db-include ), // opt/db-include ), // opt/db-include // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/db-timeout ( // opt/db-timeout PARSE_RULE_OPTION_NAME("db-timeout"), // opt/db-timeout PARSE_RULE_OPTION_TYPE(Time), // opt/db-timeout PARSE_RULE_OPTION_RESET(true), // opt/db-timeout PARSE_RULE_OPTION_REQUIRED(true), // opt/db-timeout PARSE_RULE_OPTION_SECTION(Global), // opt/db-timeout // opt/db-timeout PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/db-timeout ( // opt/db-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/db-timeout ), // opt/db-timeout // opt/db-timeout PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/db-timeout ( // opt/db-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/db-timeout ), // opt/db-timeout // opt/db-timeout PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/db-timeout ( // opt/db-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/db-timeout ), // opt/db-timeout // opt/db-timeout PARSE_RULE_OPTIONAL // opt/db-timeout ( // opt/db-timeout PARSE_RULE_OPTIONAL_GROUP // opt/db-timeout ( // opt/db-timeout PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/db-timeout ( // opt/db-timeout PARSE_RULE_VAL_TIME(100ms), // opt/db-timeout PARSE_RULE_VAL_TIME(7d), // opt/db-timeout ), // opt/db-timeout // opt/db-timeout PARSE_RULE_OPTIONAL_DEFAULT // opt/db-timeout ( // opt/db-timeout PARSE_RULE_VAL_TIME(30m), // opt/db-timeout ), // opt/db-timeout ), // opt/db-timeout ), // opt/db-timeout ), // opt/db-timeout // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/delta ( // opt/delta PARSE_RULE_OPTION_NAME("delta"), // opt/delta PARSE_RULE_OPTION_TYPE(Boolean), // opt/delta PARSE_RULE_OPTION_NEGATE(true), // opt/delta PARSE_RULE_OPTION_RESET(true), // opt/delta PARSE_RULE_OPTION_REQUIRED(true), // opt/delta PARSE_RULE_OPTION_SECTION(Global), // opt/delta // opt/delta PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/delta ( // opt/delta PARSE_RULE_OPTION_COMMAND(Backup) // opt/delta PARSE_RULE_OPTION_COMMAND(Restore) // opt/delta ), // opt/delta // opt/delta PARSE_RULE_OPTIONAL // opt/delta ( // opt/delta PARSE_RULE_OPTIONAL_GROUP // opt/delta ( // opt/delta PARSE_RULE_OPTIONAL_DEFAULT // opt/delta ( // opt/delta PARSE_RULE_VAL_BOOL_FALSE, // opt/delta ), // opt/delta ), // opt/delta ), // opt/delta ), // opt/delta // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/detail-level ( // opt/detail-level PARSE_RULE_OPTION_NAME("detail-level"), // opt/detail-level PARSE_RULE_OPTION_TYPE(StringId), // opt/detail-level PARSE_RULE_OPTION_SEQUENCE(true), // opt/detail-level PARSE_RULE_OPTION_REQUIRED(true), // opt/detail-level PARSE_RULE_OPTION_SECTION(CommandLine), // opt/detail-level // opt/detail-level PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/detail-level ( // opt/detail-level PARSE_RULE_OPTION_COMMAND(Info) // opt/detail-level ), // opt/detail-level // opt/detail-level PARSE_RULE_OPTIONAL // opt/detail-level ( // opt/detail-level PARSE_RULE_OPTIONAL_GROUP // opt/detail-level ( // opt/detail-level PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/detail-level ( // opt/detail-level PARSE_RULE_VAL_STRID(full), // opt/detail-level PARSE_RULE_VAL_STRID(progress), // opt/detail-level ), // opt/detail-level // opt/detail-level PARSE_RULE_OPTIONAL_DEFAULT // opt/detail-level ( // opt/detail-level PARSE_RULE_VAL_STRID(full), // opt/detail-level ), // opt/detail-level ), // opt/detail-level ), // opt/detail-level ), // opt/detail-level // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/dry-run ( // opt/dry-run PARSE_RULE_OPTION_NAME("dry-run"), // opt/dry-run PARSE_RULE_OPTION_TYPE(Boolean), // opt/dry-run PARSE_RULE_OPTION_REQUIRED(true), // opt/dry-run PARSE_RULE_OPTION_SECTION(CommandLine), // opt/dry-run // opt/dry-run PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/dry-run ( // opt/dry-run PARSE_RULE_OPTION_COMMAND(Expire) // opt/dry-run ), // opt/dry-run // opt/dry-run PARSE_RULE_OPTIONAL // opt/dry-run ( // opt/dry-run PARSE_RULE_OPTIONAL_GROUP // opt/dry-run ( // opt/dry-run PARSE_RULE_OPTIONAL_DEFAULT // opt/dry-run ( // opt/dry-run PARSE_RULE_VAL_BOOL_FALSE, // opt/dry-run ), // opt/dry-run ), // opt/dry-run ), // opt/dry-run ), // opt/dry-run // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/exclude ( // opt/exclude PARSE_RULE_OPTION_NAME("exclude"), // opt/exclude PARSE_RULE_OPTION_TYPE(List), // opt/exclude PARSE_RULE_OPTION_RESET(true), // opt/exclude PARSE_RULE_OPTION_REQUIRED(false), // opt/exclude PARSE_RULE_OPTION_SECTION(Global), // opt/exclude PARSE_RULE_OPTION_MULTI(true), // opt/exclude // opt/exclude PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/exclude ( // opt/exclude PARSE_RULE_OPTION_COMMAND(Backup) // opt/exclude ), // opt/exclude ), // opt/exclude // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/exec-id ( // opt/exec-id PARSE_RULE_OPTION_NAME("exec-id"), // opt/exec-id PARSE_RULE_OPTION_TYPE(String), // opt/exec-id PARSE_RULE_OPTION_INTERNAL(true) // opt/exec-id PARSE_RULE_OPTION_REQUIRED(false), // opt/exec-id PARSE_RULE_OPTION_SECTION(CommandLine), // opt/exec-id // opt/exec-id PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/exec-id ( // opt/exec-id PARSE_RULE_OPTION_COMMAND(Annotate) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Backup) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Check) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Expire) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Info) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Manifest) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Restore) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Server) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/exec-id PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/exec-id PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/exec-id PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Start) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Stop) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Verify) // opt/exec-id ), // opt/exec-id // opt/exec-id PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/exec-id ( // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/exec-id ), // opt/exec-id // opt/exec-id PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/exec-id ( // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Backup) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Restore) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Verify) // opt/exec-id ), // opt/exec-id // opt/exec-id PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/exec-id ( // opt/exec-id PARSE_RULE_OPTION_COMMAND(Annotate) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Backup) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Check) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Expire) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Info) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Manifest) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Restore) // opt/exec-id PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/exec-id PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/exec-id PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Verify) // opt/exec-id ), // opt/exec-id ), // opt/exec-id // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/expire-auto ( // opt/expire-auto PARSE_RULE_OPTION_NAME("expire-auto"), // opt/expire-auto PARSE_RULE_OPTION_TYPE(Boolean), // opt/expire-auto PARSE_RULE_OPTION_NEGATE(true), // opt/expire-auto PARSE_RULE_OPTION_RESET(true), // opt/expire-auto PARSE_RULE_OPTION_REQUIRED(true), // opt/expire-auto PARSE_RULE_OPTION_SECTION(Global), // opt/expire-auto // opt/expire-auto PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/expire-auto ( // opt/expire-auto PARSE_RULE_OPTION_COMMAND(Backup) // opt/expire-auto ), // opt/expire-auto // opt/expire-auto PARSE_RULE_OPTIONAL // opt/expire-auto ( // opt/expire-auto PARSE_RULE_OPTIONAL_GROUP // opt/expire-auto ( // opt/expire-auto PARSE_RULE_OPTIONAL_DEFAULT // opt/expire-auto ( // opt/expire-auto PARSE_RULE_VAL_BOOL_TRUE, // opt/expire-auto ), // opt/expire-auto ), // opt/expire-auto ), // opt/expire-auto ), // opt/expire-auto // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/filter ( // opt/filter PARSE_RULE_OPTION_NAME("filter"), // opt/filter PARSE_RULE_OPTION_TYPE(String), // opt/filter PARSE_RULE_OPTION_REQUIRED(false), // opt/filter PARSE_RULE_OPTION_SECTION(CommandLine), // opt/filter // opt/filter PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/filter ( // opt/filter PARSE_RULE_OPTION_COMMAND(Manifest) // opt/filter PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/filter ), // opt/filter ), // opt/filter // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/force ( // opt/force PARSE_RULE_OPTION_NAME("force"), // opt/force PARSE_RULE_OPTION_TYPE(Boolean), // opt/force PARSE_RULE_OPTION_REQUIRED(true), // opt/force PARSE_RULE_OPTION_SECTION(CommandLine), // opt/force // opt/force PARSE_RULE_OPTION_COMMAND_INTERNAL_LIST // opt/force ( // opt/force PARSE_RULE_OPTION_COMMAND_INTERNAL(StanzaCreate, true), // opt/force ) // opt/force // opt/force PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/force ( // opt/force PARSE_RULE_OPTION_COMMAND(Backup) // opt/force PARSE_RULE_OPTION_COMMAND(Restore) // opt/force PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/force PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/force PARSE_RULE_OPTION_COMMAND(Stop) // opt/force ), // opt/force // opt/force PARSE_RULE_OPTIONAL // opt/force ( // opt/force PARSE_RULE_OPTIONAL_GROUP // opt/force ( // opt/force PARSE_RULE_FILTER_CMD // opt/force ( // opt/force PARSE_RULE_VAL_CMD(Backup), // opt/force ), // opt/force // opt/force PARSE_RULE_OPTIONAL_DEPEND // opt/force ( // opt/force PARSE_RULE_VAL_OPT(Online), // opt/force PARSE_RULE_VAL_BOOL_FALSE, // opt/force ), // opt/force // opt/force PARSE_RULE_OPTIONAL_DEFAULT // opt/force ( // opt/force PARSE_RULE_VAL_BOOL_FALSE, // opt/force ), // opt/force ), // opt/force // opt/force PARSE_RULE_OPTIONAL_GROUP // opt/force ( // opt/force PARSE_RULE_OPTIONAL_DEFAULT // opt/force ( // opt/force PARSE_RULE_VAL_BOOL_FALSE, // opt/force ), // opt/force ), // opt/force ), // opt/force ), // opt/force // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/help ( // opt/help PARSE_RULE_OPTION_NAME("help"), // opt/help PARSE_RULE_OPTION_TYPE(Boolean), // opt/help PARSE_RULE_OPTION_REQUIRED(true), // opt/help PARSE_RULE_OPTION_SECTION(CommandLine), // opt/help // opt/help PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/help ( // opt/help PARSE_RULE_OPTION_COMMAND(Help) // opt/help ), // opt/help // opt/help PARSE_RULE_OPTIONAL // opt/help ( // opt/help PARSE_RULE_OPTIONAL_GROUP // opt/help ( // opt/help PARSE_RULE_OPTIONAL_DEFAULT // opt/help ( // opt/help PARSE_RULE_VAL_BOOL_FALSE, // opt/help ), // opt/help ), // opt/help ), // opt/help ), // opt/help // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/ignore-missing ( // opt/ignore-missing PARSE_RULE_OPTION_NAME("ignore-missing"), // opt/ignore-missing PARSE_RULE_OPTION_TYPE(Boolean), // opt/ignore-missing PARSE_RULE_OPTION_REQUIRED(true), // opt/ignore-missing PARSE_RULE_OPTION_SECTION(CommandLine), // opt/ignore-missing // opt/ignore-missing PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/ignore-missing ( // opt/ignore-missing PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/ignore-missing ), // opt/ignore-missing // opt/ignore-missing PARSE_RULE_OPTIONAL // opt/ignore-missing ( // opt/ignore-missing PARSE_RULE_OPTIONAL_GROUP // opt/ignore-missing ( // opt/ignore-missing PARSE_RULE_OPTIONAL_DEFAULT // opt/ignore-missing ( // opt/ignore-missing PARSE_RULE_VAL_BOOL_FALSE, // opt/ignore-missing ), // opt/ignore-missing ), // opt/ignore-missing ), // opt/ignore-missing ), // opt/ignore-missing // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTION_NAME("io-timeout"), // opt/io-timeout PARSE_RULE_OPTION_TYPE(Time), // opt/io-timeout PARSE_RULE_OPTION_RESET(true), // opt/io-timeout PARSE_RULE_OPTION_REQUIRED(true), // opt/io-timeout PARSE_RULE_OPTION_SECTION(Global), // opt/io-timeout // opt/io-timeout PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Annotate) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Expire) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Info) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Manifest) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Restore) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Server) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Verify) // opt/io-timeout ), // opt/io-timeout // opt/io-timeout PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/io-timeout ), // opt/io-timeout // opt/io-timeout PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Restore) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Verify) // opt/io-timeout ), // opt/io-timeout // opt/io-timeout PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Annotate) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Expire) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Info) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Manifest) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Restore) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Verify) // opt/io-timeout ), // opt/io-timeout // opt/io-timeout PARSE_RULE_OPTIONAL // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTIONAL_GROUP // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/io-timeout ( // opt/io-timeout PARSE_RULE_VAL_TIME(100ms), // opt/io-timeout PARSE_RULE_VAL_TIME(1h), // opt/io-timeout ), // opt/io-timeout // opt/io-timeout PARSE_RULE_OPTIONAL_DEFAULT // opt/io-timeout ( // opt/io-timeout PARSE_RULE_VAL_TIME(1m), // opt/io-timeout ), // opt/io-timeout ), // opt/io-timeout ), // opt/io-timeout ), // opt/io-timeout // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/job-retry ( // opt/job-retry PARSE_RULE_OPTION_NAME("job-retry"), // opt/job-retry PARSE_RULE_OPTION_TYPE(Integer), // opt/job-retry PARSE_RULE_OPTION_INTERNAL(true) // opt/job-retry PARSE_RULE_OPTION_RESET(true), // opt/job-retry PARSE_RULE_OPTION_REQUIRED(true), // opt/job-retry PARSE_RULE_OPTION_SECTION(Global), // opt/job-retry // opt/job-retry PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/job-retry ( // opt/job-retry PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/job-retry PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/job-retry PARSE_RULE_OPTION_COMMAND(Backup) // opt/job-retry PARSE_RULE_OPTION_COMMAND(Restore) // opt/job-retry PARSE_RULE_OPTION_COMMAND(Verify) // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/job-retry ( // opt/job-retry PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/job-retry PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/job-retry ( // opt/job-retry PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/job-retry PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/job-retry PARSE_RULE_OPTION_COMMAND(Backup) // opt/job-retry PARSE_RULE_OPTION_COMMAND(Restore) // opt/job-retry PARSE_RULE_OPTION_COMMAND(Verify) // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTIONAL // opt/job-retry ( // opt/job-retry PARSE_RULE_OPTIONAL_GROUP // opt/job-retry ( // opt/job-retry PARSE_RULE_FILTER_CMD // opt/job-retry ( // opt/job-retry PARSE_RULE_VAL_CMD(ArchiveGet), // opt/job-retry PARSE_RULE_VAL_CMD(ArchivePush), // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/job-retry ( // opt/job-retry PARSE_RULE_VAL_INT(0), // opt/job-retry PARSE_RULE_VAL_INT(360), // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTIONAL_DEFAULT // opt/job-retry ( // opt/job-retry PARSE_RULE_VAL_INT(1), // opt/job-retry ), // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTIONAL_GROUP // opt/job-retry ( // opt/job-retry PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/job-retry ( // opt/job-retry PARSE_RULE_VAL_INT(0), // opt/job-retry PARSE_RULE_VAL_INT(360), // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTIONAL_DEFAULT // opt/job-retry ( // opt/job-retry PARSE_RULE_VAL_INT(2), // opt/job-retry ), // opt/job-retry ), // opt/job-retry ), // opt/job-retry ), // opt/job-retry // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_OPTION_NAME("job-retry-interval"), // opt/job-retry-interval PARSE_RULE_OPTION_TYPE(Time), // opt/job-retry-interval PARSE_RULE_OPTION_INTERNAL(true) // opt/job-retry-interval PARSE_RULE_OPTION_RESET(true), // opt/job-retry-interval PARSE_RULE_OPTION_REQUIRED(true), // opt/job-retry-interval PARSE_RULE_OPTION_SECTION(Global), // opt/job-retry-interval // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(Backup) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(Restore) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(Verify) // opt/job-retry-interval ), // opt/job-retry-interval // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/job-retry-interval ), // opt/job-retry-interval // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(Backup) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(Restore) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(Verify) // opt/job-retry-interval ), // opt/job-retry-interval // opt/job-retry-interval PARSE_RULE_OPTIONAL // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_OPTIONAL_GROUP // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_VAL_TIME(0s), // opt/job-retry-interval PARSE_RULE_VAL_TIME(15m), // opt/job-retry-interval ), // opt/job-retry-interval // opt/job-retry-interval PARSE_RULE_OPTIONAL_DEFAULT // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_VAL_TIME(15s), // opt/job-retry-interval ), // opt/job-retry-interval ), // opt/job-retry-interval ), // opt/job-retry-interval ), // opt/job-retry-interval // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/link-all ( // opt/link-all PARSE_RULE_OPTION_NAME("link-all"), // opt/link-all PARSE_RULE_OPTION_TYPE(Boolean), // opt/link-all PARSE_RULE_OPTION_NEGATE(true), // opt/link-all PARSE_RULE_OPTION_RESET(true), // opt/link-all PARSE_RULE_OPTION_REQUIRED(true), // opt/link-all PARSE_RULE_OPTION_SECTION(Global), // opt/link-all // opt/link-all PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/link-all ( // opt/link-all PARSE_RULE_OPTION_COMMAND(Restore) // opt/link-all ), // opt/link-all // opt/link-all PARSE_RULE_OPTIONAL // opt/link-all ( // opt/link-all PARSE_RULE_OPTIONAL_GROUP // opt/link-all ( // opt/link-all PARSE_RULE_OPTIONAL_DEFAULT // opt/link-all ( // opt/link-all PARSE_RULE_VAL_BOOL_FALSE, // opt/link-all ), // opt/link-all ), // opt/link-all ), // opt/link-all ), // opt/link-all // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/link-map ( // opt/link-map PARSE_RULE_OPTION_NAME("link-map"), // opt/link-map PARSE_RULE_OPTION_TYPE(Hash), // opt/link-map PARSE_RULE_OPTION_RESET(true), // opt/link-map PARSE_RULE_OPTION_REQUIRED(false), // opt/link-map PARSE_RULE_OPTION_SECTION(Global), // opt/link-map PARSE_RULE_OPTION_MULTI(true), // opt/link-map // opt/link-map PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/link-map ( // opt/link-map PARSE_RULE_OPTION_COMMAND(Restore) // opt/link-map ), // opt/link-map ), // opt/link-map // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/lock ( // opt/lock PARSE_RULE_OPTION_NAME("lock"), // opt/lock PARSE_RULE_OPTION_TYPE(List), // opt/lock PARSE_RULE_OPTION_INTERNAL(true) // opt/lock PARSE_RULE_OPTION_REQUIRED(false), // opt/lock PARSE_RULE_OPTION_SECTION(CommandLine), // opt/lock PARSE_RULE_OPTION_MULTI(true), // opt/lock // opt/lock PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/lock ( // opt/lock PARSE_RULE_OPTION_COMMAND(Annotate) // opt/lock PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/lock PARSE_RULE_OPTION_COMMAND(Backup) // opt/lock PARSE_RULE_OPTION_COMMAND(Expire) // opt/lock PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/lock PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/lock PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/lock ), // opt/lock ), // opt/lock // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTION_NAME("lock-path"), // opt/lock-path PARSE_RULE_OPTION_TYPE(Path), // opt/lock-path PARSE_RULE_OPTION_RESET(true), // opt/lock-path PARSE_RULE_OPTION_REQUIRED(true), // opt/lock-path PARSE_RULE_OPTION_SECTION(Global), // opt/lock-path // opt/lock-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Info) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/lock-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/lock-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/lock-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Start) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Stop) // opt/lock-path ), // opt/lock-path // opt/lock-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/lock-path ), // opt/lock-path // opt/lock-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/lock-path ), // opt/lock-path // opt/lock-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Info) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/lock-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/lock-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/lock-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/lock-path ), // opt/lock-path // opt/lock-path PARSE_RULE_OPTIONAL // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTIONAL_GROUP // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTIONAL_DEFAULT // opt/lock-path ( // opt/lock-path PARSE_RULE_VAL_STR(QT_FS_tmp_FS_pgbackrest_QT), // opt/lock-path ), // opt/lock-path ), // opt/lock-path ), // opt/lock-path ), // opt/lock-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTION_NAME("log-level-console"), // opt/log-level-console PARSE_RULE_OPTION_TYPE(StringId), // opt/log-level-console PARSE_RULE_OPTION_SEQUENCE(true), // opt/log-level-console PARSE_RULE_OPTION_RESET(true), // opt/log-level-console PARSE_RULE_OPTION_REQUIRED(true), // opt/log-level-console PARSE_RULE_OPTION_SECTION(Global), // opt/log-level-console // opt/log-level-console PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Check) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Info) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Server) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Start) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Stop) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-console ), // opt/log-level-console // opt/log-level-console PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-console ), // opt/log-level-console // opt/log-level-console PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-console ), // opt/log-level-console // opt/log-level-console PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Check) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Info) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-console ), // opt/log-level-console // opt/log-level-console PARSE_RULE_OPTIONAL // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTIONAL_GROUP // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/log-level-console ( // opt/log-level-console PARSE_RULE_VAL_STRID(off), // opt/log-level-console PARSE_RULE_VAL_STRID(error), // opt/log-level-console PARSE_RULE_VAL_STRID(warn), // opt/log-level-console PARSE_RULE_VAL_STRID(info), // opt/log-level-console PARSE_RULE_VAL_STRID(detail), // opt/log-level-console PARSE_RULE_VAL_STRID(debug), // opt/log-level-console PARSE_RULE_VAL_STRID(trace), // opt/log-level-console ), // opt/log-level-console // opt/log-level-console PARSE_RULE_OPTIONAL_DEFAULT // opt/log-level-console ( // opt/log-level-console PARSE_RULE_VAL_STRID(warn), // opt/log-level-console PARSE_RULE_VAL_SEQ(2), // opt/log-level-console ), // opt/log-level-console ), // opt/log-level-console ), // opt/log-level-console ), // opt/log-level-console // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTION_NAME("log-level-file"), // opt/log-level-file PARSE_RULE_OPTION_TYPE(StringId), // opt/log-level-file PARSE_RULE_OPTION_SEQUENCE(true), // opt/log-level-file PARSE_RULE_OPTION_RESET(true), // opt/log-level-file PARSE_RULE_OPTION_REQUIRED(true), // opt/log-level-file PARSE_RULE_OPTION_SECTION(Global), // opt/log-level-file // opt/log-level-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Check) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Info) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Server) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Start) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Stop) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-file ), // opt/log-level-file // opt/log-level-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-file ), // opt/log-level-file // opt/log-level-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-file ), // opt/log-level-file // opt/log-level-file PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Check) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Info) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-file ), // opt/log-level-file // opt/log-level-file PARSE_RULE_OPTIONAL // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTIONAL_GROUP // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/log-level-file ( // opt/log-level-file PARSE_RULE_VAL_STRID(off), // opt/log-level-file PARSE_RULE_VAL_STRID(error), // opt/log-level-file PARSE_RULE_VAL_STRID(warn), // opt/log-level-file PARSE_RULE_VAL_STRID(info), // opt/log-level-file PARSE_RULE_VAL_STRID(detail), // opt/log-level-file PARSE_RULE_VAL_STRID(debug), // opt/log-level-file PARSE_RULE_VAL_STRID(trace), // opt/log-level-file ), // opt/log-level-file // opt/log-level-file PARSE_RULE_OPTIONAL_DEFAULT // opt/log-level-file ( // opt/log-level-file PARSE_RULE_VAL_STRID(info), // opt/log-level-file PARSE_RULE_VAL_SEQ(3), // opt/log-level-file ), // opt/log-level-file ), // opt/log-level-file ), // opt/log-level-file ), // opt/log-level-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTION_NAME("log-level-stderr"), // opt/log-level-stderr PARSE_RULE_OPTION_TYPE(StringId), // opt/log-level-stderr PARSE_RULE_OPTION_SEQUENCE(true), // opt/log-level-stderr PARSE_RULE_OPTION_RESET(true), // opt/log-level-stderr PARSE_RULE_OPTION_REQUIRED(true), // opt/log-level-stderr PARSE_RULE_OPTION_SECTION(Global), // opt/log-level-stderr // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Check) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Info) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Server) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Start) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Stop) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-stderr ), // opt/log-level-stderr // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-stderr ), // opt/log-level-stderr // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-stderr ), // opt/log-level-stderr // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Check) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Info) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-stderr ), // opt/log-level-stderr // opt/log-level-stderr PARSE_RULE_OPTIONAL // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTIONAL_GROUP // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_VAL_STRID(off), // opt/log-level-stderr PARSE_RULE_VAL_STRID(error), // opt/log-level-stderr PARSE_RULE_VAL_STRID(warn), // opt/log-level-stderr PARSE_RULE_VAL_STRID(info), // opt/log-level-stderr PARSE_RULE_VAL_STRID(detail), // opt/log-level-stderr PARSE_RULE_VAL_STRID(debug), // opt/log-level-stderr PARSE_RULE_VAL_STRID(trace), // opt/log-level-stderr ), // opt/log-level-stderr // opt/log-level-stderr PARSE_RULE_OPTIONAL_DEFAULT // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_VAL_STRID(off), // opt/log-level-stderr ), // opt/log-level-stderr ), // opt/log-level-stderr ), // opt/log-level-stderr ), // opt/log-level-stderr // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/log-path ( // opt/log-path PARSE_RULE_OPTION_NAME("log-path"), // opt/log-path PARSE_RULE_OPTION_TYPE(Path), // opt/log-path PARSE_RULE_OPTION_RESET(true), // opt/log-path PARSE_RULE_OPTION_REQUIRED(true), // opt/log-path PARSE_RULE_OPTION_SECTION(Global), // opt/log-path // opt/log-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/log-path ( // opt/log-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-path PARSE_RULE_OPTION_COMMAND(Check) // opt/log-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-path PARSE_RULE_OPTION_COMMAND(Info) // opt/log-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-path PARSE_RULE_OPTION_COMMAND(Server) // opt/log-path PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/log-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-path PARSE_RULE_OPTION_COMMAND(Start) // opt/log-path PARSE_RULE_OPTION_COMMAND(Stop) // opt/log-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-path ), // opt/log-path // opt/log-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/log-path ( // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-path ), // opt/log-path // opt/log-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/log-path ( // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-path ), // opt/log-path // opt/log-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/log-path ( // opt/log-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-path PARSE_RULE_OPTION_COMMAND(Check) // opt/log-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-path PARSE_RULE_OPTION_COMMAND(Info) // opt/log-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-path ), // opt/log-path // opt/log-path PARSE_RULE_OPTIONAL // opt/log-path ( // opt/log-path PARSE_RULE_OPTIONAL_GROUP // opt/log-path ( // opt/log-path PARSE_RULE_OPTIONAL_DEFAULT // opt/log-path ( // opt/log-path PARSE_RULE_VAL_STR(QT_FS_var_FS_log_FS_pgbackrest_QT), // opt/log-path ), // opt/log-path ), // opt/log-path ), // opt/log-path ), // opt/log-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTION_NAME("log-subprocess"), // opt/log-subprocess PARSE_RULE_OPTION_TYPE(Boolean), // opt/log-subprocess PARSE_RULE_OPTION_NEGATE(true), // opt/log-subprocess PARSE_RULE_OPTION_RESET(true), // opt/log-subprocess PARSE_RULE_OPTION_REQUIRED(true), // opt/log-subprocess PARSE_RULE_OPTION_SECTION(Global), // opt/log-subprocess // opt/log-subprocess PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Check) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Info) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-subprocess ), // opt/log-subprocess // opt/log-subprocess PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-subprocess ), // opt/log-subprocess // opt/log-subprocess PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-subprocess ), // opt/log-subprocess // opt/log-subprocess PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Check) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Info) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-subprocess ), // opt/log-subprocess // opt/log-subprocess PARSE_RULE_OPTIONAL // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTIONAL_GROUP // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTIONAL_DEFAULT // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_VAL_BOOL_FALSE, // opt/log-subprocess ), // opt/log-subprocess ), // opt/log-subprocess ), // opt/log-subprocess ), // opt/log-subprocess // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTION_NAME("log-timestamp"), // opt/log-timestamp PARSE_RULE_OPTION_TYPE(Boolean), // opt/log-timestamp PARSE_RULE_OPTION_NEGATE(true), // opt/log-timestamp PARSE_RULE_OPTION_RESET(true), // opt/log-timestamp PARSE_RULE_OPTION_REQUIRED(true), // opt/log-timestamp PARSE_RULE_OPTION_SECTION(Global), // opt/log-timestamp // opt/log-timestamp PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Check) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Info) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Server) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Start) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Stop) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-timestamp ), // opt/log-timestamp // opt/log-timestamp PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-timestamp ), // opt/log-timestamp // opt/log-timestamp PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-timestamp ), // opt/log-timestamp // opt/log-timestamp PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Check) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Info) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-timestamp ), // opt/log-timestamp // opt/log-timestamp PARSE_RULE_OPTIONAL // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTIONAL_GROUP // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTIONAL_DEFAULT // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_VAL_BOOL_TRUE, // opt/log-timestamp ), // opt/log-timestamp ), // opt/log-timestamp ), // opt/log-timestamp ), // opt/log-timestamp // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/manifest-save-threshold ( // opt/manifest-save-threshold PARSE_RULE_OPTION_NAME("manifest-save-threshold"), // opt/manifest-save-threshold PARSE_RULE_OPTION_TYPE(Size), // opt/manifest-save-threshold PARSE_RULE_OPTION_RESET(true), // opt/manifest-save-threshold PARSE_RULE_OPTION_REQUIRED(true), // opt/manifest-save-threshold PARSE_RULE_OPTION_SECTION(Global), // opt/manifest-save-threshold // opt/manifest-save-threshold PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/manifest-save-threshold ( // opt/manifest-save-threshold PARSE_RULE_OPTION_COMMAND(Backup) // opt/manifest-save-threshold ), // opt/manifest-save-threshold // opt/manifest-save-threshold PARSE_RULE_OPTIONAL // opt/manifest-save-threshold ( // opt/manifest-save-threshold PARSE_RULE_OPTIONAL_GROUP // opt/manifest-save-threshold ( // opt/manifest-save-threshold PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/manifest-save-threshold ( // opt/manifest-save-threshold PARSE_RULE_VAL_SIZE(1B), // opt/manifest-save-threshold PARSE_RULE_VAL_SIZE(1TiB), // opt/manifest-save-threshold ), // opt/manifest-save-threshold // opt/manifest-save-threshold PARSE_RULE_OPTIONAL_DEFAULT // opt/manifest-save-threshold ( // opt/manifest-save-threshold PARSE_RULE_VAL_SIZE(1GiB), // opt/manifest-save-threshold ), // opt/manifest-save-threshold ), // opt/manifest-save-threshold ), // opt/manifest-save-threshold ), // opt/manifest-save-threshold // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTION_NAME("neutral-umask"), // opt/neutral-umask PARSE_RULE_OPTION_TYPE(Boolean), // opt/neutral-umask PARSE_RULE_OPTION_NEGATE(true), // opt/neutral-umask PARSE_RULE_OPTION_RESET(true), // opt/neutral-umask PARSE_RULE_OPTION_REQUIRED(true), // opt/neutral-umask PARSE_RULE_OPTION_SECTION(Global), // opt/neutral-umask // opt/neutral-umask PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Annotate) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Backup) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Check) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Expire) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Restore) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Start) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Stop) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Verify) // opt/neutral-umask ), // opt/neutral-umask // opt/neutral-umask PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/neutral-umask ), // opt/neutral-umask // opt/neutral-umask PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Backup) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Restore) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Verify) // opt/neutral-umask ), // opt/neutral-umask // opt/neutral-umask PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Annotate) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Backup) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Check) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Expire) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Restore) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Verify) // opt/neutral-umask ), // opt/neutral-umask // opt/neutral-umask PARSE_RULE_OPTIONAL // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTIONAL_GROUP // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTIONAL_DEFAULT // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_VAL_BOOL_TRUE, // opt/neutral-umask ), // opt/neutral-umask ), // opt/neutral-umask ), // opt/neutral-umask ), // opt/neutral-umask // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/oldest ( // opt/oldest PARSE_RULE_OPTION_NAME("oldest"), // opt/oldest PARSE_RULE_OPTION_TYPE(Boolean), // opt/oldest PARSE_RULE_OPTION_REQUIRED(true), // opt/oldest PARSE_RULE_OPTION_SECTION(CommandLine), // opt/oldest // opt/oldest PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/oldest ( // opt/oldest PARSE_RULE_OPTION_COMMAND(Expire) // opt/oldest ), // opt/oldest // opt/oldest PARSE_RULE_OPTIONAL // opt/oldest ( // opt/oldest PARSE_RULE_OPTIONAL_GROUP // opt/oldest ( // opt/oldest PARSE_RULE_FILTER_CMD // opt/oldest ( // opt/oldest PARSE_RULE_VAL_CMD(Expire), // opt/oldest ), // opt/oldest // opt/oldest PARSE_RULE_OPTIONAL_DEPEND // opt/oldest ( // opt/oldest PARSE_RULE_VAL_OPT(Stanza), // opt/oldest ), // opt/oldest // opt/oldest PARSE_RULE_OPTIONAL_DEFAULT // opt/oldest ( // opt/oldest PARSE_RULE_VAL_BOOL_FALSE, // opt/oldest ), // opt/oldest ), // opt/oldest // opt/oldest PARSE_RULE_OPTIONAL_GROUP // opt/oldest ( // opt/oldest PARSE_RULE_OPTIONAL_DEFAULT // opt/oldest ( // opt/oldest PARSE_RULE_VAL_BOOL_FALSE, // opt/oldest ), // opt/oldest ), // opt/oldest ), // opt/oldest ), // opt/oldest // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/online ( // opt/online PARSE_RULE_OPTION_NAME("online"), // opt/online PARSE_RULE_OPTION_TYPE(Boolean), // opt/online PARSE_RULE_OPTION_NEGATE(true), // opt/online PARSE_RULE_OPTION_REQUIRED(true), // opt/online PARSE_RULE_OPTION_SECTION(CommandLine), // opt/online // opt/online PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/online ( // opt/online PARSE_RULE_OPTION_COMMAND(Backup) // opt/online PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/online PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/online ), // opt/online // opt/online PARSE_RULE_OPTIONAL // opt/online ( // opt/online PARSE_RULE_OPTIONAL_GROUP // opt/online ( // opt/online PARSE_RULE_OPTIONAL_DEFAULT // opt/online ( // opt/online PARSE_RULE_VAL_BOOL_TRUE, // opt/online ), // opt/online ), // opt/online ), // opt/online ), // opt/online // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/output ( // opt/output PARSE_RULE_OPTION_NAME("output"), // opt/output PARSE_RULE_OPTION_TYPE(StringId), // opt/output PARSE_RULE_OPTION_SEQUENCE(true), // opt/output PARSE_RULE_OPTION_REQUIRED(true), // opt/output PARSE_RULE_OPTION_SECTION(CommandLine), // opt/output // opt/output PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/output ( // opt/output PARSE_RULE_OPTION_COMMAND(Info) // opt/output PARSE_RULE_OPTION_COMMAND(Manifest) // opt/output PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/output PARSE_RULE_OPTION_COMMAND(Verify) // opt/output PARSE_RULE_OPTION_COMMAND(Version) // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL // opt/output ( // opt/output PARSE_RULE_OPTIONAL_GROUP // opt/output ( // opt/output PARSE_RULE_FILTER_CMD // opt/output ( // opt/output PARSE_RULE_VAL_CMD(Verify), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/output ( // opt/output PARSE_RULE_VAL_STRID(none), // opt/output PARSE_RULE_VAL_STRID(text), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_DEFAULT // opt/output ( // opt/output PARSE_RULE_VAL_STRID(none), // opt/output ), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_GROUP // opt/output ( // opt/output PARSE_RULE_FILTER_CMD // opt/output ( // opt/output PARSE_RULE_VAL_CMD(Version), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/output ( // opt/output PARSE_RULE_VAL_STRID(text), // opt/output PARSE_RULE_VAL_STRID(num), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_DEFAULT // opt/output ( // opt/output PARSE_RULE_VAL_STRID(text), // opt/output ), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_GROUP // opt/output ( // opt/output PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/output ( // opt/output PARSE_RULE_VAL_STRID(text), // opt/output PARSE_RULE_VAL_STRID(json), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_DEFAULT // opt/output ( // opt/output PARSE_RULE_VAL_STRID(text), // opt/output ), // opt/output ), // opt/output ), // opt/output ), // opt/output // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/page-header-check ( // opt/page-header-check PARSE_RULE_OPTION_NAME("page-header-check"), // opt/page-header-check PARSE_RULE_OPTION_TYPE(Boolean), // opt/page-header-check PARSE_RULE_OPTION_NEGATE(true), // opt/page-header-check PARSE_RULE_OPTION_RESET(true), // opt/page-header-check PARSE_RULE_OPTION_REQUIRED(true), // opt/page-header-check PARSE_RULE_OPTION_SECTION(Global), // opt/page-header-check // opt/page-header-check PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/page-header-check ( // opt/page-header-check PARSE_RULE_OPTION_COMMAND(Backup) // opt/page-header-check ), // opt/page-header-check // opt/page-header-check PARSE_RULE_OPTIONAL // opt/page-header-check ( // opt/page-header-check PARSE_RULE_OPTIONAL_GROUP // opt/page-header-check ( // opt/page-header-check PARSE_RULE_OPTIONAL_DEFAULT // opt/page-header-check ( // opt/page-header-check PARSE_RULE_VAL_BOOL_TRUE, // opt/page-header-check ), // opt/page-header-check ), // opt/page-header-check ), // opt/page-header-check ), // opt/page-header-check // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg ( // opt/pg PARSE_RULE_OPTION_NAME("pg"), // opt/pg PARSE_RULE_OPTION_TYPE(Integer), // opt/pg PARSE_RULE_OPTION_INTERNAL(true) // opt/pg PARSE_RULE_OPTION_REQUIRED(false), // opt/pg PARSE_RULE_OPTION_SECTION(CommandLine), // opt/pg // opt/pg PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg ( // opt/pg PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg ), // opt/pg // opt/pg PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg ( // opt/pg PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg ), // opt/pg // opt/pg PARSE_RULE_OPTIONAL // opt/pg ( // opt/pg PARSE_RULE_OPTIONAL_GROUP // opt/pg ( // opt/pg PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/pg ( // opt/pg PARSE_RULE_VAL_INT(1), // opt/pg PARSE_RULE_VAL_INT(256), // opt/pg ), // opt/pg ), // opt/pg ), // opt/pg ), // opt/pg // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-database ( // opt/pg-database PARSE_RULE_OPTION_NAME("pg-database"), // opt/pg-database PARSE_RULE_OPTION_TYPE(String), // opt/pg-database PARSE_RULE_OPTION_RESET(true), // opt/pg-database PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-database PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-database PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-database // opt/pg-database PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-database ( // opt/pg-database PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-database PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-database PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-database PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-database PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-database ), // opt/pg-database // opt/pg-database PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-database ( // opt/pg-database PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-database ), // opt/pg-database // opt/pg-database PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/pg-database ( // opt/pg-database PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-database PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-database PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-database PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-database PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-database ), // opt/pg-database // opt/pg-database PARSE_RULE_OPTIONAL // opt/pg-database ( // opt/pg-database PARSE_RULE_OPTIONAL_GROUP // opt/pg-database ( // opt/pg-database PARSE_RULE_OPTIONAL_DEPEND // opt/pg-database ( // opt/pg-database PARSE_RULE_VAL_OPT(PgPath), // opt/pg-database ), // opt/pg-database // opt/pg-database PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-database ( // opt/pg-database PARSE_RULE_VAL_STR(QT_postgres_QT), // opt/pg-database ), // opt/pg-database ), // opt/pg-database ), // opt/pg-database ), // opt/pg-database // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTION_NAME("pg-host"), // opt/pg-host PARSE_RULE_OPTION_TYPE(String), // opt/pg-host PARSE_RULE_OPTION_RESET(true), // opt/pg-host PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-host PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host // opt/pg-host PARSE_RULE_OPTION_COMMAND_INTERNAL_LIST // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTION_COMMAND_INTERNAL(ArchiveGet, true), // opt/pg-host PARSE_RULE_OPTION_COMMAND_INTERNAL(ArchivePush, true), // opt/pg-host PARSE_RULE_OPTION_COMMAND_INTERNAL(Restore, true), // opt/pg-host ) // opt/pg-host // opt/pg-host PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host ), // opt/pg-host // opt/pg-host PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-host ), // opt/pg-host // opt/pg-host PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-host ), // opt/pg-host // opt/pg-host PARSE_RULE_OPTIONAL // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTIONAL_GROUP // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host ( // opt/pg-host PARSE_RULE_VAL_OPT(PgLocal), // opt/pg-host PARSE_RULE_VAL_BOOL_FALSE, // opt/pg-host ), // opt/pg-host ), // opt/pg-host ), // opt/pg-host ), // opt/pg-host // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-ca-file ( // opt/pg-host-ca-file PARSE_RULE_OPTION_NAME("pg-host-ca-file"), // opt/pg-host-ca-file PARSE_RULE_OPTION_TYPE(String), // opt/pg-host-ca-file PARSE_RULE_OPTION_RESET(true), // opt/pg-host-ca-file PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-host-ca-file PARSE_RULE_OPTION_SECTION(Global), // opt/pg-host-ca-file PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-ca-file // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-ca-file ( // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-ca-file ), // opt/pg-host-ca-file // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-ca-file ( // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-ca-file ), // opt/pg-host-ca-file // opt/pg-host-ca-file PARSE_RULE_OPTIONAL // opt/pg-host-ca-file ( // opt/pg-host-ca-file PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-ca-file ( // opt/pg-host-ca-file PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-ca-file ( // opt/pg-host-ca-file PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-ca-file PARSE_RULE_VAL_STRID(tls), // opt/pg-host-ca-file ), // opt/pg-host-ca-file ), // opt/pg-host-ca-file ), // opt/pg-host-ca-file ), // opt/pg-host-ca-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-ca-path ( // opt/pg-host-ca-path PARSE_RULE_OPTION_NAME("pg-host-ca-path"), // opt/pg-host-ca-path PARSE_RULE_OPTION_TYPE(String), // opt/pg-host-ca-path PARSE_RULE_OPTION_RESET(true), // opt/pg-host-ca-path PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-host-ca-path PARSE_RULE_OPTION_SECTION(Global), // opt/pg-host-ca-path PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-ca-path // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-ca-path ( // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-ca-path ), // opt/pg-host-ca-path // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-ca-path ( // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-ca-path ), // opt/pg-host-ca-path // opt/pg-host-ca-path PARSE_RULE_OPTIONAL // opt/pg-host-ca-path ( // opt/pg-host-ca-path PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-ca-path ( // opt/pg-host-ca-path PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-ca-path ( // opt/pg-host-ca-path PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-ca-path PARSE_RULE_VAL_STRID(tls), // opt/pg-host-ca-path ), // opt/pg-host-ca-path ), // opt/pg-host-ca-path ), // opt/pg-host-ca-path ), // opt/pg-host-ca-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-cert-file ( // opt/pg-host-cert-file PARSE_RULE_OPTION_NAME("pg-host-cert-file"), // opt/pg-host-cert-file PARSE_RULE_OPTION_TYPE(String), // opt/pg-host-cert-file PARSE_RULE_OPTION_RESET(true), // opt/pg-host-cert-file PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-cert-file PARSE_RULE_OPTION_SECTION(Global), // opt/pg-host-cert-file PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-cert-file // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-cert-file ( // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-cert-file ), // opt/pg-host-cert-file // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-cert-file ( // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-cert-file ), // opt/pg-host-cert-file // opt/pg-host-cert-file PARSE_RULE_OPTIONAL // opt/pg-host-cert-file ( // opt/pg-host-cert-file PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-cert-file ( // opt/pg-host-cert-file PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-cert-file ( // opt/pg-host-cert-file PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-cert-file PARSE_RULE_VAL_STRID(tls), // opt/pg-host-cert-file ), // opt/pg-host-cert-file ), // opt/pg-host-cert-file ), // opt/pg-host-cert-file ), // opt/pg-host-cert-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_OPTION_NAME("pg-host-cmd"), // opt/pg-host-cmd PARSE_RULE_OPTION_TYPE(String), // opt/pg-host-cmd PARSE_RULE_OPTION_DEFAULT_TYPE(Dynamic), // opt/pg-host-cmd PARSE_RULE_OPTION_RESET(true), // opt/pg-host-cmd PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-cmd PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host-cmd PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-cmd // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-cmd ), // opt/pg-host-cmd // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-cmd ), // opt/pg-host-cmd // opt/pg-host-cmd PARSE_RULE_OPTIONAL // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-cmd PARSE_RULE_VAL_STRID(ssh), // opt/pg-host-cmd ), // opt/pg-host-cmd // opt/pg-host-cmd PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_DEFAULT_DYNAMIC(Bin), // opt/pg-host-cmd ), // opt/pg-host-cmd ), // opt/pg-host-cmd ), // opt/pg-host-cmd ), // opt/pg-host-cmd // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_OPTION_NAME("pg-host-config"), // opt/pg-host-config PARSE_RULE_OPTION_TYPE(Path), // opt/pg-host-config PARSE_RULE_OPTION_RESET(true), // opt/pg-host-config PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-config PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host-config PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-config // opt/pg-host-config PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-config ), // opt/pg-host-config // opt/pg-host-config PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-config ), // opt/pg-host-config // opt/pg-host-config PARSE_RULE_OPTIONAL // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_VAL_OPT(PgHost), // opt/pg-host-config ), // opt/pg-host-config // opt/pg-host-config PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_FILE), // opt/pg-host-config ), // opt/pg-host-config ), // opt/pg-host-config ), // opt/pg-host-config ), // opt/pg-host-config // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_OPTION_NAME("pg-host-config-include-path"), // opt/pg-host-config-include-path PARSE_RULE_OPTION_TYPE(Path), // opt/pg-host-config-include-path PARSE_RULE_OPTION_RESET(true), // opt/pg-host-config-include-path PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-config-include-path PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host-config-include-path PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-config-include-path // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-config-include-path ), // opt/pg-host-config-include-path // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-config-include-path ), // opt/pg-host-config-include-path // opt/pg-host-config-include-path PARSE_RULE_OPTIONAL // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_VAL_OPT(PgHost), // opt/pg-host-config-include-path ), // opt/pg-host-config-include-path // opt/pg-host-config-include-path PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_INCLUDE_PATH), ), // opt/pg-host-config-include-path ), // opt/pg-host-config-include-path ), // opt/pg-host-config-include-path ), // opt/pg-host-config-include-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_OPTION_NAME("pg-host-config-path"), // opt/pg-host-config-path PARSE_RULE_OPTION_TYPE(Path), // opt/pg-host-config-path PARSE_RULE_OPTION_RESET(true), // opt/pg-host-config-path PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-config-path PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host-config-path PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-config-path // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-config-path ), // opt/pg-host-config-path // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-config-path ), // opt/pg-host-config-path // opt/pg-host-config-path PARSE_RULE_OPTIONAL // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_VAL_OPT(PgHost), // opt/pg-host-config-path ), // opt/pg-host-config-path // opt/pg-host-config-path PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH), // opt/pg-host-config-path ), // opt/pg-host-config-path ), // opt/pg-host-config-path ), // opt/pg-host-config-path ), // opt/pg-host-config-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-key-file ( // opt/pg-host-key-file PARSE_RULE_OPTION_NAME("pg-host-key-file"), // opt/pg-host-key-file PARSE_RULE_OPTION_TYPE(String), // opt/pg-host-key-file PARSE_RULE_OPTION_RESET(true), // opt/pg-host-key-file PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-key-file PARSE_RULE_OPTION_SECTION(Global), // opt/pg-host-key-file PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-key-file // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-key-file ( // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-key-file ), // opt/pg-host-key-file // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-key-file ( // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-key-file ), // opt/pg-host-key-file // opt/pg-host-key-file PARSE_RULE_OPTIONAL // opt/pg-host-key-file ( // opt/pg-host-key-file PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-key-file ( // opt/pg-host-key-file PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-key-file ( // opt/pg-host-key-file PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-key-file PARSE_RULE_VAL_STRID(tls), // opt/pg-host-key-file ), // opt/pg-host-key-file ), // opt/pg-host-key-file ), // opt/pg-host-key-file ), // opt/pg-host-key-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_OPTION_NAME("pg-host-port"), // opt/pg-host-port PARSE_RULE_OPTION_TYPE(Integer), // opt/pg-host-port PARSE_RULE_OPTION_RESET(true), // opt/pg-host-port PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-host-port PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host-port PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-port // opt/pg-host-port PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-port ), // opt/pg-host-port // opt/pg-host-port PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-port ), // opt/pg-host-port // opt/pg-host-port PARSE_RULE_OPTIONAL // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-port PARSE_RULE_VAL_STRID(ssh), // opt/pg-host-port PARSE_RULE_VAL_STRID(tls), // opt/pg-host-port ), // opt/pg-host-port // opt/pg-host-port PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_VAL_INT(0), // opt/pg-host-port PARSE_RULE_VAL_INT(65535), // opt/pg-host-port ), // opt/pg-host-port // opt/pg-host-port PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_OPTIONAL_DEFAULT_MAP // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_VAL_STRID(tls), // opt/pg-host-port PARSE_RULE_VAL_INT(8432), // opt/pg-host-port ), // opt/pg-host-port ), // opt/pg-host-port ), // opt/pg-host-port ), // opt/pg-host-port ), // opt/pg-host-port // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_OPTION_NAME("pg-host-type"), // opt/pg-host-type PARSE_RULE_OPTION_TYPE(StringId), // opt/pg-host-type PARSE_RULE_OPTION_RESET(true), // opt/pg-host-type PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-type PARSE_RULE_OPTION_SECTION(Global), // opt/pg-host-type PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-type // opt/pg-host-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-type ), // opt/pg-host-type // opt/pg-host-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-type ), // opt/pg-host-type // opt/pg-host-type PARSE_RULE_OPTIONAL // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_VAL_OPT(PgHost), // opt/pg-host-type ), // opt/pg-host-type // opt/pg-host-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_VAL_STRID(ssh), // opt/pg-host-type PARSE_RULE_VAL_STRID(tls), // opt/pg-host-type ), // opt/pg-host-type // opt/pg-host-type PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_VAL_STRID(ssh), // opt/pg-host-type ), // opt/pg-host-type ), // opt/pg-host-type ), // opt/pg-host-type ), // opt/pg-host-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_OPTION_NAME("pg-host-user"), // opt/pg-host-user PARSE_RULE_OPTION_TYPE(String), // opt/pg-host-user PARSE_RULE_OPTION_RESET(true), // opt/pg-host-user PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-host-user PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host-user PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-user // opt/pg-host-user PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-user ), // opt/pg-host-user // opt/pg-host-user PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-user ), // opt/pg-host-user // opt/pg-host-user PARSE_RULE_OPTIONAL // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-user PARSE_RULE_VAL_STRID(ssh), // opt/pg-host-user ), // opt/pg-host-user // opt/pg-host-user PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_VAL_STR(QT_postgres_QT), // opt/pg-host-user ), // opt/pg-host-user ), // opt/pg-host-user ), // opt/pg-host-user ), // opt/pg-host-user // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-local ( // opt/pg-local PARSE_RULE_OPTION_NAME("pg-local"), // opt/pg-local PARSE_RULE_OPTION_TYPE(Boolean), // opt/pg-local PARSE_RULE_OPTION_INTERNAL(true) // opt/pg-local PARSE_RULE_OPTION_NEGATE(true), // opt/pg-local PARSE_RULE_OPTION_RESET(true), // opt/pg-local PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-local PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-local PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-local // opt/pg-local PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-local ( // opt/pg-local PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-local PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-local PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-local PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-local PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-local PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-local PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-local PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-local PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-local ), // opt/pg-local // opt/pg-local PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/pg-local ( // opt/pg-local PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-local PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-local ), // opt/pg-local // opt/pg-local PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-local ( // opt/pg-local PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-local PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-local PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-local PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-local ), // opt/pg-local // opt/pg-local PARSE_RULE_OPTIONAL // opt/pg-local ( // opt/pg-local PARSE_RULE_OPTIONAL_GROUP // opt/pg-local ( // opt/pg-local PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-local ( // opt/pg-local PARSE_RULE_VAL_BOOL_FALSE, // opt/pg-local ), // opt/pg-local ), // opt/pg-local ), // opt/pg-local ), // opt/pg-local // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-path ( // opt/pg-path PARSE_RULE_OPTION_NAME("pg-path"), // opt/pg-path PARSE_RULE_OPTION_TYPE(Path), // opt/pg-path PARSE_RULE_OPTION_RESET(true), // opt/pg-path PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-path PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-path PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-path // opt/pg-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-path ( // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/pg-path ( // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-path ( // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/pg-path ( // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTIONAL // opt/pg-path ( // opt/pg-path PARSE_RULE_OPTIONAL_GROUP // opt/pg-path ( // opt/pg-path PARSE_RULE_FILTER_CMD // opt/pg-path ( // opt/pg-path PARSE_RULE_VAL_CMD(ArchivePush), // opt/pg-path PARSE_RULE_VAL_CMD(Manifest), // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTIONAL_NOT_REQUIRED(), // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTIONAL_GROUP // opt/pg-path ( // opt/pg-path PARSE_RULE_FILTER_CMD // opt/pg-path ( // opt/pg-path PARSE_RULE_VAL_CMD(Check), // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTIONAL_DEPEND // opt/pg-path ( // opt/pg-path PARSE_RULE_VAL_OPT(Stanza), // opt/pg-path ), // opt/pg-path ), // opt/pg-path ), // opt/pg-path ), // opt/pg-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-port ( // opt/pg-port PARSE_RULE_OPTION_NAME("pg-port"), // opt/pg-port PARSE_RULE_OPTION_TYPE(Integer), // opt/pg-port PARSE_RULE_OPTION_RESET(true), // opt/pg-port PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-port PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-port PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-port // opt/pg-port PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-port ( // opt/pg-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-port PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-port ), // opt/pg-port // opt/pg-port PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-port ( // opt/pg-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-port ), // opt/pg-port // opt/pg-port PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/pg-port ( // opt/pg-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-port PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-port ), // opt/pg-port // opt/pg-port PARSE_RULE_OPTIONAL // opt/pg-port ( // opt/pg-port PARSE_RULE_OPTIONAL_GROUP // opt/pg-port ( // opt/pg-port PARSE_RULE_OPTIONAL_DEPEND // opt/pg-port ( // opt/pg-port PARSE_RULE_VAL_OPT(PgPath), // opt/pg-port ), // opt/pg-port // opt/pg-port PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/pg-port ( // opt/pg-port PARSE_RULE_VAL_INT(0), // opt/pg-port PARSE_RULE_VAL_INT(65535), // opt/pg-port ), // opt/pg-port // opt/pg-port PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-port ( // opt/pg-port PARSE_RULE_VAL_INT(5432), // opt/pg-port ), // opt/pg-port ), // opt/pg-port ), // opt/pg-port ), // opt/pg-port // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_OPTION_NAME("pg-socket-path"), // opt/pg-socket-path PARSE_RULE_OPTION_TYPE(String), // opt/pg-socket-path PARSE_RULE_OPTION_RESET(true), // opt/pg-socket-path PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-socket-path PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-socket-path PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-socket-path // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-socket-path ), // opt/pg-socket-path // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-socket-path ), // opt/pg-socket-path // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-socket-path ), // opt/pg-socket-path // opt/pg-socket-path PARSE_RULE_OPTIONAL // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_OPTIONAL_GROUP // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_OPTIONAL_DEPEND // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_VAL_OPT(PgPath), // opt/pg-socket-path ), // opt/pg-socket-path ), // opt/pg-socket-path ), // opt/pg-socket-path ), // opt/pg-socket-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-user ( // opt/pg-user PARSE_RULE_OPTION_NAME("pg-user"), // opt/pg-user PARSE_RULE_OPTION_TYPE(String), // opt/pg-user PARSE_RULE_OPTION_RESET(true), // opt/pg-user PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-user PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-user PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-user // opt/pg-user PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-user ( // opt/pg-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-user PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-user PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-user PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-user PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-user ), // opt/pg-user // opt/pg-user PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-user ( // opt/pg-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-user ), // opt/pg-user // opt/pg-user PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/pg-user ( // opt/pg-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-user PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-user PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-user PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-user PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-user ), // opt/pg-user // opt/pg-user PARSE_RULE_OPTIONAL // opt/pg-user ( // opt/pg-user PARSE_RULE_OPTIONAL_GROUP // opt/pg-user ( // opt/pg-user PARSE_RULE_OPTIONAL_DEPEND // opt/pg-user ( // opt/pg-user PARSE_RULE_VAL_OPT(PgPath), // opt/pg-user ), // opt/pg-user ), // opt/pg-user ), // opt/pg-user ), // opt/pg-user // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-version-force ( // opt/pg-version-force PARSE_RULE_OPTION_NAME("pg-version-force"), // opt/pg-version-force PARSE_RULE_OPTION_TYPE(String), // opt/pg-version-force PARSE_RULE_OPTION_RESET(true), // opt/pg-version-force PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-version-force PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-version-force // opt/pg-version-force PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-version-force ( // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Verify) // opt/pg-version-force ), // opt/pg-version-force // opt/pg-version-force PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/pg-version-force ( // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-version-force ), // opt/pg-version-force // opt/pg-version-force PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-version-force ( // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Verify) // opt/pg-version-force ), // opt/pg-version-force // opt/pg-version-force PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/pg-version-force ( // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Verify) // opt/pg-version-force ), // opt/pg-version-force ), // opt/pg-version-force // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/priority ( // opt/priority PARSE_RULE_OPTION_NAME("priority"), // opt/priority PARSE_RULE_OPTION_TYPE(Integer), // opt/priority PARSE_RULE_OPTION_RESET(true), // opt/priority PARSE_RULE_OPTION_REQUIRED(false), // opt/priority PARSE_RULE_OPTION_SECTION(Global), // opt/priority // opt/priority PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/priority ( // opt/priority PARSE_RULE_OPTION_COMMAND(Annotate) // opt/priority PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/priority PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/priority PARSE_RULE_OPTION_COMMAND(Backup) // opt/priority PARSE_RULE_OPTION_COMMAND(Check) // opt/priority PARSE_RULE_OPTION_COMMAND(Expire) // opt/priority PARSE_RULE_OPTION_COMMAND(Info) // opt/priority PARSE_RULE_OPTION_COMMAND(Manifest) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/priority PARSE_RULE_OPTION_COMMAND(Restore) // opt/priority PARSE_RULE_OPTION_COMMAND(Server) // opt/priority PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/priority PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/priority PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/priority PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/priority PARSE_RULE_OPTION_COMMAND(Start) // opt/priority PARSE_RULE_OPTION_COMMAND(Stop) // opt/priority PARSE_RULE_OPTION_COMMAND(Verify) // opt/priority ), // opt/priority // opt/priority PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/priority ( // opt/priority PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/priority PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/priority ), // opt/priority // opt/priority PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/priority ( // opt/priority PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/priority PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/priority PARSE_RULE_OPTION_COMMAND(Backup) // opt/priority PARSE_RULE_OPTION_COMMAND(Restore) // opt/priority PARSE_RULE_OPTION_COMMAND(Verify) // opt/priority ), // opt/priority // opt/priority PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/priority ( // opt/priority PARSE_RULE_OPTION_COMMAND(Annotate) // opt/priority PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/priority PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/priority PARSE_RULE_OPTION_COMMAND(Backup) // opt/priority PARSE_RULE_OPTION_COMMAND(Check) // opt/priority PARSE_RULE_OPTION_COMMAND(Expire) // opt/priority PARSE_RULE_OPTION_COMMAND(Info) // opt/priority PARSE_RULE_OPTION_COMMAND(Manifest) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/priority PARSE_RULE_OPTION_COMMAND(Restore) // opt/priority PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/priority PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/priority PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/priority PARSE_RULE_OPTION_COMMAND(Verify) // opt/priority ), // opt/priority // opt/priority PARSE_RULE_OPTIONAL // opt/priority ( // opt/priority PARSE_RULE_OPTIONAL_GROUP // opt/priority ( // opt/priority PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/priority ( // opt/priority PARSE_RULE_VAL_INT(DS_20), // opt/priority PARSE_RULE_VAL_INT(19), // opt/priority ), // opt/priority ), // opt/priority ), // opt/priority ), // opt/priority // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/process ( // opt/process PARSE_RULE_OPTION_NAME("process"), // opt/process PARSE_RULE_OPTION_TYPE(Integer), // opt/process PARSE_RULE_OPTION_INTERNAL(true) // opt/process PARSE_RULE_OPTION_REQUIRED(false), // opt/process PARSE_RULE_OPTION_SECTION(CommandLine), // opt/process // opt/process PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/process ( // opt/process PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/process PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/process PARSE_RULE_OPTION_COMMAND(Backup) // opt/process PARSE_RULE_OPTION_COMMAND(Restore) // opt/process PARSE_RULE_OPTION_COMMAND(Verify) // opt/process ), // opt/process // opt/process PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/process ( // opt/process PARSE_RULE_OPTION_COMMAND(Annotate) // opt/process PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/process PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/process PARSE_RULE_OPTION_COMMAND(Backup) // opt/process PARSE_RULE_OPTION_COMMAND(Check) // opt/process PARSE_RULE_OPTION_COMMAND(Expire) // opt/process PARSE_RULE_OPTION_COMMAND(Info) // opt/process PARSE_RULE_OPTION_COMMAND(Manifest) // opt/process PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/process PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/process PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/process PARSE_RULE_OPTION_COMMAND(Restore) // opt/process PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/process PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/process PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/process PARSE_RULE_OPTION_COMMAND(Verify) // opt/process ), // opt/process // opt/process PARSE_RULE_OPTIONAL // opt/process ( // opt/process PARSE_RULE_OPTIONAL_GROUP // opt/process ( // opt/process PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/process ( // opt/process PARSE_RULE_VAL_INT(0), // opt/process PARSE_RULE_VAL_INT(1024), // opt/process ), // opt/process ), // opt/process ), // opt/process ), // opt/process // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/process-max ( // opt/process-max PARSE_RULE_OPTION_NAME("process-max"), // opt/process-max PARSE_RULE_OPTION_TYPE(Integer), // opt/process-max PARSE_RULE_OPTION_RESET(true), // opt/process-max PARSE_RULE_OPTION_REQUIRED(true), // opt/process-max PARSE_RULE_OPTION_SECTION(Global), // opt/process-max // opt/process-max PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/process-max ( // opt/process-max PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/process-max PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/process-max PARSE_RULE_OPTION_COMMAND(Backup) // opt/process-max PARSE_RULE_OPTION_COMMAND(Restore) // opt/process-max PARSE_RULE_OPTION_COMMAND(Verify) // opt/process-max ), // opt/process-max // opt/process-max PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/process-max ( // opt/process-max PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/process-max PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/process-max ), // opt/process-max // opt/process-max PARSE_RULE_OPTIONAL // opt/process-max ( // opt/process-max PARSE_RULE_OPTIONAL_GROUP // opt/process-max ( // opt/process-max PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/process-max ( // opt/process-max PARSE_RULE_VAL_INT(1), // opt/process-max PARSE_RULE_VAL_INT(999), // opt/process-max ), // opt/process-max // opt/process-max PARSE_RULE_OPTIONAL_DEFAULT // opt/process-max ( // opt/process-max PARSE_RULE_VAL_INT(1), // opt/process-max ), // opt/process-max ), // opt/process-max ), // opt/process-max ), // opt/process-max // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTION_NAME("protocol-timeout"), // opt/protocol-timeout PARSE_RULE_OPTION_TYPE(Time), // opt/protocol-timeout PARSE_RULE_OPTION_RESET(true), // opt/protocol-timeout PARSE_RULE_OPTION_REQUIRED(true), // opt/protocol-timeout PARSE_RULE_OPTION_SECTION(Global), // opt/protocol-timeout // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Annotate) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Expire) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Info) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Manifest) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Restore) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Server) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Verify) // opt/protocol-timeout ), // opt/protocol-timeout // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/protocol-timeout ), // opt/protocol-timeout // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Restore) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Verify) // opt/protocol-timeout ), // opt/protocol-timeout // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Annotate) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Expire) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Info) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Manifest) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Restore) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Verify) // opt/protocol-timeout ), // opt/protocol-timeout // opt/protocol-timeout PARSE_RULE_OPTIONAL // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTIONAL_GROUP // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_VAL_TIME(100ms), // opt/protocol-timeout PARSE_RULE_VAL_TIME(7d), // opt/protocol-timeout ), // opt/protocol-timeout // opt/protocol-timeout PARSE_RULE_OPTIONAL_DEFAULT // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_VAL_TIME(31m), // opt/protocol-timeout ), // opt/protocol-timeout ), // opt/protocol-timeout ), // opt/protocol-timeout ), // opt/protocol-timeout // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/raw ( // opt/raw PARSE_RULE_OPTION_NAME("raw"), // opt/raw PARSE_RULE_OPTION_TYPE(Boolean), // opt/raw PARSE_RULE_OPTION_REQUIRED(true), // opt/raw PARSE_RULE_OPTION_SECTION(CommandLine), // opt/raw // opt/raw PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/raw ( // opt/raw PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/raw ), // opt/raw // opt/raw PARSE_RULE_OPTIONAL // opt/raw ( // opt/raw PARSE_RULE_OPTIONAL_GROUP // opt/raw ( // opt/raw PARSE_RULE_OPTIONAL_DEFAULT // opt/raw ( // opt/raw PARSE_RULE_VAL_BOOL_FALSE, // opt/raw ), // opt/raw ), // opt/raw ), // opt/raw ), // opt/raw // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/recovery-option ( // opt/recovery-option PARSE_RULE_OPTION_NAME("recovery-option"), // opt/recovery-option PARSE_RULE_OPTION_TYPE(Hash), // opt/recovery-option PARSE_RULE_OPTION_RESET(true), // opt/recovery-option PARSE_RULE_OPTION_REQUIRED(false), // opt/recovery-option PARSE_RULE_OPTION_SECTION(Global), // opt/recovery-option PARSE_RULE_OPTION_MULTI(true), // opt/recovery-option // opt/recovery-option PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/recovery-option ( // opt/recovery-option PARSE_RULE_OPTION_COMMAND(Restore) // opt/recovery-option ), // opt/recovery-option // opt/recovery-option PARSE_RULE_OPTIONAL // opt/recovery-option ( // opt/recovery-option PARSE_RULE_OPTIONAL_GROUP // opt/recovery-option ( // opt/recovery-option PARSE_RULE_OPTIONAL_DEPEND // opt/recovery-option ( // opt/recovery-option PARSE_RULE_VAL_OPT(Type), // opt/recovery-option PARSE_RULE_VAL_STRID(default), // opt/recovery-option PARSE_RULE_VAL_STRID(immediate), // opt/recovery-option PARSE_RULE_VAL_STRID(lsn), // opt/recovery-option PARSE_RULE_VAL_STRID(name), // opt/recovery-option PARSE_RULE_VAL_STRID(time), // opt/recovery-option PARSE_RULE_VAL_STRID(standby), // opt/recovery-option PARSE_RULE_VAL_STRID(xid), // opt/recovery-option ), // opt/recovery-option ), // opt/recovery-option ), // opt/recovery-option ), // opt/recovery-option // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/recurse ( // opt/recurse PARSE_RULE_OPTION_NAME("recurse"), // opt/recurse PARSE_RULE_OPTION_TYPE(Boolean), // opt/recurse PARSE_RULE_OPTION_REQUIRED(true), // opt/recurse PARSE_RULE_OPTION_SECTION(CommandLine), // opt/recurse // opt/recurse PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/recurse ( // opt/recurse PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/recurse PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/recurse ), // opt/recurse // opt/recurse PARSE_RULE_OPTIONAL // opt/recurse ( // opt/recurse PARSE_RULE_OPTIONAL_GROUP // opt/recurse ( // opt/recurse PARSE_RULE_OPTIONAL_DEFAULT // opt/recurse ( // opt/recurse PARSE_RULE_VAL_BOOL_FALSE, // opt/recurse ), // opt/recurse ), // opt/recurse ), // opt/recurse ), // opt/recurse // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/reference ( // opt/reference PARSE_RULE_OPTION_NAME("reference"), // opt/reference PARSE_RULE_OPTION_TYPE(String), // opt/reference PARSE_RULE_OPTION_REQUIRED(false), // opt/reference PARSE_RULE_OPTION_SECTION(CommandLine), // opt/reference // opt/reference PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/reference ( // opt/reference PARSE_RULE_OPTION_COMMAND(Manifest) // opt/reference ), // opt/reference ), // opt/reference // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/remote-type ( // opt/remote-type PARSE_RULE_OPTION_NAME("remote-type"), // opt/remote-type PARSE_RULE_OPTION_TYPE(StringId), // opt/remote-type PARSE_RULE_OPTION_INTERNAL(true) // opt/remote-type PARSE_RULE_OPTION_REQUIRED(false), // opt/remote-type PARSE_RULE_OPTION_SECTION(CommandLine), // opt/remote-type // opt/remote-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/remote-type ( // opt/remote-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/remote-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/remote-type ), // opt/remote-type // opt/remote-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/remote-type ( // opt/remote-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/remote-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/remote-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Check) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Info) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/remote-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/remote-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/remote-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/remote-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/remote-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/remote-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/remote-type ), // opt/remote-type // opt/remote-type PARSE_RULE_OPTIONAL // opt/remote-type ( // opt/remote-type PARSE_RULE_OPTIONAL_GROUP // opt/remote-type ( // opt/remote-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/remote-type ( // opt/remote-type PARSE_RULE_VAL_STRID(pg), // opt/remote-type PARSE_RULE_VAL_STRID(repo), // opt/remote-type ), // opt/remote-type ), // opt/remote-type ), // opt/remote-type ), // opt/remote-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo ( // opt/repo PARSE_RULE_OPTION_NAME("repo"), // opt/repo PARSE_RULE_OPTION_TYPE(Integer), // opt/repo PARSE_RULE_OPTION_REQUIRED(false), // opt/repo PARSE_RULE_OPTION_SECTION(CommandLine), // opt/repo // opt/repo PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo ( // opt/repo PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo PARSE_RULE_OPTION_COMMAND(Info) // opt/repo PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo ), // opt/repo // opt/repo PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo ( // opt/repo PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo ), // opt/repo // opt/repo PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo ( // opt/repo PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo ), // opt/repo // opt/repo PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo ( // opt/repo PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo PARSE_RULE_OPTION_COMMAND(Check) // opt/repo PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo PARSE_RULE_OPTION_COMMAND(Info) // opt/repo PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo ), // opt/repo // opt/repo PARSE_RULE_OPTIONAL // opt/repo ( // opt/repo PARSE_RULE_OPTIONAL_GROUP // opt/repo ( // opt/repo PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo ( // opt/repo PARSE_RULE_VAL_INT(1), // opt/repo PARSE_RULE_VAL_INT(256), // opt/repo ), // opt/repo ), // opt/repo ), // opt/repo ), // opt/repo // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTION_NAME("repo-azure-account"), // opt/repo-azure-account PARSE_RULE_OPTION_TYPE(String), // opt/repo-azure-account PARSE_RULE_OPTION_RESET(true), // opt/repo-azure-account PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-azure-account PARSE_RULE_OPTION_SECTION(Global), // opt/repo-azure-account PARSE_RULE_OPTION_SECURE(true), // opt/repo-azure-account PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-azure-account // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-account ), // opt/repo-azure-account // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-account ), // opt/repo-azure-account // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-account ), // opt/repo-azure-account // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-account ), // opt/repo-azure-account // opt/repo-azure-account PARSE_RULE_OPTIONAL // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTIONAL_GROUP // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTIONAL_DEPEND // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_VAL_OPT(RepoType), // opt/repo-azure-account PARSE_RULE_VAL_STRID(azure), // opt/repo-azure-account ), // opt/repo-azure-account ), // opt/repo-azure-account ), // opt/repo-azure-account ), // opt/repo-azure-account // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTION_NAME("repo-azure-container"), // opt/repo-azure-container PARSE_RULE_OPTION_TYPE(String), // opt/repo-azure-container PARSE_RULE_OPTION_RESET(true), // opt/repo-azure-container PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-azure-container PARSE_RULE_OPTION_SECTION(Global), // opt/repo-azure-container PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-azure-container // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-container ), // opt/repo-azure-container // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-container ), // opt/repo-azure-container // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-container ), // opt/repo-azure-container // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-container ), // opt/repo-azure-container // opt/repo-azure-container PARSE_RULE_OPTIONAL // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTIONAL_GROUP // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTIONAL_DEPEND // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_VAL_OPT(RepoType), // opt/repo-azure-container PARSE_RULE_VAL_STRID(azure), // opt/repo-azure-container ), // opt/repo-azure-container ), // opt/repo-azure-container ), // opt/repo-azure-container ), // opt/repo-azure-container // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTION_NAME("repo-azure-endpoint"), // opt/repo-azure-endpoint PARSE_RULE_OPTION_TYPE(String), // opt/repo-azure-endpoint PARSE_RULE_OPTION_RESET(true), // opt/repo-azure-endpoint PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-azure-endpoint PARSE_RULE_OPTION_SECTION(Global), // opt/repo-azure-endpoint PARSE_RULE_OPTION_SECURE(true), // opt/repo-azure-endpoint PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-azure-endpoint // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint // opt/repo-azure-endpoint PARSE_RULE_OPTIONAL // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTIONAL_GROUP // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTIONAL_DEPEND // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_VAL_OPT(RepoType), // opt/repo-azure-endpoint PARSE_RULE_VAL_STRID(azure), // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint // opt/repo-azure-endpoint PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_VAL_STR(QT_blob_DT_core_DT_windows_DT_net_QT), // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTION_NAME("repo-azure-key"), // opt/repo-azure-key PARSE_RULE_OPTION_TYPE(String), // opt/repo-azure-key PARSE_RULE_OPTION_RESET(true), // opt/repo-azure-key PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-azure-key PARSE_RULE_OPTION_SECTION(Global), // opt/repo-azure-key PARSE_RULE_OPTION_SECURE(true), // opt/repo-azure-key PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-azure-key // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-key ), // opt/repo-azure-key // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key ), // opt/repo-azure-key // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-key ), // opt/repo-azure-key // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-key ), // opt/repo-azure-key // opt/repo-azure-key PARSE_RULE_OPTIONAL // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTIONAL_GROUP // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTIONAL_DEPEND // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_VAL_OPT(RepoAzureKeyType), // opt/repo-azure-key PARSE_RULE_VAL_STRID(shared), // opt/repo-azure-key PARSE_RULE_VAL_STRID(sas), // opt/repo-azure-key ), // opt/repo-azure-key ), // opt/repo-azure-key ), // opt/repo-azure-key ), // opt/repo-azure-key // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTION_NAME("repo-azure-key-type"), // opt/repo-azure-key-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-azure-key-type PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-azure-key-type PARSE_RULE_OPTION_RESET(true), // opt/repo-azure-key-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-azure-key-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-azure-key-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-key-type ), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key-type ), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-key-type ), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-key-type ), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTIONAL // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTIONAL_DEPEND // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_VAL_OPT(RepoType), // opt/repo-azure-key-type PARSE_RULE_VAL_STRID(azure), // opt/repo-azure-key-type ), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_VAL_STRID(shared), // opt/repo-azure-key-type PARSE_RULE_VAL_STRID(sas), // opt/repo-azure-key-type PARSE_RULE_VAL_STRID(auto), // opt/repo-azure-key-type ), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_VAL_STRID(shared), // opt/repo-azure-key-type ), // opt/repo-azure-key-type ), // opt/repo-azure-key-type ), // opt/repo-azure-key-type ), // opt/repo-azure-key-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTION_NAME("repo-azure-uri-style"), // opt/repo-azure-uri-style PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-azure-uri-style PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-azure-uri-style PARSE_RULE_OPTION_RESET(true), // opt/repo-azure-uri-style PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-azure-uri-style PARSE_RULE_OPTION_SECTION(Global), // opt/repo-azure-uri-style PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTIONAL // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTIONAL_GROUP // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTIONAL_DEPEND // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_VAL_OPT(RepoType), // opt/repo-azure-uri-style PARSE_RULE_VAL_STRID(azure), // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_VAL_STRID(host), // opt/repo-azure-uri-style PARSE_RULE_VAL_STRID(path), // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_VAL_STRID(host), // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-block ( // opt/repo-block PARSE_RULE_OPTION_NAME("repo-block"), // opt/repo-block PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-block PARSE_RULE_OPTION_NEGATE(true), // opt/repo-block PARSE_RULE_OPTION_RESET(true), // opt/repo-block PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-block PARSE_RULE_OPTION_SECTION(Global), // opt/repo-block PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-block // opt/repo-block PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-block ( // opt/repo-block PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-block ), // opt/repo-block // opt/repo-block PARSE_RULE_OPTIONAL // opt/repo-block ( // opt/repo-block PARSE_RULE_OPTIONAL_GROUP // opt/repo-block ( // opt/repo-block PARSE_RULE_OPTIONAL_DEPEND // opt/repo-block ( // opt/repo-block PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_BOOL_FALSE), // opt/repo-block PARSE_RULE_VAL_OPT(RepoBundle), // opt/repo-block PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-block ), // opt/repo-block // opt/repo-block PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-block ( // opt/repo-block PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-block ), // opt/repo-block ), // opt/repo-block ), // opt/repo-block ), // opt/repo-block // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-block-age-map ( // opt/repo-block-age-map PARSE_RULE_OPTION_NAME("repo-block-age-map"), // opt/repo-block-age-map PARSE_RULE_OPTION_TYPE(Hash), // opt/repo-block-age-map PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-block-age-map PARSE_RULE_OPTION_RESET(true), // opt/repo-block-age-map PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-block-age-map PARSE_RULE_OPTION_SECTION(Global), // opt/repo-block-age-map PARSE_RULE_OPTION_MULTI(true), // opt/repo-block-age-map PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-block-age-map // opt/repo-block-age-map PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-block-age-map ( // opt/repo-block-age-map PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-block-age-map ), // opt/repo-block-age-map // opt/repo-block-age-map PARSE_RULE_OPTIONAL // opt/repo-block-age-map ( // opt/repo-block-age-map PARSE_RULE_OPTIONAL_GROUP // opt/repo-block-age-map ( // opt/repo-block-age-map PARSE_RULE_OPTIONAL_DEPEND // opt/repo-block-age-map ( // opt/repo-block-age-map PARSE_RULE_VAL_OPT(RepoBlock), // opt/repo-block-age-map PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-block-age-map ), // opt/repo-block-age-map ), // opt/repo-block-age-map ), // opt/repo-block-age-map ), // opt/repo-block-age-map // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-block-checksum-size-map ( // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_NAME("repo-block-checksum-size-map"), // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_TYPE(Hash), // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_RESET(true), // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_SECTION(Global), // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_MULTI(true), // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-block-checksum-size-map // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-block-checksum-size-map ( // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-block-checksum-size-map ), // opt/repo-block-checksum-size-map // opt/repo-block-checksum-size-map PARSE_RULE_OPTIONAL // opt/repo-block-checksum-size-map ( // opt/repo-block-checksum-size-map PARSE_RULE_OPTIONAL_GROUP // opt/repo-block-checksum-size-map ( // opt/repo-block-checksum-size-map PARSE_RULE_OPTIONAL_DEPEND // opt/repo-block-checksum-size-map ( // opt/repo-block-checksum-size-map PARSE_RULE_VAL_OPT(RepoBlock), // opt/repo-block-checksum-size-map PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-block-checksum-size-map ), // opt/repo-block-checksum-size-map ), // opt/repo-block-checksum-size-map ), // opt/repo-block-checksum-size-map ), // opt/repo-block-checksum-size-map // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-block-size-map ( // opt/repo-block-size-map PARSE_RULE_OPTION_NAME("repo-block-size-map"), // opt/repo-block-size-map PARSE_RULE_OPTION_TYPE(Hash), // opt/repo-block-size-map PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-block-size-map PARSE_RULE_OPTION_RESET(true), // opt/repo-block-size-map PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-block-size-map PARSE_RULE_OPTION_SECTION(Global), // opt/repo-block-size-map PARSE_RULE_OPTION_MULTI(true), // opt/repo-block-size-map PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-block-size-map // opt/repo-block-size-map PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-block-size-map ( // opt/repo-block-size-map PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-block-size-map ), // opt/repo-block-size-map // opt/repo-block-size-map PARSE_RULE_OPTIONAL // opt/repo-block-size-map ( // opt/repo-block-size-map PARSE_RULE_OPTIONAL_GROUP // opt/repo-block-size-map ( // opt/repo-block-size-map PARSE_RULE_OPTIONAL_DEPEND // opt/repo-block-size-map ( // opt/repo-block-size-map PARSE_RULE_VAL_OPT(RepoBlock), // opt/repo-block-size-map PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-block-size-map ), // opt/repo-block-size-map ), // opt/repo-block-size-map ), // opt/repo-block-size-map ), // opt/repo-block-size-map // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_OPTION_NAME("repo-block-size-super"), // opt/repo-block-size-super PARSE_RULE_OPTION_TYPE(Size), // opt/repo-block-size-super PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-block-size-super PARSE_RULE_OPTION_RESET(true), // opt/repo-block-size-super PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-block-size-super PARSE_RULE_OPTION_SECTION(Global), // opt/repo-block-size-super PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-block-size-super // opt/repo-block-size-super PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-block-size-super ), // opt/repo-block-size-super // opt/repo-block-size-super PARSE_RULE_OPTIONAL // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_OPTIONAL_GROUP // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_OPTIONAL_DEPEND // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_VAL_OPT(RepoBlock), // opt/repo-block-size-super PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-block-size-super ), // opt/repo-block-size-super // opt/repo-block-size-super PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_VAL_SIZE(32KiB), // opt/repo-block-size-super PARSE_RULE_VAL_SIZE(16MiB), // opt/repo-block-size-super ), // opt/repo-block-size-super // opt/repo-block-size-super PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_VAL_SIZE(256KiB), // opt/repo-block-size-super ), // opt/repo-block-size-super ), // opt/repo-block-size-super ), // opt/repo-block-size-super ), // opt/repo-block-size-super // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_OPTION_NAME("repo-block-size-super-full"), // opt/repo-block-size-super-full PARSE_RULE_OPTION_TYPE(Size), // opt/repo-block-size-super-full PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-block-size-super-full PARSE_RULE_OPTION_RESET(true), // opt/repo-block-size-super-full PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-block-size-super-full PARSE_RULE_OPTION_SECTION(Global), // opt/repo-block-size-super-full PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-block-size-super-full // opt/repo-block-size-super-full PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full // opt/repo-block-size-super-full PARSE_RULE_OPTIONAL // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_OPTIONAL_GROUP // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_OPTIONAL_DEPEND // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_VAL_OPT(RepoBlock), // opt/repo-block-size-super-full PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full // opt/repo-block-size-super-full PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_VAL_SIZE(32KiB), // opt/repo-block-size-super-full PARSE_RULE_VAL_SIZE(16MiB), // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full // opt/repo-block-size-super-full PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_VAL_SIZE(1MiB), // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-bundle ( // opt/repo-bundle PARSE_RULE_OPTION_NAME("repo-bundle"), // opt/repo-bundle PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-bundle PARSE_RULE_OPTION_NEGATE(true), // opt/repo-bundle PARSE_RULE_OPTION_RESET(true), // opt/repo-bundle PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-bundle PARSE_RULE_OPTION_SECTION(Global), // opt/repo-bundle PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-bundle // opt/repo-bundle PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-bundle ( // opt/repo-bundle PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-bundle ), // opt/repo-bundle // opt/repo-bundle PARSE_RULE_OPTIONAL // opt/repo-bundle ( // opt/repo-bundle PARSE_RULE_OPTIONAL_GROUP // opt/repo-bundle ( // opt/repo-bundle PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-bundle ( // opt/repo-bundle PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-bundle ), // opt/repo-bundle ), // opt/repo-bundle ), // opt/repo-bundle ), // opt/repo-bundle // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_OPTION_NAME("repo-bundle-limit"), // opt/repo-bundle-limit PARSE_RULE_OPTION_TYPE(Size), // opt/repo-bundle-limit PARSE_RULE_OPTION_RESET(true), // opt/repo-bundle-limit PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-bundle-limit PARSE_RULE_OPTION_SECTION(Global), // opt/repo-bundle-limit PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-bundle-limit // opt/repo-bundle-limit PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-bundle-limit ), // opt/repo-bundle-limit // opt/repo-bundle-limit PARSE_RULE_OPTIONAL // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_OPTIONAL_GROUP // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_OPTIONAL_DEPEND // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_VAL_OPT(RepoBundle), // opt/repo-bundle-limit PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-bundle-limit ), // opt/repo-bundle-limit // opt/repo-bundle-limit PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_VAL_SIZE(8KiB), // opt/repo-bundle-limit PARSE_RULE_VAL_SIZE(1PiB), // opt/repo-bundle-limit ), // opt/repo-bundle-limit // opt/repo-bundle-limit PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_VAL_SIZE(2MiB), // opt/repo-bundle-limit ), // opt/repo-bundle-limit ), // opt/repo-bundle-limit ), // opt/repo-bundle-limit ), // opt/repo-bundle-limit // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_OPTION_NAME("repo-bundle-size"), // opt/repo-bundle-size PARSE_RULE_OPTION_TYPE(Size), // opt/repo-bundle-size PARSE_RULE_OPTION_RESET(true), // opt/repo-bundle-size PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-bundle-size PARSE_RULE_OPTION_SECTION(Global), // opt/repo-bundle-size PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-bundle-size // opt/repo-bundle-size PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-bundle-size ), // opt/repo-bundle-size // opt/repo-bundle-size PARSE_RULE_OPTIONAL // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_OPTIONAL_GROUP // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_OPTIONAL_DEPEND // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_VAL_OPT(RepoBundle), // opt/repo-bundle-size PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-bundle-size ), // opt/repo-bundle-size // opt/repo-bundle-size PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_VAL_SIZE(1MiB), // opt/repo-bundle-size PARSE_RULE_VAL_SIZE(1PiB), // opt/repo-bundle-size ), // opt/repo-bundle-size // opt/repo-bundle-size PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_VAL_SIZE(20MiB), // opt/repo-bundle-size ), // opt/repo-bundle-size ), // opt/repo-bundle-size ), // opt/repo-bundle-size ), // opt/repo-bundle-size // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTION_NAME("repo-cipher-pass"), // opt/repo-cipher-pass PARSE_RULE_OPTION_TYPE(String), // opt/repo-cipher-pass PARSE_RULE_OPTION_RESET(true), // opt/repo-cipher-pass PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-cipher-pass PARSE_RULE_OPTION_SECTION(Global), // opt/repo-cipher-pass PARSE_RULE_OPTION_SECURE(true), // opt/repo-cipher-pass PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-cipher-pass PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-cipher-pass // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-cipher-pass ), // opt/repo-cipher-pass // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-pass ), // opt/repo-cipher-pass // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-cipher-pass ), // opt/repo-cipher-pass // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-cipher-pass ), // opt/repo-cipher-pass // opt/repo-cipher-pass PARSE_RULE_OPTIONAL // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTIONAL_GROUP // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTIONAL_DEPEND // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_VAL_OPT(RepoCipherType), // opt/repo-cipher-pass PARSE_RULE_VAL_STRID(aes_DS_256_DS_cbc), // opt/repo-cipher-pass ), // opt/repo-cipher-pass ), // opt/repo-cipher-pass ), // opt/repo-cipher-pass ), // opt/repo-cipher-pass // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-cipher-rotate ( // opt/repo-cipher-rotate PARSE_RULE_OPTION_NAME("repo-cipher-rotate"), // opt/repo-cipher-rotate PARSE_RULE_OPTION_TYPE(Time), // opt/repo-cipher-rotate PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-cipher-rotate PARSE_RULE_OPTION_RESET(true), // opt/repo-cipher-rotate PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-cipher-rotate PARSE_RULE_OPTION_SECTION(Global), // opt/repo-cipher-rotate PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-cipher-rotate // opt/repo-cipher-rotate PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-cipher-rotate ( // opt/repo-cipher-rotate PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-cipher-rotate PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-cipher-rotate ), // opt/repo-cipher-rotate // opt/repo-cipher-rotate PARSE_RULE_OPTIONAL // opt/repo-cipher-rotate ( // opt/repo-cipher-rotate PARSE_RULE_OPTIONAL_GROUP // opt/repo-cipher-rotate ( // opt/repo-cipher-rotate PARSE_RULE_OPTIONAL_DEPEND // opt/repo-cipher-rotate ( // opt/repo-cipher-rotate PARSE_RULE_VAL_OPT(RepoCipherType), // opt/repo-cipher-rotate PARSE_RULE_VAL_STRID(aes_DS_256_DS_cbc), // opt/repo-cipher-rotate ), // opt/repo-cipher-rotate // opt/repo-cipher-rotate PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-cipher-rotate ( // opt/repo-cipher-rotate PARSE_RULE_VAL_TIME(15d), // opt/repo-cipher-rotate PARSE_RULE_VAL_TIME(9999999d), // opt/repo-cipher-rotate ), // opt/repo-cipher-rotate // opt/repo-cipher-rotate PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-cipher-rotate ( // opt/repo-cipher-rotate PARSE_RULE_VAL_TIME(90d), // opt/repo-cipher-rotate ), // opt/repo-cipher-rotate ), // opt/repo-cipher-rotate ), // opt/repo-cipher-rotate ), // opt/repo-cipher-rotate // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTION_NAME("repo-cipher-type"), // opt/repo-cipher-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-cipher-type PARSE_RULE_OPTION_RESET(true), // opt/repo-cipher-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-cipher-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-cipher-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-cipher-type PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-cipher-type // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-cipher-type ), // opt/repo-cipher-type // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-type ), // opt/repo-cipher-type // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-cipher-type ), // opt/repo-cipher-type // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-cipher-type ), // opt/repo-cipher-type // opt/repo-cipher-type PARSE_RULE_OPTIONAL // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_VAL_STRID(none), // opt/repo-cipher-type PARSE_RULE_VAL_STRID(aes_DS_256_DS_cbc), // opt/repo-cipher-type ), // opt/repo-cipher-type // opt/repo-cipher-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_VAL_STRID(none), // opt/repo-cipher-type ), // opt/repo-cipher-type ), // opt/repo-cipher-type ), // opt/repo-cipher-type ), // opt/repo-cipher-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-format ( // opt/repo-format PARSE_RULE_OPTION_NAME("repo-format"), // opt/repo-format PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-format PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-format PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-format PARSE_RULE_OPTION_SECTION(CommandLine), // opt/repo-format PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-format // opt/repo-format PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-format ( // opt/repo-format PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-format PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-format ), // opt/repo-format // opt/repo-format PARSE_RULE_OPTIONAL // opt/repo-format ( // opt/repo-format PARSE_RULE_OPTIONAL_GROUP // opt/repo-format ( // opt/repo-format PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-format ( // opt/repo-format PARSE_RULE_VAL_INT(5), // opt/repo-format PARSE_RULE_VAL_INT(6), // opt/repo-format ), // opt/repo-format // opt/repo-format PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-format ( // opt/repo-format PARSE_RULE_VAL_INT(5), // opt/repo-format ), // opt/repo-format ), // opt/repo-format ), // opt/repo-format ), // opt/repo-format // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTION_NAME("repo-gcs-bucket"), // opt/repo-gcs-bucket PARSE_RULE_OPTION_TYPE(String), // opt/repo-gcs-bucket PARSE_RULE_OPTION_RESET(true), // opt/repo-gcs-bucket PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-gcs-bucket PARSE_RULE_OPTION_SECTION(Global), // opt/repo-gcs-bucket PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-gcs-bucket // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket // opt/repo-gcs-bucket PARSE_RULE_OPTIONAL // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTIONAL_GROUP // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTIONAL_DEPEND // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_VAL_OPT(RepoGcsKeyType), // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTION_NAME("repo-gcs-endpoint"), // opt/repo-gcs-endpoint PARSE_RULE_OPTION_TYPE(String), // opt/repo-gcs-endpoint PARSE_RULE_OPTION_RESET(true), // opt/repo-gcs-endpoint PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-gcs-endpoint PARSE_RULE_OPTION_SECTION(Global), // opt/repo-gcs-endpoint PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-gcs-endpoint // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint // opt/repo-gcs-endpoint PARSE_RULE_OPTIONAL // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTIONAL_GROUP // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTIONAL_DEPEND // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_VAL_OPT(RepoGcsKeyType), // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint // opt/repo-gcs-endpoint PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_VAL_STR(QT_storage_DT_googleapis_DT_com_QT), // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTION_NAME("repo-gcs-key"), // opt/repo-gcs-key PARSE_RULE_OPTION_TYPE(String), // opt/repo-gcs-key PARSE_RULE_OPTION_RESET(true), // opt/repo-gcs-key PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-gcs-key PARSE_RULE_OPTION_SECTION(Global), // opt/repo-gcs-key PARSE_RULE_OPTION_SECURE(true), // opt/repo-gcs-key PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-gcs-key // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-key ), // opt/repo-gcs-key // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key ), // opt/repo-gcs-key // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-key ), // opt/repo-gcs-key // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-key ), // opt/repo-gcs-key // opt/repo-gcs-key PARSE_RULE_OPTIONAL // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTIONAL_GROUP // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTIONAL_DEPEND // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_VAL_OPT(RepoGcsKeyType), // opt/repo-gcs-key PARSE_RULE_VAL_STRID(service), // opt/repo-gcs-key PARSE_RULE_VAL_STRID(token), // opt/repo-gcs-key PARSE_RULE_VAL_STRID(web_DS_id), // opt/repo-gcs-key ), // opt/repo-gcs-key ), // opt/repo-gcs-key ), // opt/repo-gcs-key ), // opt/repo-gcs-key // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTION_NAME("repo-gcs-key-type"), // opt/repo-gcs-key-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-gcs-key-type PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-gcs-key-type PARSE_RULE_OPTION_RESET(true), // opt/repo-gcs-key-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-gcs-key-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-gcs-key-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTIONAL // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTIONAL_DEPEND // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_VAL_OPT(RepoType), // opt/repo-gcs-key-type PARSE_RULE_VAL_STRID(gcs), // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_VAL_STRID(service), // opt/repo-gcs-key-type PARSE_RULE_VAL_STRID(token), // opt/repo-gcs-key-type PARSE_RULE_VAL_STRID(auto), // opt/repo-gcs-key-type PARSE_RULE_VAL_STRID(web_DS_id), // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_VAL_STRID(service), // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-gcs-user-project ( // opt/repo-gcs-user-project PARSE_RULE_OPTION_NAME("repo-gcs-user-project"), // opt/repo-gcs-user-project PARSE_RULE_OPTION_TYPE(String), // opt/repo-gcs-user-project PARSE_RULE_OPTION_RESET(true), // opt/repo-gcs-user-project PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-gcs-user-project PARSE_RULE_OPTION_SECTION(Global), // opt/repo-gcs-user-project PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-gcs-user-project // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-gcs-user-project ( // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-user-project ), // opt/repo-gcs-user-project // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-gcs-user-project ( // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-user-project ), // opt/repo-gcs-user-project // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-gcs-user-project ( // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-user-project ), // opt/repo-gcs-user-project // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-gcs-user-project ( // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-user-project ), // opt/repo-gcs-user-project ), // opt/repo-gcs-user-project // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-hardlink ( // opt/repo-hardlink PARSE_RULE_OPTION_NAME("repo-hardlink"), // opt/repo-hardlink PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-hardlink PARSE_RULE_OPTION_NEGATE(true), // opt/repo-hardlink PARSE_RULE_OPTION_RESET(true), // opt/repo-hardlink PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-hardlink PARSE_RULE_OPTION_SECTION(Global), // opt/repo-hardlink PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-hardlink // opt/repo-hardlink PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-hardlink ( // opt/repo-hardlink PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-hardlink ), // opt/repo-hardlink // opt/repo-hardlink PARSE_RULE_OPTIONAL // opt/repo-hardlink ( // opt/repo-hardlink PARSE_RULE_OPTIONAL_GROUP // opt/repo-hardlink ( // opt/repo-hardlink PARSE_RULE_OPTIONAL_DEPEND // opt/repo-hardlink ( // opt/repo-hardlink PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_BOOL_FALSE), // opt/repo-hardlink PARSE_RULE_VAL_OPT(RepoBundle), // opt/repo-hardlink PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-hardlink ), // opt/repo-hardlink // opt/repo-hardlink PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-hardlink ( // opt/repo-hardlink PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-hardlink ), // opt/repo-hardlink ), // opt/repo-hardlink ), // opt/repo-hardlink ), // opt/repo-hardlink // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host ( // opt/repo-host PARSE_RULE_OPTION_NAME("repo-host"), // opt/repo-host PARSE_RULE_OPTION_TYPE(String), // opt/repo-host PARSE_RULE_OPTION_RESET(true), // opt/repo-host PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-host PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host // opt/repo-host PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host ( // opt/repo-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host ), // opt/repo-host // opt/repo-host PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host ( // opt/repo-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host ), // opt/repo-host // opt/repo-host PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host ( // opt/repo-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host ), // opt/repo-host // opt/repo-host PARSE_RULE_OPTIONAL // opt/repo-host ( // opt/repo-host PARSE_RULE_OPTIONAL_GROUP // opt/repo-host ( // opt/repo-host PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host ( // opt/repo-host PARSE_RULE_VAL_OPT(RepoLocal), // opt/repo-host PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-host ), // opt/repo-host ), // opt/repo-host ), // opt/repo-host ), // opt/repo-host // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_OPTION_NAME("repo-host-ca-file"), // opt/repo-host-ca-file PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-ca-file PARSE_RULE_OPTION_RESET(true), // opt/repo-host-ca-file PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-host-ca-file PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-ca-file PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-ca-file // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-ca-file ), // opt/repo-host-ca-file // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-ca-file ), // opt/repo-host-ca-file // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-ca-file ), // opt/repo-host-ca-file // opt/repo-host-ca-file PARSE_RULE_OPTIONAL // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-ca-file PARSE_RULE_VAL_STRID(tls), // opt/repo-host-ca-file ), // opt/repo-host-ca-file ), // opt/repo-host-ca-file ), // opt/repo-host-ca-file ), // opt/repo-host-ca-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_OPTION_NAME("repo-host-ca-path"), // opt/repo-host-ca-path PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-ca-path PARSE_RULE_OPTION_RESET(true), // opt/repo-host-ca-path PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-host-ca-path PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-ca-path PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-ca-path // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-ca-path ), // opt/repo-host-ca-path // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-ca-path ), // opt/repo-host-ca-path // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-ca-path ), // opt/repo-host-ca-path // opt/repo-host-ca-path PARSE_RULE_OPTIONAL // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-ca-path PARSE_RULE_VAL_STRID(tls), // opt/repo-host-ca-path ), // opt/repo-host-ca-path ), // opt/repo-host-ca-path ), // opt/repo-host-ca-path ), // opt/repo-host-ca-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_OPTION_NAME("repo-host-cert-file"), // opt/repo-host-cert-file PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-cert-file PARSE_RULE_OPTION_RESET(true), // opt/repo-host-cert-file PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-cert-file PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-cert-file PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-cert-file // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-cert-file ), // opt/repo-host-cert-file // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-cert-file ), // opt/repo-host-cert-file // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-cert-file ), // opt/repo-host-cert-file // opt/repo-host-cert-file PARSE_RULE_OPTIONAL // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-cert-file PARSE_RULE_VAL_STRID(tls), // opt/repo-host-cert-file ), // opt/repo-host-cert-file ), // opt/repo-host-cert-file ), // opt/repo-host-cert-file ), // opt/repo-host-cert-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_OPTION_NAME("repo-host-cmd"), // opt/repo-host-cmd PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-cmd PARSE_RULE_OPTION_DEFAULT_TYPE(Dynamic), // opt/repo-host-cmd PARSE_RULE_OPTION_RESET(true), // opt/repo-host-cmd PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-cmd PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-cmd PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-cmd // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-cmd ), // opt/repo-host-cmd // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-cmd ), // opt/repo-host-cmd // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-cmd ), // opt/repo-host-cmd // opt/repo-host-cmd PARSE_RULE_OPTIONAL // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-cmd PARSE_RULE_VAL_STRID(ssh), // opt/repo-host-cmd ), // opt/repo-host-cmd // opt/repo-host-cmd PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_DEFAULT_DYNAMIC(Bin), // opt/repo-host-cmd ), // opt/repo-host-cmd ), // opt/repo-host-cmd ), // opt/repo-host-cmd ), // opt/repo-host-cmd // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_OPTION_NAME("repo-host-config"), // opt/repo-host-config PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-config PARSE_RULE_OPTION_RESET(true), // opt/repo-host-config PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-config PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-config PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-config // opt/repo-host-config PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-config ), // opt/repo-host-config // opt/repo-host-config PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config ), // opt/repo-host-config // opt/repo-host-config PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-config ), // opt/repo-host-config // opt/repo-host-config PARSE_RULE_OPTIONAL // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_VAL_OPT(RepoHost), // opt/repo-host-config ), // opt/repo-host-config // opt/repo-host-config PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_FILE), // opt/repo-host-config ), // opt/repo-host-config ), // opt/repo-host-config ), // opt/repo-host-config ), // opt/repo-host-config // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_OPTION_NAME("repo-host-config-include-path"), // opt/repo-host-config-include-path PARSE_RULE_OPTION_TYPE(Path), // opt/repo-host-config-include-path PARSE_RULE_OPTION_RESET(true), // opt/repo-host-config-include-path PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-config-include-path PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-config-include-path PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-config-include-path // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path // opt/repo-host-config-include-path PARSE_RULE_OPTIONAL // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_VAL_OPT(RepoHost), // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path // opt/repo-host-config-include-path PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_INCLUDE_PATH), ), // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_OPTION_NAME("repo-host-config-path"), // opt/repo-host-config-path PARSE_RULE_OPTION_TYPE(Path), // opt/repo-host-config-path PARSE_RULE_OPTION_RESET(true), // opt/repo-host-config-path PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-config-path PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-config-path PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-config-path // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-config-path ), // opt/repo-host-config-path // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config-path ), // opt/repo-host-config-path // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-config-path ), // opt/repo-host-config-path // opt/repo-host-config-path PARSE_RULE_OPTIONAL // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_VAL_OPT(RepoHost), // opt/repo-host-config-path ), // opt/repo-host-config-path // opt/repo-host-config-path PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH), // opt/repo-host-config-path ), // opt/repo-host-config-path ), // opt/repo-host-config-path ), // opt/repo-host-config-path ), // opt/repo-host-config-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_OPTION_NAME("repo-host-key-file"), // opt/repo-host-key-file PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-key-file PARSE_RULE_OPTION_RESET(true), // opt/repo-host-key-file PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-key-file PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-key-file PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-key-file // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-key-file ), // opt/repo-host-key-file // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-key-file ), // opt/repo-host-key-file // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-key-file ), // opt/repo-host-key-file // opt/repo-host-key-file PARSE_RULE_OPTIONAL // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-key-file PARSE_RULE_VAL_STRID(tls), // opt/repo-host-key-file ), // opt/repo-host-key-file ), // opt/repo-host-key-file ), // opt/repo-host-key-file ), // opt/repo-host-key-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTION_NAME("repo-host-port"), // opt/repo-host-port PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-host-port PARSE_RULE_OPTION_RESET(true), // opt/repo-host-port PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-host-port PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-port PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-port // opt/repo-host-port PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-port ), // opt/repo-host-port // opt/repo-host-port PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-port ), // opt/repo-host-port // opt/repo-host-port PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-port ), // opt/repo-host-port // opt/repo-host-port PARSE_RULE_OPTIONAL // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-port PARSE_RULE_VAL_STRID(ssh), // opt/repo-host-port PARSE_RULE_VAL_STRID(tls), // opt/repo-host-port ), // opt/repo-host-port // opt/repo-host-port PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_VAL_INT(0), // opt/repo-host-port PARSE_RULE_VAL_INT(65535), // opt/repo-host-port ), // opt/repo-host-port // opt/repo-host-port PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTIONAL_DEFAULT_MAP // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_VAL_STRID(tls), // opt/repo-host-port PARSE_RULE_VAL_INT(8432), // opt/repo-host-port ), // opt/repo-host-port ), // opt/repo-host-port ), // opt/repo-host-port ), // opt/repo-host-port ), // opt/repo-host-port // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_OPTION_NAME("repo-host-type"), // opt/repo-host-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-host-type PARSE_RULE_OPTION_RESET(true), // opt/repo-host-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-type // opt/repo-host-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-type ), // opt/repo-host-type // opt/repo-host-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-type ), // opt/repo-host-type // opt/repo-host-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-type ), // opt/repo-host-type // opt/repo-host-type PARSE_RULE_OPTIONAL // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_VAL_OPT(RepoHost), // opt/repo-host-type ), // opt/repo-host-type // opt/repo-host-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_VAL_STRID(ssh), // opt/repo-host-type PARSE_RULE_VAL_STRID(tls), // opt/repo-host-type ), // opt/repo-host-type // opt/repo-host-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_VAL_STRID(ssh), // opt/repo-host-type ), // opt/repo-host-type ), // opt/repo-host-type ), // opt/repo-host-type ), // opt/repo-host-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_OPTION_NAME("repo-host-user"), // opt/repo-host-user PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-user PARSE_RULE_OPTION_RESET(true), // opt/repo-host-user PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-host-user PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-user PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-user // opt/repo-host-user PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-user ), // opt/repo-host-user // opt/repo-host-user PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-user ), // opt/repo-host-user // opt/repo-host-user PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-user ), // opt/repo-host-user // opt/repo-host-user PARSE_RULE_OPTIONAL // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-user PARSE_RULE_VAL_STRID(ssh), // opt/repo-host-user ), // opt/repo-host-user // opt/repo-host-user PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_VAL_STR(QT_pgbackrest_QT), // opt/repo-host-user ), // opt/repo-host-user ), // opt/repo-host-user ), // opt/repo-host-user ), // opt/repo-host-user // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-local ( // opt/repo-local PARSE_RULE_OPTION_NAME("repo-local"), // opt/repo-local PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-local PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-local PARSE_RULE_OPTION_NEGATE(true), // opt/repo-local PARSE_RULE_OPTION_RESET(true), // opt/repo-local PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-local PARSE_RULE_OPTION_SECTION(Global), // opt/repo-local PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-local // opt/repo-local PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-local ( // opt/repo-local PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-local PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-local PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-local PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-local PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-local PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-local PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-local PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-local PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-local ), // opt/repo-local // opt/repo-local PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-local ( // opt/repo-local PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-local PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-local ), // opt/repo-local // opt/repo-local PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-local ( // opt/repo-local PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-local PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-local ), // opt/repo-local // opt/repo-local PARSE_RULE_OPTIONAL // opt/repo-local ( // opt/repo-local PARSE_RULE_OPTIONAL_GROUP // opt/repo-local ( // opt/repo-local PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-local ( // opt/repo-local PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-local ), // opt/repo-local ), // opt/repo-local ), // opt/repo-local ), // opt/repo-local // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTION_NAME("repo-path"), // opt/repo-path PARSE_RULE_OPTION_TYPE(Path), // opt/repo-path PARSE_RULE_OPTION_RESET(true), // opt/repo-path PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-path PARSE_RULE_OPTION_SECTION(Global), // opt/repo-path PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-path PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-path // opt/repo-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-path ), // opt/repo-path // opt/repo-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-path ), // opt/repo-path // opt/repo-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-path ), // opt/repo-path // opt/repo-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-path ), // opt/repo-path // opt/repo-path PARSE_RULE_OPTIONAL // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTIONAL_GROUP // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-path ( // opt/repo-path PARSE_RULE_VAL_STR(QT_FS_var_FS_lib_FS_pgbackrest_QT), // opt/repo-path ), // opt/repo-path ), // opt/repo-path ), // opt/repo-path ), // opt/repo-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-retention-archive ( // opt/repo-retention-archive PARSE_RULE_OPTION_NAME("repo-retention-archive"), // opt/repo-retention-archive PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-retention-archive PARSE_RULE_OPTION_RESET(true), // opt/repo-retention-archive PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-retention-archive PARSE_RULE_OPTION_SECTION(Global), // opt/repo-retention-archive PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-retention-archive // opt/repo-retention-archive PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-retention-archive ( // opt/repo-retention-archive PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-retention-archive PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-retention-archive ), // opt/repo-retention-archive // opt/repo-retention-archive PARSE_RULE_OPTIONAL // opt/repo-retention-archive ( // opt/repo-retention-archive PARSE_RULE_OPTIONAL_GROUP // opt/repo-retention-archive ( // opt/repo-retention-archive PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-retention-archive ( // opt/repo-retention-archive PARSE_RULE_VAL_INT(1), // opt/repo-retention-archive PARSE_RULE_VAL_INT(9999999), // opt/repo-retention-archive ), // opt/repo-retention-archive ), // opt/repo-retention-archive ), // opt/repo-retention-archive ), // opt/repo-retention-archive // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-retention-archive-type ( // opt/repo-retention-archive-type PARSE_RULE_OPTION_NAME("repo-retention-archive-type"), // opt/repo-retention-archive-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-retention-archive-type PARSE_RULE_OPTION_RESET(true), // opt/repo-retention-archive-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-retention-archive-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-retention-archive-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-retention-archive-type // opt/repo-retention-archive-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-retention-archive-type ( // opt/repo-retention-archive-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-retention-archive-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-retention-archive-type ), // opt/repo-retention-archive-type // opt/repo-retention-archive-type PARSE_RULE_OPTIONAL // opt/repo-retention-archive-type ( // opt/repo-retention-archive-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-retention-archive-type ( // opt/repo-retention-archive-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-retention-archive-type ( // opt/repo-retention-archive-type PARSE_RULE_VAL_STRID(full), // opt/repo-retention-archive-type PARSE_RULE_VAL_STRID(diff), // opt/repo-retention-archive-type PARSE_RULE_VAL_STRID(incr), // opt/repo-retention-archive-type ), // opt/repo-retention-archive-type // opt/repo-retention-archive-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-retention-archive-type ( // opt/repo-retention-archive-type PARSE_RULE_VAL_STRID(full), // opt/repo-retention-archive-type ), // opt/repo-retention-archive-type ), // opt/repo-retention-archive-type ), // opt/repo-retention-archive-type ), // opt/repo-retention-archive-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-retention-diff ( // opt/repo-retention-diff PARSE_RULE_OPTION_NAME("repo-retention-diff"), // opt/repo-retention-diff PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-retention-diff PARSE_RULE_OPTION_RESET(true), // opt/repo-retention-diff PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-retention-diff PARSE_RULE_OPTION_SECTION(Global), // opt/repo-retention-diff PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-retention-diff // opt/repo-retention-diff PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-retention-diff ( // opt/repo-retention-diff PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-retention-diff PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-retention-diff ), // opt/repo-retention-diff // opt/repo-retention-diff PARSE_RULE_OPTIONAL // opt/repo-retention-diff ( // opt/repo-retention-diff PARSE_RULE_OPTIONAL_GROUP // opt/repo-retention-diff ( // opt/repo-retention-diff PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-retention-diff ( // opt/repo-retention-diff PARSE_RULE_VAL_INT(1), // opt/repo-retention-diff PARSE_RULE_VAL_INT(9999999), // opt/repo-retention-diff ), // opt/repo-retention-diff ), // opt/repo-retention-diff ), // opt/repo-retention-diff ), // opt/repo-retention-diff // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-retention-full ( // opt/repo-retention-full PARSE_RULE_OPTION_NAME("repo-retention-full"), // opt/repo-retention-full PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-retention-full PARSE_RULE_OPTION_RESET(true), // opt/repo-retention-full PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-retention-full PARSE_RULE_OPTION_SECTION(Global), // opt/repo-retention-full PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-retention-full // opt/repo-retention-full PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-retention-full ( // opt/repo-retention-full PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-retention-full PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-retention-full ), // opt/repo-retention-full // opt/repo-retention-full PARSE_RULE_OPTIONAL // opt/repo-retention-full ( // opt/repo-retention-full PARSE_RULE_OPTIONAL_GROUP // opt/repo-retention-full ( // opt/repo-retention-full PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-retention-full ( // opt/repo-retention-full PARSE_RULE_VAL_INT(1), // opt/repo-retention-full PARSE_RULE_VAL_INT(9999999), // opt/repo-retention-full ), // opt/repo-retention-full ), // opt/repo-retention-full ), // opt/repo-retention-full ), // opt/repo-retention-full // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-retention-full-type ( // opt/repo-retention-full-type PARSE_RULE_OPTION_NAME("repo-retention-full-type"), // opt/repo-retention-full-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-retention-full-type PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-retention-full-type PARSE_RULE_OPTION_RESET(true), // opt/repo-retention-full-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-retention-full-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-retention-full-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-retention-full-type // opt/repo-retention-full-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-retention-full-type ( // opt/repo-retention-full-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-retention-full-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-retention-full-type ), // opt/repo-retention-full-type // opt/repo-retention-full-type PARSE_RULE_OPTIONAL // opt/repo-retention-full-type ( // opt/repo-retention-full-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-retention-full-type ( // opt/repo-retention-full-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-retention-full-type ( // opt/repo-retention-full-type PARSE_RULE_VAL_STRID(count), // opt/repo-retention-full-type PARSE_RULE_VAL_STRID(time), // opt/repo-retention-full-type ), // opt/repo-retention-full-type // opt/repo-retention-full-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-retention-full-type ( // opt/repo-retention-full-type PARSE_RULE_VAL_STRID(count), // opt/repo-retention-full-type ), // opt/repo-retention-full-type ), // opt/repo-retention-full-type ), // opt/repo-retention-full-type ), // opt/repo-retention-full-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-retention-history ( // opt/repo-retention-history PARSE_RULE_OPTION_NAME("repo-retention-history"), // opt/repo-retention-history PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-retention-history PARSE_RULE_OPTION_RESET(true), // opt/repo-retention-history PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-retention-history PARSE_RULE_OPTION_SECTION(Global), // opt/repo-retention-history PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-retention-history // opt/repo-retention-history PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-retention-history ( // opt/repo-retention-history PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-retention-history PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-retention-history ), // opt/repo-retention-history // opt/repo-retention-history PARSE_RULE_OPTIONAL // opt/repo-retention-history ( // opt/repo-retention-history PARSE_RULE_OPTIONAL_GROUP // opt/repo-retention-history ( // opt/repo-retention-history PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-retention-history ( // opt/repo-retention-history PARSE_RULE_VAL_INT(0), // opt/repo-retention-history PARSE_RULE_VAL_INT(9999999), // opt/repo-retention-history ), // opt/repo-retention-history ), // opt/repo-retention-history ), // opt/repo-retention-history ), // opt/repo-retention-history // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTION_NAME("repo-s3-bucket"), // opt/repo-s3-bucket PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-bucket PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-bucket PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-bucket PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-bucket PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-bucket PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-s3-bucket // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-bucket ), // opt/repo-s3-bucket // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-bucket ), // opt/repo-s3-bucket // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-bucket ), // opt/repo-s3-bucket // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-bucket ), // opt/repo-s3-bucket // opt/repo-s3-bucket PARSE_RULE_OPTIONAL // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-bucket PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-bucket ), // opt/repo-s3-bucket ), // opt/repo-s3-bucket ), // opt/repo-s3-bucket ), // opt/repo-s3-bucket // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTION_NAME("repo-s3-endpoint"), // opt/repo-s3-endpoint PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-endpoint PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-endpoint PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-endpoint PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-endpoint PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-endpoint PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-s3-endpoint // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint // opt/repo-s3-endpoint PARSE_RULE_OPTIONAL // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-endpoint PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTION_NAME("repo-s3-key"), // opt/repo-s3-key PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-key PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-key PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-key PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-key PARSE_RULE_OPTION_SECURE(true), // opt/repo-s3-key PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-key PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-s3-key // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key ), // opt/repo-s3-key // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key ), // opt/repo-s3-key // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key ), // opt/repo-s3-key // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key ), // opt/repo-s3-key // opt/repo-s3-key PARSE_RULE_OPTIONAL // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_VAL_OPT(RepoS3KeyType), // opt/repo-s3-key PARSE_RULE_VAL_STRID(shared), // opt/repo-s3-key ), // opt/repo-s3-key ), // opt/repo-s3-key ), // opt/repo-s3-key ), // opt/repo-s3-key // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTION_NAME("repo-s3-key-secret"), // opt/repo-s3-key-secret PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-key-secret PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-key-secret PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-key-secret PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-key-secret PARSE_RULE_OPTION_SECURE(true), // opt/repo-s3-key-secret PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-key-secret PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-s3-key-secret // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret // opt/repo-s3-key-secret PARSE_RULE_OPTIONAL // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_VAL_OPT(RepoS3KeyType), // opt/repo-s3-key-secret PARSE_RULE_VAL_STRID(shared), // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTION_NAME("repo-s3-key-type"), // opt/repo-s3-key-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-s3-key-type PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-s3-key-type PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-key-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-key-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-key-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key-type ), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-type ), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key-type ), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key-type ), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTIONAL // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-key-type ), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(shared), // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(auto), // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(web_DS_id), // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(pod_DS_id), // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(process), // opt/repo-s3-key-type ), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(shared), // opt/repo-s3-key-type ), // opt/repo-s3-key-type ), // opt/repo-s3-key-type ), // opt/repo-s3-key-type ), // opt/repo-s3-key-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_NAME("repo-s3-kms-key-id"), // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-kms-key-id // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id // opt/repo-s3-kms-key-id PARSE_RULE_OPTIONAL // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-kms-key-id PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTION_NAME("repo-s3-process-cmd"), // opt/repo-s3-process-cmd PARSE_RULE_OPTION_TYPE(List), // opt/repo-s3-process-cmd PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-process-cmd PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-process-cmd PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-process-cmd PARSE_RULE_OPTION_MULTI(true), // opt/repo-s3-process-cmd PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-process-cmd // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd // opt/repo-s3-process-cmd PARSE_RULE_OPTIONAL // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_VAL_OPT(RepoS3KeyType), // opt/repo-s3-process-cmd PARSE_RULE_VAL_STRID(process), // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTION_NAME("repo-s3-region"), // opt/repo-s3-region PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-region PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-region PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-region PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-region PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-region PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-s3-region // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-region ), // opt/repo-s3-region // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-region ), // opt/repo-s3-region // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-region ), // opt/repo-s3-region // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-region ), // opt/repo-s3-region // opt/repo-s3-region PARSE_RULE_OPTIONAL // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-region PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-region ), // opt/repo-s3-region ), // opt/repo-s3-region ), // opt/repo-s3-region ), // opt/repo-s3-region // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTION_NAME("repo-s3-requester-pays"), // opt/repo-s3-requester-pays PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-s3-requester-pays PARSE_RULE_OPTION_NEGATE(true), // opt/repo-s3-requester-pays PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-requester-pays PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-requester-pays PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-requester-pays PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-requester-pays // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays // opt/repo-s3-requester-pays PARSE_RULE_OPTIONAL // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-requester-pays PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays // opt/repo-s3-requester-pays PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTION_NAME("repo-s3-role"), // opt/repo-s3-role PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-role PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-role PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-s3-role PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-role PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-role // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-role ), // opt/repo-s3-role // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-role ), // opt/repo-s3-role // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-role ), // opt/repo-s3-role // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-role ), // opt/repo-s3-role // opt/repo-s3-role PARSE_RULE_OPTIONAL // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_VAL_OPT(RepoS3KeyType), // opt/repo-s3-role PARSE_RULE_VAL_STRID(auto), // opt/repo-s3-role ), // opt/repo-s3-role ), // opt/repo-s3-role ), // opt/repo-s3-role ), // opt/repo-s3-role // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTION_NAME("repo-s3-service"), // opt/repo-s3-service PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-service PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-service PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-s3-service PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-service PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-service // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-service ), // opt/repo-s3-service // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-service ), // opt/repo-s3-service // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-service ), // opt/repo-s3-service // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-service ), // opt/repo-s3-service // opt/repo-s3-service PARSE_RULE_OPTIONAL // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-service PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-service ), // opt/repo-s3-service // opt/repo-s3-service PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_VAL_STR(QT_s3_QT), // opt/repo-s3-service ), // opt/repo-s3-service ), // opt/repo-s3-service ), // opt/repo-s3-service ), // opt/repo-s3-service // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_NAME("repo-s3-sse-customer-key"), // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_SECURE(true), // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-sse-customer-key // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key // opt/repo-s3-sse-customer-key PARSE_RULE_OPTIONAL // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-sse-customer-key PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTION_NAME("repo-s3-sts-host"), // opt/repo-s3-sts-host PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-sts-host PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-sts-host PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-sts-host PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-sts-host PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-sts-host // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host // opt/repo-s3-sts-host PARSE_RULE_OPTIONAL // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_VAL_OPT(RepoS3KeyType), // opt/repo-s3-sts-host PARSE_RULE_VAL_STRID(web_DS_id), // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host // opt/repo-s3-sts-host PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_VAL_STR(QT_sts_DT_amazonaws_DT_com_QT), // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTION_NAME("repo-s3-token"), // opt/repo-s3-token PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-token PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-token PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-s3-token PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-token PARSE_RULE_OPTION_SECURE(true), // opt/repo-s3-token PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-token // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-token ), // opt/repo-s3-token // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-token ), // opt/repo-s3-token // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-token ), // opt/repo-s3-token // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-token ), // opt/repo-s3-token // opt/repo-s3-token PARSE_RULE_OPTIONAL // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_VAL_OPT(RepoS3KeyType), // opt/repo-s3-token PARSE_RULE_VAL_STRID(shared), // opt/repo-s3-token ), // opt/repo-s3-token ), // opt/repo-s3-token ), // opt/repo-s3-token ), // opt/repo-s3-token // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTION_NAME("repo-s3-uri-style"), // opt/repo-s3-uri-style PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-s3-uri-style PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-s3-uri-style PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-uri-style PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-uri-style PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-uri-style PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTIONAL // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-uri-style PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_VAL_STRID(host), // opt/repo-s3-uri-style PARSE_RULE_VAL_STRID(path), // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_VAL_STRID(host), // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTION_NAME("repo-sftp-host"), // opt/repo-sftp-host PARSE_RULE_OPTION_TYPE(String), // opt/repo-sftp-host PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-host PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-sftp-host PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-host PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-host // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host ), // opt/repo-sftp-host // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host ), // opt/repo-sftp-host // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host ), // opt/repo-sftp-host // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host ), // opt/repo-sftp-host // opt/repo-sftp-host PARSE_RULE_OPTIONAL // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-host PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-host ), // opt/repo-sftp-host ), // opt/repo-sftp-host ), // opt/repo-sftp-host ), // opt/repo-sftp-host // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_NAME("repo-sftp-host-fingerprint"), // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_TYPE(String), // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-host-fingerprint // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTIONAL // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_VAL_OPT(RepoSftpHostKeyCheckType), // opt/repo-sftp-host-fingerprint PARSE_RULE_VAL_STRID(fingerprint), // opt/repo-sftp-host-fingerprint PARSE_RULE_VAL_STRID(strict), // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_NAME("repo-sftp-host-key-check-type"), // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTIONAL // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_STRID(accept_DS_new), // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_STRID(fingerprint), // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_STRID(none), // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_STRID(strict), // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_STRID(strict), // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_NAME("repo-sftp-host-key-hash-type"), // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-host-key-hash-type // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTIONAL // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-host-key-hash-type PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_VAL_STRID(md5), // opt/repo-sftp-host-key-hash-type PARSE_RULE_VAL_STRID(sha1), // opt/repo-sftp-host-key-hash-type PARSE_RULE_VAL_STRID(sha256), // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTION_NAME("repo-sftp-host-port"), // opt/repo-sftp-host-port PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-sftp-host-port PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-host-port PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-sftp-host-port PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-host-port PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTIONAL // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-host-port PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_VAL_INT(1), // opt/repo-sftp-host-port PARSE_RULE_VAL_INT(65535), // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_VAL_INT(22), // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTION_NAME("repo-sftp-host-user"), // opt/repo-sftp-host-user PARSE_RULE_OPTION_TYPE(String), // opt/repo-sftp-host-user PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-host-user PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-sftp-host-user PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-host-user PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-host-user // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user // opt/repo-sftp-host-user PARSE_RULE_OPTIONAL // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-host-user PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTION_NAME("repo-sftp-known-host"), // opt/repo-sftp-known-host PARSE_RULE_OPTION_TYPE(List), // opt/repo-sftp-known-host PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-known-host PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-sftp-known-host PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-known-host PARSE_RULE_OPTION_MULTI(true), // opt/repo-sftp-known-host PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-known-host // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host // opt/repo-sftp-known-host PARSE_RULE_OPTIONAL // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_VAL_OPT(RepoSftpHostKeyCheckType), // opt/repo-sftp-known-host PARSE_RULE_VAL_STRID(accept_DS_new), // opt/repo-sftp-known-host PARSE_RULE_VAL_STRID(strict), // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_NAME("repo-sftp-private-key-file"), // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_TYPE(String), // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-private-key-file // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file // opt/repo-sftp-private-key-file PARSE_RULE_OPTIONAL // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-private-key-file PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_NAME("repo-sftp-private-key-passphrase"), // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_TYPE(String), // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_SECURE(true), // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-private-key-passphrase // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTIONAL // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-private-key-passphrase PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_NAME("repo-sftp-public-key-file"), // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_TYPE(String), // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-public-key-file // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file // opt/repo-sftp-public-key-file PARSE_RULE_OPTIONAL // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-public-key-file PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTION_NAME("repo-storage-ca-file"), // opt/repo-storage-ca-file PARSE_RULE_OPTION_TYPE(String), // opt/repo-storage-ca-file PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-ca-file PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-storage-ca-file PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-ca-file PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-ca-file // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file // opt/repo-storage-ca-file PARSE_RULE_OPTIONAL // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-ca-file PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-ca-file PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-ca-file PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTION_NAME("repo-storage-ca-path"), // opt/repo-storage-ca-path PARSE_RULE_OPTION_TYPE(Path), // opt/repo-storage-ca-path PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-ca-path PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-storage-ca-path PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-ca-path PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-ca-path // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path // opt/repo-storage-ca-path PARSE_RULE_OPTIONAL // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-ca-path PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-ca-path PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-ca-path PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTION_NAME("repo-storage-host"), // opt/repo-storage-host PARSE_RULE_OPTION_TYPE(String), // opt/repo-storage-host PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-host PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-storage-host PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-host PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-host // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-host ), // opt/repo-storage-host // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-host ), // opt/repo-storage-host // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-host ), // opt/repo-storage-host // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-host ), // opt/repo-storage-host // opt/repo-storage-host PARSE_RULE_OPTIONAL // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-host PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-host PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-host ), // opt/repo-storage-host ), // opt/repo-storage-host ), // opt/repo-storage-host ), // opt/repo-storage-host // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTION_NAME("repo-storage-port"), // opt/repo-storage-port PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-storage-port PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-port PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-storage-port PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-port PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-port ), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-port ), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-port ), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-port ), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTIONAL // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-port PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-port PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-port ), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_VAL_INT(1), // opt/repo-storage-port PARSE_RULE_VAL_INT(65535), // opt/repo-storage-port ), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_VAL_INT(443), // opt/repo-storage-port ), // opt/repo-storage-port ), // opt/repo-storage-port ), // opt/repo-storage-port ), // opt/repo-storage-port // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-prefetch ( // opt/repo-storage-prefetch PARSE_RULE_OPTION_NAME("repo-storage-prefetch"), // opt/repo-storage-prefetch PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-storage-prefetch PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-prefetch PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-storage-prefetch PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-prefetch PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-prefetch // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-prefetch ( // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-prefetch ), // opt/repo-storage-prefetch // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-prefetch ( // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-prefetch ), // opt/repo-storage-prefetch // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-prefetch ( // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-prefetch ), // opt/repo-storage-prefetch // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-prefetch ( // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-prefetch PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-prefetch ), // opt/repo-storage-prefetch // opt/repo-storage-prefetch PARSE_RULE_OPTIONAL // opt/repo-storage-prefetch ( // opt/repo-storage-prefetch PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-prefetch ( // opt/repo-storage-prefetch PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-prefetch ( // opt/repo-storage-prefetch PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-prefetch PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-prefetch PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-prefetch PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-prefetch ), // opt/repo-storage-prefetch // opt/repo-storage-prefetch PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-storage-prefetch ( // opt/repo-storage-prefetch PARSE_RULE_VAL_INT(0), // opt/repo-storage-prefetch PARSE_RULE_VAL_INT(1024), // opt/repo-storage-prefetch ), // opt/repo-storage-prefetch // opt/repo-storage-prefetch PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-storage-prefetch ( // opt/repo-storage-prefetch PARSE_RULE_VAL_INT(3), // opt/repo-storage-prefetch ), // opt/repo-storage-prefetch ), // opt/repo-storage-prefetch ), // opt/repo-storage-prefetch ), // opt/repo-storage-prefetch // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-read-over ( // opt/repo-storage-read-over PARSE_RULE_OPTION_NAME("repo-storage-read-over"), // opt/repo-storage-read-over PARSE_RULE_OPTION_TYPE(Size), // opt/repo-storage-read-over PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-read-over PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-storage-read-over PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-read-over PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-read-over // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-read-over ( // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-read-over ), // opt/repo-storage-read-over // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-read-over ( // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-read-over ), // opt/repo-storage-read-over // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-read-over ( // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-read-over ), // opt/repo-storage-read-over // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-read-over ( // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-read-over PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-read-over ), // opt/repo-storage-read-over // opt/repo-storage-read-over PARSE_RULE_OPTIONAL // opt/repo-storage-read-over ( // opt/repo-storage-read-over PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-read-over ( // opt/repo-storage-read-over PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-read-over ( // opt/repo-storage-read-over PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-read-over PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-read-over PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-read-over PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-read-over ), // opt/repo-storage-read-over // opt/repo-storage-read-over PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-storage-read-over ( // opt/repo-storage-read-over PARSE_RULE_VAL_SIZE(0B), // opt/repo-storage-read-over PARSE_RULE_VAL_SIZE(1MiB), // opt/repo-storage-read-over ), // opt/repo-storage-read-over // opt/repo-storage-read-over PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-storage-read-over ( // opt/repo-storage-read-over PARSE_RULE_VAL_SIZE(64KiB), // opt/repo-storage-read-over ), // opt/repo-storage-read-over ), // opt/repo-storage-read-over ), // opt/repo-storage-read-over ), // opt/repo-storage-read-over // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTION_NAME("repo-storage-tag"), // opt/repo-storage-tag PARSE_RULE_OPTION_TYPE(Hash), // opt/repo-storage-tag PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-tag PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-storage-tag PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-tag PARSE_RULE_OPTION_MULTI(true), // opt/repo-storage-tag PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-tag // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-tag ), // opt/repo-storage-tag // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-tag ), // opt/repo-storage-tag // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-tag ), // opt/repo-storage-tag // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-tag ), // opt/repo-storage-tag // opt/repo-storage-tag PARSE_RULE_OPTIONAL // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-tag PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-tag PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-tag PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-tag ), // opt/repo-storage-tag ), // opt/repo-storage-tag ), // opt/repo-storage-tag ), // opt/repo-storage-tag // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_NAME("repo-storage-upload-chunk-size"), // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_TYPE(Size), // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL_ALLOW_RANGE_MAP // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(4MiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(1GiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(4MiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(1GiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(5MiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(1GiB), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL_DEFAULT_MAP // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(4MiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(4MiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(5MiB), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTION_NAME("repo-storage-verify-tls"), // opt/repo-storage-verify-tls PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-storage-verify-tls PARSE_RULE_OPTION_NEGATE(true), // opt/repo-storage-verify-tls PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-verify-tls PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-storage-verify-tls PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-verify-tls PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-verify-tls // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls // opt/repo-storage-verify-tls PARSE_RULE_OPTIONAL // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-verify-tls PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-verify-tls PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-verify-tls PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls // opt/repo-storage-verify-tls PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-symlink ( // opt/repo-symlink PARSE_RULE_OPTION_NAME("repo-symlink"), // opt/repo-symlink PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-symlink PARSE_RULE_OPTION_NEGATE(true), // opt/repo-symlink PARSE_RULE_OPTION_RESET(true), // opt/repo-symlink PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-symlink PARSE_RULE_OPTION_SECTION(Global), // opt/repo-symlink PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-symlink // opt/repo-symlink PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-symlink ( // opt/repo-symlink PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-symlink PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-symlink ), // opt/repo-symlink // opt/repo-symlink PARSE_RULE_OPTIONAL // opt/repo-symlink ( // opt/repo-symlink PARSE_RULE_OPTIONAL_GROUP // opt/repo-symlink ( // opt/repo-symlink PARSE_RULE_OPTIONAL_DEPEND // opt/repo-symlink ( // opt/repo-symlink PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_BOOL_FALSE), // opt/repo-symlink PARSE_RULE_VAL_OPT(RepoType), // opt/repo-symlink PARSE_RULE_VAL_STRID(posix), // opt/repo-symlink ), // opt/repo-symlink // opt/repo-symlink PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-symlink ( // opt/repo-symlink PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-symlink ), // opt/repo-symlink ), // opt/repo-symlink ), // opt/repo-symlink ), // opt/repo-symlink // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-target-time ( // opt/repo-target-time PARSE_RULE_OPTION_NAME("repo-target-time"), // opt/repo-target-time PARSE_RULE_OPTION_TYPE(String), // opt/repo-target-time PARSE_RULE_OPTION_RESET(true), // opt/repo-target-time PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-target-time PARSE_RULE_OPTION_SECTION(Global), // opt/repo-target-time // opt/repo-target-time PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-target-time ( // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-target-time ), // opt/repo-target-time // opt/repo-target-time PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-target-time ( // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-target-time ), // opt/repo-target-time // opt/repo-target-time PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-target-time ( // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-target-time ), // opt/repo-target-time ), // opt/repo-target-time // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTION_NAME("repo-type"), // opt/repo-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-type PARSE_RULE_OPTION_RESET(true), // opt/repo-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-type PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-type // opt/repo-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-type ), // opt/repo-type // opt/repo-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-type ), // opt/repo-type // opt/repo-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-type ), // opt/repo-type // opt/repo-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-type ), // opt/repo-type // opt/repo-type PARSE_RULE_OPTIONAL // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-type ( // opt/repo-type PARSE_RULE_VAL_STRID(azure), // opt/repo-type PARSE_RULE_VAL_STRID(cifs), // opt/repo-type PARSE_RULE_VAL_STRID(gcs), // opt/repo-type PARSE_RULE_VAL_STRID(posix), // opt/repo-type PARSE_RULE_VAL_STRID(s3), // opt/repo-type PARSE_RULE_VAL_STRID(sftp), // opt/repo-type #ifndef HAVE_LIBSSH2 // opt/repo-type PARSE_RULE_BOOL_FALSE, // opt/repo-type #endif // opt/repo-type ), // opt/repo-type // opt/repo-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-type ( // opt/repo-type PARSE_RULE_VAL_STRID(posix), // opt/repo-type ), // opt/repo-type ), // opt/repo-type ), // opt/repo-type ), // opt/repo-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/report ( // opt/report PARSE_RULE_OPTION_NAME("report"), // opt/report PARSE_RULE_OPTION_TYPE(Boolean), // opt/report PARSE_RULE_OPTION_INTERNAL(true) // opt/report PARSE_RULE_OPTION_REQUIRED(true), // opt/report PARSE_RULE_OPTION_SECTION(CommandLine), // opt/report // opt/report PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/report ( // opt/report PARSE_RULE_OPTION_COMMAND(Check) // opt/report ), // opt/report // opt/report PARSE_RULE_OPTIONAL // opt/report ( // opt/report PARSE_RULE_OPTIONAL_GROUP // opt/report ( // opt/report PARSE_RULE_OPTIONAL_DEFAULT // opt/report ( // opt/report PARSE_RULE_VAL_BOOL_FALSE, // opt/report ), // opt/report ), // opt/report ), // opt/report ), // opt/report // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/resume ( // opt/resume PARSE_RULE_OPTION_NAME("resume"), // opt/resume PARSE_RULE_OPTION_TYPE(Boolean), // opt/resume PARSE_RULE_OPTION_NEGATE(true), // opt/resume PARSE_RULE_OPTION_RESET(true), // opt/resume PARSE_RULE_OPTION_REQUIRED(true), // opt/resume PARSE_RULE_OPTION_SECTION(Global), // opt/resume // opt/resume PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/resume ( // opt/resume PARSE_RULE_OPTION_COMMAND(Backup) // opt/resume ), // opt/resume // opt/resume PARSE_RULE_OPTIONAL // opt/resume ( // opt/resume PARSE_RULE_OPTIONAL_GROUP // opt/resume ( // opt/resume PARSE_RULE_OPTIONAL_DEFAULT // opt/resume ( // opt/resume PARSE_RULE_VAL_BOOL_TRUE, // opt/resume ), // opt/resume ), // opt/resume ), // opt/resume ), // opt/resume // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTION_NAME("sck-block"), // opt/sck-block PARSE_RULE_OPTION_TYPE(Boolean), // opt/sck-block PARSE_RULE_OPTION_INTERNAL(true) // opt/sck-block PARSE_RULE_OPTION_NEGATE(true), // opt/sck-block PARSE_RULE_OPTION_RESET(true), // opt/sck-block PARSE_RULE_OPTION_REQUIRED(true), // opt/sck-block PARSE_RULE_OPTION_SECTION(Global), // opt/sck-block // opt/sck-block PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTION_COMMAND(Annotate) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Backup) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Check) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Expire) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Info) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Manifest) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Restore) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Server) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/sck-block PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/sck-block PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/sck-block PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Verify) // opt/sck-block ), // opt/sck-block // opt/sck-block PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-block ), // opt/sck-block // opt/sck-block PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Backup) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Restore) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Verify) // opt/sck-block ), // opt/sck-block // opt/sck-block PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTION_COMMAND(Annotate) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Backup) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Check) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Expire) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Info) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Manifest) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Restore) // opt/sck-block PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/sck-block PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/sck-block PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Verify) // opt/sck-block ), // opt/sck-block // opt/sck-block PARSE_RULE_OPTIONAL // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTIONAL_GROUP // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTIONAL_DEFAULT // opt/sck-block ( // opt/sck-block PARSE_RULE_VAL_BOOL_FALSE, // opt/sck-block ), // opt/sck-block ), // opt/sck-block ), // opt/sck-block ), // opt/sck-block // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTION_NAME("sck-keep-alive"), // opt/sck-keep-alive PARSE_RULE_OPTION_TYPE(Boolean), // opt/sck-keep-alive PARSE_RULE_OPTION_NEGATE(true), // opt/sck-keep-alive PARSE_RULE_OPTION_RESET(true), // opt/sck-keep-alive PARSE_RULE_OPTION_REQUIRED(true), // opt/sck-keep-alive PARSE_RULE_OPTION_SECTION(Global), // opt/sck-keep-alive // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Annotate) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Backup) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Check) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Expire) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Info) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Manifest) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Restore) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Server) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Verify) // opt/sck-keep-alive ), // opt/sck-keep-alive // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-keep-alive ), // opt/sck-keep-alive // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Backup) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Restore) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Verify) // opt/sck-keep-alive ), // opt/sck-keep-alive // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Annotate) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Backup) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Check) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Expire) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Info) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Manifest) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Restore) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Verify) // opt/sck-keep-alive ), // opt/sck-keep-alive // opt/sck-keep-alive PARSE_RULE_OPTIONAL // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTIONAL_GROUP // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTIONAL_DEFAULT // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_VAL_BOOL_TRUE, // opt/sck-keep-alive ), // opt/sck-keep-alive ), // opt/sck-keep-alive ), // opt/sck-keep-alive ), // opt/sck-keep-alive // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/set ( // opt/set PARSE_RULE_OPTION_NAME("set"), // opt/set PARSE_RULE_OPTION_TYPE(String), // opt/set PARSE_RULE_OPTION_REQUIRED(false), // opt/set PARSE_RULE_OPTION_SECTION(CommandLine), // opt/set // opt/set PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/set ( // opt/set PARSE_RULE_OPTION_COMMAND(Annotate) // opt/set PARSE_RULE_OPTION_COMMAND(Expire) // opt/set PARSE_RULE_OPTION_COMMAND(Info) // opt/set PARSE_RULE_OPTION_COMMAND(Manifest) // opt/set PARSE_RULE_OPTION_COMMAND(Restore) // opt/set PARSE_RULE_OPTION_COMMAND(Verify) // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL // opt/set ( // opt/set PARSE_RULE_OPTIONAL_GROUP // opt/set ( // opt/set PARSE_RULE_FILTER_CMD // opt/set ( // opt/set PARSE_RULE_VAL_CMD(Annotate), // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL_REQUIRED(), // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL_GROUP // opt/set ( // opt/set PARSE_RULE_FILTER_CMD // opt/set ( // opt/set PARSE_RULE_VAL_CMD(Info), // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL_DEPEND // opt/set ( // opt/set PARSE_RULE_VAL_OPT(Stanza), // opt/set ), // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL_GROUP // opt/set ( // opt/set PARSE_RULE_FILTER_CMD // opt/set ( // opt/set PARSE_RULE_VAL_CMD(Manifest), // opt/set PARSE_RULE_VAL_CMD(Restore), // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL_DEFAULT // opt/set ( // opt/set PARSE_RULE_VAL_STR(QT_latest_QT), // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL_REQUIRED(), // opt/set ), // opt/set ), // opt/set ), // opt/set // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/sort ( // opt/sort PARSE_RULE_OPTION_NAME("sort"), // opt/sort PARSE_RULE_OPTION_TYPE(StringId), // opt/sort PARSE_RULE_OPTION_SEQUENCE(true), // opt/sort PARSE_RULE_OPTION_REQUIRED(true), // opt/sort PARSE_RULE_OPTION_SECTION(CommandLine), // opt/sort // opt/sort PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/sort ( // opt/sort PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/sort ), // opt/sort // opt/sort PARSE_RULE_OPTIONAL // opt/sort ( // opt/sort PARSE_RULE_OPTIONAL_GROUP // opt/sort ( // opt/sort PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/sort ( // opt/sort PARSE_RULE_VAL_STRID(none), // opt/sort PARSE_RULE_VAL_STRID(asc), // opt/sort PARSE_RULE_VAL_STRID(desc), // opt/sort ), // opt/sort // opt/sort PARSE_RULE_OPTIONAL_DEFAULT // opt/sort ( // opt/sort PARSE_RULE_VAL_STRID(asc), // opt/sort PARSE_RULE_VAL_SEQ(1), // opt/sort ), // opt/sort ), // opt/sort ), // opt/sort ), // opt/sort // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTION_NAME("spool-path"), // opt/spool-path PARSE_RULE_OPTION_TYPE(Path), // opt/spool-path PARSE_RULE_OPTION_RESET(true), // opt/spool-path PARSE_RULE_OPTION_REQUIRED(true), // opt/spool-path PARSE_RULE_OPTION_SECTION(Global), // opt/spool-path // opt/spool-path PARSE_RULE_OPTION_COMMAND_INTERNAL_LIST // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTION_COMMAND_INTERNAL(Restore, true), // opt/spool-path ) // opt/spool-path // opt/spool-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/spool-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/spool-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/spool-path ), // opt/spool-path // opt/spool-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/spool-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/spool-path ), // opt/spool-path // opt/spool-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/spool-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/spool-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/spool-path ), // opt/spool-path // opt/spool-path PARSE_RULE_OPTIONAL // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTIONAL_GROUP // opt/spool-path ( // opt/spool-path PARSE_RULE_FILTER_CMD // opt/spool-path ( // opt/spool-path PARSE_RULE_VAL_CMD(ArchiveGet), // opt/spool-path PARSE_RULE_VAL_CMD(ArchivePush), // opt/spool-path ), // opt/spool-path // opt/spool-path PARSE_RULE_OPTIONAL_DEPEND // opt/spool-path ( // opt/spool-path PARSE_RULE_VAL_OPT(ArchiveAsync), // opt/spool-path PARSE_RULE_VAL_BOOL_TRUE, // opt/spool-path ), // opt/spool-path // opt/spool-path PARSE_RULE_OPTIONAL_DEFAULT // opt/spool-path ( // opt/spool-path PARSE_RULE_VAL_STR(QT_FS_var_FS_spool_FS_pgbackrest_QT), // opt/spool-path ), // opt/spool-path ), // opt/spool-path // opt/spool-path PARSE_RULE_OPTIONAL_GROUP // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTIONAL_DEFAULT // opt/spool-path ( // opt/spool-path PARSE_RULE_VAL_STR(QT_FS_var_FS_spool_FS_pgbackrest_QT), // opt/spool-path ), // opt/spool-path ), // opt/spool-path ), // opt/spool-path ), // opt/spool-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/stanza ( // opt/stanza PARSE_RULE_OPTION_NAME("stanza"), // opt/stanza PARSE_RULE_OPTION_TYPE(String), // opt/stanza PARSE_RULE_OPTION_REQUIRED(true), // opt/stanza PARSE_RULE_OPTION_SECTION(CommandLine), // opt/stanza // opt/stanza PARSE_RULE_OPTION_COMMAND_INTERNAL_LIST // opt/stanza ( // opt/stanza PARSE_RULE_OPTION_COMMAND_INTERNAL(RepoGet, true), // opt/stanza PARSE_RULE_OPTION_COMMAND_INTERNAL(RepoLs, true), // opt/stanza PARSE_RULE_OPTION_COMMAND_INTERNAL(RepoRm, true), // opt/stanza ) // opt/stanza // opt/stanza PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/stanza ( // opt/stanza PARSE_RULE_OPTION_COMMAND(Annotate) // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/stanza PARSE_RULE_OPTION_COMMAND(Backup) // opt/stanza PARSE_RULE_OPTION_COMMAND(Check) // opt/stanza PARSE_RULE_OPTION_COMMAND(Expire) // opt/stanza PARSE_RULE_OPTION_COMMAND(Info) // opt/stanza PARSE_RULE_OPTION_COMMAND(Manifest) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/stanza PARSE_RULE_OPTION_COMMAND(Restore) // opt/stanza PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/stanza PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/stanza PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/stanza PARSE_RULE_OPTION_COMMAND(Start) // opt/stanza PARSE_RULE_OPTION_COMMAND(Stop) // opt/stanza PARSE_RULE_OPTION_COMMAND(Verify) // opt/stanza ), // opt/stanza // opt/stanza PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/stanza ( // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/stanza ), // opt/stanza // opt/stanza PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/stanza ( // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/stanza PARSE_RULE_OPTION_COMMAND(Backup) // opt/stanza PARSE_RULE_OPTION_COMMAND(Restore) // opt/stanza PARSE_RULE_OPTION_COMMAND(Verify) // opt/stanza ), // opt/stanza // opt/stanza PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/stanza ( // opt/stanza PARSE_RULE_OPTION_COMMAND(Annotate) // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/stanza PARSE_RULE_OPTION_COMMAND(Backup) // opt/stanza PARSE_RULE_OPTION_COMMAND(Check) // opt/stanza PARSE_RULE_OPTION_COMMAND(Expire) // opt/stanza PARSE_RULE_OPTION_COMMAND(Info) // opt/stanza PARSE_RULE_OPTION_COMMAND(Manifest) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/stanza PARSE_RULE_OPTION_COMMAND(Restore) // opt/stanza PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/stanza PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/stanza PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/stanza PARSE_RULE_OPTION_COMMAND(Verify) // opt/stanza ), // opt/stanza // opt/stanza PARSE_RULE_OPTIONAL // opt/stanza ( // opt/stanza PARSE_RULE_OPTIONAL_GROUP // opt/stanza ( // opt/stanza PARSE_RULE_FILTER_CMD // opt/stanza ( // opt/stanza PARSE_RULE_VAL_CMD(Check), // opt/stanza PARSE_RULE_VAL_CMD(Info), // opt/stanza PARSE_RULE_VAL_CMD(RepoGet), // opt/stanza PARSE_RULE_VAL_CMD(RepoLs), // opt/stanza PARSE_RULE_VAL_CMD(RepoRm), // opt/stanza PARSE_RULE_VAL_CMD(Start), // opt/stanza PARSE_RULE_VAL_CMD(Stop), // opt/stanza ), // opt/stanza // opt/stanza PARSE_RULE_OPTIONAL_NOT_REQUIRED(), // opt/stanza ), // opt/stanza ), // opt/stanza ), // opt/stanza // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/start-fast ( // opt/start-fast PARSE_RULE_OPTION_NAME("start-fast"), // opt/start-fast PARSE_RULE_OPTION_TYPE(Boolean), // opt/start-fast PARSE_RULE_OPTION_NEGATE(true), // opt/start-fast PARSE_RULE_OPTION_RESET(true), // opt/start-fast PARSE_RULE_OPTION_REQUIRED(true), // opt/start-fast PARSE_RULE_OPTION_SECTION(Global), // opt/start-fast // opt/start-fast PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/start-fast ( // opt/start-fast PARSE_RULE_OPTION_COMMAND(Backup) // opt/start-fast ), // opt/start-fast // opt/start-fast PARSE_RULE_OPTIONAL // opt/start-fast ( // opt/start-fast PARSE_RULE_OPTIONAL_GROUP // opt/start-fast ( // opt/start-fast PARSE_RULE_OPTIONAL_DEFAULT // opt/start-fast ( // opt/start-fast PARSE_RULE_VAL_BOOL_FALSE, // opt/start-fast ), // opt/start-fast ), // opt/start-fast ), // opt/start-fast ), // opt/start-fast // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/stop-auto ( // opt/stop-auto PARSE_RULE_OPTION_NAME("stop-auto"), // opt/stop-auto PARSE_RULE_OPTION_TYPE(Boolean), // opt/stop-auto PARSE_RULE_OPTION_INTERNAL(true) // opt/stop-auto PARSE_RULE_OPTION_NEGATE(true), // opt/stop-auto PARSE_RULE_OPTION_RESET(true), // opt/stop-auto PARSE_RULE_OPTION_REQUIRED(true), // opt/stop-auto PARSE_RULE_OPTION_SECTION(Global), // opt/stop-auto // opt/stop-auto PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/stop-auto ( // opt/stop-auto PARSE_RULE_OPTION_COMMAND(Backup) // opt/stop-auto ), // opt/stop-auto // opt/stop-auto PARSE_RULE_OPTIONAL // opt/stop-auto ( // opt/stop-auto PARSE_RULE_OPTIONAL_GROUP // opt/stop-auto ( // opt/stop-auto PARSE_RULE_OPTIONAL_DEFAULT // opt/stop-auto ( // opt/stop-auto PARSE_RULE_VAL_BOOL_FALSE, // opt/stop-auto ), // opt/stop-auto ), // opt/stop-auto ), // opt/stop-auto ), // opt/stop-auto // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tablespace-map ( // opt/tablespace-map PARSE_RULE_OPTION_NAME("tablespace-map"), // opt/tablespace-map PARSE_RULE_OPTION_TYPE(Hash), // opt/tablespace-map PARSE_RULE_OPTION_RESET(true), // opt/tablespace-map PARSE_RULE_OPTION_REQUIRED(false), // opt/tablespace-map PARSE_RULE_OPTION_SECTION(Global), // opt/tablespace-map PARSE_RULE_OPTION_MULTI(true), // opt/tablespace-map // opt/tablespace-map PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tablespace-map ( // opt/tablespace-map PARSE_RULE_OPTION_COMMAND(Restore) // opt/tablespace-map ), // opt/tablespace-map ), // opt/tablespace-map // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tablespace-map-all ( // opt/tablespace-map-all PARSE_RULE_OPTION_NAME("tablespace-map-all"), // opt/tablespace-map-all PARSE_RULE_OPTION_TYPE(String), // opt/tablespace-map-all PARSE_RULE_OPTION_RESET(true), // opt/tablespace-map-all PARSE_RULE_OPTION_REQUIRED(false), // opt/tablespace-map-all PARSE_RULE_OPTION_SECTION(Global), // opt/tablespace-map-all // opt/tablespace-map-all PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tablespace-map-all ( // opt/tablespace-map-all PARSE_RULE_OPTION_COMMAND(Restore) // opt/tablespace-map-all ), // opt/tablespace-map-all ), // opt/tablespace-map-all // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/target ( // opt/target PARSE_RULE_OPTION_NAME("target"), // opt/target PARSE_RULE_OPTION_TYPE(String), // opt/target PARSE_RULE_OPTION_REQUIRED(true), // opt/target PARSE_RULE_OPTION_SECTION(CommandLine), // opt/target // opt/target PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/target ( // opt/target PARSE_RULE_OPTION_COMMAND(Restore) // opt/target ), // opt/target // opt/target PARSE_RULE_OPTIONAL // opt/target ( // opt/target PARSE_RULE_OPTIONAL_GROUP // opt/target ( // opt/target PARSE_RULE_OPTIONAL_DEPEND // opt/target ( // opt/target PARSE_RULE_VAL_OPT(Type), // opt/target PARSE_RULE_VAL_STRID(lsn), // opt/target PARSE_RULE_VAL_STRID(name), // opt/target PARSE_RULE_VAL_STRID(time), // opt/target PARSE_RULE_VAL_STRID(xid), // opt/target ), // opt/target ), // opt/target ), // opt/target ), // opt/target // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/target-action ( // opt/target-action PARSE_RULE_OPTION_NAME("target-action"), // opt/target-action PARSE_RULE_OPTION_TYPE(StringId), // opt/target-action PARSE_RULE_OPTION_REQUIRED(true), // opt/target-action PARSE_RULE_OPTION_SECTION(CommandLine), // opt/target-action // opt/target-action PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/target-action ( // opt/target-action PARSE_RULE_OPTION_COMMAND(Restore) // opt/target-action ), // opt/target-action // opt/target-action PARSE_RULE_OPTIONAL // opt/target-action ( // opt/target-action PARSE_RULE_OPTIONAL_GROUP // opt/target-action ( // opt/target-action PARSE_RULE_OPTIONAL_DEPEND // opt/target-action ( // opt/target-action PARSE_RULE_VAL_OPT(Type), // opt/target-action PARSE_RULE_VAL_STRID(immediate), // opt/target-action PARSE_RULE_VAL_STRID(lsn), // opt/target-action PARSE_RULE_VAL_STRID(name), // opt/target-action PARSE_RULE_VAL_STRID(time), // opt/target-action PARSE_RULE_VAL_STRID(xid), // opt/target-action ), // opt/target-action // opt/target-action PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/target-action ( // opt/target-action PARSE_RULE_VAL_STRID(pause), // opt/target-action PARSE_RULE_VAL_STRID(promote), // opt/target-action PARSE_RULE_VAL_STRID(shutdown), // opt/target-action ), // opt/target-action // opt/target-action PARSE_RULE_OPTIONAL_DEFAULT // opt/target-action ( // opt/target-action PARSE_RULE_VAL_STRID(pause), // opt/target-action ), // opt/target-action ), // opt/target-action ), // opt/target-action ), // opt/target-action // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/target-exclusive ( // opt/target-exclusive PARSE_RULE_OPTION_NAME("target-exclusive"), // opt/target-exclusive PARSE_RULE_OPTION_TYPE(Boolean), // opt/target-exclusive PARSE_RULE_OPTION_REQUIRED(true), // opt/target-exclusive PARSE_RULE_OPTION_SECTION(CommandLine), // opt/target-exclusive // opt/target-exclusive PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/target-exclusive ( // opt/target-exclusive PARSE_RULE_OPTION_COMMAND(Restore) // opt/target-exclusive ), // opt/target-exclusive // opt/target-exclusive PARSE_RULE_OPTIONAL // opt/target-exclusive ( // opt/target-exclusive PARSE_RULE_OPTIONAL_GROUP // opt/target-exclusive ( // opt/target-exclusive PARSE_RULE_OPTIONAL_DEPEND // opt/target-exclusive ( // opt/target-exclusive PARSE_RULE_VAL_OPT(Type), // opt/target-exclusive PARSE_RULE_VAL_STRID(lsn), // opt/target-exclusive PARSE_RULE_VAL_STRID(time), // opt/target-exclusive PARSE_RULE_VAL_STRID(xid), // opt/target-exclusive ), // opt/target-exclusive // opt/target-exclusive PARSE_RULE_OPTIONAL_DEFAULT // opt/target-exclusive ( // opt/target-exclusive PARSE_RULE_VAL_BOOL_FALSE, // opt/target-exclusive ), // opt/target-exclusive ), // opt/target-exclusive ), // opt/target-exclusive ), // opt/target-exclusive // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/target-timeline ( // opt/target-timeline PARSE_RULE_OPTION_NAME("target-timeline"), // opt/target-timeline PARSE_RULE_OPTION_TYPE(String), // opt/target-timeline PARSE_RULE_OPTION_REQUIRED(false), // opt/target-timeline PARSE_RULE_OPTION_SECTION(CommandLine), // opt/target-timeline // opt/target-timeline PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/target-timeline ( // opt/target-timeline PARSE_RULE_OPTION_COMMAND(Restore) // opt/target-timeline ), // opt/target-timeline // opt/target-timeline PARSE_RULE_OPTIONAL // opt/target-timeline ( // opt/target-timeline PARSE_RULE_OPTIONAL_GROUP // opt/target-timeline ( // opt/target-timeline PARSE_RULE_OPTIONAL_DEPEND // opt/target-timeline ( // opt/target-timeline PARSE_RULE_VAL_OPT(Type), // opt/target-timeline PARSE_RULE_VAL_STRID(default), // opt/target-timeline PARSE_RULE_VAL_STRID(lsn), // opt/target-timeline PARSE_RULE_VAL_STRID(name), // opt/target-timeline PARSE_RULE_VAL_STRID(standby), // opt/target-timeline PARSE_RULE_VAL_STRID(time), // opt/target-timeline PARSE_RULE_VAL_STRID(xid), // opt/target-timeline ), // opt/target-timeline ), // opt/target-timeline ), // opt/target-timeline ), // opt/target-timeline // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTION_NAME("tcp-keep-alive-count"), // opt/tcp-keep-alive-count PARSE_RULE_OPTION_TYPE(Integer), // opt/tcp-keep-alive-count PARSE_RULE_OPTION_RESET(true), // opt/tcp-keep-alive-count PARSE_RULE_OPTION_REQUIRED(false), // opt/tcp-keep-alive-count PARSE_RULE_OPTION_SECTION(Global), // opt/tcp-keep-alive-count // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Check) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Expire) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Info) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Server) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Check) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Expire) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Info) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count // opt/tcp-keep-alive-count PARSE_RULE_OPTIONAL // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTIONAL_GROUP // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTIONAL_DEPEND // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_VAL_OPT(SckKeepAlive), // opt/tcp-keep-alive-count PARSE_RULE_VAL_BOOL_TRUE, // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count // opt/tcp-keep-alive-count PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_VAL_INT(1), // opt/tcp-keep-alive-count PARSE_RULE_VAL_INT(32), // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_NAME("tcp-keep-alive-idle"), // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_TYPE(Integer), // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_RESET(true), // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_REQUIRED(false), // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_SECTION(Global), // opt/tcp-keep-alive-idle // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Check) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Expire) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Info) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Server) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Check) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Expire) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Info) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle // opt/tcp-keep-alive-idle PARSE_RULE_OPTIONAL // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTIONAL_GROUP // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTIONAL_DEPEND // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_VAL_OPT(SckKeepAlive), // opt/tcp-keep-alive-idle PARSE_RULE_VAL_BOOL_TRUE, // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle // opt/tcp-keep-alive-idle PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_VAL_INT(1), // opt/tcp-keep-alive-idle PARSE_RULE_VAL_INT(3600), // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_NAME("tcp-keep-alive-interval"), // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_TYPE(Integer), // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_RESET(true), // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_REQUIRED(false), // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_SECTION(Global), // opt/tcp-keep-alive-interval // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Check) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Expire) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Info) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Server) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Check) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Expire) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Info) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval // opt/tcp-keep-alive-interval PARSE_RULE_OPTIONAL // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTIONAL_GROUP // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTIONAL_DEPEND // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_VAL_OPT(SckKeepAlive), // opt/tcp-keep-alive-interval PARSE_RULE_VAL_BOOL_TRUE, // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval // opt/tcp-keep-alive-interval PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_VAL_INT(1), // opt/tcp-keep-alive-interval PARSE_RULE_VAL_INT(900), // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-cipher-12 ( // opt/tls-cipher-12 PARSE_RULE_OPTION_NAME("tls-cipher-12"), // opt/tls-cipher-12 PARSE_RULE_OPTION_TYPE(String), // opt/tls-cipher-12 PARSE_RULE_OPTION_RESET(true), // opt/tls-cipher-12 PARSE_RULE_OPTION_REQUIRED(false), // opt/tls-cipher-12 PARSE_RULE_OPTION_SECTION(Global), // opt/tls-cipher-12 // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-cipher-12 ( // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Backup) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Check) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Expire) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Info) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Restore) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Verify) // opt/tls-cipher-12 ), // opt/tls-cipher-12 // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/tls-cipher-12 ( // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-12 ), // opt/tls-cipher-12 // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/tls-cipher-12 ( // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Backup) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Restore) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Verify) // opt/tls-cipher-12 ), // opt/tls-cipher-12 // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/tls-cipher-12 ( // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Backup) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Check) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Expire) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Info) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Restore) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Verify) // opt/tls-cipher-12 ), // opt/tls-cipher-12 ), // opt/tls-cipher-12 // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-cipher-13 ( // opt/tls-cipher-13 PARSE_RULE_OPTION_NAME("tls-cipher-13"), // opt/tls-cipher-13 PARSE_RULE_OPTION_TYPE(String), // opt/tls-cipher-13 PARSE_RULE_OPTION_RESET(true), // opt/tls-cipher-13 PARSE_RULE_OPTION_REQUIRED(false), // opt/tls-cipher-13 PARSE_RULE_OPTION_SECTION(Global), // opt/tls-cipher-13 // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-cipher-13 ( // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Backup) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Check) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Expire) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Info) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Restore) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Verify) // opt/tls-cipher-13 ), // opt/tls-cipher-13 // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/tls-cipher-13 ( // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-13 ), // opt/tls-cipher-13 // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/tls-cipher-13 ( // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Backup) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Restore) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Verify) // opt/tls-cipher-13 ), // opt/tls-cipher-13 // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/tls-cipher-13 ( // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Backup) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Check) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Expire) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Info) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Restore) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Verify) // opt/tls-cipher-13 ), // opt/tls-cipher-13 ), // opt/tls-cipher-13 // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-server-address ( // opt/tls-server-address PARSE_RULE_OPTION_NAME("tls-server-address"), // opt/tls-server-address PARSE_RULE_OPTION_TYPE(String), // opt/tls-server-address PARSE_RULE_OPTION_RESET(true), // opt/tls-server-address PARSE_RULE_OPTION_REQUIRED(true), // opt/tls-server-address PARSE_RULE_OPTION_SECTION(Global), // opt/tls-server-address // opt/tls-server-address PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-server-address ( // opt/tls-server-address PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-server-address PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tls-server-address ), // opt/tls-server-address // opt/tls-server-address PARSE_RULE_OPTIONAL // opt/tls-server-address ( // opt/tls-server-address PARSE_RULE_OPTIONAL_GROUP // opt/tls-server-address ( // opt/tls-server-address PARSE_RULE_OPTIONAL_DEFAULT // opt/tls-server-address ( // opt/tls-server-address PARSE_RULE_VAL_STR(QT_localhost_QT), // opt/tls-server-address ), // opt/tls-server-address ), // opt/tls-server-address ), // opt/tls-server-address ), // opt/tls-server-address // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-server-auth ( // opt/tls-server-auth PARSE_RULE_OPTION_NAME("tls-server-auth"), // opt/tls-server-auth PARSE_RULE_OPTION_TYPE(Hash), // opt/tls-server-auth PARSE_RULE_OPTION_RESET(true), // opt/tls-server-auth PARSE_RULE_OPTION_REQUIRED(true), // opt/tls-server-auth PARSE_RULE_OPTION_SECTION(Global), // opt/tls-server-auth PARSE_RULE_OPTION_MULTI(true), // opt/tls-server-auth // opt/tls-server-auth PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-server-auth ( // opt/tls-server-auth PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-server-auth ), // opt/tls-server-auth ), // opt/tls-server-auth // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-server-ca-file ( // opt/tls-server-ca-file PARSE_RULE_OPTION_NAME("tls-server-ca-file"), // opt/tls-server-ca-file PARSE_RULE_OPTION_TYPE(Path), // opt/tls-server-ca-file PARSE_RULE_OPTION_RESET(true), // opt/tls-server-ca-file PARSE_RULE_OPTION_REQUIRED(true), // opt/tls-server-ca-file PARSE_RULE_OPTION_SECTION(Global), // opt/tls-server-ca-file // opt/tls-server-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-server-ca-file ( // opt/tls-server-ca-file PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-server-ca-file ), // opt/tls-server-ca-file ), // opt/tls-server-ca-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-server-cert-file ( // opt/tls-server-cert-file PARSE_RULE_OPTION_NAME("tls-server-cert-file"), // opt/tls-server-cert-file PARSE_RULE_OPTION_TYPE(Path), // opt/tls-server-cert-file PARSE_RULE_OPTION_RESET(true), // opt/tls-server-cert-file PARSE_RULE_OPTION_REQUIRED(true), // opt/tls-server-cert-file PARSE_RULE_OPTION_SECTION(Global), // opt/tls-server-cert-file // opt/tls-server-cert-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-server-cert-file ( // opt/tls-server-cert-file PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-server-cert-file ), // opt/tls-server-cert-file ), // opt/tls-server-cert-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-server-key-file ( // opt/tls-server-key-file PARSE_RULE_OPTION_NAME("tls-server-key-file"), // opt/tls-server-key-file PARSE_RULE_OPTION_TYPE(Path), // opt/tls-server-key-file PARSE_RULE_OPTION_RESET(true), // opt/tls-server-key-file PARSE_RULE_OPTION_REQUIRED(true), // opt/tls-server-key-file PARSE_RULE_OPTION_SECTION(Global), // opt/tls-server-key-file // opt/tls-server-key-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-server-key-file ( // opt/tls-server-key-file PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-server-key-file ), // opt/tls-server-key-file ), // opt/tls-server-key-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-server-port ( // opt/tls-server-port PARSE_RULE_OPTION_NAME("tls-server-port"), // opt/tls-server-port PARSE_RULE_OPTION_TYPE(Integer), // opt/tls-server-port PARSE_RULE_OPTION_RESET(true), // opt/tls-server-port PARSE_RULE_OPTION_REQUIRED(true), // opt/tls-server-port PARSE_RULE_OPTION_SECTION(Global), // opt/tls-server-port // opt/tls-server-port PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-server-port ( // opt/tls-server-port PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-server-port PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tls-server-port ), // opt/tls-server-port // opt/tls-server-port PARSE_RULE_OPTIONAL // opt/tls-server-port ( // opt/tls-server-port PARSE_RULE_OPTIONAL_GROUP // opt/tls-server-port ( // opt/tls-server-port PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/tls-server-port ( // opt/tls-server-port PARSE_RULE_VAL_INT(1), // opt/tls-server-port PARSE_RULE_VAL_INT(65535), // opt/tls-server-port ), // opt/tls-server-port // opt/tls-server-port PARSE_RULE_OPTIONAL_DEFAULT // opt/tls-server-port ( // opt/tls-server-port PARSE_RULE_VAL_INT(8432), // opt/tls-server-port ), // opt/tls-server-port ), // opt/tls-server-port ), // opt/tls-server-port ), // opt/tls-server-port // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/type ( // opt/type PARSE_RULE_OPTION_NAME("type"), // opt/type PARSE_RULE_OPTION_TYPE(StringId), // opt/type PARSE_RULE_OPTION_REQUIRED(true), // opt/type PARSE_RULE_OPTION_SECTION(CommandLine), // opt/type // opt/type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/type ( // opt/type PARSE_RULE_OPTION_COMMAND(Backup) // opt/type PARSE_RULE_OPTION_COMMAND(Info) // opt/type PARSE_RULE_OPTION_COMMAND(Restore) // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL // opt/type ( // opt/type PARSE_RULE_OPTIONAL_GROUP // opt/type ( // opt/type PARSE_RULE_FILTER_CMD // opt/type ( // opt/type PARSE_RULE_VAL_CMD(Backup), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/type ( // opt/type PARSE_RULE_VAL_STRID(full), // opt/type PARSE_RULE_VAL_STRID(diff), // opt/type PARSE_RULE_VAL_STRID(incr), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_DEFAULT // opt/type ( // opt/type PARSE_RULE_VAL_STRID(incr), // opt/type ), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_GROUP // opt/type ( // opt/type PARSE_RULE_FILTER_CMD // opt/type ( // opt/type PARSE_RULE_VAL_CMD(Info), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/type ( // opt/type PARSE_RULE_VAL_STRID(full), // opt/type PARSE_RULE_VAL_STRID(diff), // opt/type PARSE_RULE_VAL_STRID(incr), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_NOT_REQUIRED(), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_GROUP // opt/type ( // opt/type PARSE_RULE_FILTER_CMD // opt/type ( // opt/type PARSE_RULE_VAL_CMD(Restore), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_SEQUENCE(), // opt/type // opt/type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/type ( // opt/type PARSE_RULE_VAL_STRID(default), // opt/type PARSE_RULE_VAL_STRID(preserve), // opt/type PARSE_RULE_VAL_STRID(immediate), // opt/type PARSE_RULE_VAL_STRID(standby), // opt/type PARSE_RULE_VAL_STRID(none), // opt/type PARSE_RULE_VAL_STRID(name), // opt/type PARSE_RULE_VAL_STRID(time), // opt/type PARSE_RULE_VAL_STRID(lsn), // opt/type PARSE_RULE_VAL_STRID(xid), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_DEFAULT // opt/type ( // opt/type PARSE_RULE_VAL_STRID(default), // opt/type ), // opt/type ), // opt/type ), // opt/type ), // opt/type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/verbose ( // opt/verbose PARSE_RULE_OPTION_NAME("verbose"), // opt/verbose PARSE_RULE_OPTION_TYPE(Boolean), // opt/verbose PARSE_RULE_OPTION_REQUIRED(true), // opt/verbose PARSE_RULE_OPTION_SECTION(CommandLine), // opt/verbose // opt/verbose PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/verbose ( // opt/verbose PARSE_RULE_OPTION_COMMAND(Verify) // opt/verbose ), // opt/verbose // opt/verbose PARSE_RULE_OPTIONAL // opt/verbose ( // opt/verbose PARSE_RULE_OPTIONAL_GROUP // opt/verbose ( // opt/verbose PARSE_RULE_OPTIONAL_DEFAULT // opt/verbose ( // opt/verbose PARSE_RULE_VAL_BOOL_FALSE, // opt/verbose ), // opt/verbose ), // opt/verbose ), // opt/verbose ), // opt/verbose // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/version ( // opt/version PARSE_RULE_OPTION_NAME("version"), // opt/version PARSE_RULE_OPTION_TYPE(Boolean), // opt/version PARSE_RULE_OPTION_REQUIRED(true), // opt/version PARSE_RULE_OPTION_SECTION(CommandLine), // opt/version // opt/version PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/version ( // opt/version PARSE_RULE_OPTION_COMMAND(Help) // opt/version ), // opt/version // opt/version PARSE_RULE_OPTIONAL // opt/version ( // opt/version PARSE_RULE_OPTIONAL_GROUP // opt/version ( // opt/version PARSE_RULE_OPTIONAL_DEFAULT // opt/version ( // opt/version PARSE_RULE_VAL_BOOL_FALSE, // opt/version ), // opt/version ), // opt/version ), // opt/version ), // opt/version }; /*********************************************************************************************************************************** Option deprecations ***********************************************************************************************************************************/ #define CFG_OPTION_DEPRECATE_TOTAL 39 static const ParseRuleOptionDeprecate parseRuleOptionDeprecate[CFG_OPTION_DEPRECATE_TOTAL] = { { // opt-deprecate/archive-push-queue-max .name = "archive-queue-max", // opt-deprecate/archive-push-queue-max .id = cfgOptArchivePushQueueMax, // opt-deprecate/archive-push-queue-max .unindexed = true, // opt-deprecate/archive-push-queue-max }, // opt-deprecate/archive-push-queue-max // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-host-cmd .name = "backup-cmd", // opt-deprecate/repo-host-cmd .id = cfgOptRepoHostCmd, // opt-deprecate/repo-host-cmd .unindexed = true, // opt-deprecate/repo-host-cmd }, // opt-deprecate/repo-host-cmd // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-host-config .name = "backup-config", // opt-deprecate/repo-host-config .id = cfgOptRepoHostConfig, // opt-deprecate/repo-host-config .unindexed = true, // opt-deprecate/repo-host-config }, // opt-deprecate/repo-host-config // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-host .name = "backup-host", // opt-deprecate/repo-host .id = cfgOptRepoHost, // opt-deprecate/repo-host .unindexed = true, // opt-deprecate/repo-host }, // opt-deprecate/repo-host // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-host-port .name = "backup-ssh-port", // opt-deprecate/repo-host-port .id = cfgOptRepoHostPort, // opt-deprecate/repo-host-port .unindexed = true, // opt-deprecate/repo-host-port }, // opt-deprecate/repo-host-port // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-host-user .name = "backup-user", // opt-deprecate/repo-host-user .id = cfgOptRepoHostUser, // opt-deprecate/repo-host-user .unindexed = true, // opt-deprecate/repo-host-user }, // opt-deprecate/repo-host-user // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-host-cmd .name = "db-cmd", // opt-deprecate/pg-host-cmd .id = cfgOptPgHostCmd, // opt-deprecate/pg-host-cmd .indexed = true, // opt-deprecate/pg-host-cmd .unindexed = true, // opt-deprecate/pg-host-cmd }, // opt-deprecate/pg-host-cmd // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-host-config .name = "db-config", // opt-deprecate/pg-host-config .id = cfgOptPgHostConfig, // opt-deprecate/pg-host-config .indexed = true, // opt-deprecate/pg-host-config .unindexed = true, // opt-deprecate/pg-host-config }, // opt-deprecate/pg-host-config // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-host .name = "db-host", // opt-deprecate/pg-host .id = cfgOptPgHost, // opt-deprecate/pg-host .indexed = true, // opt-deprecate/pg-host .unindexed = true, // opt-deprecate/pg-host }, // opt-deprecate/pg-host // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-path .name = "db-path", // opt-deprecate/pg-path .id = cfgOptPgPath, // opt-deprecate/pg-path .indexed = true, // opt-deprecate/pg-path .unindexed = true, // opt-deprecate/pg-path }, // opt-deprecate/pg-path // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-port .name = "db-port", // opt-deprecate/pg-port .id = cfgOptPgPort, // opt-deprecate/pg-port .indexed = true, // opt-deprecate/pg-port .unindexed = true, // opt-deprecate/pg-port }, // opt-deprecate/pg-port // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-socket-path .name = "db-socket-path", // opt-deprecate/pg-socket-path .id = cfgOptPgSocketPath, // opt-deprecate/pg-socket-path .indexed = true, // opt-deprecate/pg-socket-path .unindexed = true, // opt-deprecate/pg-socket-path }, // opt-deprecate/pg-socket-path // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-host-port .name = "db-ssh-port", // opt-deprecate/pg-host-port .id = cfgOptPgHostPort, // opt-deprecate/pg-host-port .indexed = true, // opt-deprecate/pg-host-port .unindexed = true, // opt-deprecate/pg-host-port }, // opt-deprecate/pg-host-port // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-host-user .name = "db-user", // opt-deprecate/pg-host-user .id = cfgOptPgHostUser, // opt-deprecate/pg-host-user .indexed = true, // opt-deprecate/pg-host-user .unindexed = true, // opt-deprecate/pg-host-user }, // opt-deprecate/pg-host-user // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-hardlink .name = "hardlink", // opt-deprecate/repo-hardlink .id = cfgOptRepoHardlink, // opt-deprecate/repo-hardlink .unindexed = true, // opt-deprecate/repo-hardlink }, // opt-deprecate/repo-hardlink // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-ca-file .name = "repo-azure-ca-file", // opt-deprecate/repo-storage-ca-file .id = cfgOptRepoStorageCaFile, // opt-deprecate/repo-storage-ca-file .indexed = true, // opt-deprecate/repo-storage-ca-file }, // opt-deprecate/repo-storage-ca-file // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-ca-path .name = "repo-azure-ca-path", // opt-deprecate/repo-storage-ca-path .id = cfgOptRepoStorageCaPath, // opt-deprecate/repo-storage-ca-path .indexed = true, // opt-deprecate/repo-storage-ca-path }, // opt-deprecate/repo-storage-ca-path // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-host .name = "repo-azure-host", // opt-deprecate/repo-storage-host .id = cfgOptRepoStorageHost, // opt-deprecate/repo-storage-host .indexed = true, // opt-deprecate/repo-storage-host }, // opt-deprecate/repo-storage-host // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-port .name = "repo-azure-port", // opt-deprecate/repo-storage-port .id = cfgOptRepoStoragePort, // opt-deprecate/repo-storage-port .indexed = true, // opt-deprecate/repo-storage-port }, // opt-deprecate/repo-storage-port // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-verify-tls .name = "repo-azure-verify-tls", // opt-deprecate/repo-storage-verify-tls .id = cfgOptRepoStorageVerifyTls, // opt-deprecate/repo-storage-verify-tls .indexed = true, // opt-deprecate/repo-storage-verify-tls }, // opt-deprecate/repo-storage-verify-tls // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-cipher-pass .name = "repo-cipher-pass", // opt-deprecate/repo-cipher-pass .id = cfgOptRepoCipherPass, // opt-deprecate/repo-cipher-pass .unindexed = true, // opt-deprecate/repo-cipher-pass }, // opt-deprecate/repo-cipher-pass // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-cipher-type .name = "repo-cipher-type", // opt-deprecate/repo-cipher-type .id = cfgOptRepoCipherType, // opt-deprecate/repo-cipher-type .unindexed = true, // opt-deprecate/repo-cipher-type }, // opt-deprecate/repo-cipher-type // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-path .name = "repo-path", // opt-deprecate/repo-path .id = cfgOptRepoPath, // opt-deprecate/repo-path .unindexed = true, // opt-deprecate/repo-path }, // opt-deprecate/repo-path // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-s3-bucket .name = "repo-s3-bucket", // opt-deprecate/repo-s3-bucket .id = cfgOptRepoS3Bucket, // opt-deprecate/repo-s3-bucket .unindexed = true, // opt-deprecate/repo-s3-bucket }, // opt-deprecate/repo-s3-bucket // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-ca-file .name = "repo-s3-ca-file", // opt-deprecate/repo-storage-ca-file .id = cfgOptRepoStorageCaFile, // opt-deprecate/repo-storage-ca-file .indexed = true, // opt-deprecate/repo-storage-ca-file .unindexed = true, // opt-deprecate/repo-storage-ca-file }, // opt-deprecate/repo-storage-ca-file // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-ca-path .name = "repo-s3-ca-path", // opt-deprecate/repo-storage-ca-path .id = cfgOptRepoStorageCaPath, // opt-deprecate/repo-storage-ca-path .indexed = true, // opt-deprecate/repo-storage-ca-path .unindexed = true, // opt-deprecate/repo-storage-ca-path }, // opt-deprecate/repo-storage-ca-path // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-s3-endpoint .name = "repo-s3-endpoint", // opt-deprecate/repo-s3-endpoint .id = cfgOptRepoS3Endpoint, // opt-deprecate/repo-s3-endpoint .unindexed = true, // opt-deprecate/repo-s3-endpoint }, // opt-deprecate/repo-s3-endpoint // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-host .name = "repo-s3-host", // opt-deprecate/repo-storage-host .id = cfgOptRepoStorageHost, // opt-deprecate/repo-storage-host .indexed = true, // opt-deprecate/repo-storage-host .unindexed = true, // opt-deprecate/repo-storage-host }, // opt-deprecate/repo-storage-host // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-s3-key .name = "repo-s3-key", // opt-deprecate/repo-s3-key .id = cfgOptRepoS3Key, // opt-deprecate/repo-s3-key .unindexed = true, // opt-deprecate/repo-s3-key }, // opt-deprecate/repo-s3-key // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-s3-key-secret .name = "repo-s3-key-secret", // opt-deprecate/repo-s3-key-secret .id = cfgOptRepoS3KeySecret, // opt-deprecate/repo-s3-key-secret .unindexed = true, // opt-deprecate/repo-s3-key-secret }, // opt-deprecate/repo-s3-key-secret // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-port .name = "repo-s3-port", // opt-deprecate/repo-storage-port .id = cfgOptRepoStoragePort, // opt-deprecate/repo-storage-port .indexed = true, // opt-deprecate/repo-storage-port }, // opt-deprecate/repo-storage-port // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-s3-region .name = "repo-s3-region", // opt-deprecate/repo-s3-region .id = cfgOptRepoS3Region, // opt-deprecate/repo-s3-region .unindexed = true, // opt-deprecate/repo-s3-region }, // opt-deprecate/repo-s3-region // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-verify-tls .name = "repo-s3-verify-ssl", // opt-deprecate/repo-storage-verify-tls .id = cfgOptRepoStorageVerifyTls, // opt-deprecate/repo-storage-verify-tls .indexed = true, // opt-deprecate/repo-storage-verify-tls .unindexed = true, // opt-deprecate/repo-storage-verify-tls }, // opt-deprecate/repo-storage-verify-tls // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-verify-tls .name = "repo-s3-verify-tls", // opt-deprecate/repo-storage-verify-tls .id = cfgOptRepoStorageVerifyTls, // opt-deprecate/repo-storage-verify-tls .indexed = true, // opt-deprecate/repo-storage-verify-tls }, // opt-deprecate/repo-storage-verify-tls // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-type .name = "repo-type", // opt-deprecate/repo-type .id = cfgOptRepoType, // opt-deprecate/repo-type .unindexed = true, // opt-deprecate/repo-type }, // opt-deprecate/repo-type // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-retention-archive .name = "retention-archive", // opt-deprecate/repo-retention-archive .id = cfgOptRepoRetentionArchive, // opt-deprecate/repo-retention-archive .unindexed = true, // opt-deprecate/repo-retention-archive }, // opt-deprecate/repo-retention-archive // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-retention-archive-type .name = "retention-archive-type", // opt-deprecate/repo-retention-archive-type .id = cfgOptRepoRetentionArchiveType, // opt-deprecate/repo-retention-archive-type .unindexed = true, // opt-deprecate/repo-retention-archive-type }, // opt-deprecate/repo-retention-archive-type // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-retention-diff .name = "retention-diff", // opt-deprecate/repo-retention-diff .id = cfgOptRepoRetentionDiff, // opt-deprecate/repo-retention-diff .unindexed = true, // opt-deprecate/repo-retention-diff }, // opt-deprecate/repo-retention-diff // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-retention-full .name = "retention-full", // opt-deprecate/repo-retention-full .id = cfgOptRepoRetentionFull, // opt-deprecate/repo-retention-full .unindexed = true, // opt-deprecate/repo-retention-full }, // opt-deprecate/repo-retention-full }; /*********************************************************************************************************************************** Order for option parse resolution ***********************************************************************************************************************************/ static const uint8_t optionResolveOrder[] = { cfgOptStanza, // opt-resolve-order cfgOptAllowRoot, // opt-resolve-order cfgOptAnnotation, // opt-resolve-order cfgOptArchiveAsync, // opt-resolve-order cfgOptArchiveExpireBefore, // opt-resolve-order cfgOptArchiveGetQueueMax, // opt-resolve-order cfgOptArchiveHeaderCheck, // opt-resolve-order cfgOptArchiveMissingRetry, // opt-resolve-order cfgOptArchiveMode, // opt-resolve-order cfgOptArchivePushBatchSize, // opt-resolve-order cfgOptArchivePushQueueMax, // opt-resolve-order cfgOptArchiveTimeout, // opt-resolve-order cfgOptBackupStandby, // opt-resolve-order cfgOptBeta, // opt-resolve-order cfgOptBufferSize, // opt-resolve-order cfgOptChecksumPage, // opt-resolve-order cfgOptChecksumPageError, // opt-resolve-order cfgOptCmd, // opt-resolve-order cfgOptCmdSsh, // opt-resolve-order cfgOptCompress, // opt-resolve-order cfgOptCompressLevelNetwork, // opt-resolve-order cfgOptCompressType, // opt-resolve-order cfgOptConfig, // opt-resolve-order cfgOptConfigIncludePath, // opt-resolve-order cfgOptConfigPath, // opt-resolve-order cfgOptDbExclude, // opt-resolve-order cfgOptDbInclude, // opt-resolve-order cfgOptDbTimeout, // opt-resolve-order cfgOptDelta, // opt-resolve-order cfgOptDetailLevel, // opt-resolve-order cfgOptDryRun, // opt-resolve-order cfgOptExclude, // opt-resolve-order cfgOptExecId, // opt-resolve-order cfgOptExpireAuto, // opt-resolve-order cfgOptFilter, // opt-resolve-order cfgOptHelp, // opt-resolve-order cfgOptIgnoreMissing, // opt-resolve-order cfgOptIoTimeout, // opt-resolve-order cfgOptJobRetry, // opt-resolve-order cfgOptJobRetryInterval, // opt-resolve-order cfgOptLinkAll, // opt-resolve-order cfgOptLinkMap, // opt-resolve-order cfgOptLock, // opt-resolve-order cfgOptLockPath, // opt-resolve-order cfgOptLogLevelConsole, // opt-resolve-order cfgOptLogLevelFile, // opt-resolve-order cfgOptLogLevelStderr, // opt-resolve-order cfgOptLogPath, // opt-resolve-order cfgOptLogSubprocess, // opt-resolve-order cfgOptLogTimestamp, // opt-resolve-order cfgOptManifestSaveThreshold, // opt-resolve-order cfgOptNeutralUmask, // opt-resolve-order cfgOptOldest, // opt-resolve-order cfgOptOnline, // opt-resolve-order cfgOptOutput, // opt-resolve-order cfgOptPageHeaderCheck, // opt-resolve-order cfgOptPg, // opt-resolve-order cfgOptPgLocal, // opt-resolve-order cfgOptPgPath, // opt-resolve-order cfgOptPgPort, // opt-resolve-order cfgOptPgSocketPath, // opt-resolve-order cfgOptPgUser, // opt-resolve-order cfgOptPgVersionForce, // opt-resolve-order cfgOptPriority, // opt-resolve-order cfgOptProcess, // opt-resolve-order cfgOptProcessMax, // opt-resolve-order cfgOptProtocolTimeout, // opt-resolve-order cfgOptRaw, // opt-resolve-order cfgOptRecurse, // opt-resolve-order cfgOptReference, // opt-resolve-order cfgOptRemoteType, // opt-resolve-order cfgOptRepo, // opt-resolve-order cfgOptRepoBundle, // opt-resolve-order cfgOptRepoBundleLimit, // opt-resolve-order cfgOptRepoBundleSize, // opt-resolve-order cfgOptRepoCipherType, // opt-resolve-order cfgOptRepoFormat, // opt-resolve-order cfgOptRepoGcsUserProject, // opt-resolve-order cfgOptRepoHardlink, // opt-resolve-order cfgOptRepoLocal, // opt-resolve-order cfgOptRepoPath, // opt-resolve-order cfgOptRepoRetentionArchive, // opt-resolve-order cfgOptRepoRetentionArchiveType, // opt-resolve-order cfgOptRepoRetentionDiff, // opt-resolve-order cfgOptRepoRetentionFull, // opt-resolve-order cfgOptRepoRetentionFullType, // opt-resolve-order cfgOptRepoRetentionHistory, // opt-resolve-order cfgOptRepoTargetTime, // opt-resolve-order cfgOptRepoType, // opt-resolve-order cfgOptReport, // opt-resolve-order cfgOptResume, // opt-resolve-order cfgOptSckBlock, // opt-resolve-order cfgOptSckKeepAlive, // opt-resolve-order cfgOptSet, // opt-resolve-order cfgOptSort, // opt-resolve-order cfgOptSpoolPath, // opt-resolve-order cfgOptStartFast, // opt-resolve-order cfgOptStopAuto, // opt-resolve-order cfgOptTablespaceMap, // opt-resolve-order cfgOptTablespaceMapAll, // opt-resolve-order cfgOptTcpKeepAliveCount, // opt-resolve-order cfgOptTcpKeepAliveIdle, // opt-resolve-order cfgOptTcpKeepAliveInterval, // opt-resolve-order cfgOptTlsCipher12, // opt-resolve-order cfgOptTlsCipher13, // opt-resolve-order cfgOptTlsServerAddress, // opt-resolve-order cfgOptTlsServerAuth, // opt-resolve-order cfgOptTlsServerCaFile, // opt-resolve-order cfgOptTlsServerCertFile, // opt-resolve-order cfgOptTlsServerKeyFile, // opt-resolve-order cfgOptTlsServerPort, // opt-resolve-order cfgOptType, // opt-resolve-order cfgOptVerbose, // opt-resolve-order cfgOptVersion, // opt-resolve-order cfgOptArchiveCheck, // opt-resolve-order cfgOptArchiveCopy, // opt-resolve-order cfgOptArchiveModeCheck, // opt-resolve-order cfgOptCompressLevel, // opt-resolve-order cfgOptForce, // opt-resolve-order cfgOptPgDatabase, // opt-resolve-order cfgOptPgHost, // opt-resolve-order cfgOptPgHostConfig, // opt-resolve-order cfgOptPgHostConfigIncludePath, // opt-resolve-order cfgOptPgHostConfigPath, // opt-resolve-order cfgOptPgHostType, // opt-resolve-order cfgOptPgHostUser, // opt-resolve-order cfgOptRecoveryOption, // opt-resolve-order cfgOptRepoAzureAccount, // opt-resolve-order cfgOptRepoAzureContainer, // opt-resolve-order cfgOptRepoAzureEndpoint, // opt-resolve-order cfgOptRepoAzureKeyType, // opt-resolve-order cfgOptRepoAzureUriStyle, // opt-resolve-order cfgOptRepoBlock, // opt-resolve-order cfgOptRepoBlockAgeMap, // opt-resolve-order cfgOptRepoBlockChecksumSizeMap, // opt-resolve-order cfgOptRepoBlockSizeMap, // opt-resolve-order cfgOptRepoBlockSizeSuper, // opt-resolve-order cfgOptRepoBlockSizeSuperFull, // opt-resolve-order cfgOptRepoCipherPass, // opt-resolve-order cfgOptRepoCipherRotate, // opt-resolve-order cfgOptRepoGcsKeyType, // opt-resolve-order cfgOptRepoHost, // opt-resolve-order cfgOptRepoHostConfig, // opt-resolve-order cfgOptRepoHostConfigIncludePath, // opt-resolve-order cfgOptRepoHostConfigPath, // opt-resolve-order cfgOptRepoHostType, // opt-resolve-order cfgOptRepoHostUser, // opt-resolve-order cfgOptRepoS3Bucket, // opt-resolve-order cfgOptRepoS3Endpoint, // opt-resolve-order cfgOptRepoS3KeyType, // opt-resolve-order cfgOptRepoS3KmsKeyId, // opt-resolve-order cfgOptRepoS3ProcessCmd, // opt-resolve-order cfgOptRepoS3Region, // opt-resolve-order cfgOptRepoS3RequesterPays, // opt-resolve-order cfgOptRepoS3Role, // opt-resolve-order cfgOptRepoS3Service, // opt-resolve-order cfgOptRepoS3SseCustomerKey, // opt-resolve-order cfgOptRepoS3StsHost, // opt-resolve-order cfgOptRepoS3Token, // opt-resolve-order cfgOptRepoS3UriStyle, // opt-resolve-order cfgOptRepoSftpHost, // opt-resolve-order cfgOptRepoSftpHostKeyCheckType, // opt-resolve-order cfgOptRepoSftpHostKeyHashType, // opt-resolve-order cfgOptRepoSftpHostPort, // opt-resolve-order cfgOptRepoSftpHostUser, // opt-resolve-order cfgOptRepoSftpKnownHost, // opt-resolve-order cfgOptRepoSftpPrivateKeyFile, // opt-resolve-order cfgOptRepoSftpPrivateKeyPassphrase, // opt-resolve-order cfgOptRepoSftpPublicKeyFile, // opt-resolve-order cfgOptRepoStorageCaFile, // opt-resolve-order cfgOptRepoStorageCaPath, // opt-resolve-order cfgOptRepoStorageHost, // opt-resolve-order cfgOptRepoStoragePort, // opt-resolve-order cfgOptRepoStoragePrefetch, // opt-resolve-order cfgOptRepoStorageReadOver, // opt-resolve-order cfgOptRepoStorageTag, // opt-resolve-order cfgOptRepoStorageUploadChunkSize, // opt-resolve-order cfgOptRepoStorageVerifyTls, // opt-resolve-order cfgOptRepoSymlink, // opt-resolve-order cfgOptTarget, // opt-resolve-order cfgOptTargetAction, // opt-resolve-order cfgOptTargetExclusive, // opt-resolve-order cfgOptTargetTimeline, // opt-resolve-order cfgOptPgHostCaFile, // opt-resolve-order cfgOptPgHostCaPath, // opt-resolve-order cfgOptPgHostCertFile, // opt-resolve-order cfgOptPgHostCmd, // opt-resolve-order cfgOptPgHostKeyFile, // opt-resolve-order cfgOptPgHostPort, // opt-resolve-order cfgOptRepoAzureKey, // opt-resolve-order cfgOptRepoGcsBucket, // opt-resolve-order cfgOptRepoGcsEndpoint, // opt-resolve-order cfgOptRepoGcsKey, // opt-resolve-order cfgOptRepoHostCaFile, // opt-resolve-order cfgOptRepoHostCaPath, // opt-resolve-order cfgOptRepoHostCertFile, // opt-resolve-order cfgOptRepoHostCmd, // opt-resolve-order cfgOptRepoHostKeyFile, // opt-resolve-order cfgOptRepoHostPort, // opt-resolve-order cfgOptRepoS3Key, // opt-resolve-order cfgOptRepoS3KeySecret, // opt-resolve-order cfgOptRepoSftpHostFingerprint, // opt-resolve-order }; ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/parse.c0000664000175100017510000041126115265101745016721 0ustar00runnerrunner/*********************************************************************************************************************************** Command and Option Parse ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/log.h" #include "common/macro.h" #include "common/memContext.h" #include "common/regExp.h" #include "config/common.h" #include "config/config.intern.h" #include "config/parse.h" #include "version.h" /*********************************************************************************************************************************** The maximum number of keys that an indexed option can have, e.g. pg256-path would be the maximum pg-path option ***********************************************************************************************************************************/ #define CFG_OPTION_KEY_MAX 256 /*********************************************************************************************************************************** Section enum - defines which sections of the config an option can appear in ***********************************************************************************************************************************/ typedef enum { cfgSectionCommandLine, // Command-line only cfgSectionGlobal, // Command-line or in any config section cfgSectionStanza, // Command-line or in any config stanza section } ConfigSection; /*********************************************************************************************************************************** Default type enum ***********************************************************************************************************************************/ typedef enum { cfgDefaultTypeStatic, // A stored static string cfgDefaultTypeDynamic, // Determined at runtime } ConfigDefaultType; /*********************************************************************************************************************************** Standard config file name and old default path and name ***********************************************************************************************************************************/ #define PGBACKREST_CONFIG_ORIG_PATH_FILE "/etc/" PROJECT_CONFIG_FILE STRING_STATIC(PGBACKREST_CONFIG_ORIG_PATH_FILE_STR, PGBACKREST_CONFIG_ORIG_PATH_FILE); /*********************************************************************************************************************************** Mem context and local variables ***********************************************************************************************************************************/ static struct ConfigParseLocal { MemContext *memContext; // Mem context Ini *ini; // Parsed ini data } configParseLocal; /*********************************************************************************************************************************** Define how a command is parsed ***********************************************************************************************************************************/ typedef struct ParseRuleCommand { const char *name; // Name unsigned int commandRoleValid : CFG_COMMAND_ROLE_TOTAL; // Valid for the command role? bool lockRequired : 1; // Is an immediate lock required? bool lockRemoteRequired : 1; // Is a lock required on the remote? unsigned int lockType : 3; // Lock type required bool logFile : 1; // Will the command log to a file? unsigned int logLevelDefault : 4; // Default log level bool parameterAllowed : 1; // Command-line parameters are allowed } ParseRuleCommand; // Macros used to define parse rules in parse.auto.c.inc in a format that diffs well #define PARSE_RULE_COMMAND(...) \ {__VA_ARGS__} #define PARSE_RULE_COMMAND_NAME(nameParam) \ .name = nameParam #define PARSE_RULE_COMMAND_ROLE_VALID_LIST(...) \ .commandRoleValid = 0 __VA_ARGS__ #define PARSE_RULE_COMMAND_ROLE(commandRoleParam) \ | (1 << cfgCmdRole##commandRoleParam) #define PARSE_RULE_COMMAND_INTERNAL(internalParam) #define PARSE_RULE_COMMAND_LOCK_REQUIRED(lockRequiredParam) \ .lockRequired = lockRequiredParam #define PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(lockRemoteRequiredParam) \ .lockRemoteRequired = lockRemoteRequiredParam #define PARSE_RULE_COMMAND_LOCK_TYPE(lockTypeParam) \ .lockType = lockType##lockTypeParam #define PARSE_RULE_COMMAND_LOG_FILE(logFileParam) \ .logFile = logFileParam #define PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(logLevelDefaultParam) \ .logLevelDefault = logLevel##logLevelDefaultParam #define PARSE_RULE_COMMAND_PARAMETER_ALLOWED(parameterAllowedParam) \ .parameterAllowed = parameterAllowedParam /*********************************************************************************************************************************** Define how an option group is parsed ***********************************************************************************************************************************/ typedef struct ParseRuleOptionGroup { const char *name; // All options in the group must be prefixed with this name } ParseRuleOptionGroup; // Macros used to define parse rules in parse.auto.c.inc in a format that diffs well #define PARSE_RULE_OPTION_GROUP(...) \ {__VA_ARGS__} #define PARSE_RULE_OPTION_GROUP_NAME(nameParam) \ .name = nameParam /*********************************************************************************************************************************** Define how an option is parsed and interacts with other options ***********************************************************************************************************************************/ typedef struct ParseRuleOption { const char *name; // Name unsigned int type : 4; // e.g. string, int, boolean unsigned int defaultType : 1; // e.g. static, dynamic bool boolLike : 1; // Option accepts y/n and can be treated as bool? bool beta : 1; // Is the option a beta feature? bool negate : 1; // Can the option be negated on the command line? bool reset : 1; // Can the option be reset on the command line? bool required : 1; // Is the option required? bool sequence : 1; // StringId contains sequence? unsigned int section : 2; // e.g. global, stanza, cmd-line bool secure : 1; // Needs to be redacted in logs and cmd-line? bool multi : 1; // Can be specified multiple times? bool group : 1; // In a group? unsigned int groupId : 1; // Id if in a group bool deprecateMatch : 1; // Does a deprecated name exactly match the option name? unsigned int packSize : 7; // Size of optional data in pack format uint32_t commandRoleValid[CFG_COMMAND_ROLE_TOTAL]; // Valid for the command role? const uint8_t *pack; // Optional data in pack format } ParseRuleOption; // Define additional types of data that can be associated with an option. Because these types are rare they are not given dedicated // fields and are instead packed and read at runtime. This may seem inefficient but they are only accessed a single time during // parse so space efficiency is more important than performance. typedef enum { parseRuleOptionalTypeValid = 1, parseRuleOptionalTypeSequence, parseRuleOptionalTypeAllowRange, parseRuleOptionalTypeAllowList, parseRuleOptionalTypeDefault, parseRuleOptionalTypeRequired, } ParseRuleOptionalType; // Optional rule filter types typedef enum { parseRuleFilterTypeCommand = 1, } ParseRuleFilterType; // Macros used to define parse rules in parse.auto.c.inc in a format that diffs well #define PARSE_RULE_OPTION(...) \ {__VA_ARGS__} #define PARSE_RULE_OPTION_NAME(nameParam) \ .name = nameParam #define PARSE_RULE_OPTION_TYPE(typeParam) \ .type = cfgOptType##typeParam #define PARSE_RULE_OPTION_INTERNAL(typeParam) #define PARSE_RULE_OPTION_COMMAND_INTERNAL_LIST(...) #define PARSE_RULE_OPTION_DEFAULT_TYPE(defaultTypeParam) \ .defaultType = cfgDefaultType##defaultTypeParam #define PARSE_RULE_OPTION_BOOL_LIKE(boolLikeParam) \ .boolLike = boolLikeParam #define PARSE_RULE_OPTION_BETA(betaParam) \ .beta = betaParam #define PARSE_RULE_OPTION_NEGATE(negateParam) \ .negate = negateParam #define PARSE_RULE_OPTION_RESET(resetParam) \ .reset = resetParam #define PARSE_RULE_OPTION_REQUIRED(requiredParam) \ .required = requiredParam #define PARSE_RULE_OPTION_SECTION(sectionParam) \ .section = cfgSection##sectionParam #define PARSE_RULE_OPTION_SECURE(secureParam) \ .secure = secureParam #define PARSE_RULE_OPTION_SEQUENCE(sequenceParam) \ .sequence = sequenceParam #define PARSE_RULE_OPTION_MULTI(typeMulti) \ .multi = typeMulti #define PARSE_RULE_OPTION_GROUP_ID(groupIdParam) \ .group = true, .groupId = cfgOptGrp##groupIdParam #define PARSE_RULE_OPTION_DEPRECATE_MATCH(deprecateMatchParam) \ .deprecateMatch = deprecateMatchParam #define PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST(...) \ .commandRoleValid[cfgCmdRoleMain] = 0 __VA_ARGS__ #define PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST(...) \ .commandRoleValid[cfgCmdRoleAsync] = 0 __VA_ARGS__ #define PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST(...) \ .commandRoleValid[cfgCmdRoleLocal] = 0 __VA_ARGS__ #define PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST(...) \ .commandRoleValid[cfgCmdRoleRemote] = 0 __VA_ARGS__ #define PARSE_RULE_OPTION_COMMAND(commandParam) \ | (1 << cfgCmd##commandParam) #define PARSE_RULE_STRPUB(value) {.buffer = (const char *)value, .size = sizeof(value) - 1} // Macros used to define optional parse rules in pack format #define PARSE_RULE_VARINT_01(value) \ value #define PARSE_RULE_VARINT_02(value) \ 0x80 | (value & 0x7f), (value >> 7) & 0x7f #define PARSE_RULE_BOOL_TRUE 0x28 #define PARSE_RULE_BOOL_FALSE 0x20 #define PARSE_RULE_U32_1(value) 0x88, PARSE_RULE_VARINT_01(value) #define PARSE_RULE_U32_2(value) 0x88, PARSE_RULE_VARINT_02(value) #define PARSE_RULE_PACK(...) __VA_ARGS__ 0x00 #define PARSE_RULE_PACK_SIZE(...) \ 0xf0, 0x02, sizeof((const uint8_t []){PARSE_RULE_PACK(__VA_ARGS__)}), \ PARSE_RULE_PACK(__VA_ARGS__) #define PARSE_RULE_VAL_BOOL_TRUE PARSE_RULE_BOOL_TRUE #define PARSE_RULE_VAL_BOOL_FALSE PARSE_RULE_BOOL_FALSE #define PARSE_RULE_VAL_STR_IDX(valueIdx) ((const String *)&parseRuleValueStr[valueIdx]) #define PARSE_RULE_VAL_SEQ(value) PARSE_RULE_U32_1(value) #define PARSE_RULE_OPTIONAL(...) \ .packSize = sizeof((const uint8_t []){PARSE_RULE_PACK(__VA_ARGS__)}), \ .pack = (const uint8_t []){PARSE_RULE_PACK(__VA_ARGS__)} #define PARSE_RULE_OPTIONAL_GROUP(...) PARSE_RULE_PACK_SIZE(__VA_ARGS__) #define PARSE_RULE_FILTER_CMD(...) \ PARSE_RULE_PACK_SIZE(PARSE_RULE_U32_1(parseRuleFilterTypeCommand), __VA_ARGS__) #define PARSE_RULE_OPTIONAL_DEPEND(...) \ PARSE_RULE_U32_1(parseRuleOptionalTypeValid), PARSE_RULE_PACK_SIZE(__VA_ARGS__) #define PARSE_RULE_OPTIONAL_SEQUENCE(...) \ PARSE_RULE_U32_1(parseRuleOptionalTypeSequence) #define PARSE_RULE_OPTIONAL_NO_SEQUENCE(...) PARSE_RULE_OPTIONAL_SEQUENCE(__VA_ARGS__) #define PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(value) PARSE_RULE_BOOL_TRUE, value #define PARSE_RULE_OPTIONAL_ALLOW_LIST(...) \ PARSE_RULE_U32_1(parseRuleOptionalTypeAllowList), PARSE_RULE_PACK_SIZE(__VA_ARGS__) #define PARSE_RULE_OPTIONAL_ALLOW_RANGE(...) \ PARSE_RULE_U32_1(parseRuleOptionalTypeAllowRange), PARSE_RULE_PACK_SIZE(__VA_ARGS__) #define PARSE_RULE_OPTIONAL_ALLOW_RANGE_MAP(...) \ 0x10, __VA_ARGS__ 0x00 #define PARSE_RULE_OPTIONAL_DEFAULT(...) \ PARSE_RULE_U32_1(parseRuleOptionalTypeDefault), PARSE_RULE_PACK_SIZE(__VA_ARGS__) #define PARSE_RULE_OPTIONAL_DEFAULT_MAP(...) \ 0x10, __VA_ARGS__ 0x00 #define PARSE_RULE_OPTIONAL_REQUIRED(...) \ PARSE_RULE_U32_1(parseRuleOptionalTypeRequired) #define PARSE_RULE_OPTIONAL_NOT_REQUIRED(...) PARSE_RULE_OPTIONAL_REQUIRED(__VA_ARGS__) /*********************************************************************************************************************************** Define option deprecations ***********************************************************************************************************************************/ typedef struct ParseRuleOptionDeprecate { const char *name; // Deprecated name ConfigOption id; // Option Id bool indexed; // Can the deprecation be indexed? bool unindexed; // Can the deprecation be unindexed? } ParseRuleOptionDeprecate; /*********************************************************************************************************************************** Include automatically generated parse data ***********************************************************************************************************************************/ #include "config/parse.auto.c.inc" /*********************************************************************************************************************************** Struct to hold options parsed from the command line ***********************************************************************************************************************************/ typedef struct ParseOptionValue { bool found : 1; // Was the option found? bool negate : 1; // Was the option negated on the command line? bool reset : 1; // Was the option reset on the command line? unsigned int source : 2; // Where was the option found? StringList *valueList; // List of values found } ParseOptionValue; typedef struct ParseOption { unsigned int indexListTotal; // Total options in indexed list ParseOptionValue *indexList; // List of indexed option values } ParseOption; #define FUNCTION_LOG_PARSE_OPTION_FORMAT(value, buffer, bufferSize) \ typeToLog("ParseOption", buffer, bufferSize) /*********************************************************************************************************************************** Get the indexed value, creating the array to contain it if needed ***********************************************************************************************************************************/ static ParseOptionValue * parseOptionIdxValue(ParseOption *const optionList, const unsigned int optionId, const unsigned int optionKeyIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(PARSE_OPTION, optionList); // Structure containing all options being parsed FUNCTION_TEST_PARAM(UINT, optionId); // Unique ID which also identifies the option in the parse list FUNCTION_TEST_PARAM(UINT, optionKeyIdx); // Zero-based key index (e.g. pg3-path => 2), 0 for non-indexed FUNCTION_TEST_END(); // If the requested index is beyond what has already been allocated if (optionKeyIdx >= optionList[optionId].indexListTotal) { // If the option is in a group if (parseRuleOption[optionId].group) { unsigned int optionOffset = 0; // Allocate enough memory to include the requested indexed or a fixed amount to avoid too many allocations if (optionList[optionId].indexListTotal == 0) { optionList[optionId].indexListTotal = optionKeyIdx >= (LIST_INITIAL_SIZE / 2) ? optionKeyIdx + 1 : (LIST_INITIAL_SIZE / 2); optionList[optionId].indexList = memNew(sizeof(ParseOptionValue) * optionList[optionId].indexListTotal); } // Allocate more memory when needed. This could be more efficient but the limited number of indexes currently allowed // makes it difficult to get coverage on a better implementation. else { optionOffset = optionList[optionId].indexListTotal; optionList[optionId].indexListTotal = optionKeyIdx + 1; optionList[optionId].indexList = memResize( optionList[optionId].indexList, sizeof(ParseOptionValue) * optionList[optionId].indexListTotal); } // Initialize the newly allocated memory for (unsigned int optKeyIdx = optionOffset; optKeyIdx < optionList[optionId].indexListTotal; optKeyIdx++) optionList[optionId].indexList[optKeyIdx] = (ParseOptionValue){0}; } // Else the option is not in a group so there can only be one value else { optionList[optionId].indexList = memNew(sizeof(ParseOptionValue)); optionList[optionId].indexListTotal = 1; optionList[optionId].indexList[0] = (ParseOptionValue){0}; } } // Return the indexed value FUNCTION_TEST_RETURN_TYPE_P(ParseOptionValue, &optionList[optionId].indexList[optionKeyIdx]); } /*********************************************************************************************************************************** Check that --beta is set if a beta option is used ***********************************************************************************************************************************/ static void parseOptionBeta( const unsigned int optionId, const unsigned int optionKeyIdx, const bool beta, const ParseOption *const parseOptionBeta) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, optionId); FUNCTION_TEST_PARAM(UINT, optionKeyIdx); FUNCTION_TEST_PARAM(BOOL, beta); FUNCTION_TEST_PARAM_P(PARSE_OPTION, parseOptionBeta); FUNCTION_TEST_END(); if (beta && optionId != cfgOptBeta && (parseOptionBeta->indexListTotal == 0 || parseOptionBeta->indexList[0].negate)) { THROW_FMT( OptionInvalidError, "option '%s' is not valid without option '" CFGOPT_BETA "'\n" "HINT: beta features require the --" CFGOPT_BETA " option to prevent accidental usage.", cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Get command id by name ***********************************************************************************************************************************/ static ConfigCommand cfgParseCommandId(const char *const commandName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, commandName); FUNCTION_TEST_END(); ASSERT(commandName != NULL); // If command has a colon only search part before the colon const char *const colonPtr = strchr(commandName, ':'); const size_t commandSize = colonPtr == NULL ? (size_t)strlen(commandName) : (size_t)(colonPtr - commandName); // Get command ConfigCommand commandId; for (commandId = 0; commandId < CFG_COMMAND_TOTAL; commandId++) { if (strncmp(commandName, parseRuleCommand[commandId].name, commandSize) == 0) break; } if (commandId == CFG_COMMAND_TOTAL) THROW_FMT(CommandInvalidError, "invalid command '%s'", commandName); FUNCTION_TEST_RETURN(ENUM, commandId); } /**********************************************************************************************************************************/ FN_EXTERN const char * cfgParseCommandName(const ConfigCommand commandId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, commandId); FUNCTION_TEST_END(); ASSERT(commandId < CFG_COMMAND_TOTAL); FUNCTION_TEST_RETURN_CONST(STRINGZ, parseRuleCommand[commandId].name); } /*********************************************************************************************************************************** Convert command role from String to enum and vice versa ***********************************************************************************************************************************/ STRING_STATIC(CONFIG_COMMAND_ROLE_ASYNC_STR, CONFIG_COMMAND_ROLE_ASYNC); STRING_STATIC(CONFIG_COMMAND_ROLE_LOCAL_STR, CONFIG_COMMAND_ROLE_LOCAL); STRING_STATIC(CONFIG_COMMAND_ROLE_REMOTE_STR, CONFIG_COMMAND_ROLE_REMOTE); static ConfigCommandRole cfgParseCommandRoleEnum(const char *const commandRole) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, commandRole); FUNCTION_TEST_END(); if (commandRole == NULL) FUNCTION_TEST_RETURN(ENUM, cfgCmdRoleMain); else if (strEqZ(CONFIG_COMMAND_ROLE_ASYNC_STR, commandRole)) FUNCTION_TEST_RETURN(ENUM, cfgCmdRoleAsync); else if (strEqZ(CONFIG_COMMAND_ROLE_LOCAL_STR, commandRole)) FUNCTION_TEST_RETURN(ENUM, cfgCmdRoleLocal); else if (strEqZ(CONFIG_COMMAND_ROLE_REMOTE_STR, commandRole)) FUNCTION_TEST_RETURN(ENUM, cfgCmdRoleRemote); THROW_FMT(CommandInvalidError, "invalid command role '%s'", commandRole); } FN_EXTERN const String * cfgParseCommandRoleStr(const ConfigCommandRole commandRole) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, commandRole); FUNCTION_TEST_END(); const String *result = NULL; switch (commandRole) { case cfgCmdRoleMain: break; case cfgCmdRoleAsync: result = CONFIG_COMMAND_ROLE_ASYNC_STR; break; case cfgCmdRoleLocal: result = CONFIG_COMMAND_ROLE_LOCAL_STR; break; case cfgCmdRoleRemote: result = CONFIG_COMMAND_ROLE_REMOTE_STR; break; } FUNCTION_TEST_RETURN_CONST(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * cfgParseCommandRoleName(const ConfigCommand commandId, const ConfigCommandRole commandRoleId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, commandId); FUNCTION_TEST_PARAM(ENUM, commandRoleId); FUNCTION_TEST_END(); String *const result = strCatZ(strNew(), cfgParseCommandName(commandId)); if (commandRoleId != cfgCmdRoleMain) strCatFmt(result, ":%s", strZ(cfgParseCommandRoleStr(commandRoleId))); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN const Ini * cfgParseIni(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(INI, configParseLocal.ini); } /**********************************************************************************************************************************/ FN_EXTERN StringList * cfgParseStanzaList(void) { FUNCTION_TEST_VOID(); StringList *const result = strLstNew(); if (configParseLocal.ini != NULL) { MEM_CONTEXT_TEMP_BEGIN() { const StringList *const sectionList = strLstSort(iniSectionList(configParseLocal.ini), sortOrderAsc); for (unsigned int sectionIdx = 0; sectionIdx < strLstSize(sectionList); sectionIdx++) { const String *const section = strLstGet(sectionList, sectionIdx); // Skip global sections if (strEqZ(section, CFGDEF_SECTION_GLOBAL) || strBeginsWithZ(section, CFGDEF_SECTION_GLOBAL ":")) continue; // Extract stanza const StringList *const sectionPart = strLstNewSplitZ(section, ":"); ASSERT(strLstSize(sectionPart) <= 2); strLstAddIfMissing(result, strLstGet(sectionPart, 0)); } } MEM_CONTEXT_TEMP_END(); } strLstSort(result, sortOrderAsc); FUNCTION_TEST_RETURN(STRING_LIST, result); } /*********************************************************************************************************************************** Find an option by name in the option list ***********************************************************************************************************************************/ #define OPTION_PREFIX_NEGATE "no-" #define OPTION_PREFIX_RESET "reset-" #define OPTION_NAME_SIZE_MAX 64 FN_EXTERN CfgParseOptionResult cfgParseOption(const String *const optionCandidate, const CfgParseOptionParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, optionCandidate); FUNCTION_TEST_PARAM(BOOL, param.prefixMatch); FUNCTION_TEST_PARAM(BOOL, param.ignoreMissingIndex); FUNCTION_TEST_END(); ASSERT(optionCandidate != NULL); CfgParseOptionResult result = {0}; // Copy the option to a buffer so it can be efficiently manipulated char optionName[OPTION_NAME_SIZE_MAX + 1]; size_t optionNameSize = strSize(optionCandidate); if (optionNameSize > sizeof(optionName) - 1) { THROW_FMT( OptionInvalidError, "option '%s' exceeds maximum size of " STRINGIFY(OPTION_NAME_SIZE_MAX), strZ(optionCandidate)); } memcpy(optionName, strZ(optionCandidate), optionNameSize + 1); // If this looks like negate if (strncmp(optionName, OPTION_PREFIX_NEGATE, sizeof(OPTION_PREFIX_NEGATE) - 1) == 0) { result.negate = true; // Strip the negate prefix optionNameSize -= sizeof(OPTION_PREFIX_NEGATE) - 1; memmove(optionName, optionName + (sizeof(OPTION_PREFIX_NEGATE) - 1), optionNameSize + 1); } // Else if looks like reset else if (strncmp(optionName, OPTION_PREFIX_RESET, sizeof(OPTION_PREFIX_RESET) - 1) == 0) { result.reset = true; // Strip the reset prefix optionNameSize -= sizeof(OPTION_PREFIX_RESET) - 1; memmove(optionName, optionName + (sizeof(OPTION_PREFIX_RESET) - 1), optionNameSize + 1); } // Indexed options must have at least one dash char *const dashPtr = strchr(optionName, '-'); bool indexed = false; if (dashPtr != NULL) { if (dashPtr == optionName) THROW_FMT(OptionInvalidError, "option '%s' cannot begin with a dash", strZ(optionCandidate)); // Check if the first dash is preceded by a numeric key and keep a tally of the key char *numberPtr = dashPtr; unsigned int multiplier = 1; while (numberPtr > optionName && isdigit(*(numberPtr - 1))) { numberPtr--; result.keyIdx += (unsigned int)(*numberPtr - '0') * multiplier; multiplier *= 10; } if (numberPtr == optionName) THROW_FMT(OptionInvalidError, "option '%s' cannot begin with a number", strZ(optionCandidate)); // If there was a number then the option is indexed if (numberPtr != dashPtr) { indexed = true; // Strip the key to get the base option name optionNameSize -= (size_t)(dashPtr - numberPtr); memmove(numberPtr, dashPtr, optionNameSize + 1); // Check that the index is within bounds if (result.keyIdx < 1 || result.keyIdx > CFG_OPTION_KEY_MAX) { THROW_FMT( OptionInvalidError, "option '%s' key must be between 1 and " STRINGIFY(CFG_OPTION_KEY_MAX), strZ(optionCandidate)); } // Subtract one to represent a key index result.keyIdx--; } } // Search for an exact match. A copy of the option name must be made because bsearch() requires a reference. const char *const optionNamePtr = optionName; const ParseRuleOption *optionFound = bsearch( &optionNamePtr, parseRuleOption, CFG_OPTION_TOTAL, sizeof(ParseRuleOption), lstComparatorZ); // If the option was not found if (optionFound == NULL) { // Search for a single partial match (if requested) if (param.prefixMatch) { unsigned int findPartialIdx = 0; unsigned int findPartialTotal = 0; for (unsigned int findIdx = 0; findIdx < CFG_OPTION_TOTAL; findIdx++) { if (strncmp(parseRuleOption[findIdx].name, optionName, optionNameSize) == 0) { findPartialIdx = findIdx; findPartialTotal++; if (findPartialTotal > 1) break; } } // If a single partial match was found if (findPartialTotal == 1) optionFound = &parseRuleOption[findPartialIdx]; } // If the option was not found search deprecations if (optionFound == NULL) { // Search deprecations for an exact match const ParseRuleOptionDeprecate *deprecate = bsearch( &optionNamePtr, parseRuleOptionDeprecate, CFG_OPTION_DEPRECATE_TOTAL, sizeof(ParseRuleOptionDeprecate), lstComparatorZ); // If the option was not found then search deprecations for a single partial match (if requested) if (deprecate == NULL && param.prefixMatch) { unsigned int findPartialIdx = 0; unsigned int findPartialTotal = 0; for (unsigned int deprecateIdx = 0; deprecateIdx < CFG_OPTION_DEPRECATE_TOTAL; deprecateIdx++) { if (strncmp(parseRuleOptionDeprecate[deprecateIdx].name, optionName, optionNameSize) == 0) { findPartialIdx = deprecateIdx; findPartialTotal++; if (findPartialTotal > 1) break; } } // If a single partial match was found if (findPartialTotal == 1) deprecate = &parseRuleOptionDeprecate[findPartialIdx]; } // Deprecation was found if (deprecate != NULL) { // Error if the option is indexed but the deprecation is not if (indexed && !deprecate->indexed) THROW_FMT(OptionInvalidError, "deprecated option '%s' cannot have an index", strZ(optionCandidate)); // Error if the option is unindexed but the deprecation is not optionFound = &parseRuleOption[deprecate->id]; if (!indexed && !deprecate->unindexed && !param.ignoreMissingIndex) { ASSERT(dashPtr != NULL); const char *const groupName = parseRuleOptionGroup[optionFound->groupId].name; THROW_FMT( OptionInvalidError, "deprecated option '%s' requires an index\n" "HINT: add the required index, e.g. %.*s1%s.\n" "HINT: consider using the non-deprecated name, e.g. %s1%s.", strZ(optionCandidate), (int)(dashPtr - optionName), optionName, dashPtr, groupName, optionFound->name + strlen(groupName)); } result.deprecated = true; } } } // Option was found if (optionFound != NULL) { result.found = true; result.id = (unsigned int)(optionFound - parseRuleOption); // Error if negate is not allowed if (result.negate && !optionFound->negate) THROW_FMT(OptionInvalidError, "option '%s' cannot be negated", strZ(optionCandidate)); // Error if reset is not allowed if (result.reset && !optionFound->reset) THROW_FMT(OptionInvalidError, "option '%s' cannot be reset", strZ(optionCandidate)); // It is possible for an unindexed deprecation to match an indexed option name (without the index) exactly. For example, the // deprecated repo-path option now maps to repo-path index 0, which will yield an exact match. In this case we still need to // mark the option as deprecated. if (indexed == false && optionFound->deprecateMatch) result.deprecated = true; // If not deprecated make sure indexing matches. Deprecation indexing has already been checked because the rules are per // deprecation. if (!result.deprecated) { // Error if the option is indexed but should not be if (indexed && !optionFound->group) THROW_FMT(OptionInvalidError, "option '%s' cannot have an index", strZ(optionCandidate)); // Error if the option is unindexed but an index is required if (!indexed && optionFound->group && !param.ignoreMissingIndex) { const char *const groupName = parseRuleOptionGroup[optionFound->groupId].name; THROW_FMT( OptionInvalidError, "option '%s' requires an index\n" "HINT: add the required index, e.g. %s1%s.", strZ(optionCandidate), groupName, optionFound->name + strlen(groupName)); } } // Set flags result.beta = optionFound->beta; result.multi = optionFound->multi; FUNCTION_TEST_RETURN_TYPE(CfgParseOptionResult, result); } FUNCTION_TEST_RETURN_TYPE(CfgParseOptionResult, (CfgParseOptionResult){0}); } /**********************************************************************************************************************************/ FN_EXTERN ConfigOptionDataType cfgParseOptionDataType(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); switch (parseRuleOption[optionId].type) { case cfgOptTypeBoolean: FUNCTION_TEST_RETURN(ENUM, cfgOptDataTypeBoolean); case cfgOptTypeHash: FUNCTION_TEST_RETURN(ENUM, cfgOptDataTypeHash); case cfgOptTypeInteger: case cfgOptTypeSize: case cfgOptTypeTime: FUNCTION_TEST_RETURN(ENUM, cfgOptDataTypeInteger); case cfgOptTypeList: FUNCTION_TEST_RETURN(ENUM, cfgOptDataTypeList); case cfgOptTypeStringId: FUNCTION_TEST_RETURN(ENUM, cfgOptDataTypeStringId); default: break; } ASSERT(parseRuleOption[optionId].type == cfgOptTypePath || parseRuleOption[optionId].type == cfgOptTypeString); FUNCTION_TEST_RETURN(ENUM, cfgOptDataTypeString); } /*********************************************************************************************************************************** Get a value string by index ***********************************************************************************************************************************/ static const String * cfgParseOptionValueStr(const ConfigOptionType type, unsigned int valueIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(UINT, valueIdx); FUNCTION_TEST_END(); switch (type) { case cfgOptTypeInteger: valueIdx = parseRuleValueIntStrMap[valueIdx]; break; case cfgOptTypeSize: valueIdx = parseRuleValueSizeStrMap[valueIdx]; break; case cfgOptTypeTime: valueIdx = parseRuleValueTimeStrMap[valueIdx]; break; default: ASSERT(type == cfgOptTypePath || type == cfgOptTypeString || type == cfgOptTypeStringId); break; } FUNCTION_TEST_RETURN_CONST(STRING, PARSE_RULE_VAL_STR_IDX(valueIdx)); } /*********************************************************************************************************************************** Get an integer value by index ***********************************************************************************************************************************/ static int64_t cfgParseOptionValue(const ConfigOptionType type, unsigned int valueIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(UINT, valueIdx); FUNCTION_TEST_END(); switch (type) { case cfgOptTypeInteger: FUNCTION_TEST_RETURN(INT64, parseRuleValueInt[valueIdx]); case cfgOptTypeSize: FUNCTION_TEST_RETURN(INT64, parseRuleValueSize[valueIdx]); default: ASSERT(type == cfgOptTypeTime); break; } FUNCTION_TEST_RETURN(INT64, parseRuleValueTime[valueIdx]); } /*********************************************************************************************************************************** Get default value from pack ***********************************************************************************************************************************/ static void cfgParseOptionValuePack( PackRead *const ruleData, const ConfigOptionType type, ConfigOptionValueType *value, const String **raw) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, ruleData); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM_P(VOID, value); FUNCTION_TEST_PARAM_PP(VOID, raw); FUNCTION_TEST_END(); switch (pckReadType(ruleData)) { case pckTypeBool: { value->boolean = pckReadBoolP(ruleData); if (raw != NULL) *raw = value->boolean ? Y_STR : N_STR; break; } default: { const unsigned int valueIdx = pckReadU32P(ruleData); switch (type) { case cfgOptTypeInteger: case cfgOptTypeSize: case cfgOptTypeTime: value->integer = cfgParseOptionValue(type, valueIdx); break; case cfgOptTypePath: case cfgOptTypeString: value->string = PARSE_RULE_VAL_STR_IDX(valueIdx); break; default: ASSERT(type == cfgOptTypeStringId); break; } if (raw != NULL) *raw = cfgParseOptionValueStr(type, valueIdx); } } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Find an optional rule ***********************************************************************************************************************************/ typedef struct CfgParseOptionalRuleState { PackRead *pack; unsigned int typeNext; bool done; // Valid const uint8_t *valid; size_t validSize; unsigned int matchValue; // Sequence bool sequence; // Allow range int64_t allowRangeMin; int64_t allowRangeMax; unsigned int allowRangeMinIdx; unsigned int allowRangeMaxIdx; bool allowRangeMatch; // Allow list const uint8_t *allowList; size_t allowListSize; // Default const String *const defaultDynamicBin; // Binary for dynamic default const String *defaultRaw; ConfigOptionValueType defaultValue; uint8_t defaultSequence; // Sequence for the default // Required bool required; } CfgParseOptionalRuleState; static bool cfgParseOptionalRule( CfgParseOptionalRuleState *const optionalRules, const ParseRuleOptionalType optionalRuleType, const ConfigCommand commandId, const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, optionalRules); FUNCTION_TEST_PARAM(ENUM, optionalRuleType); FUNCTION_TEST_PARAM(ENUM, commandId); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(optionalRuleType != 0); ASSERT(commandId < CFG_COMMAND_TOTAL); ASSERT(optionId < CFG_OPTION_TOTAL); bool result = false; bool match = false; // Check for optional rules const ParseRuleOption *const ruleOption = &parseRuleOption[optionId]; if (!optionalRules->done && ruleOption->pack != NULL) { // Initialize optional rules if (optionalRules->pack == NULL) { MEM_CONTEXT_TEMP_BEGIN() { PackRead *const groupList = pckReadNewC(ruleOption->pack, ruleOption->packSize); // Search for a matching group do { // Get the group pack PackRead *group = pckReadPackReadConstP(groupList); // Process filters if any pckReadNext(group); if (pckReadType(group) == pckTypePack) { // Check for filter match bool match = false; PackRead *const filter = pckReadPackReadConstP(group); const ParseRuleFilterType filterType = (ParseRuleFilterType)pckReadU32P(filter); switch (filterType) { default: { ASSERT(filterType == parseRuleFilterTypeCommand); while (pckReadNext(filter)) { if ((ConfigCommand)pckReadU32P(filter) == commandId) { match = true; break; } } break; } } // Filter did not match if (!match) group = NULL; } // If the group matched if (group != NULL) { // Get first optional rule type. This is needed since pckReadNext() has already been called and it cannot // be called again until data is read. optionalRules->typeNext = pckReadU32P(group); ASSERT(optionalRules->typeNext != 0); // Move group to prior context and stop searching optionalRules->pack = pckReadMove(group, memContextPrior()); break; } } while (pckReadNext(groupList)); } MEM_CONTEXT_TEMP_END(); // If no group matched then done if (optionalRules->pack == NULL) { optionalRules->done = true; FUNCTION_TEST_RETURN(BOOL, false); } } // Search for the specified optional rule do { // Read the next optional rule type if it has not already been read if (optionalRules->typeNext == 0) { // If there are no more rules then done if (!pckReadNext(optionalRules->pack)) { optionalRules->done = true; FUNCTION_TEST_RETURN(BOOL, false); } optionalRules->typeNext = pckReadU32P(optionalRules->pack); ASSERT(optionalRules->typeNext != 0); } // If this is the requested optional rule if (optionalRules->typeNext == optionalRuleType) { // Optional rule was found and dependency was matched result = true; match = true; // Process optional rule switch (optionalRuleType) { case parseRuleOptionalTypeValid: { pckReadNext(optionalRules->pack); optionalRules->valid = pckReadBufPtr(optionalRules->pack); optionalRules->validSize = pckReadSize(optionalRules->pack); pckReadConsume(optionalRules->pack); break; } case parseRuleOptionalTypeSequence: optionalRules->sequence = !ruleOption->sequence; break; case parseRuleOptionalTypeAllowList: { pckReadNext(optionalRules->pack); optionalRules->allowList = pckReadBufPtr(optionalRules->pack); optionalRules->allowListSize = pckReadSize(optionalRules->pack); pckReadConsume(optionalRules->pack); break; } case parseRuleOptionalTypeAllowRange: { PackRead *const ruleData = pckReadPackReadConstP(optionalRules->pack); pckReadNext(ruleData); if (pckReadType(ruleData) == pckTypeArray) { pckReadArrayBeginP(ruleData); while (!pckReadNullP(ruleData)) // {uncovered_branch - no options with partial allow range maps} { const unsigned int map = pckReadU32P(ruleData); optionalRules->allowRangeMinIdx = pckReadU32P(ruleData); optionalRules->allowRangeMaxIdx = pckReadU32P(ruleData); if (map == optionalRules->matchValue) { optionalRules->allowRangeMatch = true; break; } } match = optionalRules->allowRangeMatch; } else { optionalRules->allowRangeMinIdx = pckReadU32P(ruleData); optionalRules->allowRangeMaxIdx = pckReadU32P(ruleData); } optionalRules->allowRangeMin = cfgParseOptionValue(ruleOption->type, optionalRules->allowRangeMinIdx); optionalRules->allowRangeMax = cfgParseOptionValue(ruleOption->type, optionalRules->allowRangeMaxIdx); break; } case parseRuleOptionalTypeDefault: { PackRead *const ruleData = pckReadPackReadConstP(optionalRules->pack); if (ruleOption->defaultType == cfgDefaultTypeDynamic) { ASSERT(ruleOption->type == cfgOptTypeString); ASSERT(!ruleOption->sequence); // No need to check the value until there is more than one pckReadU32P(ruleData); optionalRules->defaultValue.string = optionalRules->defaultDynamicBin; optionalRules->defaultRaw = optionalRules->defaultDynamicBin; } else { ASSERT(ruleOption->defaultType == cfgDefaultTypeStatic); pckReadNext(ruleData); if (pckReadType(ruleData) == pckTypeArray) { ASSERT(!ruleOption->sequence); pckReadArrayBeginP(ruleData); match = false; while (!pckReadNullP(ruleData)) { const unsigned int map = pckReadU32P(ruleData); cfgParseOptionValuePack( ruleData, ruleOption->type, &optionalRules->defaultValue, &optionalRules->defaultRaw); if (map == optionalRules->matchValue) { match = true; break; } } } else { cfgParseOptionValuePack( ruleData, ruleOption->type, &optionalRules->defaultValue, &optionalRules->defaultRaw); if (optionalRules->sequence) optionalRules->defaultSequence = (uint8_t)pckReadU32P(ruleData); } } break; } default: { ASSERT(optionalRuleType == parseRuleOptionalTypeRequired); optionalRules->required = !ruleOption->required; } } } // Else not the requested optional rule else { // If the optional rule type is greater than requested then return. The optional rule may be requested later. if (optionalRules->typeNext > optionalRuleType) FUNCTION_TEST_RETURN(BOOL, false); // Consume the unused optional rule pckReadConsume(optionalRules->pack); } // Type will need to be read again on next iteration optionalRules->typeNext = 0; } while (!result); } FUNCTION_TEST_RETURN(BOOL, result && match); } /*********************************************************************************************************************************** Resolve an option dependency ***********************************************************************************************************************************/ typedef struct CfgParseOptionalFilterDependResult { bool valid; // Is the dependency valid? bool defaultExists; // Is there a default of the dependency is not valid? ConfigOption dependId; // Option id for the dependency ConfigOptionValueType defaultValue; // Default value if dependency is not valid unsigned int matchValue; // Match value if the dependency is valid } CfgParseOptionalFilterDependResult; static CfgParseOptionalFilterDependResult cfgParseOptionalFilterDepend( PackRead *const filter, const ConfigOptionType optionType, const Config *const config, const unsigned int optionListIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, filter); FUNCTION_TEST_PARAM(ENUM, optionType); FUNCTION_TEST_PARAM_P(VOID, config); FUNCTION_TEST_PARAM(UINT, optionListIdx); FUNCTION_TEST_END(); CfgParseOptionalFilterDependResult result = {.valid = false}; // Default when the dependency is not resolved, if it exists pckReadNext(filter); if (pckReadType(filter) == pckTypeBool) { pckReadBoolP(filter); pckReadNext(filter); result.defaultExists = true; cfgParseOptionValuePack(filter, optionType, &result.defaultValue, NULL); } // Get the depend option value result.dependId = (ConfigOption)pckReadU32P(filter); ASSERT(config->option[result.dependId].index != NULL); const ConfigOptionValue *const dependValue = &config->option[result.dependId].index[parseRuleOption[result.dependId].group ? optionListIdx : 0]; // Is the dependency resolved? if (dependValue->set) { // If a depend list exists, make sure the value is in the list if (pckReadNext(filter)) { do { switch (cfgParseOptionDataType(result.dependId)) { case cfgOptDataTypeBoolean: result.valid = pckReadBoolP(filter) == dependValue->value.boolean; break; default: { ASSERT(cfgParseOptionDataType(result.dependId) == cfgOptDataTypeStringId); result.matchValue = pckReadU32P(filter); if (strEq(PARSE_RULE_VAL_STR_IDX(result.matchValue), dependValue->display)) result.valid = true; break; } } if (result.valid) break; } while (pckReadNext(filter)); } else result.valid = true; } FUNCTION_TEST_RETURN_TYPE(CfgParseOptionalFilterDependResult, result); } /**********************************************************************************************************************************/ FN_EXTERN const char * cfgParseOptionName(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN_CONST(STRINGZ, parseRuleOption[optionId].name); } /**********************************************************************************************************************************/ FN_EXTERN const char * cfgParseOptionKeyIdxName(const ConfigOption optionId, const unsigned int keyIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, keyIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT((!parseRuleOption[optionId].group && keyIdx == 0) || parseRuleOption[optionId].group); // If the option is in a group then construct the name const ParseRuleOption *const ruleOption = &parseRuleOption[optionId]; if (ruleOption->group) { FUNCTION_TEST_RETURN_CONST( STRINGZ, zNewFmt( "%s%u%s", parseRuleOptionGroup[ruleOption->groupId].name, keyIdx + 1, ruleOption->name + strlen(parseRuleOptionGroup[ruleOption->groupId].name))); } // Else return the stored name FUNCTION_TEST_RETURN_CONST(STRINGZ, ruleOption->name); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgParseOptionGroup(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN(BOOL, parseRuleOption[optionId].group); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgParseOptionGroupId(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(parseRuleOption[optionId].group); FUNCTION_TEST_RETURN(UINT, parseRuleOption[optionId].groupId); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgParseOptionSecure(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN(BOOL, parseRuleOption[optionId].secure); } /**********************************************************************************************************************************/ FN_EXTERN ConfigOptionType cfgParseOptionType(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN(ENUM, parseRuleOption[optionId].type); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgParseOptionValid(const ConfigCommand commandId, const ConfigCommandRole commandRoleId, const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, commandId); FUNCTION_TEST_PARAM(ENUM, commandRoleId); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(commandId < CFG_COMMAND_TOTAL); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN(BOOL, parseRuleOption[optionId].commandRoleValid[commandRoleId] & ((uint32_t)1 << commandId)); } /*********************************************************************************************************************************** Load the configuration file(s) The parent mem context is used. Defaults are passed to make testing easier. Rules: - config and config-include-path are default. In this case, the config file will be loaded, if it exists, and *.conf files in the config-include-path will be appended, if they exist. A missing/empty dir will be ignored except that the original default for the config file will be attempted to be loaded if the current default is not found. - config only is specified. Only the specified config file will be loaded and is required. The default config-include-path will be ignored. - config and config-path are specified. The specified config file will be loaded and is required. The overridden default of the config-include-path (/conf.d) will be loaded if exists but is not required. - config-include-path only is specified. *.conf files in the config-include-path will be loaded and the path is required to exist. The default config will be be loaded if it exists. - config-include-path and config-path are specified. The *.conf files in the config-include-path will be loaded and the directory passed must exist. The overridden default of the config file path (/pgbackrest.conf) will be loaded if exists but is not required. - If the config and config-include-path are specified. The config file will be loaded and is expected to exist and *.conf files in the config-include-path will be appended and at least one is expected to exist. - If --no-config is specified and --config-include-path is specified then only *.conf files in the config-include-path will be loaded; the directory is required. - If --no-config is specified and --config-path is specified then only *.conf files in the overridden default config-include-path (/conf.d) will be loaded if exist but not required. - If --no-config is specified and neither --config-include-path nor --config-path are specified then no configs will be loaded. - If --config-path only, the defaults for config and config-include-path will be changed to use that as a base path but the files will not be required to exist since this is a default override. ***********************************************************************************************************************************/ static void cfgFileLoadPart(String **const config, const Buffer *const configPart) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(STRING, config); FUNCTION_LOG_PARAM(BUFFER, configPart); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); if (configPart != NULL) { // Validate the file by parsing it as an Ini object. If the file is not properly formed, an error will occur. if (bufUsed(configPart) > 0) { iniValid(iniNewP(ioBufferReadNew(configPart))); // Create the result config file if (*config == NULL) { *config = strNew(); } // Else add an LF in case the previous file did not end with one else strCat(*config, LF_STR); // Add the config part to the result config file strCat(*config, strNewBuf(configPart)); } } FUNCTION_LOG_RETURN_VOID(); } static String * cfgFileLoad( const Storage *const storage, // Storage to load configs const ParseOption *const optionList, // All options and their current settings // NOTE: Passing defaults to enable more complete test coverage const String *optConfigDefault, // Current default for --config option const String *optConfigIncludePathDefault, // Current default for --config-include-path option const String *const origConfigDefault) // Original --config option default (/etc/pgbackrest.conf) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM_P(PARSE_OPTION, optionList); FUNCTION_LOG_PARAM(STRING, optConfigDefault); FUNCTION_LOG_PARAM(STRING, optConfigIncludePathDefault); FUNCTION_LOG_PARAM(STRING, origConfigDefault); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(optionList != NULL); ASSERT(optConfigDefault != NULL); ASSERT(optConfigIncludePathDefault != NULL); ASSERT(origConfigDefault != NULL); bool loadConfig = true; bool loadConfigInclude = true; // If the option is specified on the command line, then found will be true meaning the file is required to exist, else it is // optional const bool configFound = optionList[cfgOptConfig].indexList != NULL && optionList[cfgOptConfig].indexList[0].found; bool configRequired = configFound; const bool configPathRequired = optionList[cfgOptConfigPath].indexList != NULL && optionList[cfgOptConfigPath].indexList[0].found; bool configIncludeRequired = optionList[cfgOptConfigIncludePath].indexList != NULL && optionList[cfgOptConfigIncludePath].indexList[0].found; // Save default for later determining if must check old original default config path const String *const optConfigDefaultCurrent = optConfigDefault; // If the config-path option is found on the command line, then its value will override the base path defaults for config and // config-include-path if (configPathRequired) { optConfigDefault = strNewFmt( "%s/%s", strZ(strLstGet(optionList[cfgOptConfigPath].indexList[0].valueList, 0)), strBaseZ(optConfigDefault)); optConfigIncludePathDefault = strNewFmt( "%s/%s", strZ(strLstGet(optionList[cfgOptConfigPath].indexList[0].valueList, 0)), PROJECT_CONFIG_INCLUDE_PATH); } // If the --no-config option was passed then do not load the config file if (optionList[cfgOptConfig].indexList != NULL && optionList[cfgOptConfig].indexList[0].negate) { loadConfig = false; configRequired = false; } // If --config option is specified on the command line but neither the --config-include-path nor the config-path are passed, // then do not attempt to load the include files if (configFound && !(configPathRequired || configIncludeRequired)) { loadConfigInclude = false; configIncludeRequired = false; } String *result = NULL; // Load the main config file if (loadConfig) { const String *configFileName = NULL; // Get the config file name from the command-line if it exists else default if (configRequired) configFileName = strLstGet(optionList[cfgOptConfig].indexList[0].valueList, 0); else configFileName = optConfigDefault; // Load the config file const Buffer *buffer = storageGetP(storageNewReadP(storage, configFileName, .ignoreMissing = !configRequired)); // Convert the contents of the file buffer to the config string object if (buffer != NULL) result = strCatBuf(strNew(), buffer); else if (strEq(configFileName, optConfigDefaultCurrent)) { // If config is current default and it was not found, attempt to load the config file from the old default location buffer = storageGetP(storageNewReadP(storage, origConfigDefault, .ignoreMissing = !configRequired)); if (buffer != NULL) result = strCatBuf(strNew(), buffer); } } // Load *.conf files from the include directory if (loadConfigInclude) { // Validate the file by parsing it as an Ini object. If the file is not properly formed, an error will occur. if (result != NULL) iniValid(iniNewP(ioBufferReadNew(BUFSTR(result)))); const String *configIncludePath = NULL; // Get the config include path from the command-line if it exists else default if (configIncludeRequired) configIncludePath = strLstGet(optionList[cfgOptConfigIncludePath].indexList[0].valueList, 0); else configIncludePath = optConfigIncludePathDefault; // Get a list of conf files from the specified path -error on missing directory if the option was passed on the command line StringList *const list = storageListP( storage, configIncludePath, .expression = STRDEF(".+\\.conf$"), .errorOnMissing = configIncludeRequired, .nullOnMissing = !configIncludeRequired); // If conf files are found, then add them to the config string if (list != NULL && !strLstEmpty(list)) { // Sort the list for reproducibility only -- order does not matter strLstSort(list, sortOrderAsc); for (unsigned int listIdx = 0; listIdx < strLstSize(list); listIdx++) { cfgFileLoadPart( &result, storageGetP( storageNewReadP( storage, strNewFmt("%s/%s", strZ(configIncludePath), strZ(strLstGet(list, listIdx))), .ignoreMissing = true))); } } } FUNCTION_LOG_RETURN(STRING, result); } /*********************************************************************************************************************************** ??? Add validation of section names and check all sections for invalid options in the check command. It's too expensive to add the logic to this critical path code. ***********************************************************************************************************************************/ // Helper to check that option values are valid for conditional builds. This is a bit tricky since the distros used for unit testing // have all possible features enabled, so this is split out to allow it to be tested independently. The loop variable is // intentionally integrated into this function to make it obvious if it is omitted from the caller. static bool cfgParseOptionValueCondition( bool more, PackRead *const allowList, const bool allowListFound, const unsigned int optionId, const unsigned int optionKeyIdx, const String *const valueAllow) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, more); FUNCTION_TEST_PARAM(PACK_READ, allowList); FUNCTION_TEST_PARAM(BOOL, allowListFound); FUNCTION_TEST_PARAM(UINT, optionId); FUNCTION_TEST_PARAM(UINT, optionKeyIdx); FUNCTION_TEST_PARAM(STRING, valueAllow); FUNCTION_TEST_END(); ASSERT(allowList != NULL); ASSERT(valueAllow != NULL); if (more && pckReadType(allowList) == pckTypeBool) { pckReadBoolP(allowList); if (allowListFound) { THROW_FMT( OptionInvalidValueError, PROJECT_NAME " not built with '%s=%s' support\n" "HINT: if " PROJECT_NAME " was installed from a package, does the package support this feature?\n" "HINT: if " PROJECT_NAME " was built from source, were the required development packages installed?", cfgParseOptionKeyIdxName(optionId, optionKeyIdx), strZ(valueAllow)); } more = pckReadNext(allowList); } FUNCTION_TEST_RETURN(BOOL, more); } FN_EXTERN void cfgParse(const Storage *const storage, const unsigned int argListSize, const char *argList[], const CfgParseParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(UINT, argListSize); FUNCTION_LOG_PARAM(CHARPY, argList); FUNCTION_LOG_PARAM(BOOL, param.noResetLogLevel); FUNCTION_LOG_PARAM(BOOL, param.noConfigLoad); FUNCTION_LOG_END(); // Initialize local mem context if (configParseLocal.memContext == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(ConfigParse, .childQty = MEM_CONTEXT_QTY_MAX) { configParseLocal.memContext = MEM_CONTEXT_NEW(); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_BEGIN() { // Create the config struct Config *config; OBJ_NEW_BASE_BEGIN(Config, .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX) { config = OBJ_NEW_ALLOC(); *config = (Config) { .memContext = MEM_CONTEXT_NEW(), .command = cfgCmdHelp, .bin = strNewZ(argList[0]), }; } OBJ_NEW_END(); // Phase 1: parse command line parameters // ------------------------------------------------------------------------------------------------------------------------- // List of parsed options ParseOption parseOptionList[CFG_OPTION_TOTAL] = {{0}}; // Only the first non-option parameter should be treated as a command so track if the command has been set bool commandSet = false; bool commandHelpSet = false; for (unsigned int argListIdx = 1; argListIdx < argListSize; argListIdx++) { const char *arg = argList[argListIdx]; // If an option if (arg[0] == '-') { // Options must start with -- if (arg[1] != '-') THROW_FMT(OptionInvalidError, "option '%s' must begin with --", arg); // Consume -- arg += 2; // Get the option name and the value when separated by = const String *optionName; const String *optionArg = NULL; const char *const equalPtr = strchr(arg, '='); if (equalPtr) { optionName = strNewZN(arg, (size_t)(equalPtr - arg)); optionArg = strNewZ(equalPtr + 1); } else optionName = strNewZ(arg); // Lookup the option name CfgParseOptionResult option = cfgParseOptionP(optionName, .prefixMatch = true); if (!option.found) THROW_FMT(OptionInvalidError, "invalid option '--%s'", arg); // If the option may have an argument (arguments are optional for boolean options) const ParseRuleOption *const ruleOption = &parseRuleOption[option.id]; if (!option.negate && !option.reset) { // Handle boolean (only y/n allowed as argument) if (ruleOption->type == cfgOptTypeBoolean) { // Validate argument/set negate when argument present if (optionArg != NULL) { if (strEqZ(optionArg, "n")) option.negate = true; else if (!strEqZ(optionArg, "y")) { THROW_FMT( OptionInvalidValueError, "boolean option '--%s' argument must be 'y' or 'n'", strZ(optionName)); } } } // If no argument was found with the option then try the next argument (unless bool-like) else if (optionArg == NULL) { // If bool-like then set arg to y if (ruleOption->boolLike) { optionArg = Y_STR; } // Else if there are no more arguments in the list else if (argListIdx == argListSize - 1) { THROW_FMT(OptionInvalidError, "option '--%s' requires an argument", strZ(optionName)); } // Else get arg else optionArg = strNewZ(argList[++argListIdx]); } } // Else error if an argument was found with the option else if (optionArg != NULL) THROW_FMT(OptionInvalidError, "option '%s' does not allow an argument", strZ(optionName)); // if negated and bool-like then set to n if (option.negate && ruleOption->boolLike) { option.negate = false; optionArg = N_STR; } // Error if this option is secure and cannot be passed on the command line if (cfgParseOptionSecure(option.id)) { THROW_FMT( OptionInvalidError, "option '%s' is not allowed on the command-line\n" "HINT: this option could expose secrets in the process list.\n" "HINT: specify the option in a configuration file or an environment variable instead.", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } // If the option has not been found yet then set it ParseOptionValue *const optionValue = parseOptionIdxValue(parseOptionList, option.id, option.keyIdx); if (!optionValue->found) { *optionValue = (ParseOptionValue) { .found = true, .negate = option.negate, .reset = option.reset, .source = cfgSourceParam, }; // Only set the argument if the option has one if (optionArg != NULL) { optionValue->valueList = strLstNew(); strLstAdd(optionValue->valueList, optionArg); } } else { // Make sure option is not negated more than once. It probably wouldn't hurt anything to accept this case but // there's no point in allowing the user to be sloppy. if (optionValue->negate && option.negate) { THROW_FMT( OptionInvalidError, "option '%s' is negated multiple times", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } // Make sure option is not reset more than once. Same justification as negate. if (optionValue->reset && option.reset) { THROW_FMT( OptionInvalidError, "option '%s' is reset multiple times", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } // Don't allow an option to be both negated and reset if ((optionValue->reset && option.negate) || (optionValue->negate && option.reset)) { THROW_FMT( OptionInvalidError, "option '%s' cannot be negated and reset", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } // Don't allow an option to be both set and negated if (optionValue->negate != option.negate) { THROW_FMT( OptionInvalidError, "option '%s' cannot be set and negated", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } // Don't allow an option to be both set and reset if (optionValue->reset != option.reset) { THROW_FMT( OptionInvalidError, "option '%s' cannot be set and reset", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } // Add the argument if (optionArg != NULL && option.multi) { strLstAdd(optionValue->valueList, optionArg); } // Error if the option does not accept multiple arguments else { THROW_FMT( OptionInvalidError, "option '%s' cannot be set multiple times", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } } } // Else command or parameter else { // The first argument should be the command if (!commandSet) { // Try getting the command from the valid command list config->command = cfgParseCommandId(arg); // Set help if help command if (config->command == cfgCmdHelp && !commandHelpSet) { commandHelpSet = true; } // Else parse command role if appended else { const char *const colonPtr = strchr(arg, ':'); if (colonPtr != NULL) config->commandRole = cfgParseCommandRoleEnum(colonPtr + 1); else config->commandRole = cfgCmdRoleMain; // Error when role is not valid for the command if (!(parseRuleCommand[config->command].commandRoleValid & ((unsigned int)1 << config->commandRole))) THROW_FMT(CommandInvalidError, "invalid command/role combination '%s'", arg); // Error if help requested and role is not main if (commandHelpSet && config->commandRole != cfgCmdRoleMain) THROW_FMT(CommandInvalidError, "help is not available for the '%s' role", colonPtr + 1); config->help = commandHelpSet; commandSet = true; } } // Additional arguments are command arguments else { if (config->paramList == NULL) { MEM_CONTEXT_BEGIN(config->memContext) { config->paramList = strLstNew(); } MEM_CONTEXT_END(); } strLstAddZ(config->paramList, arg); } } } // Handle command not found if (!commandSet && !commandHelpSet) { ASSERT(config->command == cfgCmdHelp); for (unsigned int optionIdx = 0; optionIdx < CFG_OPTION_TOTAL; optionIdx++) { if (parseOptionList[optionIdx].indexListTotal != 0 && !cfgParseOptionValid(cfgCmdHelp, cfgCmdRoleMain, optionIdx)) THROW_FMT(CommandRequiredError, "no command found"); } } // Set command options const ParseRuleCommand *const ruleCommand = &parseRuleCommand[config->command]; config->lockRequired = ruleCommand->lockRequired; config->lockRemoteRequired = ruleCommand->lockRemoteRequired; config->lockType = (LockType)ruleCommand->lockType; config->logFile = ruleCommand->logFile; config->logLevelDefault = (LogLevel)ruleCommand->logLevelDefault; // Error when parameters found but the command does not allow parameters if (config->paramList != NULL && !config->help && !ruleCommand->parameterAllowed) THROW(ParamInvalidError, "command does not allow parameters"); // Enable logging for main role so config file warnings will be output if (!param.noResetLogLevel && config->commandRole == cfgCmdRoleMain) logInit(logLevelWarn, logLevelOff, logLevelOff, false, 0, 1, false); // Determine whether any options should be loaded from the environment or config files. The help and version commands do // not use any non command-line options so they can skip this step. This prevents basic help or version from failing if, // e.g. the process does not have access to load the config file or there is an invalid option in the environment. const bool optionLoad = config->command != cfgCmdHelp && config->command != cfgCmdVersion; // Phase 2: parse environment variables // ------------------------------------------------------------------------------------------------------------------------- if (optionLoad) { unsigned int environIdx = 0; // Loop through all environment variables and look for our env vars by matching the prefix while (environ[environIdx] != NULL) { const char *keyValue = environ[environIdx]; environIdx++; if (strstr(keyValue, PGBACKREST_ENV) == keyValue) { // Find the first = char const char *const equalPtr = strchr(keyValue, '='); ASSERT(equalPtr != NULL); // Get key and value const String *const key = strReplaceChr( strLower(strNewZN(keyValue + PGBACKREST_ENV_SIZE, (size_t)(equalPtr - (keyValue + PGBACKREST_ENV_SIZE)))), '_', '-'); const String *const value = STR(equalPtr + 1); // Find the option CfgParseOptionResult option = cfgParseOptionP(key); // Warn if the option not found if (!option.found) { LOG_WARN_FMT("environment contains invalid option '%s'", strZ(key)); continue; } // Warn if negate option found in env else if (option.negate) { LOG_WARN_FMT("environment contains invalid negate option '%s'", strZ(key)); continue; } // Warn if reset option found in env else if (option.reset) { LOG_WARN_FMT("environment contains invalid reset option '%s'", strZ(key)); continue; } // Continue if the option is not valid for this command if (!cfgParseOptionValid(config->command, config->commandRole, option.id)) continue; if (strSize(value) == 0) THROW_FMT(OptionInvalidValueError, "environment variable '%s' must have a value", strZ(key)); // Continue if the option has already been specified on the command line ParseOptionValue *optionValue = parseOptionIdxValue(parseOptionList, option.id, option.keyIdx); if (optionValue->found) continue; optionValue->found = true; optionValue->source = cfgSourceConfig; // Convert boolean to string if (cfgParseOptionType(option.id) == cfgOptTypeBoolean) { if (strEqZ(value, "n")) optionValue->negate = true; else if (!strEqZ(value, "y")) THROW_FMT(OptionInvalidValueError, "environment boolean option '%s' must be 'y' or 'n'", strZ(key)); } // Else split list/hash into separate values else if (option.multi) { optionValue->valueList = strLstNewSplitZ(value, ":"); } // Else add the string value else { optionValue->valueList = strLstNew(); strLstAdd(optionValue->valueList, value); } } } } // Phase 3: parse config file unless --no-config passed // ------------------------------------------------------------------------------------------------------------------------- // Load the configuration file(s) if (!param.noConfigLoad && optionLoad) { const String *const configString = cfgFileLoad( storage, parseOptionList, PARSE_RULE_VAL_STR_IDX(parseRuleValStrCFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_FILE), PARSE_RULE_VAL_STR_IDX(parseRuleValStrCFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_INCLUDE_PATH), PGBACKREST_CONFIG_ORIG_PATH_FILE_STR); iniFree(configParseLocal.ini); configParseLocal.ini = NULL; if (configString != NULL) { MEM_CONTEXT_BEGIN(configParseLocal.memContext) { configParseLocal.ini = iniNewP(ioBufferReadNew(BUFSTR(configString)), .store = true); } MEM_CONTEXT_END(); } } if (configParseLocal.ini != NULL) { // Get the stanza name String *stanza = NULL; if (parseOptionList[cfgOptStanza].indexList != NULL) stanza = strLstGet(parseOptionList[cfgOptStanza].indexList[0].valueList, 0); // Build list of sections to search for options StringList *const sectionList = strLstNew(); if (stanza != NULL) { strLstAddFmt(sectionList, "%s:%s", strZ(stanza), cfgParseCommandName(config->command)); strLstAdd(sectionList, stanza); } strLstAddFmt(sectionList, CFGDEF_SECTION_GLOBAL ":%s", cfgParseCommandName(config->command)); strLstAddZ(sectionList, CFGDEF_SECTION_GLOBAL); // Loop through sections to search for options for (unsigned int sectionIdx = 0; sectionIdx < strLstSize(sectionList); sectionIdx++) { const String *const section = strLstGet(sectionList, sectionIdx); const StringList *const keyList = iniSectionKeyList(configParseLocal.ini, section); KeyValue *const optionFound = kvNew(); // Loop through keys to search for options for (unsigned int keyIdx = 0; keyIdx < strLstSize(keyList); keyIdx++) { String *const key = strLstGet(keyList, keyIdx); // Find the optionName in the main list const CfgParseOptionResult option = cfgParseOptionP(key); const ParseRuleOption *const ruleOption = &parseRuleOption[option.id]; // Warn if the option not found if (!option.found) { LOG_WARN_FMT("configuration file contains invalid option '%s'", strZ(key)); continue; } // Warn if negate option found in config else if (option.negate) { LOG_WARN_FMT("configuration file contains negate option '%s'", strZ(key)); continue; } // Warn if reset option found in config else if (option.reset) { LOG_WARN_FMT("configuration file contains reset option '%s'", strZ(key)); continue; } // Warn if this option should be command-line only if (ruleOption->section == cfgSectionCommandLine) { LOG_WARN_FMT("configuration file contains command-line only option '%s'", strZ(key)); continue; } // Make sure this option does not appear in the same section with an alternate name const Variant *const optionFoundKey = VARUINT64(option.id * CFG_OPTION_KEY_MAX + option.keyIdx); const Variant *const optionFoundName = kvGet(optionFound, optionFoundKey); if (optionFoundName != NULL) { THROW_FMT( OptionInvalidError, "configuration file contains duplicate options ('%s', '%s') in section '[%s]'", strZ(key), strZ(varStr(optionFoundName)), strZ(section)); } else kvPut(optionFound, optionFoundKey, VARSTR(key)); // Continue if the option is not valid for this command if (!cfgParseOptionValid(config->command, config->commandRole, option.id)) { // Warn if it is in a command section if (sectionIdx % 2 == 0) { LOG_WARN_FMT( "configuration file contains option '%s' invalid for section '%s'", strZ(key), strZ(section)); } continue; } // Continue if stanza option is in a global section if (ruleOption->section == cfgSectionStanza && (strEqZ(section, CFGDEF_SECTION_GLOBAL) || strBeginsWithZ(section, CFGDEF_SECTION_GLOBAL ":"))) { LOG_WARN_FMT( "configuration file contains stanza-only option '%s' in global section '%s'", strZ(key), strZ(section)); continue; } // Continue if this option has already been found in another section or command-line/environment ParseOptionValue *const optionValue = parseOptionIdxValue(parseOptionList, option.id, option.keyIdx); if (optionValue->found) continue; optionValue->found = true; optionValue->source = cfgSourceConfig; // Process list if (iniSectionKeyIsList(configParseLocal.ini, section, key)) { // Error if the option cannot be specified multiple times if (!option.multi) { THROW_FMT( OptionInvalidError, "option '%s' cannot be set multiple times", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } optionValue->valueList = iniGetList(configParseLocal.ini, section, key); } else { // Get the option value const String *const value = iniGet(configParseLocal.ini, section, key); if (strSize(value) == 0) { THROW_FMT( OptionInvalidValueError, "section '%s', key '%s' must have a value", strZ(section), strZ(key)); } if (cfgParseOptionType(option.id) == cfgOptTypeBoolean) { if (strEqZ(value, "n")) optionValue->negate = true; else if (!strEqZ(value, "y")) THROW_FMT(OptionInvalidValueError, "boolean option '%s' must be 'y' or 'n'", strZ(key)); } // Else add the string value else { optionValue->valueList = strLstNew(); strLstAdd(optionValue->valueList, value); } } } } } // Phase 4: create the config and resolve indexed options for each group // ------------------------------------------------------------------------------------------------------------------------- // Determine how many indexes are used in each group bool groupIdxMap[CFG_OPTION_GROUP_TOTAL][CFG_OPTION_KEY_MAX] = {{0}}; for (unsigned int optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++) { // Always assign name since it may be needed for error messages const ParseRuleOption *const ruleOption = &parseRuleOption[optionId]; config->option[optionId].name = ruleOption->name; // Is the option valid for this command? if (cfgParseOptionValid(config->command, config->commandRole, optionId)) { config->option[optionId].valid = true; config->option[optionId].sequence = ruleOption->sequence; config->option[optionId].dataType = cfgParseOptionDataType(optionId); config->option[optionId].group = ruleOption->group; config->option[optionId].groupId = ruleOption->groupId; } else { // Error if the invalid option was explicitly set on the command-line if (parseOptionList[optionId].indexList != NULL) { THROW_FMT( OptionInvalidError, "option '%s' not valid for command '%s'", cfgParseOptionName(optionId), cfgParseCommandName(config->command)); } // Continue to the next option continue; } // If the option is in a group if (ruleOption->group) { const unsigned int groupId = ruleOption->groupId; config->optionGroup[groupId].valid = true; // Scan the option values to determine which indexes are in use. Store them in a map that will later be scanned to // create a list of just the used indexes. for (unsigned int optionKeyIdx = 0; optionKeyIdx < parseOptionList[optionId].indexListTotal; optionKeyIdx++) { if (parseOptionList[optionId].indexList[optionKeyIdx].found && !parseOptionList[optionId].indexList[optionKeyIdx].reset) { if (!groupIdxMap[groupId][optionKeyIdx]) { config->optionGroup[groupId].indexTotal++; groupIdxMap[groupId][optionKeyIdx] = true; } } } } } // Write the indexes into the group in order for (unsigned int groupId = 0; groupId < CFG_OPTION_GROUP_TOTAL; groupId++) { // Set group name config->optionGroup[groupId].name = parseRuleOptionGroup[groupId].name; // Skip the group if it is not valid if (!config->optionGroup[groupId].valid) continue; // Allocate memory for the index to key index map. The pg group always reserves index 0 for key 1 (see below) even when // key 1 was not configured, so allocate an extra slot in that case to avoid writing past the end of the map. unsigned int optionIdxTotal = config->optionGroup[groupId].indexTotal; if (groupId == cfgOptGrpPg && optionIdxTotal > 0 && !groupIdxMap[groupId][0]) optionIdxTotal++; MEM_CONTEXT_BEGIN(config->memContext) { config->optionGroup[groupId].indexMap = memNew(sizeof(unsigned int) * (optionIdxTotal == 0 ? 1 : optionIdxTotal)); } MEM_CONTEXT_END(); // If no values were found in any index then use index 0 since all valid groups must have at least one index. This may // lead to an error unless all options in the group have defaults but that will be resolved later. if (optionIdxTotal == 0) { config->optionGroup[groupId].indexTotal = 1; config->optionGroup[groupId].indexMap[0] = 0; } // Else write the key to index map for the group. This allows translation from keys to indexes and vice versa. else { unsigned int optionIdxMax = 0; unsigned int optionKeyIdx = 0; // ??? For the pg group, key 1 is required to maintain compatibility with older versions. Before removing this // constraint the pg group remap to key 1 for remotes will need to be dealt with in the protocol/helper module. if (groupId == cfgOptGrpPg) { optionKeyIdx = 1; optionIdxMax = 1; config->optionGroup[groupId].indexMap[0] = 0; } // Write keys into the index map for (; optionKeyIdx < CFG_OPTION_KEY_MAX; optionKeyIdx++) { if (groupIdxMap[groupId][optionKeyIdx]) { config->optionGroup[groupId].indexMap[optionIdxMax] = optionKeyIdx; optionIdxMax++; } } // Update the index total to include the reserved pg index 0 when key 1 was not configured config->optionGroup[groupId].indexTotal = optionIdxMax; } } // Phase 5: validate option definitions and load into configuration // ------------------------------------------------------------------------------------------------------------------------- // Determine whether a group index will be kept based on non-default values bool optionGroupIndexKeep[CFG_OPTION_GROUP_TOTAL][CFG_OPTION_KEY_MAX] = {{false}}; for (unsigned int optionOrderIdx = 0; optionOrderIdx < CFG_OPTION_TOTAL; optionOrderIdx++) { // Validate options based on the option resolve order. This allows resolving all options in a single pass. const ConfigOption optionId = optionResolveOrder[optionOrderIdx]; // Skip this option if it is not valid if (!config->option[optionId].valid) continue; // Determine the option index total. For options that are not indexed the index total is 1. const ParseRuleOption *const ruleOption = &parseRuleOption[optionId]; const bool optionGroup = ruleOption->group; const unsigned int optionGroupId = optionGroup ? ruleOption->groupId : UINT_MAX; const unsigned int optionListIndexTotal = optionGroup ? config->optionGroup[optionGroupId].indexTotal : 1; MEM_CONTEXT_BEGIN(config->memContext) { config->option[optionId].index = memNew(sizeof(ConfigOptionValue) * optionListIndexTotal); } MEM_CONTEXT_END(); // Loop through the option indexes const ConfigOptionType optionType = cfgParseOptionType(optionId); for (unsigned int optionListIdx = 0; optionListIdx < optionListIndexTotal; optionListIdx++) { // Get the key index by looking it up in the group or by defaulting to 0 for ungrouped options const unsigned optionKeyIdx = optionGroup ? config->optionGroup[optionGroupId].indexMap[optionListIdx] : 0; // Get the parsed value using the key index. Provide a default structure when the value was not found. const ParseOptionValue *const parseOptionValue = optionKeyIdx < parseOptionList[optionId].indexListTotal ? &parseOptionList[optionId].indexList[optionKeyIdx] : &(ParseOptionValue){0}; // Get the location where the value will be stored in the configuration ConfigOptionValue *const configOptionValue = &config->option[optionId].index[optionListIdx]; // Is the value set for this option? const bool optionSet = parseOptionValue->found && (optionType == cfgOptTypeBoolean || !parseOptionValue->negate) && !parseOptionValue->reset; // Initialize option value and set negate and reset flag *configOptionValue = (ConfigOptionValue){.negate = parseOptionValue->negate, .reset = parseOptionValue->reset}; // Is the option valid? CfgParseOptionalRuleState optionalRules = {.defaultDynamicBin = config->bin}; CfgParseOptionalFilterDependResult dependResult = {.valid = true}; const char *dependOptionName = NULL; if (cfgParseOptionalRule(&optionalRules, parseRuleOptionalTypeValid, config->command, optionId)) { PackRead *const filter = pckReadNewC(optionalRules.valid, optionalRules.validSize); dependResult = cfgParseOptionalFilterDepend(filter, optionType, config, optionListIdx); dependOptionName = cfgParseOptionKeyIdxName( dependResult.dependId, parseRuleOption[dependResult.dependId].group ? optionKeyIdx : 0); // If depend not resolved and option value is set on the command-line then error. It is OK to have unresolved // options in the config file because they may be there for another command. For instance, spool-path is only // loaded for the archive-push command when archive-async=y, and the presence of spool-path in the config file // should not cause an error here, it will just end up null. if (!dependResult.valid && optionSet && parseOptionValue->source == cfgSourceParam) { PackRead *const filter = pckReadNewC(optionalRules.valid, optionalRules.validSize); // If there is a boolean default value just consume it since it is not needed here pckReadNext(filter); if (pckReadType(filter) == pckTypeBool) { pckReadBoolP(filter); pckReadConsume(filter); } // Get depend option id and name const ConfigOption dependId = pckReadU32P(filter); // If depend value is not set ASSERT(config->option[dependId].index != NULL); if (!config->option[dependId].index[optionListIdx].set) { THROW_FMT( OptionInvalidError, "option '%s' not valid without option '%s'", cfgParseOptionKeyIdxName(optionId, optionKeyIdx), dependOptionName); } // Build type dependent error data const String *errorValue = EMPTY_STR; switch (parseRuleOption[dependId].type) { case cfgOptTypeBoolean: { if (!pckReadBoolP(filter)) dependOptionName = zNewFmt("no-%s", dependOptionName); break; } default: { String *const errorList = strNew(); unsigned int validSize = 0; while (pckReadNext(filter)) { strCatFmt( errorList, "%s'%s'", validSize != 0 ? ", " : "", strZ(cfgParseOptionValueStr(parseRuleOption[dependId].type, pckReadU32P(filter)))); validSize++; } ASSERT(validSize > 0); if (validSize == 1) errorValue = strNewFmt(" = %s", strZ(errorList)); else errorValue = strNewFmt(" in (%s)", strZ(errorList)); } } THROW_FMT( OptionInvalidError, "option '%s' not valid without option '%s'%s", cfgParseOptionKeyIdxName(optionId, optionKeyIdx), dependOptionName, strZ(errorValue)); } // Store the match value in case we need it to find the default, range, etc. optionalRules.matchValue = dependResult.matchValue; pckReadFree(filter); } // Does the command override sequence? if (cfgParseOptionalRule(&optionalRules, parseRuleOptionalTypeSequence, config->command, optionId)) config->option[optionId].sequence = optionalRules.sequence; else optionalRules.sequence = config->option[optionId].sequence; if (dependResult.valid) { // Is the option set? if (optionSet) { configOptionValue->set = true; configOptionValue->source = parseOptionValue->source; // Check beta status parseOptionBeta(optionId, optionKeyIdx, ruleOption->beta, &parseOptionList[cfgOptBeta]); if (optionType == cfgOptTypeBoolean) { configOptionValue->value.boolean = !parseOptionValue->negate; } else if (optionType == cfgOptTypeHash) { KeyValue *value = NULL; MEM_CONTEXT_BEGIN(config->memContext) { value = kvNew(); } MEM_CONTEXT_END(); for (unsigned int listIdx = 0; listIdx < strLstSize(parseOptionValue->valueList); listIdx++) { const char *const pair = strZ(strLstGet(parseOptionValue->valueList, listIdx)); const char *const equal = strchr(pair, '='); if (equal == NULL) { THROW_FMT( OptionInvalidError, "key/value '%s' not valid for '%s' option", strZ(strLstGet(parseOptionValue->valueList, listIdx)), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } // Warn if a value will be overwritten const Variant *const key = VARSTR(strNewZN(pair, (size_t)(equal - pair))); const String *const old = varStr(kvGet(value, key)); const Variant *const new = VARSTRZ(equal + 1); if (old != NULL) { LOG_WARN_FMT( "key '%s' value '%s' is overwritten with '%s' for '%s' option", strZ(varStr(key)), strZ(old), strZ(varStr(new)), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } // Put the value (overwriting if the key already exists) kvPut(value, key, new); } configOptionValue->value.keyValue = value; } else if (optionType == cfgOptTypeList) { MEM_CONTEXT_BEGIN(config->memContext) { configOptionValue->value.list = varLstNewStrLst(parseOptionValue->valueList); } MEM_CONTEXT_END(); } else { String *const value = strLstGet(parseOptionValue->valueList, 0); const String *const valueAllow = value; // Preserve original value to display MEM_CONTEXT_BEGIN(config->memContext) { configOptionValue->display = strDup(value); } MEM_CONTEXT_END(); // If a numeric type check that the value is valid if (optionType == cfgOptTypeInteger || optionType == cfgOptTypeSize || optionType == cfgOptTypeTime) { // Check that the value can be converted TRY_BEGIN() { switch (optionType) { case cfgOptTypeInteger: configOptionValue->value.integer = cvtZToInt64(strZ(value)); break; case cfgOptTypeSize: configOptionValue->value.integer = cfgParseSize(value); break; default: { ASSERT(optionType == cfgOptTypeTime); configOptionValue->value.integer = cfgParseTime(value); break; } } } CATCH_ANY() { THROW_FMT( OptionInvalidValueError, "'%s' is not valid for '%s' option", strZ(value), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } TRY_END(); if (cfgParseOptionalRule( &optionalRules, parseRuleOptionalTypeAllowRange, config->command, optionId) && (configOptionValue->value.integer < optionalRules.allowRangeMin || configOptionValue->value.integer > optionalRules.allowRangeMax)) { THROW_FMT( OptionInvalidValueError, "'%s' is out of range for '%s' option%s\n" "HINT: allowed range is %s to %s inclusive", strZ(value), cfgParseOptionKeyIdxName(optionId, optionKeyIdx), optionalRules.allowRangeMatch ? zNewFmt( " when '%s' option = '%s'", dependOptionName, strZ(PARSE_RULE_VAL_STR_IDX(optionalRules.matchValue))) : "", strZ(cfgParseOptionValueStr(optionType, optionalRules.allowRangeMinIdx)), strZ(cfgParseOptionValueStr(optionType, optionalRules.allowRangeMaxIdx))); } } // Else if string make sure it is valid else if (optionType != cfgOptTypeStringId) { ASSERT(optionType == cfgOptTypePath || optionType == cfgOptTypeString); // Set string value to display value configOptionValue->value.string = configOptionValue->display; // Empty strings are not valid if (strSize(value) == 0) { THROW_FMT( OptionInvalidValueError, "'%s' must be >= 1 character for '%s' option", strZ(value), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } // If path make sure it is valid if (optionType == cfgOptTypePath) { // Make sure it starts with / if (!strBeginsWithZ(value, "/")) { THROW_FMT( OptionInvalidValueError, "'%s' must begin with / for '%s' option", strZ(value), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } // Make sure there are no occurrences of // if (strstr(strZ(value), "//") != NULL) { THROW_FMT( OptionInvalidValueError, "'%s' cannot contain // for '%s' option", strZ(value), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } // If the path ends with a / we'll strip it off (unless the value is just /) if (strEndsWithZ(value, "/") && strSize(value) != 1) { strTruncIdx(value, (int)strSize(value) - 1); // Reset string value since it was modified MEM_CONTEXT_BEGIN(config->memContext) { configOptionValue->value.string = strDup(value); } MEM_CONTEXT_END(); } } } // If the option has an allow list then check it if (cfgParseOptionalRule(&optionalRules, parseRuleOptionalTypeAllowList, config->command, optionId)) { PackRead *const allowList = pckReadNewC(optionalRules.allowList, optionalRules.allowListSize); bool allowListFound = false; unsigned int sequence = 0; pckReadNext(allowList); while (true) { // Compare based on option type const unsigned int valueIdx = pckReadU32P(allowList); switch (ruleOption->type) { case cfgOptTypeStringId: { if (strEq(PARSE_RULE_VAL_STR_IDX(valueIdx), configOptionValue->display)) { allowListFound = true; if (optionalRules.sequence) configOptionValue->sequence = (uint8_t)sequence; } sequence++; break; } default: allowListFound = cfgParseOptionValue(ruleOption->type, valueIdx) == configOptionValue->value.integer; break; } // Stop when allow list is exhausted or value is found if (!cfgParseOptionValueCondition( pckReadNext(allowList), allowList, allowListFound, optionId, optionKeyIdx, valueAllow) || allowListFound) { break; } } pckReadFree(allowList); if (!allowListFound) { PackRead *const allowList = pckReadNewC(optionalRules.allowList, optionalRules.allowListSize); String *const hintList = strNew(); while (!pckReadNullP(allowList)) { if (!strEmpty(hintList)) strCatZ(hintList, ", "); strCatFmt( hintList, "'%s'", strZ(cfgParseOptionValueStr(ruleOption->type, pckReadU32P(allowList)))); } THROW_FMT( OptionInvalidValueError, "'%s' is not allowed for '%s' option\n" "HINT: allowed values are %s", strZ(valueAllow), cfgParseOptionKeyIdxName(optionId, optionKeyIdx), strZ(hintList)); } } } } // Else set source when negated (value is already false) else if (parseOptionValue->negate) { configOptionValue->source = parseOptionValue->source; } if ((!configOptionValue->set && !parseOptionValue->negate) || config->help) { // If the option has a default if (cfgParseOptionalRule(&optionalRules, parseRuleOptionalTypeDefault, config->command, optionId)) { if (!configOptionValue->set) { configOptionValue->set = true; configOptionValue->value = optionalRules.defaultValue; configOptionValue->sequence = optionalRules.defaultSequence; configOptionValue->display = optionalRules.defaultRaw; } configOptionValue->defaultValue = optionalRules.defaultRaw; } // Else error if option is required and help was not requested else if (!config->help) { const bool required = cfgParseOptionalRule(&optionalRules, parseRuleOptionalTypeRequired, config->command, optionId) ? optionalRules.required : ruleOption->required; if (required) { THROW_FMT( OptionRequiredError, "%s command requires option: %s%s", cfgParseCommandName(config->command), cfgParseOptionKeyIdxName(optionId, optionKeyIdx), ruleOption->section == cfgSectionStanza ? "\nHINT: does this stanza exist?" : ""); } } } // If a non-default group option, keep the group index if (optionGroup && configOptionValue->source != cfgSourceDefault) optionGroupIndexKeep[optionGroupId][optionListIdx] = true; } // Else apply the default for the unresolved dependency, if it exists else if (dependResult.defaultExists) { // Fully reinitialize since it might have been left partially set if dependency was not resolved *configOptionValue = (ConfigOptionValue) { .set = true, .value = dependResult.defaultValue, .defaultValue = optionalRules.defaultRaw, .display = optionalRules.defaultRaw, }; } pckReadFree(optionalRules.pack); } } // Phase 6: Remove any group indexes that have all default values (unless there is only one) // // It is possible that a group index was created for an option that was later found to not meet dependencies. In this case // all values will be default leading to a phantom group, which can be quite confusing. Remove all group indexes that are // all default (except the final one) and make sure the key for the final all default group index is 1. // ------------------------------------------------------------------------------------------------------------------------- for (unsigned int optionGroupIdx = 0; optionGroupIdx < CFG_OPTION_GROUP_TOTAL; optionGroupIdx++) { ConfigOptionGroupData *const optionGroup = &config->optionGroup[optionGroupIdx]; // Iterate group indexes for (unsigned int keyIdx = optionGroup->indexTotal - 1; keyIdx + 1 > 0; keyIdx--) { // Break if there is only one index since each group must have at least one if (optionGroup->indexTotal == 1) break; // If the group index does not have a non-default value if (!optionGroupIndexKeep[optionGroupIdx][keyIdx]) { // Remove the value if it is not last if (keyIdx < optionGroup->indexTotal - 1) { // Remove index key memmove( optionGroup->indexMap + keyIdx, optionGroup->indexMap + (keyIdx + 1), sizeof(unsigned int) * (optionGroup->indexTotal - keyIdx - 1)); // Iterate all options for (unsigned int optionIdx = 0; optionIdx < CFG_OPTION_TOTAL; optionIdx++) { ConfigOptionData *const option = &config->option[optionIdx]; // Remove the value if in the correct group if (option->group && option->groupId == optionGroupIdx) { memmove( option->index + keyIdx, option->index + (keyIdx + 1), sizeof(ConfigOptionValue) * (optionGroup->indexTotal - keyIdx - 1)); } } } // Decrement index total optionGroup->indexTotal--; } } // If the remaining index contains all default values and is not key 1 then make it key 1. This prevents the key from // being determined by the key of an unused option. if (optionGroup->indexTotal == 1 && !optionGroupIndexKeep[optionGroupIdx][0] && optionGroup->indexMap[0] != 0) optionGroup->indexMap[0] = 0; } // Initialize config cfgInit(config); // Set option group default index. The first index in the group is automatically set unless the group option, e.g. pg, is // set. For now the group default options are hard-coded but they could be dynamic. An assert has been added to make sure // the code breaks if a new group is added. for (unsigned int groupId = 0; groupId < CFG_OPTION_GROUP_TOTAL; groupId++) { ASSERT(groupId == cfgOptGrpPg || groupId == cfgOptGrpRepo); // Get the group default option const unsigned int defaultOptionId = groupId == cfgOptGrpPg ? cfgOptPg : cfgOptRepo; // Does a default always exist? config->optionGroup[groupId].indexDefaultExists = // A default always exists for the pg group groupId == cfgOptGrpPg || // The repo group allows a default when the repo option is valid, i.e. either repo1 is the only key set or a repo // is specified cfgOptionValid(cfgOptRepo); // Does the group default option exist? if (cfgOptionTest(defaultOptionId)) { // Search for the key const unsigned int optionKeyIdx = cfgOptionUInt(defaultOptionId) - 1; unsigned int index = 0; for (; index < cfgOptionGroupIdxTotal(groupId); index++) { if (config->optionGroup[groupId].indexMap[index] == optionKeyIdx) break; } // Error if the key was not found if (index == cfgOptionGroupIdxTotal(groupId)) { THROW_FMT( OptionInvalidValueError, "key '%s' is not valid for '%s' option", strZ(cfgOptionDisplay(defaultOptionId)), cfgOptionName(defaultOptionId)); } // Set the default config->optionGroup[groupId].indexDefault = index; config->optionGroup[groupId].indexDefaultExists = true; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/parse.h0000664000175100017510000001705615265101745016732 0ustar00runnerrunner/*********************************************************************************************************************************** Parse Configuration ***********************************************************************************************************************************/ #ifndef CONFIG_PARSE_H #define CONFIG_PARSE_H #include "common/ini.h" #include "config/config.h" #include "storage/storage.h" /*********************************************************************************************************************************** Define global section name ***********************************************************************************************************************************/ #define CFGDEF_SECTION_GLOBAL "global" /*********************************************************************************************************************************** Prefix for environment variables ***********************************************************************************************************************************/ #define PGBACKREST_ENV "PGBACKREST_" #define PGBACKREST_ENV_SIZE (sizeof(PGBACKREST_ENV) - 1) /*********************************************************************************************************************************** In some environments this will not be extern'd ***********************************************************************************************************************************/ extern char **environ; /*********************************************************************************************************************************** Option type enum ***********************************************************************************************************************************/ typedef enum { cfgOptTypeInteger, // Signed 64-bit integer cfgOptTypeSize, // Size, e.g. 1m, 2gb cfgOptTypeTime, // Time in seconds, e.g. 23, 1.5 cfgOptTypeStringId, // StringId cfgOptTypeString, // String cfgOptTypePath, // Path string with validation cfgOptTypeBoolean, // Boolean cfgOptTypeHash, // Associative array, e.g. key1=val1,key2=val2 cfgOptTypeList, // String list, e.g. val1,val2 } ConfigOptionType; /*********************************************************************************************************************************** Underlying data type for an option ***********************************************************************************************************************************/ typedef enum { cfgOptDataTypeBoolean, // Boolean cfgOptDataTypeHash, // Hash cfgOptDataTypeInteger, // Signed 64-bit integer cfgOptDataTypeList, // List cfgOptDataTypeString, // String cfgOptDataTypeStringId, // StringId } ConfigOptionDataType; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Parse the command-line arguments and config file to produce final config data typedef struct CfgParseParam { VAR_PARAM_HEADER; bool noResetLogLevel; // Do not reset log level bool noConfigLoad; // Do not reload the config file const String *stanza; // Load config as stanza } CfgParseParam; #define cfgParseP(storage, argListSize, argList, ...) \ cfgParse(storage, argListSize, argList, (CfgParseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void cfgParse(const Storage *storage, unsigned int argListSize, const char *argList[], CfgParseParam param); // Get command name by id FN_EXTERN const char *cfgParseCommandName(ConfigCommand commandId); // Get command/role name with custom separator FN_EXTERN String *cfgParseCommandRoleName(const ConfigCommand commandId, const ConfigCommandRole commandRoleId); // Convert command role enum to String FN_EXTERN const String *cfgParseCommandRoleStr(ConfigCommandRole commandRole); // Return the parsed ini config FN_EXTERN const Ini *cfgParseIni(void); // Parse option name and return option info typedef struct CfgParseOptionParam { VAR_PARAM_HEADER; bool prefixMatch; // Allow prefix matches, e.g. 'stanz' for 'stanza' bool ignoreMissingIndex; // Help requires the base option name to be searchable } CfgParseOptionParam; typedef struct CfgParseOptionResult { bool found; // Was the option found? ConfigOption id; // Option ID unsigned int keyIdx; // Option key index, i.e. option key - 1 bool negate; // Was the option negated? bool reset; // Was the option reset? bool deprecated; // Is the option deprecated? bool beta; // Is the option in beta? bool multi; // Can the option be specified multiple times? } CfgParseOptionResult; #define cfgParseOptionP(optionName, ...) \ cfgParseOption(optionName, (CfgParseOptionParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN CfgParseOptionResult cfgParseOption(const String *const optionName, const CfgParseOptionParam param); // Option name from id FN_EXTERN const char *cfgParseOptionName(ConfigOption optionId); // Option name from id and key FN_EXTERN const char *cfgParseOptionKeyIdxName(ConfigOption optionId, unsigned int keyIdx); // Is the option in a group? Unlike cfgOptionGroup() this works for an option that is not valid for the current command FN_EXTERN bool cfgParseOptionGroup(ConfigOption optionId); // Option group id FN_EXTERN unsigned int cfgParseOptionGroupId(ConfigOption optionId); // Does the option need to be protected from showing up in logs, command lines, etc? FN_EXTERN bool cfgParseOptionSecure(ConfigOption optionId); // Option data type FN_EXTERN ConfigOptionType cfgParseOptionType(ConfigOption optionId); // Get the underlying data type for an option FN_EXTERN ConfigOptionDataType cfgParseOptionDataType(ConfigOption optionId); // Get list of stanzas in the configuration FN_EXTERN StringList *cfgParseStanzaList(void); // Is the option valid for the command? FN_EXTERN bool cfgParseOptionValid(ConfigCommand commandId, ConfigCommandRole commandRoleId, ConfigOption optionId); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/protocol.c0000664000175100017510000000511615265101745017446 0ustar00runnerrunner/*********************************************************************************************************************************** Configuration Protocol Handler ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "common/type/json.h" #include "config/config.intern.h" #include "config/parse.h" #include "config/protocol.h" /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * configOptionProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { VariantList *const optionList = varLstNew(); while (pckReadNext(param)) { CfgParseOptionResult option = cfgParseOptionP(pckReadStrP(param)); CHECK(AssertError, option.found, "option not found"); varLstAdd(optionList, varDup(cfgOptionIdxVar(option.id, cfgOptionKeyToIdx(option.id, option.keyIdx + 1)))); } pckWriteStrP(protocolServerResultData(result), jsonFromVar(varNewVarLst(optionList))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ FN_EXTERN VariantList * configOptionRemote(ProtocolClient *const client, const VariantList *const paramList) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client); FUNCTION_LOG_PARAM(VARIANT_LIST, paramList); FUNCTION_LOG_END(); VariantList *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const param = protocolPackNew(); for (unsigned int paramIdx = 0; paramIdx < varLstSize(paramList); paramIdx++) pckWriteStrP(param, varStr(varLstGet(paramList, paramIdx))); const VariantList *const list = varVarLst( jsonToVar(pckReadStrP(protocolClientRequestP(client, PROTOCOL_COMMAND_CONFIG_OPTION, .param = param)))); MEM_CONTEXT_PRIOR_BEGIN() { result = varLstDup(list); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(VARIANT_LIST, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/config/protocol.h0000664000175100017510000000314215265101745017450 0ustar00runnerrunner/*********************************************************************************************************************************** Configuration Protocol Handler ***********************************************************************************************************************************/ #ifndef CONFIG_PROTOCOL_H #define CONFIG_PROTOCOL_H #include "common/type/pack.h" #include "protocol/client.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process config protocol requests FN_EXTERN ProtocolServerResult *configOptionProtocol(PackRead *param); // Get option values from a remote process FN_EXTERN VariantList *configOptionRemote(ProtocolClient *client, const VariantList *paramList); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_CONFIG_OPTION STRID5("opt-g", 0x7dd20f0) #define PROTOCOL_SERVER_HANDLER_OPTION_LIST \ {.command = PROTOCOL_COMMAND_CONFIG_OPTION, .process = configOptionProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/db/0000775000175100017510000000000015265101745014556 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/db/db.c0000664000175100017510000007346315265101745015324 0ustar00runnerrunner/*********************************************************************************************************************************** Database Client ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/type/json.h" #include "common/wait.h" #include "config/config.h" #include "config/protocol.h" #include "db/db.h" #include "db/protocol.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "version.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define PG_BACKUP_ADVISORY_LOCK "12340078987004321" #define DB_PING_SEC 30 /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Db { DbPub pub; // Publicly accessible variables PgClient *client; // Local PostgreSQL client ProtocolClient *remoteClient; // Protocol client for remote db queries ProtocolClientSession *session; // Protocol session for requests const Storage *storage; // PostgreSQL storage const String *applicationName; // Used to identify this connection in PostgreSQL time_t pingTimeLast; // Last time cluster was pinged }; /*********************************************************************************************************************************** Close protocol connection. No need to close a locally created PgClient since it has its own destructor. ***********************************************************************************************************************************/ static void dbFreeResource(THIS_VOID) { THIS(Db); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); ASSERT(this != NULL); protocolClientSessionFree(this->session); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Db * dbNew(PgClient *const client, ProtocolClient *const remoteClient, const Storage *const storage, const String *const applicationName) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PG_CLIENT, client); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, remoteClient); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, applicationName); FUNCTION_LOG_END(); ASSERT((client != NULL && remoteClient == NULL) || (client == NULL && remoteClient != NULL)); ASSERT(storage != NULL); ASSERT(applicationName != NULL); OBJ_NEW_BEGIN(Db, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (Db) { .pub = { .memContext = memContextCurrent(), }, .remoteClient = remoteClient, .session = remoteClient != NULL ? protocolClientSessionNewP(remoteClient, PROTOCOL_COMMAND_DB) : NULL, .storage = storage, .applicationName = strDup(applicationName), }; this->client = pgClientMove(client, this->pub.memContext); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(DB, this); } /*********************************************************************************************************************************** Execute a query ***********************************************************************************************************************************/ static Pack * dbQuery(Db *const this, const PgClientQueryResult resultType, const String *const query) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(STRING_ID, resultType); FUNCTION_LOG_PARAM(STRING, query); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(resultType != 0); ASSERT(query != NULL); Pack *result; // Query remotely if (this->remoteClient != NULL) { MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const param = protocolPackNew(); pckWriteStrIdP(param, resultType); pckWriteStrP(param, query); PackRead *const read = protocolClientSessionRequestP(this->session, .param = param); MEM_CONTEXT_PRIOR_BEGIN() { result = pckReadPackP(read); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); } // Else locally else result = pgClientQuery(this->client, query, resultType); FUNCTION_LOG_RETURN(PACK, result); } /*********************************************************************************************************************************** Execute a command that expects no output ***********************************************************************************************************************************/ static void dbExec(Db *const this, const String *const command) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(STRING, command); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(command != NULL); CHECK(AssertError, dbQuery(this, pgClientQueryResultNone, command) == NULL, "exec returned data"); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Execute a query that returns a single row and column ***********************************************************************************************************************************/ static PackRead * dbQueryColumn(Db *const this, const String *const query) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(STRING, query); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(query != NULL); Pack *const pack = dbQuery(this, pgClientQueryResultColumn, query); PackRead *const result = pckReadNew(pack); pckMove(pack, objMemContext(result)); FUNCTION_LOG_RETURN(PACK_READ, result); } /*********************************************************************************************************************************** Execute a query that returns a single row ***********************************************************************************************************************************/ static PackRead * dbQueryRow(Db *const this, const String *const query) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(STRING, query); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(query != NULL); Pack *const pack = dbQuery(this, pgClientQueryResultRow, query); PackRead *const result = pckReadNew(pack); pckMove(pack, objMemContext(result)); FUNCTION_LOG_RETURN(PACK_READ, result); } /*********************************************************************************************************************************** Is the cluster in recovery? ***********************************************************************************************************************************/ static bool dbIsInRecovery(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result; MEM_CONTEXT_TEMP_BEGIN() { result = pckReadBoolP(dbQueryColumn(this, STRDEF("select pg_catalog.pg_is_in_recovery()"))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN void dbOpen(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Open the connection if (this->remoteClient != NULL) { protocolClientSessionOpenP(this->session); // Set a callback to notify the remote when a connection is closed memContextCallbackSet(this->pub.memContext, dbFreeResource, this); // Get db-timeout from the remote since it might be different than the local value this->pub.dbTimeout = varUInt64Force( varLstGet(configOptionRemote(this->remoteClient, varLstAdd(varLstNew(), varNewStrZ(CFGOPT_DB_TIMEOUT))), 0)); } else { pgClientOpen(this->client); this->pub.dbTimeout = cfgOptionUInt64(cfgOptDbTimeout); } // Set search_path to prevent overrides of the functions we expect to call. All queries should also be schema-qualified, // but this is an extra level protection. dbExec(this, STRDEF("set search_path = 'pg_catalog'")); // Set client encoding to UTF8. This is the only encoding (other than ASCII) that we can safely work with. dbExec(this, STRDEF("set client_encoding = 'UTF8'")); // Query the version and data_directory. Be sure the update the total in the null check below when adding/removing columns. const Pack *const row = dbQuery( this, pgClientQueryResultRow, STRDEF( "select (select setting from pg_catalog.pg_settings where name = 'server_version_num')::int4," " (select setting from pg_catalog.pg_settings where name = 'data_directory')::text," " (select setting from pg_catalog.pg_settings where name = 'archive_mode')::text," " (select setting from pg_catalog.pg_settings where name = 'archive_command')::text," " (select setting from pg_catalog.pg_settings where name = 'checkpoint_timeout')::int4")); // Check that none of the return values are null, which indicates the user cannot select some rows in pg_settings PackRead *read = pckReadNew(row); for (unsigned int columnIdx = 0; columnIdx < 5; columnIdx++) { if (pckReadNullP(read, .id = columnIdx + 1)) { THROW( DbQueryError, "unable to select some rows from pg_settings\n" "HINT: is the backup running as the postgres user?\n" "HINT: is the pg_read_all_settings role assigned for " PG_NAME " >= " PG_VERSION_10_Z "?"); } } // Restart the read to get the data read = pckReadNew(row); // Strip the minor version off since we don't need it. In the future it might be a good idea to warn users when they are // running an old minor version. this->pub.pgVersion = (unsigned int)pckReadI32P(read) / 100 * 100; // Store the data directory that PostgreSQL is running in, the archive mode, and archive command. These can be compared to // the configured pgBackRest directory, and archive settings checked for validity, when validating the configuration. // Also store the checkpoint timeout to warn in case a backup is requested without using the start-fast option. MEM_CONTEXT_BEGIN(this->pub.memContext) { this->pub.pgDataPath = pckReadStrP(read); this->pub.archiveMode = pckReadStrP(read); this->pub.archiveCommand = pckReadStrP(read); this->pub.checkpointTimeout = (TimeMSec)pckReadI32P(read) * MSEC_PER_SEC; } MEM_CONTEXT_END(); // Set application name to help identify the backup session dbExec(this, strNewFmt("set application_name = '%s'", strZ(this->applicationName))); // There is no need to have parallelism enabled in a backup session. In particular, 9.6 marks pg_stop_backup() as // parallel-safe but an error will be thrown if pg_stop_backup() is run in a worker. dbExec(this, STRDEF("set max_parallel_workers_per_gather = 0")); // Is the cluster a standby? this->pub.standby = dbIsInRecovery(this); // Get control file this->pub.pgControl = pgControlFromFile(this->storage, cfgOptionStrNull(cfgOptPgVersionForce)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ // Helper to build start backup query static String * dbBackupStartQuery(const unsigned int pgVersion, const bool startFast) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_PARAM(BOOL, startFast); FUNCTION_TEST_END(); // Build query to return start lsn and WAL segment name String *result = strCatFmt( strNew(), "select lsn::text as lsn,\n" " pg_catalog.pg_%sfile_name(lsn)::text as wal_segment_name\n" " from pg_catalog.pg_%s('" PROJECT_NAME " backup started at ' || current_timestamp::text", strZ(pgWalName(pgVersion)), pgVersion >= PG_VERSION_15 ? "backup_start" : "start_backup"); // Start backup after immediate checkpoint if (startFast) { strCatZ(result, ", " TRUE_Z); } // Else start backup at the next scheduled checkpoint else strCatZ(result, ", " FALSE_Z); // Use non-exclusive backup mode when available if (pgVersion <= PG_VERSION_14) strCatZ(result, ", " FALSE_Z); // Complete query strCatZ(result, ") as lsn"); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN DbBackupStartResult dbBackupStart(Db *const this, const bool startFast, const bool archiveCheck) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(BOOL, startFast); FUNCTION_LOG_PARAM(BOOL, archiveCheck); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); DbBackupStartResult result = {.lsn = NULL}; MEM_CONTEXT_TEMP_BEGIN() { // When the start-fast option is disabled and db-timeout is smaller than checkpoint_timeout, the command may timeout // before the backup actually starts if (!startFast && dbDbTimeout(this) <= dbCheckpointTimeout(this)) { LOG_WARN_FMT( CFGOPT_START_FAST " is disabled and " CFGOPT_DB_TIMEOUT " (%" PRIu64 "s) is smaller than the " PG_NAME " checkpoint_timeout (%" PRIu64 "s) - timeout may occur before the backup starts", dbDbTimeout(this) / MSEC_PER_SEC, dbCheckpointTimeout(this) / MSEC_PER_SEC); } // If archive check then get the current WAL segment const String *walSegmentCheck = NULL; if (archiveCheck) { walSegmentCheck = pckReadStrP( dbQueryColumn( this, strNewFmt( "select pg_catalog.pg_%sfile_name(pg_catalog.pg_current_%s_insert_%s())::text", strZ(pgWalName(dbPgVersion(this))), strZ(pgWalName(dbPgVersion(this))), strZ(pgLsnName(dbPgVersion(this)))))); } // Start backup PackRead *const read = dbQueryRow(this, dbBackupStartQuery(dbPgVersion(this), startFast)); // Make sure the backup start checkpoint was written to pg_control. This helps ensure that we have a consistent view of the // storage with PostgreSQL. const PgControl pgControl = pgControlFromFile(this->storage, cfgOptionStrNull(cfgOptPgVersionForce)); const String *const lsnStart = pckReadStrP(read); if (pgControl.checkpoint < pgLsnFromStr(lsnStart)) { THROW_FMT( DbMismatchError, "current checkpoint '%s' is less than backup start '%s'", strZ(pgLsnToStr(pgControl.checkpoint)), strZ(lsnStart)); } // If archive check then make sure WAL segment was switched on start backup. The WAL segment might not have been switched if // a switch occurred before the backup and nothing was written since then. In this case, force a switch so we can verify // that archiving is functional before starting the backup. const String *const walSegmentName = pckReadStrP(read); if (archiveCheck && strEq(walSegmentCheck, walSegmentName)) dbWalSwitch(this); // Check that the WAL timeline matches what is in pg_control if (pgTimelineFromWalSegment(walSegmentName) != dbPgControl(this).timeline) { THROW_FMT( DbMismatchError, "WAL timeline %u does not match " PG_FILE_PGCONTROL " timeline %u", pgTimelineFromWalSegment(walSegmentName), dbPgControl(this).timeline); } // Return results MEM_CONTEXT_PRIOR_BEGIN() { result.lsn = strDup(lsnStart); result.walSegmentName = strDup(walSegmentName); result.walSegmentCheck = strDup(walSegmentCheck); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ // Helper to build stop backup query static String * dbBackupStopQuery(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); // Build query to return start lsn and WAL segment name String *const result = strCatFmt( strNew(), "select lsn::text as lsn,\n" " pg_catalog.pg_%sfile_name(lsn)::text as wal_segment_name,\n" " labelfile::text as backuplabel_file,\n" " spcmapfile::text as tablespacemap_file\n" " from pg_catalog.pg_%s(", strZ(pgWalName(pgVersion)), pgVersion >= PG_VERSION_15 ? "backup_stop" : "stop_backup"); // Use non-exclusive backup mode when available if (pgVersion <= PG_VERSION_14) strCatZ(result, FALSE_Z); // Disable archive checking since we do this elsewhere if (pgVersion >= PG_VERSION_10) { if (pgVersion <= PG_VERSION_14) strCatZ(result, ", "); strCatZ(result, FALSE_Z); } // Complete query strCatZ(result, ")"); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN DbBackupStopResult dbBackupStop(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); DbBackupStopResult result = {.lsn = NULL}; MEM_CONTEXT_TEMP_BEGIN() { // Stop backup PackRead *const read = dbQueryRow(this, dbBackupStopQuery(dbPgVersion(this))); // Return results MEM_CONTEXT_PRIOR_BEGIN() { result.lsn = pckReadStrP(read); result.walSegmentName = pckReadStrP(read); result.backupLabel = pckReadStrP(read); // Return the tablespace map if it is not empty String *const tablespaceMap = pckReadStrP(read); String *const tablespaceMapTrim = strTrim(strDup(tablespaceMap)); if (!strEmpty(tablespaceMapTrim)) result.tablespaceMap = tablespaceMap; else strFree(tablespaceMap); strFree(tablespaceMapTrim); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN Pack * dbList(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN( PACK, dbQuery( this, pgClientQueryResultAny, STRDEF( "select oid::oid, datname::text, (select oid::oid from pg_catalog.pg_database where datname = 'template0')" " as datlastsysoid from pg_catalog.pg_database"))); } /**********************************************************************************************************************************/ FN_EXTERN void dbReplayWait(Db *const this, const String *const targetLsn, const uint32_t targetTimeline, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(STRING, targetLsn); FUNCTION_LOG_PARAM(UINT, targetTimeline); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(targetLsn != NULL); ASSERT(targetTimeline != 0); ASSERT(timeout > 0); MEM_CONTEXT_TEMP_BEGIN() { // Standby checkpoint before the backup started must be <= the target LSN. If not, it indicates that the standby was ahead // of the primary and cannot be following it. if (dbPgControl(this).checkpoint > pgLsnFromStr(targetLsn)) { THROW_FMT( DbMismatchError, "standby checkpoint '%s' is ahead of target '%s'", strZ(pgLsnToStr(dbPgControl(this).checkpoint)), strZ(targetLsn)); } // Loop until lsn has been reached or timeout Wait *wait = waitNew(timeout); bool targetReached = false; const char *const lsnName = strZ(pgLsnName(dbPgVersion(this))); const String *const replayLsnFunction = strNewFmt( "pg_catalog.pg_last_%s_replay_%s()", strZ(pgWalName(dbPgVersion(this))), lsnName); const String *replayLsn = NULL; do { // Build the query String *const query = strCatFmt( strNew(), "select replayLsn::text,\n" " (replayLsn > '%s')::bool as targetReached", strZ(targetLsn)); if (replayLsn != NULL) { strCatFmt( query, ",\n" " (replayLsn > '%s')::bool as replayProgress", strZ(replayLsn)); } strCatFmt( query, "\n" " from %s as replayLsn", strZ(replayLsnFunction)); // Execute the query and get replayLsn PackRead *const read = dbQueryRow(this, query); replayLsn = pckReadStrP(read); // Error when replayLsn is null which indicates that this is not a standby. This should have been sorted out before we // connected but it's possible that the standby was promoted in the meantime. if (replayLsn == NULL) { THROW_FMT( ArchiveTimeoutError, "unable to query replay lsn on the standby using '%s'\n" "HINT: Is this a standby?", strZ(replayLsnFunction)); } targetReached = pckReadBoolP(read); // If the target has not been reached but progress is being made then reset the timer if (!targetReached && pckReadBoolP(read, .defaultValue = true)) wait = waitNew(timeout); protocolKeepAlive(); } while (!targetReached && waitMore(wait)); // Error if a timeout occurred before the target lsn was reached if (!targetReached) { THROW_FMT( ArchiveTimeoutError, "timeout before standby replayed to %s - only reached %s\n" "HINT: is replication running and current on the standby?\n" "HINT: disable the 'backup-standby' option to backup directly from the primary.", strZ(targetLsn), strZ(replayLsn)); } // Perform a checkpoint dbExec(this, STRDEF("checkpoint")); // Loop until the checkpoint has reached the target lsn (or timeout) wait = waitNew(timeout); targetReached = false; const String *checkpointLsn = NULL; do { // Build the query const String *const query = strNewFmt( "select (checkpoint_%s >= '%s')::bool as targetReached,\n" " checkpoint_%s::text as checkpointLsn\n" " from pg_catalog.pg_control_checkpoint()", lsnName, strZ(targetLsn), lsnName); // Execute the query and get checkpointLsn PackRead *const read = dbQueryRow(this, query); targetReached = pckReadBoolP(read); checkpointLsn = pckReadStrP(read); protocolKeepAlive(); } while (!targetReached && waitMore(wait)); // Error if a timeout occurred before the target lsn was reached if (!targetReached) { THROW_FMT( ArchiveTimeoutError, "timeout before standby checkpoint lsn reached %s - only reached %s", strZ(targetLsn), strZ(checkpointLsn)); } // Reload pg_control in case timeline was updated by the checkpoint this->pub.pgControl = pgControlFromFile(this->storage, cfgOptionStrNull(cfgOptPgVersionForce)); // Check that the timeline matches the primary if (dbPgControl(this).timeline != targetTimeline) THROW_FMT(DbMismatchError, "standby is on timeline %u but expected %u", dbPgControl(this).timeline, targetTimeline); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void dbPing(Db *const this, const bool force) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(BOOL, force); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Ping if forced or interval has elapsed const time_t timeNow = time(NULL); if (force || timeNow - this->pingTimeLast > DB_PING_SEC) { // Make sure recovery state has not changed if (dbIsInRecovery(this) != dbIsStandby(this)) { // If this is the standby then it must have been promoted if (dbIsStandby(this)) { THROW( DbMismatchError, "standby is no longer in recovery\n" "HINT: was the standby promoted during the backup?"); } // Else if a primary then something has gone seriously wrong else THROW(AssertError, "primary has switched to recovery"); } this->pingTimeLast = timeNow; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Pack * dbTablespaceList(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN( PACK, dbQuery(this, pgClientQueryResultAny, STRDEF("select oid::oid, spcname::text from pg_catalog.pg_tablespace"))); } /**********************************************************************************************************************************/ FN_EXTERN TimeMSec dbTimeMSec(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); TimeMSec result; MEM_CONTEXT_TEMP_BEGIN() { result = (TimeMSec)pckReadI64P( dbQueryColumn(this, STRDEF("select (extract(epoch from clock_timestamp()) * 1000)::bigint"))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(TIME_MSEC, result); } /**********************************************************************************************************************************/ FN_EXTERN String * dbWalSwitch(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); ASSERT(this != NULL); String *result; MEM_CONTEXT_TEMP_BEGIN() { // Create a restore point to ensure current WAL will be archived dbQueryColumn(this, STRDEF("select pg_catalog.pg_create_restore_point('" PROJECT_NAME " Archive Check')::text")); // Request a WAL segment switch const char *const walName = strZ(pgWalName(dbPgVersion(this))); const String *const walFileName = pckReadStrP( dbQueryColumn(this, strNewFmt("select pg_catalog.pg_%sfile_name(pg_catalog.pg_switch_%s())::text", walName, walName))); // Copy WAL segment name to the prior context MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(walFileName); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void dbToLog(const Db *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{client: "); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT(this->client, pgClientToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", remoteClient: "); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT( this->remoteClient, protocolClientToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/db/db.h0000664000175100017510000001523115265101745015316 0ustar00runnerrunner/*********************************************************************************************************************************** Database Client Implements the required PostgreSQL queries and commands. Notice that there is no general purpose query function -- all queries are expected to be embedded in this object. ***********************************************************************************************************************************/ #ifndef DB_DB_H #define DB_DB_H #include "common/type/object.h" #include "postgres/client.h" #include "postgres/interface.h" #include "protocol/client.h" #include "storage/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Db Db; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Db *dbNew(PgClient *client, ProtocolClient *remoteClient, const Storage *storage, const String *applicationName); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct DbPub { MemContext *memContext; // Mem context PgControl pgControl; // Control info const String *archiveMode; // The archive_mode reported by the database const String *archiveCommand; // The archive_command reported by the database const String *pgDataPath; // Data directory reported by the database bool standby; // Is the cluster a standby? unsigned int pgVersion; // Version as reported by the database TimeMSec checkpointTimeout; // The checkpoint timeout reported by the database TimeMSec dbTimeout; // Database timeout for statements/queries from PG client } DbPub; // Archive mode loaded from the archive_mode GUC FN_INLINE_ALWAYS const String * dbArchiveMode(const Db *const this) { return THIS_PUB(Db)->archiveMode; } // Archive command loaded from the archive_command GUC FN_INLINE_ALWAYS const String * dbArchiveCommand(const Db *const this) { return THIS_PUB(Db)->archiveCommand; } // Control data FN_INLINE_ALWAYS PgControl dbPgControl(const Db *const this) { return THIS_PUB(Db)->pgControl; } // Data path loaded from the data_directory GUC FN_INLINE_ALWAYS const String * dbPgDataPath(const Db *const this) { return THIS_PUB(Db)->pgDataPath; } // Is the cluster a standby? FN_INLINE_ALWAYS bool dbIsStandby(const Db *const this) { return THIS_PUB(Db)->standby; } // Version loaded from the server_version_num GUC FN_INLINE_ALWAYS unsigned int dbPgVersion(const Db *const this) { return THIS_PUB(Db)->pgVersion; } // Checkpoint timeout loaded from the checkpoint_timeout GUC FN_INLINE_ALWAYS TimeMSec dbCheckpointTimeout(const Db *const this) { return THIS_PUB(Db)->checkpointTimeout; } // Database timeout from main/remote process FN_INLINE_ALWAYS TimeMSec dbDbTimeout(const Db *const this) { return THIS_PUB(Db)->dbTimeout; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Open the db connection FN_EXTERN void dbOpen(Db *this); // Start backup and return starting lsn and wal segment name typedef struct DbBackupStartResult { String *lsn; String *walSegmentName; String *walSegmentCheck; // Segment used to check archiving, may be NULL } DbBackupStartResult; FN_EXTERN DbBackupStartResult dbBackupStart(Db *this, bool startFast, bool archiveCheck); // Stop backup and return starting lsn, wal segment name, backup label, and tablespace map typedef struct DbBackupStopResult { String *lsn; String *walSegmentName; String *backupLabel; String *tablespaceMap; } DbBackupStopResult; FN_EXTERN DbBackupStopResult dbBackupStop(Db *this); // Get list of databases in the cluster: select oid, datname, datlastsysoid from pg_database FN_EXTERN Pack *dbList(Db *this); // Waits for replay on the standby to equal the target LSN FN_EXTERN void dbReplayWait(Db *this, const String *targetLsn, uint32_t targetTimeline, TimeMSec timeout); // Check that the cluster is alive and correctly configured during the backup FN_EXTERN void dbPing(Db *this, bool force); // Epoch time on the PostgreSQL host in ms FN_EXTERN TimeMSec dbTimeMSec(Db *this); // Get list of tablespaces in the cluster: select oid, datname, datlastsysoid from pg_database FN_EXTERN Pack *dbTablespaceList(Db *this); // Switch the WAL segment and return the segment that should have been archived FN_EXTERN String *dbWalSwitch(Db *this); // Move to a new parent mem context FN_INLINE_ALWAYS Db * dbMove(Db *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void dbFree(Db *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void dbToLog(const Db *this, StringStatic *debugLog); #define FUNCTION_LOG_DB_TYPE \ Db * #define FUNCTION_LOG_DB_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, dbToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/db/helper.c0000664000175100017510000001164315265101745016206 0ustar00runnerrunner/*********************************************************************************************************************************** Database Helper ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "config/config.h" #include "db/helper.h" #include "postgres/interface.h" #include "protocol/helper.h" #include "storage/helper.h" #include "version.h" /**********************************************************************************************************************************/ // Helper to get a connection to the specified pg cluster static Db * dbGetIdx(const unsigned int pgIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, pgIdx); FUNCTION_LOG_END(); ASSERT(pgIdx < cfgOptionGroupIdxTotal(cfgOptGrpPg)); Db *result; MEM_CONTEXT_TEMP_BEGIN() { const String *applicationName = strNewFmt(PROJECT_NAME " [%s]", cfgCommandName()); if (pgIsLocal(pgIdx)) { result = dbNew( pgClientNew( cfgOptionIdxStrNull(cfgOptPgSocketPath, pgIdx), cfgOptionIdxUInt(cfgOptPgPort, pgIdx), cfgOptionIdxStr(cfgOptPgDatabase, pgIdx), cfgOptionIdxStrNull(cfgOptPgUser, pgIdx), cfgOptionUInt64(cfgOptDbTimeout)), NULL, storagePgIdx(pgIdx), applicationName); } else result = dbNew(NULL, protocolRemoteGet(protocolStorageTypePg, pgIdx, true), storagePgIdx(pgIdx), applicationName); dbMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(DB, result); } FN_EXTERN DbGetResult dbGet(const bool primaryOnly, const bool primaryRequired, const StringId standbyRequired) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BOOL, primaryOnly); FUNCTION_LOG_PARAM(BOOL, primaryRequired); FUNCTION_LOG_PARAM(STRING_ID, standbyRequired); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(!(primaryOnly && standbyRequired == CFGOPTVAL_BACKUP_STANDBY_Y)); DbGetResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { // Loop through to look for primary and standby (if required) for (unsigned int pgIdx = 0; pgIdx < cfgOptionGroupIdxTotal(cfgOptGrpPg); pgIdx++) { Db *db = NULL; bool standby = false; TRY_BEGIN() { db = dbGetIdx(pgIdx); // This needs to be nested because db can be reset to NULL on an error in the outer try but we need the pointer // to be able to free it. TRY_BEGIN() { dbOpen(db); standby = dbIsStandby(db); } CATCH_ANY() { dbFree(db); RETHROW(); } TRY_END(); } CATCH_ANY() { LOG_WARN_FMT( "unable to check %s: [%s] %s", cfgOptionGroupName(cfgOptGrpPg, pgIdx), errorTypeName(errorType()), errorMessage()); db = NULL; } TRY_END(); // Was the connection successful if (db != NULL) { // Is this cluster a standby if (standby) { // If a standby has not already been found then assign it if (result.standby == NULL && !primaryOnly) { result.standbyIdx = pgIdx; result.standby = db; } // Else close the connection since we don't need it else dbFree(db); } // Else is a primary else { // Error if more than one primary was found if (result.primary != NULL) THROW(DbConnectError, "more than one primary cluster found"); result.primaryIdx = pgIdx; result.primary = db; } } } // Error if no primary was found if (result.primary == NULL && primaryRequired) { THROW( DbConnectError, "unable to find primary cluster - cannot proceed\n" "HINT: are all available clusters in recovery?"); } // Error if no standby was found if (result.standby == NULL && standbyRequired == CFGOPTVAL_BACKUP_STANDBY_Y) THROW(DbConnectError, "unable to find standby cluster - cannot proceed"); dbMove(result.primary, memContextPrior()); dbMove(result.standby, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/db/helper.h0000664000175100017510000000363315265101745016213 0ustar00runnerrunner/*********************************************************************************************************************************** Database Helper Helper functions for getting connections to PostgreSQL. ***********************************************************************************************************************************/ #ifndef DB_HELPER_H #define DB_HELPER_H #include #include "db/db.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Get specified cluster(s) typedef struct DbGetResult { unsigned int primaryIdx; // cfgOptGrpPg index of the primary Db *primary; // Primary db object (NULL if none requested) unsigned int standbyIdx; // cfgOptGrpPg index of the standby Db *standby; // Standby db object (NULL if none requested) } DbGetResult; FN_EXTERN DbGetResult dbGet(bool primaryOnly, bool primaryRequired, StringId standbyRequired); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_DB_GET_RESULT_TYPE \ DbGetResult #define FUNCTION_LOG_DB_GET_RESULT_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "DbGetResult", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/db/protocol.c0000664000175100017510000000452615265101745016572 0ustar00runnerrunner/*********************************************************************************************************************************** Db Protocol Handler ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "common/type/json.h" #include "common/type/list.h" #include "config/config.h" #include "db/protocol.h" #include "postgres/client.h" #include "postgres/interface.h" /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * dbOpenProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); (void)param; FUNCTION_LOG_END(); ASSERT(param == NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { PgClient *const pgClient = pgClientNew( cfgOptionStrNull(cfgOptPgSocketPath), cfgOptionUInt(cfgOptPgPort), cfgOptionStr(cfgOptPgDatabase), cfgOptionStrNull(cfgOptPgUser), cfgOptionUInt64(cfgOptDbTimeout)); pgClientOpen(pgClient); // Set session data protocolServerResultSessionDataSet(result, pgClient); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * dbQueryProtocol(PackRead *const param, void *const pgClient) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_PARAM(PG_CLIENT, pgClient); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(param != NULL); ASSERT(pgClient != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { const PgClientQueryResult resultType = (PgClientQueryResult)pckReadStrIdP(param); const String *const query = pckReadStrP(param); pckWritePackP(protocolServerResultData(result), pgClientQuery(pgClient, query, resultType)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/db/protocol.h0000664000175100017510000000277015265101745016576 0ustar00runnerrunner/*********************************************************************************************************************************** Db Protocol Handler ***********************************************************************************************************************************/ #ifndef DB_PROTOCOL_H #define DB_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process db protocol requests FN_EXTERN ProtocolServerResult *dbOpenProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *dbQueryProtocol(PackRead *param, void *sessionData); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_DB STRID5("db", 0x440) #define PROTOCOL_SERVER_HANDLER_DB_LIST \ {.command = PROTOCOL_COMMAND_DB, .open = dbOpenProtocol, .processSession = dbQueryProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/0000775000175100017510000000000015265101745015124 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/info.c0000664000175100017510000010260115265101745016223 0ustar00runnerrunner/*********************************************************************************************************************************** Info Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/crypto/hash.h" #include "common/debug.h" #include "common/format/cipherBlockFormat.h" #include "common/format/format.h" #include "common/ini.h" #include "common/io/filter/filter.h" #include "common/io/io.h" #include "common/log.h" #include "common/type/convert.h" #include "common/type/json.h" #include "common/type/object.h" #include "info/info.h" #include "storage/helper.h" #include "version.h" /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ struct Info { InfoPub pub; // Publicly accessible variables const CipherSpec *cipherSpecNone; // Returned by infoCipherSpec() when there are no keys }; struct InfoSave { MemContext *memContext; // Mem context IoWrite *write; // Write object IoFilter *checksum; // hash to generate file checksum String *sectionLast; // The last section seen }; /*********************************************************************************************************************************** Macros and buffer constants for checksum generation ***********************************************************************************************************************************/ #define INFO_CHECKSUM_BEGIN(checksum) \ do \ { \ ioFilterProcessIn(checksum, BRACEL_BUF); \ } \ while (0) BUFFER_STRDEF_STATIC(INFO_CHECKSUM_SECTION_END_BUF, "\":{"); #define INFO_CHECKSUM_SECTION(checksum, section) \ do \ { \ ioFilterProcessIn(checksum, QUOTED_BUF); \ ioFilterProcessIn(checksum, BUFSTR(section)); \ ioFilterProcessIn(checksum, INFO_CHECKSUM_SECTION_END_BUF); \ } \ while (0) BUFFER_STRDEF_STATIC(INFO_CHECKSUM_SECTION_NEXT_END_BUF, "},"); #define INFO_CHECKSUM_SECTION_NEXT(checksum) \ do \ { \ ioFilterProcessIn(checksum, INFO_CHECKSUM_SECTION_NEXT_END_BUF); \ } \ while (0) BUFFER_STRDEF_STATIC(INFO_CHECKSUM_KEY_VALUE_END_BUF, ":"); #define INFO_CHECKSUM_KEY_VALUE(checksum, key, value) \ do \ { \ ioFilterProcessIn(checksum, BUFSTR(jsonFromVar(VARSTR(key)))); \ ioFilterProcessIn(checksum, INFO_CHECKSUM_KEY_VALUE_END_BUF); \ ioFilterProcessIn(checksum, BUFSTR(value)); \ } \ while (0) #define INFO_CHECKSUM_KEY_VALUE_NEXT(checksum) \ do \ { \ ioFilterProcessIn(checksum, COMMA_BUF); \ } \ while (0) BUFFER_STRDEF_STATIC(INFO_CHECKSUM_END_BUF, "}}"); #define INFO_CHECKSUM_END(checksum) \ do \ { \ ioFilterProcessIn(checksum, INFO_CHECKSUM_END_BUF); \ } \ while (0) /**********************************************************************************************************************************/ FN_EXTERN Info * infoNew(const unsigned int format, const CipherSpec *const cipherSpecSub) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, format); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpecSub); FUNCTION_LOG_END(); ASSERT(format >= REPOSITORY_FORMAT_MIN && format <= REPOSITORY_FORMAT_MAX); OBJ_NEW_BEGIN(Info, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (Info){}; // Cipher used to encrypt/decrypt subsequent dependent files. Value may be NULL. infoCipherSpecSet(this, cipherSpecSub); this->pub.format = format; this->pub.backrestVersion = STRDEF(PROJECT_VERSION); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO, this); } /**********************************************************************************************************************************/ #define INFO_SECTION_BACKREST "backrest" #define INFO_KEY_CHECKSUM "backrest-checksum" #define INFO_SECTION_CIPHER "cipher" #define INFO_KEY_CIPHER_PASS "cipher-pass" #define INFO_KEY_CIPHER_PASS_CURRENT "cipher-pass-current" #define INFO_KEY_CIPHER_PASS_ROTATE "cipher-pass-rotate" #define INFO_KEY_CIPHER_DIGEST "digest" #define INFO_KEY_CIPHER_KEY "key" FN_EXTERN Info * infoNewLoad( IoRead *const read, const CipherSpec *const cipherSpec, InfoLoadNewCallback *const callbackFunction, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_PARAM(FUNCTIONP, callbackFunction); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(read != NULL); ASSERT(cipherSpec != NULL); ASSERT(callbackFunction != NULL); ASSERT(callbackData != NULL); OBJ_NEW_BEGIN(Info, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (Info){}; MEM_CONTEXT_TEMP_BEGIN() { String *const sectionLast = strNew(); // The last section seen during load IoFilter *const checksumActualFilter = cryptoHashNew(hashTypeSha1); // Checksum calculated from the file const String *checksumExpected = NULL; // Checksum found in ini file INFO_CHECKSUM_BEGIN(checksumActualFilter); TRY_BEGIN() { Ini *const ini = iniNewP(read, .strict = true); MEM_CONTEXT_TEMP_RESET_BEGIN() { const IniValue *value = iniValueNext(ini); while (value != NULL) { // Calculate checksum if (!(strEqZ(value->section, INFO_SECTION_BACKREST) && strEqZ(value->key, INFO_KEY_CHECKSUM))) { if (strEmpty(sectionLast) || !strEq(value->section, sectionLast)) { if (!strEmpty(sectionLast)) INFO_CHECKSUM_SECTION_NEXT(checksumActualFilter); INFO_CHECKSUM_SECTION(checksumActualFilter, value->section); strCat(strTrunc(sectionLast), value->section); } else INFO_CHECKSUM_KEY_VALUE_NEXT(checksumActualFilter); INFO_CHECKSUM_KEY_VALUE(checksumActualFilter, value->key, value->value); } // Process backrest section if (strEqZ(value->section, INFO_SECTION_BACKREST)) { // Validate and store format if (strEqZ(value->key, INFO_KEY_FORMAT)) { const unsigned int format = jsonReadUInt(jsonReadNew(value->value)); repoFormatValidate(format); this->pub.format = format; } // Store pgBackRest version else if (strEqZ(value->key, INFO_KEY_VERSION)) { MEM_CONTEXT_OBJ_BEGIN(this) { this->pub.backrestVersion = varStr(jsonToVar(value->value)); } MEM_CONTEXT_OBJ_END(); } // Store checksum to be validated later else if (strEqZ(value->key, INFO_KEY_CHECKSUM)) { MEM_CONTEXT_OBJ_BEGIN(this) { checksumExpected = varStr(jsonToVar(value->value)); } MEM_CONTEXT_OBJ_END(); } } // Process cipher section else if (strEqZ(value->section, INFO_SECTION_CIPHER)) { // No validation needed for cipher-pass, just store it if (strEqZ(value->key, INFO_KEY_CIPHER_PASS)) { MEM_CONTEXT_TEMP_BEGIN() { // Dependent files are encrypted with the same cipher type as this one const CipherType cipherType = cipherSpecType(cipherSpec); CipherSpecMap *const cipherSpecMap = cipherSpecMapNew(); const Variant *const cipherPass = jsonToVar(value->value); // Format >= 6 stores every key by id along with the digest it derives with if (infoFormat(this) >= REPOSITORY_FORMAT_6) { const KeyValue *const keyKv = varKv(cipherPass); const VariantList *const idList = kvKeyList(keyKv); for (unsigned int idIdx = 0; idIdx < varLstSize(idList); idIdx++) { const Variant *const id = varLstGet(idList, idIdx); const KeyValue *const keyData = varKv(kvGet(keyKv, id)); cipherSpecMapAdd( cipherSpecMap, varStr(id), cipherSpecNewP( cipherType, BUFSTR(varStr(kvGet(keyData, VARSTRDEF(INFO_KEY_CIPHER_KEY)))), .digest = strIdFromZ( strZ(varStr(kvGet(keyData, VARSTRDEF(INFO_KEY_CIPHER_DIGEST))))))); } } // Else there is one key with no id and no digest, which every repository derives with SHA-1 else { cipherSpecMapAdd( cipherSpecMap, CIPHER_SPEC_MAP_ID_DEFAULT_STR, cipherSpecNewP(cipherType, BUFSTR(varStr(cipherPass)), .digest = hashTypeSha1)); } infoCipherSpecMapSet(this, cipherSpecMap); } MEM_CONTEXT_TEMP_END(); } // Current key id else if (strEqZ(value->key, INFO_KEY_CIPHER_PASS_CURRENT)) { MEM_CONTEXT_TEMP_BEGIN() { cipherSpecMapIdCurrentSet( this->pub.cipherSpecMap, jsonReadStr(jsonReadNew(value->value))); } MEM_CONTEXT_TEMP_END(); } // Time of the last key rotation else if (strEqZ(value->key, INFO_KEY_CIPHER_PASS_ROTATE)) this->pub.cipherRotateTime = (time_t)jsonReadInt64(jsonReadNew(value->value)); } // Else pass to callback for processing else callbackFunction(callbackData, value->section, value->key, jsonReadNew(value->value)); value = iniValueNext(ini); MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } CATCH(CryptoError) { THROW_FMT(CryptoError, "%s\nHINT: is or was the repo encrypted?", errorMessage()); } TRY_END(); INFO_CHECKSUM_END(checksumActualFilter); // Verify the checksum first so a file that is empty or not an info file at all is reported as a checksum failure // rather than as a missing format const String *const checksumActual = strNewEncode( encodingHex, pckReadBinP(pckReadNew(ioFilterResult(checksumActualFilter)))); if (checksumExpected == NULL) THROW_FMT(ChecksumError, "invalid checksum, actual '%s' but no checksum found", strZ(checksumActual)); else if (!strEq(checksumExpected, checksumActual)) { THROW_FMT( ChecksumError, "invalid checksum, actual '%s' but expected '%s'", strZ(checksumActual), strZ(checksumExpected)); } // The format defines how everything else is read, so a file without one is not an info file this version can use. The // format is zero until the key is found and the value stored, so if we got here then the key was not found. if (infoFormat(this) == 0) THROW(FormatError, "repository format not found\nHINT: is this a valid " PROJECT_NAME " info file?"); // Only a read that looked for a header reports a format, so a result here means the file had one. The header is written // from the same format as the content, so a file where they disagree has been damaged or put together from parts of two // files. PackRead *const cipherResult = ioFilterGroupResultP(ioReadFilterGroup(read), CIPHER_BLOCK_FORMAT_FILTER_TYPE); if (cipherResult != NULL) { const unsigned int formatHeader = cipherBlockFormatResult(cipherResult); if (this->pub.format != formatHeader) { THROW_FMT( FormatError, "repository format %u does not match header format %u", this->pub.format, formatHeader); } } } MEM_CONTEXT_TEMP_END(); // A file with no cipher section has no encrypted dependent files if (this->pub.cipherSpecMap == NULL) infoCipherSpecMapSet(this, NULL); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO, this); } /**********************************************************************************************************************************/ FN_EXTERN bool infoSaveSection(InfoSave *const infoSaveData, const char *const section, const String *const sectionNext) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO_SAVE, infoSaveData); FUNCTION_TEST_PARAM(STRINGZ, section); FUNCTION_TEST_PARAM(STRING, sectionNext); FUNCTION_TEST_END(); ASSERT(infoSaveData != NULL); ASSERT(section != NULL); FUNCTION_TEST_RETURN( BOOL, (infoSaveData->sectionLast == NULL || strCmpZ(infoSaveData->sectionLast, section) < 0) && (sectionNext == NULL || strCmpZ(sectionNext, section) > 0)); } /**********************************************************************************************************************************/ FN_EXTERN void infoSaveValue(InfoSave *const infoSaveData, const char *const section, const char *const key, const String *const jsonValue) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO_SAVE, infoSaveData); FUNCTION_TEST_PARAM(STRINGZ, section); FUNCTION_TEST_PARAM(STRINGZ, key); FUNCTION_TEST_PARAM(STRING, jsonValue); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(infoSaveData != NULL); ASSERT(section != NULL); ASSERT(key != NULL); ASSERT(jsonValue != NULL); ASSERT(strSize(jsonValue) != 0); // The JSON value must not be an array because this may be confused with a section in the ini file ASSERT(strZ(jsonValue)[0] != '['); // Save section if (infoSaveData->sectionLast == NULL || !strEqZ(infoSaveData->sectionLast, section)) { if (infoSaveData->sectionLast != NULL) { INFO_CHECKSUM_SECTION_NEXT(infoSaveData->checksum); ioWriteLine(infoSaveData->write, BUFSTRDEF("")); } INFO_CHECKSUM_SECTION(infoSaveData->checksum, STR(section)); ioWrite(infoSaveData->write, BRACKETL_BUF); ioWrite(infoSaveData->write, BUFSTRZ(section)); ioWriteLine(infoSaveData->write, BRACKETR_BUF); MEM_CONTEXT_BEGIN(infoSaveData->memContext) { infoSaveData->sectionLast = strNewZ(section); } MEM_CONTEXT_END(); } else INFO_CHECKSUM_KEY_VALUE_NEXT(infoSaveData->checksum); // Save key/value INFO_CHECKSUM_KEY_VALUE(infoSaveData->checksum, STR(key), jsonValue); ioWrite(infoSaveData->write, BUFSTRZ(key)); ioWrite(infoSaveData->write, EQ_BUF); ioWriteLine(infoSaveData->write, BUFSTR(jsonValue)); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void infoSave(Info *const this, IoWrite *const write, InfoSaveCallback *const callbackFunction, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO, this); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_PARAM(FUNCTIONP, callbackFunction); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(write != NULL); ASSERT(callbackFunction != NULL); ASSERT(callbackData != NULL); MEM_CONTEXT_TEMP_BEGIN() { InfoSave data = { .memContext = MEM_CONTEXT_TEMP(), .write = write, }; ioWriteOpen(data.write); // Begin checksum calculation data.checksum = cryptoHashNew(hashTypeSha1); INFO_CHECKSUM_BEGIN(data.checksum); // Add version and format callbackFunction(callbackData, STRDEF(INFO_SECTION_BACKREST), &data); infoSaveValue(&data, INFO_SECTION_BACKREST, INFO_KEY_FORMAT, jsonFromVar(VARUINT(infoFormat(this)))); infoSaveValue(&data, INFO_SECTION_BACKREST, INFO_KEY_VERSION, jsonFromVar(VARSTRDEF(PROJECT_VERSION))); // Add cipher keys if defined const CipherSpecMap *const cipherSpecMap = infoCipherSpecMap(this); if (cipherSpecMapSize(cipherSpecMap) != 0) { callbackFunction(callbackData, STRDEF(INFO_SECTION_CIPHER), &data); MEM_CONTEXT_TEMP_BEGIN() { JsonWrite *const json = jsonWriteNewP(); // Format >= 6 stores every key by id along with the digest it derives with if (infoFormat(this) >= REPOSITORY_FORMAT_6) { jsonWriteObjectBegin(json); for (unsigned int keyIdx = 0; keyIdx < cipherSpecMapSize(cipherSpecMap); keyIdx++) { const CipherSpecMapItem *const item = cipherSpecMapGetIdx(cipherSpecMap, keyIdx); // A key must have a digest to store. Release builds check this too, instead of silently writing a key // that no reader can derive. CHECK(AssertError, cipherSpecDigest(item->cipherSpec) != 0, "cipher key must have a digest from format 6"); jsonWriteObjectBegin(jsonWriteKey(json, item->id)); jsonWriteStrId(jsonWriteKeyZ(json, INFO_KEY_CIPHER_DIGEST), cipherSpecDigest(item->cipherSpec)); jsonWriteStr( jsonWriteKeyZ(json, INFO_KEY_CIPHER_KEY), strNewBuf(cipherSpecPass(item->cipherSpec))); jsonWriteObjectEnd(json); } jsonWriteObjectEnd(json); } // Else store only the key else { const CipherSpec *const cipherSpec = infoCipherSpec(this); CHECK(AssertError, cipherSpecMapSize(cipherSpecMap) == 1, "only one cipher key can be stored before format 6"); CHECK( AssertError, cipherSpecDigest(cipherSpec) == hashTypeSha1, "cipher key must derive with sha1 before format 6"); jsonWriteStr(json, strNewBuf(cipherSpecPass(cipherSpec))); } infoSaveValue(&data, INFO_SECTION_CIPHER, INFO_KEY_CIPHER_PASS, jsonWriteResult(json)); // Record the current key id, which only exists at format >= 6 if (cipherSpecMapIdCurrent(cipherSpecMap) != NULL) { infoSaveValue( &data, INFO_SECTION_CIPHER, INFO_KEY_CIPHER_PASS_CURRENT, jsonFromVar(VARSTR(cipherSpecMapIdCurrent(cipherSpecMap)))); } // Record the time of the last key rotation if (infoCipherRotateTime(this) != 0) { infoSaveValue( &data, INFO_SECTION_CIPHER, INFO_KEY_CIPHER_PASS_ROTATE, jsonFromVar(VARINT64(infoCipherRotateTime(this)))); } } MEM_CONTEXT_TEMP_END(); } // Flush out any additional sections callbackFunction(callbackData, NULL, &data); // Add checksum (this must be set after all other values or it will not be valid) INFO_CHECKSUM_END(data.checksum); ioWrite(data.write, BUFSTRDEF("\n[" INFO_SECTION_BACKREST "]\n" INFO_KEY_CHECKSUM "=")); ioWriteLine( data.write, BUFSTR(jsonFromVar(VARSTR(strNewEncode(encodingHex, pckReadBinP(pckReadNew(ioFilterResult(data.checksum)))))))); // Close the file ioWriteClose(data.write); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ FN_EXTERN void infoFormatSet(Info *const this, const unsigned int format) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO, this); FUNCTION_TEST_PARAM(UINT, format); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(format >= REPOSITORY_FORMAT_MIN && format <= REPOSITORY_FORMAT_MAX); this->pub.format = format; FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN const CipherSpec * infoCipherSpec(const Info *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO, this); FUNCTION_TEST_END(); ASSERT(this != NULL); if (this->cipherSpecNone != NULL) FUNCTION_TEST_RETURN_CONST(CIPHER_SPEC, this->cipherSpecNone); FUNCTION_TEST_RETURN_CONST(CIPHER_SPEC, cipherSpecMapGet(infoCipherSpecMap(this), CIPHER_SPEC_MAP_ID_DEFAULT_STR)); } /**********************************************************************************************************************************/ FN_EXTERN void infoCipherSpecMapSet(Info *const this, const CipherSpecMap *const cipherSpecMap) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO, this); FUNCTION_TEST_PARAM(CIPHER_SPEC_MAP, cipherSpecMap); FUNCTION_TEST_END(); FUNCTION_AUDIT_IF(memContextCurrent() != objMemContext(this)); // Do not audit calls from within the object ASSERT(this != NULL); MEM_CONTEXT_OBJ_BEGIN(this) { // NULL becomes an empty map so the getter never returns NULL this->pub.cipherSpecMap = cipherSpecMap == NULL ? cipherSpecMapNew() : cipherSpecMapDup(cipherSpecMap); // The default key is looked up on demand, so only a caller that asks for one key errors when there is no default this->cipherSpecNone = cipherSpecMapSize(this->pub.cipherSpecMap) == 0 ? cipherSpecNewNone() : NULL; } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void infoCipherSpecAdd(Info *const this, const String *const id, const CipherSpec *const cipherSpec) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO, this); FUNCTION_TEST_PARAM(STRING, id); FUNCTION_TEST_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_TEST_END(); FUNCTION_AUDIT_IF(memContextCurrent() != objMemContext(this)); // Do not audit calls from within the object ASSERT(this != NULL); ASSERT(id != NULL); ASSERT(cipherSpec != NULL && cipherSpecType(cipherSpec) != cipherTypeNone); ASSERT(cipherSpecDigest(cipherSpec) != 0); MEM_CONTEXT_OBJ_BEGIN(this) { CipherSpecMap *const cipherSpecMap = cipherSpecMapDup(this->pub.cipherSpecMap); cipherSpecMapAdd(cipherSpecMap, id, cipherSpec); this->pub.cipherSpecMap = cipherSpecMap; this->cipherSpecNone = NULL; } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void infoCipherRotateTimeSet(Info *const this, const time_t cipherRotateTime) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO, this); FUNCTION_TEST_PARAM(TIME, cipherRotateTime); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(cipherRotateTime > 0); this->pub.cipherRotateTime = cipherRotateTime; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void infoCipherSpecSet(Info *const this, const CipherSpec *const cipherSpec) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO, this); FUNCTION_TEST_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_TEST_END(); FUNCTION_AUDIT_IF(memContextCurrent() != objMemContext(this)); // Do not audit calls from within the object ASSERT(this != NULL); // No keys needed when there is nothing to encrypt if (cipherSpec == NULL || cipherSpecType(cipherSpec) == cipherTypeNone) { infoCipherSpecMapSet(this, NULL); } // Else the one key, stored under the default id else { ASSERT(cipherSpecDigest(cipherSpec) != 0); MEM_CONTEXT_TEMP_BEGIN() { CipherSpecMap *const cipherSpecMap = cipherSpecMapNew(); cipherSpecMapAdd(cipherSpecMap, CIPHER_SPEC_MAP_ID_DEFAULT_STR, cipherSpec); infoCipherSpecMapSet(this, cipherSpecMap); } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void infoLoad(const String *const error, InfoLoadCallback *const callbackFunction, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, error); FUNCTION_LOG_PARAM(FUNCTIONP, callbackFunction); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); ASSERT(error != NULL); ASSERT(callbackFunction != NULL); ASSERT(callbackData != NULL); MEM_CONTEXT_TEMP_BEGIN() { unsigned int try = 0; volatile bool done = false; // Are all files tried? Must be preserved even on error. volatile bool loaded = false; // Was a file loaded? Must be preserved even on error. const ErrorType *loadErrorType = NULL; String *loadErrorMessage = NULL; do { // Attempt to load the file TRY_BEGIN() { loaded = callbackFunction(callbackData, try); done = true; CHECK(AssertError, loaded || try > 0, "file load must be attempted"); } CATCH_ANY() { // Set error type if none has been set if (loadErrorType == NULL) { loadErrorType = errorType(); loadErrorMessage = strCatFmt(strNew(), "%s:", strZ(error)); } // Else if the error type is different else if (loadErrorType != errorType()) { // Set type that is not file missing (which is likely the most common error) if (loadErrorType == &FileMissingError) { loadErrorType = errorType(); } // Else set a generic error else if (errorType() != &FileMissingError) loadErrorType = &FileOpenError; } // Append new error strCatFmt(loadErrorMessage, "\n%s: %s", errorTypeName(errorType()), errorMessage()); // Try again try++; } TRY_END(); } while (!done); // Error when no file was loaded if (!loaded) THROWP(loadErrorType, strZ(loadErrorMessage)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/info.h0000664000175100017510000001636615265101745016244 0ustar00runnerrunner/*********************************************************************************************************************************** Info Handler ***********************************************************************************************************************************/ #ifndef INFO_INFO_H #define INFO_INFO_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Info Info; typedef struct InfoSave InfoSave; #include "common/crypto/spec.h" #include "common/format/cipherSpecMap.h" #include "common/ini.h" #include "common/type/json.h" #include "storage/storage.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define INFO_COPY_EXT ".copy" #define INFO_KEY_FORMAT "backrest-format" #define INFO_KEY_VERSION "backrest-version" /*********************************************************************************************************************************** Function types for loading and saving ***********************************************************************************************************************************/ // The purpose of this callback is to attempt a load (from file or otherwise). Return true when the load is successful or throw an // error. Return false when there are no more loads to try, but always make at least one load attempt. The try parameter will start // at 0 and be incremented on each call. // {uncrustify_off - uncrustify unable to parse this statement} typedef bool InfoLoadCallback(void *data, unsigned int try); // {uncrustify_on} typedef void InfoLoadNewCallback(void *data, const String *section, const String *key, JsonRead *value); typedef void InfoSaveCallback(void *data, const String *sectionNext, InfoSave *infoSaveData); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Info *infoNew(unsigned int format, const CipherSpec *cipherSpecSub); // Create new object and load contents from a file. Decryption is added by the caller since which filter a file needs depends on // whether it begins with a header. The cipher spec the file is read with supplies the type for the cipher spec built from the pass // stored in it. FN_EXTERN Info *infoNewLoad( IoRead *read, const CipherSpec *cipherSpec, InfoLoadNewCallback *callbackFunction, void *callbackData); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct InfoPub { unsigned int format; // Repository format the file was written with const String *backrestVersion; // pgBackRest version CipherSpecMap *cipherSpecMap; // Cipher keys for dependent files time_t cipherRotateTime; // Time of the last cipher key rotation } InfoPub; // Repository format FN_INLINE_ALWAYS unsigned int infoFormat(const Info *const this) { return THIS_PUB(Info)->format; } FN_EXTERN void infoFormatSet(Info *this, unsigned int format); // Cipher keys for the files that require multiple keys, e.g. manifests. Never NULL, empty when there are no keys. FN_INLINE_ALWAYS const CipherSpecMap * infoCipherSpecMap(const Info *const this) { return THIS_PUB(Info)->cipherSpecMap; } // Default cipher spec or none when there are no keys FN_EXTERN const CipherSpec *infoCipherSpec(const Info *this); // Set cipher keys for dependent files. NULL means they are not encrypted. FN_EXTERN void infoCipherSpecMapSet(Info *this, const CipherSpecMap *cipherSpecMap); // Set a single cipher spec for dependent files under the default id. NULL means they are not encrypted. FN_EXTERN void infoCipherSpecSet(Info *this, const CipherSpec *cipherSpec); // Add a cipher key for dependent files under an id and make it current FN_EXTERN void infoCipherSpecAdd(Info *this, const String *id, const CipherSpec *cipherSpec); // Time of the last cipher key rotation, zero when none FN_INLINE_ALWAYS time_t infoCipherRotateTime(const Info *const this) { return THIS_PUB(Info)->cipherRotateTime; } FN_EXTERN void infoCipherRotateTimeSet(Info *this, time_t cipherRotateTime); // pgBackRest version FN_INLINE_ALWAYS const String * infoBackrestVersion(const Info *const this) { return THIS_PUB(Info)->backrestVersion; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Save to file FN_EXTERN void infoSave(Info *this, IoWrite *write, InfoSaveCallback *callbackFunction, void *callbackData); // Check if the section should be saved FN_EXTERN bool infoSaveSection(InfoSave *infoSaveData, const char *section, const String *sectionNext); // Save a JSON formatted value and update checksum FN_EXTERN void infoSaveValue(InfoSave *infoSaveData, const char *section, const char *key, const String *jsonValue); /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Load info file(s) and throw error for each attempt if none are successful FN_EXTERN void infoLoad(const String *error, InfoLoadCallback *callbackFunction, void *callbackData); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_INFO_TYPE \ Info * #define FUNCTION_LOG_INFO_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Info", buffer, bufferSize) #define FUNCTION_LOG_INFO_SAVE_TYPE \ InfoSave * #define FUNCTION_LOG_INFO_SAVE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoSave", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/infoArchive.c0000664000175100017510000003302015265101745017523 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Info Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/debug.h" #include "common/format/cipherBlockFormat.h" #include "common/format/format.h" #include "common/ini.h" #include "common/io/bufferWrite.h" #include "common/io/io.h" #include "common/log.h" #include "common/type/convert.h" #include "info/infoArchive.h" #include "info/infoPg.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/helper.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ STRING_EXTERN(INFO_ARCHIVE_PATH_FILE_STR, INFO_ARCHIVE_PATH_FILE); STRING_EXTERN(INFO_ARCHIVE_PATH_FILE_COPY_STR, INFO_ARCHIVE_PATH_FILE_COPY); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct InfoArchive { InfoArchivePub pub; // Publicly accessible variables }; /*********************************************************************************************************************************** Internal constructor ***********************************************************************************************************************************/ static InfoArchive * infoArchiveNewInternal(void) { FUNCTION_TEST_VOID(); InfoArchive *this = OBJ_NEW_ALLOC(); *this = (InfoArchive) { .pub = { .memContext = memContextCurrent(), }, }; FUNCTION_TEST_RETURN(INFO_ARCHIVE, this); } /**********************************************************************************************************************************/ FN_EXTERN InfoArchive * infoArchiveNew( const unsigned int pgVersion, const uint64_t pgSystemId, const unsigned int format, const CipherSpecMap *const cipherSpecMapSub) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_PARAM(UINT, format); FUNCTION_LOG_PARAM(CIPHER_SPEC_MAP, cipherSpecMapSub); FUNCTION_LOG_END(); ASSERT(pgVersion > 0 && pgSystemId > 0); InfoArchive *this; OBJ_NEW_BASE_BEGIN(InfoArchive, .childQty = MEM_CONTEXT_QTY_MAX) { this = infoArchiveNewInternal(); // Initialize the pg data this->pub.infoPg = infoPgNew(infoPgArchive, format, NULL); infoCipherSpecMapSet(infoPgInfo(this->pub.infoPg), cipherSpecMapSub); infoArchivePgSet(this, pgVersion, pgSystemId); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO_ARCHIVE, this); } /**********************************************************************************************************************************/ FN_EXTERN InfoArchive * infoArchiveNewLoad(IoRead *const read, const CipherSpec *const cipherSpec) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_END(); ASSERT(read != NULL); InfoArchive *this; OBJ_NEW_BASE_BEGIN(InfoArchive, .childQty = MEM_CONTEXT_QTY_MAX) { this = infoArchiveNewInternal(); this->pub.infoPg = infoPgNewLoad(read, infoPgArchive, cipherSpec, NULL, NULL); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO_ARCHIVE, this); } /**********************************************************************************************************************************/ FN_EXTERN void infoArchiveCipherRotate(InfoArchive *const this, const CipherSpec *const cipherSpec, const time_t rotateTime) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_ARCHIVE, this); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_PARAM(TIME, rotateTime); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(infoArchiveFormat(this) >= REPOSITORY_FORMAT_6); MEM_CONTEXT_TEMP_BEGIN() { // The next key id follows the current key id, or is the first key id when there is no current key const String *const idCurrent = cipherSpecMapIdCurrent(infoArchiveCipherSpecMap(this)); const String *const id = strNewFmt("%u", idCurrent == NULL ? 1 : cvtZToUInt(strZ(idCurrent)) + 1); Info *const info = infoPgInfo(infoArchivePg(this)); infoCipherSpecAdd(info, id, cipherSpec); infoCipherRotateTimeSet(info, rotateTime); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ const String * infoArchiveIdHistoryMatch( const InfoArchive *const this, const unsigned int historyId, const unsigned int pgVersion, const uint64_t pgSystemId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_ARCHIVE, this); FUNCTION_LOG_PARAM(UINT, historyId); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_END(); ASSERT(this != NULL); String *archiveId = NULL; const InfoPg *const infoPg = infoArchivePg(this); // Search the history list, from newest to oldest for (unsigned int pgIdx = 0; pgIdx < infoPgDataTotal(infoPg); pgIdx++) { const InfoPgData pgDataArchive = infoPgData(infoPg, pgIdx); // If there is an exact match with the history, system and version then get the archiveId and stop if (historyId == pgDataArchive.id && pgSystemId == pgDataArchive.systemId && pgVersion == pgDataArchive.version) { archiveId = infoPgArchiveId(infoPg, pgIdx); break; } } // If there was not an exact match, then search for the first matching database system-id and version if (archiveId == NULL) { for (unsigned int pgIdx = 0; pgIdx < infoPgDataTotal(infoPg); pgIdx++) { const InfoPgData pgDataArchive = infoPgData(infoPg, pgIdx); if (pgSystemId == pgDataArchive.systemId && pgVersion == pgDataArchive.version) { archiveId = infoPgArchiveId(infoPg, pgIdx); break; } } } // If the archive id has not been found, then error if (archiveId == NULL) { THROW_FMT( ArchiveMismatchError, "unable to retrieve the archive id for database version '%s' and system-id '%" PRIu64 "'", strZ(pgVersionToStr(pgVersion)), pgSystemId); } FUNCTION_LOG_RETURN(STRING, archiveId); } /*********************************************************************************************************************************** Save to file ***********************************************************************************************************************************/ static void infoArchiveSave(InfoArchive *const this, IoWrite *const write) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_ARCHIVE, this); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(write != NULL); MEM_CONTEXT_TEMP_BEGIN() { infoPgSave(infoArchivePg(this), write, NULL, NULL); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN InfoArchive * infoArchivePgSet(InfoArchive *const this, const unsigned int pgVersion, const uint64_t pgSystemId) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_ARCHIVE, this); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_END(); ASSERT(this != NULL); this->pub.infoPg = infoPgSet(infoArchivePg(this), pgVersion, pgSystemId, 0); FUNCTION_LOG_RETURN(INFO_ARCHIVE, this); } /**********************************************************************************************************************************/ FN_EXTERN void infoArchiveFormatSet(InfoArchive *const this, const unsigned int format) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_ARCHIVE, this); FUNCTION_LOG_PARAM(UINT, format); FUNCTION_LOG_END(); ASSERT(this != NULL); infoFormatSet(infoPgInfo(infoArchivePg(this)), format); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ typedef struct InfoArchiveLoadFileData { MemContext *memContext; // Mem context const Storage *storage; // Storage to load from const String *fileName; // Base filename const CipherSpec *cipherSpec; // Cipher spec InfoArchive *infoArchive; // Loaded infoArchive object } InfoArchiveLoadFileData; static bool infoArchiveLoadFileCallback(void *const data, const unsigned int try) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, data); FUNCTION_LOG_PARAM(UINT, try); FUNCTION_LOG_END(); ASSERT(data != NULL); InfoArchiveLoadFileData *const loadData = data; bool result = false; if (try < 2) { MEM_CONTEXT_TEMP_BEGIN() { // Construct filename based on try const String *const fileName = try == 0 ? loadData->fileName : strNewFmt("%s" INFO_COPY_EXT, strZ(loadData->fileName)); // Attempt to load the file IoRead *const read = storageReadIo(storageNewReadP(loadData->storage, fileName)); cipherBlockFormatFilterGroupReadAdd(ioReadFilterGroup(read), loadData->cipherSpec); MEM_CONTEXT_BEGIN(loadData->memContext) { loadData->infoArchive = infoArchiveNewLoad(read, loadData->cipherSpec); result = true; } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(BOOL, result); } FN_EXTERN InfoArchive * infoArchiveLoadFile( const Storage *const storage, const String *const fileName, const CipherSpec *const cipherSpec) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(fileName != NULL); ASSERT(cipherSpec != NULL); InfoArchiveLoadFileData data = { .memContext = memContextCurrent(), .storage = storage, .fileName = fileName, .cipherSpec = cipherSpec, }; MEM_CONTEXT_TEMP_BEGIN() { const char *const fileNamePath = strZ(storagePathP(storage, fileName)); TRY_BEGIN() { infoLoad( strNewFmt("unable to load info file '%s' or '%s" INFO_COPY_EXT "'", fileNamePath, fileNamePath), infoArchiveLoadFileCallback, &data); } CATCH_ANY() { THROWP_FMT( errorType(), "%s\n" "HINT: archive.info cannot be opened but is required to push/get WAL segments.\n" "HINT: is archive_command configured correctly in postgresql.conf?\n" "HINT: has a stanza-create been performed?\n" "HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving scheme.", errorMessage()); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INFO_ARCHIVE, data.infoArchive); } /**********************************************************************************************************************************/ FN_EXTERN void infoArchiveSaveFile( InfoArchive *const infoArchive, const Storage *const storage, const String *const fileName, const CipherSpec *const cipherSpec) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_ARCHIVE, infoArchive); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_END(); ASSERT(infoArchive != NULL); ASSERT(storage != NULL); ASSERT(fileName != NULL); ASSERT(cipherSpec != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Write output into a buffer since it needs to be saved to storage twice Buffer *const buffer = bufNew(ioBufferSize()); IoWrite *const write = ioBufferWriteNew(buffer); cipherBlockFormatFilterGroupWriteAddP(ioWriteFilterGroup(write), cipherSpec, infoArchiveFormat(infoArchive)); infoArchiveSave(infoArchive, write); // Save the file and make a copy storagePutP(storageNewWriteP(storage, fileName), buffer); storagePutP(storageNewWriteP(storage, strNewFmt("%s" INFO_COPY_EXT, strZ(fileName))), buffer); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/infoArchive.h0000664000175100017510000001470015265101745017534 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Info Handler ***********************************************************************************************************************************/ #ifndef INFO_INFOARCHIVE_H #define INFO_INFOARCHIVE_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct InfoArchive InfoArchive; #include "common/crypto/spec.h" #include "common/type/object.h" #include "common/type/string.h" #include "info/infoPg.h" #include "storage/storage.h" /*********************************************************************************************************************************** Archive info filename ***********************************************************************************************************************************/ #define INFO_ARCHIVE_FILE "archive.info" #define REGEX_ARCHIVE_DIR_DB_VERSION "^[0-9]+(\\.[0-9]+)*-[0-9]+$" #define INFO_ARCHIVE_PATH_FILE STORAGE_REPO_ARCHIVE "/" INFO_ARCHIVE_FILE STRING_DECLARE(INFO_ARCHIVE_PATH_FILE_STR); #define INFO_ARCHIVE_PATH_FILE_COPY INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT STRING_DECLARE(INFO_ARCHIVE_PATH_FILE_COPY_STR); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN InfoArchive *infoArchiveNew( const unsigned int pgVersion, const uint64_t pgSystemId, unsigned int format, const CipherSpecMap *cipherSpecMapSub); // Create new object and load contents from IoRead FN_EXTERN InfoArchive *infoArchiveNewLoad(IoRead *read, const CipherSpec *cipherSpec); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct InfoArchivePub { MemContext *memContext; // Mem context InfoPg *infoPg; // Contents of the DB data } InfoArchivePub; // PostgreSQL info FN_INLINE_ALWAYS InfoPg * infoArchivePg(const InfoArchive *const this) { return THIS_PUB(InfoArchive)->infoPg; } FN_EXTERN InfoArchive *infoArchivePgSet(InfoArchive *this, unsigned int pgVersion, uint64_t pgSystemId); // Current archive id FN_INLINE_ALWAYS const String * infoArchiveId(const InfoArchive *const this) { return infoPgArchiveId(infoArchivePg(this), infoPgDataCurrentId(infoArchivePg(this))); } // Cipher spec for dependent files FN_INLINE_ALWAYS const CipherSpec * infoArchiveCipherSpec(const InfoArchive *const this) { return infoPgCipherSpec(infoArchivePg(this)); } // Cipher keys for dependent files FN_INLINE_ALWAYS const CipherSpecMap * infoArchiveCipherSpecMap(const InfoArchive *const this) { return infoCipherSpecMap(infoPgInfo(infoArchivePg(this))); } // Time of the last archive key rotation, zero when none FN_INLINE_ALWAYS time_t infoArchiveCipherRotateTime(const InfoArchive *const this) { return infoCipherRotateTime(infoPgInfo(infoArchivePg(this))); } // Repository format FN_INLINE_ALWAYS unsigned int infoArchiveFormat(const InfoArchive *const this) { return infoPgFormat(infoArchivePg(this)); } FN_EXTERN void infoArchiveFormatSet(InfoArchive *this, unsigned int format); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add a cipher key for WAL at the next key id, make it current, and record the time of the rotation. Key ids are sequential from // one and are never reused. Id zero is reserved for a key migrated from format 5. FN_EXTERN void infoArchiveCipherRotate(InfoArchive *this, const CipherSpec *cipherSpec, time_t rotateTime); // Given a backrest history id and postgres systemId and version, return the archiveId of the best match FN_EXTERN const String *infoArchiveIdHistoryMatch( const InfoArchive *this, const unsigned int historyId, const unsigned int pgVersion, const uint64_t pgSystemId); // Move to a new parent mem context FN_INLINE_ALWAYS InfoArchive * infoArchiveMove(InfoArchive *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void infoArchiveFree(InfoArchive *const this) { objFree(this); } /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Load archive info FN_EXTERN InfoArchive *infoArchiveLoadFile(const Storage *storage, const String *fileName, const CipherSpec *cipherSpec); // Save archive info FN_EXTERN void infoArchiveSaveFile( InfoArchive *infoArchive, const Storage *storage, const String *fileName, const CipherSpec *cipherSpec); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_INFO_ARCHIVE_TYPE \ InfoArchive * #define FUNCTION_LOG_INFO_ARCHIVE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoArchive", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/infoBackup.c0000664000175100017510000012234115265101745017354 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Info Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/backup/common.h" #include "common/debug.h" #include "common/format/cipherBlockFormat.h" #include "common/ini.h" #include "common/io/bufferWrite.h" #include "common/io/io.h" #include "common/log.h" #include "common/regExp.h" #include "common/type/json.h" #include "common/type/list.h" #include "info/infoBackup.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/helper.h" #include "version.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ STRING_EXTERN(INFO_BACKUP_PATH_FILE_STR, INFO_BACKUP_PATH_FILE); STRING_EXTERN(INFO_BACKUP_PATH_FILE_COPY_STR, INFO_BACKUP_PATH_FILE_COPY); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct InfoBackup { InfoBackupPub pub; // Publicly accessible variables }; /*********************************************************************************************************************************** Internal constructor ***********************************************************************************************************************************/ static InfoBackup * infoBackupNewInternal(void) { FUNCTION_TEST_VOID(); FUNCTION_AUDIT_HELPER(); InfoBackup *this = OBJ_NEW_ALLOC(); *this = (InfoBackup) { .pub = { .memContext = memContextCurrent(), .backup = lstNewP(sizeof(InfoBackupData), .comparator = lstComparatorStr), .updated = true, }, }; FUNCTION_TEST_RETURN(INFO_BACKUP, this); } /**********************************************************************************************************************************/ FN_EXTERN InfoBackup * infoBackupNew( const unsigned int pgVersion, const uint64_t pgSystemId, const unsigned int pgCatalogVersion, const unsigned int format, const CipherSpec *const cipherSpecSub) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_PARAM(UINT, pgCatalogVersion); FUNCTION_LOG_PARAM(UINT, format); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpecSub); FUNCTION_LOG_END(); ASSERT(pgVersion > 0 && pgSystemId > 0 && pgCatalogVersion > 0); InfoBackup *this; OBJ_NEW_BASE_BEGIN(InfoBackup, .childQty = MEM_CONTEXT_QTY_MAX) { this = infoBackupNewInternal(); // Initialize the pg data this->pub.infoPg = infoPgNew(infoPgBackup, format, cipherSpecSub); infoBackupPgSet(this, pgVersion, pgSystemId, pgCatalogVersion); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO_BACKUP, this); } /*********************************************************************************************************************************** Create new object and load contents from a file ***********************************************************************************************************************************/ #define INFO_BACKUP_SECTION "backup" #define INFO_BACKUP_SECTION_BACKUP_CURRENT INFO_BACKUP_SECTION ":current" #define INFO_BACKUP_KEY_BACKUP_ANNOTATION "backup-annotation" #define INFO_BACKUP_KEY_BACKUP_ARCHIVE_START "backup-archive-start" #define INFO_BACKUP_KEY_BACKUP_ARCHIVE_STOP "backup-archive-stop" #define INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE "backup-info-repo-size" #define INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_DELTA "backup-info-repo-size-delta" #define INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_MAP "backup-info-repo-size-map" #define INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_MAP_DELTA "backup-info-repo-size-map-delta" #define INFO_BACKUP_KEY_BACKUP_INFO_SIZE "backup-info-size" #define INFO_BACKUP_KEY_BACKUP_INFO_SIZE_DELTA "backup-info-size-delta" #define INFO_BACKUP_KEY_BACKUP_LSN_START "backup-lsn-start" #define INFO_BACKUP_KEY_BACKUP_LSN_STOP "backup-lsn-stop" #define INFO_BACKUP_KEY_BACKUP_PRIOR STRID5("backup-prior", 0x93d3286e1558c220) #define INFO_BACKUP_KEY_BACKUP_REFERENCE "backup-reference" #define INFO_BACKUP_KEY_BACKUP_TIMESTAMP_START "backup-timestamp-start" #define INFO_BACKUP_KEY_BACKUP_TIMESTAMP_STOP "backup-timestamp-stop" #define INFO_BACKUP_KEY_BACKUP_TYPE STRID5("backup-type", 0x1619a6e1558c220) #define INFO_BACKUP_KEY_BACKUP_ERROR STRID5("backup-error", 0x93e522ee1558c220) #define INFO_BACKUP_KEY_OPT_ARCHIVE_CHECK "option-archive-check" #define INFO_BACKUP_KEY_OPT_ARCHIVE_COPY "option-archive-copy" #define INFO_BACKUP_KEY_OPT_BACKUP_STANDBY "option-backup-standby" #define INFO_BACKUP_KEY_OPT_CHECKSUM_PAGE "option-checksum-page" #define INFO_BACKUP_KEY_OPT_COMPRESS "option-compress" #define INFO_BACKUP_KEY_OPT_HARDLINK "option-hardlink" #define INFO_BACKUP_KEY_OPT_ONLINE "option-online" static void infoBackupLoadCallback(void *const data, const String *const section, const String *const key, JsonRead *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(JSON_READ, json); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(data != NULL); ASSERT(section != NULL); ASSERT(key != NULL); ASSERT(json != NULL); InfoBackup *const infoBackup = (InfoBackup *)data; // Process current backup list if (strEqZ(section, INFO_BACKUP_SECTION_BACKUP_CURRENT)) { MEM_CONTEXT_BEGIN(lstMemContext(infoBackup->pub.backup)) { jsonReadObjectBegin(json); InfoBackupData info = { .backupLabel = strDup(key), }; // Format and version info.backrestFormat = jsonReadUInt(jsonReadKeyRequireZ(json, INFO_KEY_FORMAT)); info.backrestVersion = jsonReadStr(jsonReadKeyRequireZ(json, INFO_KEY_VERSION)); // Annotation if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_ANNOTATION)) info.backupAnnotation = jsonReadVar(json); // Archive start/stop if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_ARCHIVE_START)) info.backupArchiveStart = jsonReadStr(json); if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_ARCHIVE_STOP)) info.backupArchiveStop = jsonReadStr(json); // Report errors detected during the backup. The key may not exist in older versions. if (jsonReadKeyExpectStrId(json, INFO_BACKUP_KEY_BACKUP_ERROR)) info.backupError = varNewBool(jsonReadBool(json)); // Size info info.backupInfoRepoSize = jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE)); info.backupInfoRepoSizeDelta = jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_DELTA)); if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_MAP)) { info.backupInfoRepoSizeMap = varNewUInt64(jsonReadUInt64(json)); info.backupInfoRepoSizeMapDelta = varNewUInt64( jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_MAP_DELTA))); } info.backupInfoSize = jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_INFO_SIZE)); info.backupInfoSizeDelta = jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_INFO_SIZE_DELTA)); // Lsn start/stop if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_LSN_START)) info.backupLsnStart = jsonReadStr(json); if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_LSN_STOP)) info.backupLsnStop = jsonReadStr(json); // Prior backup if (jsonReadKeyExpectStrId(json, INFO_BACKUP_KEY_BACKUP_PRIOR)) info.backupPrior = jsonReadStr(json); // Reference if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_REFERENCE)) info.backupReference = jsonReadStrLst(json); // Read timestamps as uint64 to ensure a positive value (guarantee no backups before 1970) info.backupTimestampStart = (time_t)jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_TIMESTAMP_START)); info.backupTimestampStop = (time_t)jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_TIMESTAMP_STOP)); // backup type info.backupType = (BackupType)jsonReadStrId(jsonReadKeyRequireStrId(json, INFO_BACKUP_KEY_BACKUP_TYPE)); ASSERT(info.backupType == backupTypeFull || info.backupType == backupTypeDiff || info.backupType == backupTypeIncr); // Database id info.backupPgId = jsonReadUInt(jsonReadKeyRequireZ(json, INFO_KEY_DB_ID)); // Options info.optionArchiveCheck = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_ARCHIVE_CHECK)); info.optionArchiveCopy = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_ARCHIVE_COPY)); info.optionBackupStandby = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_BACKUP_STANDBY)); info.optionChecksumPage = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_CHECKSUM_PAGE)); info.optionCompress = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_COMPRESS)); info.optionHardlink = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_HARDLINK)); info.optionOnline = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_ONLINE)); // Add the backup data to the list lstAdd(infoBackup->pub.backup, &info); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN InfoBackup * infoBackupNewLoad(IoRead *const read, const CipherSpec *const cipherSpec) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_END(); ASSERT(read != NULL); InfoBackup *this; OBJ_NEW_BASE_BEGIN(InfoBackup, .childQty = MEM_CONTEXT_QTY_MAX) { this = infoBackupNewInternal(); this->pub.infoPg = infoPgNewLoad(read, infoPgBackup, cipherSpec, infoBackupLoadCallback, this); this->pub.updated = false; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO_BACKUP, this); } /*********************************************************************************************************************************** Save to file ***********************************************************************************************************************************/ static void infoBackupSaveCallback(void *const data, const String *const sectionNext, InfoSave *const infoSaveData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(STRING, sectionNext); FUNCTION_TEST_PARAM(INFO_SAVE, infoSaveData); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(data != NULL); ASSERT(infoSaveData != NULL); InfoBackup *const infoBackup = (InfoBackup *)data; if (infoSaveSection(infoSaveData, INFO_BACKUP_SECTION_BACKUP_CURRENT, sectionNext)) { // Set the backup current section for (unsigned int backupIdx = 0; backupIdx < infoBackupDataTotal(infoBackup); backupIdx++) { const InfoBackupData backupData = infoBackupData(infoBackup, backupIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); jsonWriteUInt(jsonWriteKeyZ(json, INFO_KEY_FORMAT), backupData.backrestFormat); jsonWriteStr(jsonWriteKeyZ(json, INFO_KEY_VERSION), backupData.backrestVersion); if (backupData.backupAnnotation != NULL) jsonWriteVar(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_ANNOTATION), backupData.backupAnnotation); jsonWriteStr(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_ARCHIVE_START), backupData.backupArchiveStart); jsonWriteStr(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_ARCHIVE_STOP), backupData.backupArchiveStop); // Do not save backup-error if it was not loaded. This prevents backups that were added before the backup-error flag // was introduced from being saved with an incorrect value. if (backupData.backupError != NULL) jsonWriteBool(jsonWriteKeyStrId(json, INFO_BACKUP_KEY_BACKUP_ERROR), varBool(backupData.backupError)); jsonWriteUInt64(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE), backupData.backupInfoRepoSize); jsonWriteUInt64(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_DELTA), backupData.backupInfoRepoSizeDelta); ASSERT( (backupData.backupInfoRepoSizeMap != NULL && backupData.backupInfoRepoSizeMap != NULL) || (backupData.backupInfoRepoSizeMap == NULL && backupData.backupInfoRepoSizeMap == NULL)); if (backupData.backupInfoRepoSizeMap != NULL) { jsonWriteUInt64( jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_MAP), varUInt64(backupData.backupInfoRepoSizeMap)); jsonWriteUInt64( jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_MAP_DELTA), varUInt64(backupData.backupInfoRepoSizeMapDelta)); } jsonWriteUInt64(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_INFO_SIZE), backupData.backupInfoSize); jsonWriteUInt64(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_INFO_SIZE_DELTA), backupData.backupInfoSizeDelta); if (backupData.backupLsnStart != NULL) jsonWriteStr(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_LSN_START), backupData.backupLsnStart); if (backupData.backupLsnStop != NULL) jsonWriteStr(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_LSN_STOP), backupData.backupLsnStop); if (backupData.backupPrior != NULL) jsonWriteStr(jsonWriteKeyStrId(json, INFO_BACKUP_KEY_BACKUP_PRIOR), backupData.backupPrior); if (backupData.backupReference != NULL) jsonWriteStrLst(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_REFERENCE), backupData.backupReference); // When storing time_t treat as signed int to avoid casting jsonWriteInt64(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_TIMESTAMP_START), backupData.backupTimestampStart); jsonWriteInt64(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_TIMESTAMP_STOP), backupData.backupTimestampStop); jsonWriteStrId(jsonWriteKeyStrId(json, INFO_BACKUP_KEY_BACKUP_TYPE), backupData.backupType); jsonWriteUInt(jsonWriteKeyZ(json, INFO_KEY_DB_ID), backupData.backupPgId); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_ARCHIVE_CHECK), backupData.optionArchiveCheck); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_ARCHIVE_COPY), backupData.optionArchiveCopy); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_BACKUP_STANDBY), backupData.optionBackupStandby); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_CHECKSUM_PAGE), backupData.optionChecksumPage); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_COMPRESS), backupData.optionCompress); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_HARDLINK), backupData.optionHardlink); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_ONLINE), backupData.optionOnline); infoSaveValue( infoSaveData, INFO_BACKUP_SECTION_BACKUP_CURRENT, strZ(backupData.backupLabel), jsonWriteResult(jsonWriteObjectEnd(json))); } } FUNCTION_TEST_RETURN_VOID(); } static void infoBackupSave(InfoBackup *const this, IoWrite *const write) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(write != NULL); MEM_CONTEXT_TEMP_BEGIN() { infoPgSave(infoBackupPg(this), write, infoBackupSaveCallback, this); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN InfoBackup * infoBackupPgSet( InfoBackup *const this, const unsigned int pgVersion, const uint64_t pgSystemId, const unsigned int pgCatalogVersion) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_PARAM(UINT, pgCatalogVersion); FUNCTION_LOG_END(); this->pub.infoPg = infoPgSet(infoBackupPg(this), pgVersion, pgSystemId, pgCatalogVersion); this->pub.updated = true; FUNCTION_LOG_RETURN(INFO_BACKUP, this); } /**********************************************************************************************************************************/ FN_EXTERN void infoBackupFormatSet(InfoBackup *const this, const unsigned int format) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(UINT, format); FUNCTION_LOG_END(); ASSERT(this != NULL); infoFormatSet(infoPgInfo(infoBackupPg(this)), format); this->pub.updated = true; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN InfoBackupData infoBackupData(const InfoBackup *const this, const unsigned int backupDataIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(UINT, backupDataIdx); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(INFO_BACKUP_DATA, *(InfoBackupData *)lstGet(this->pub.backup, backupDataIdx)); } /**********************************************************************************************************************************/ FN_EXTERN void infoBackupDataAdd(InfoBackup *const this, const Manifest *const manifest) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(manifest != NULL); MEM_CONTEXT_TEMP_BEGIN() { const ManifestData *const manData = manifestData(manifest); // Calculate backup sizes, references and report errors uint64_t backupSize = 0; uint64_t backupSizeDelta = 0; uint64_t backupRepoSize = 0; uint64_t backupRepoSizeDelta = 0; uint64_t backupRepoSizeMap = 0; uint64_t backupRepoSizeMapDelta = 0; bool backupError = false; for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFile file = manifestFile(manifest, fileIdx); backupSize += file.size; backupRepoSize += file.sizeRepo > 0 ? file.sizeRepo : file.size; backupRepoSizeMap += file.blockIncrMapSize; // If a reference to a file exists, then it is in a previous backup and the delta calculation was already done if (file.reference == NULL) { backupSizeDelta += file.size; backupRepoSizeDelta += file.sizeRepo > 0 ? file.sizeRepo : file.size; backupRepoSizeMapDelta += file.blockIncrMapSize; } // Is there an error in the file? if (file.checksumPageError) backupError = true; } MEM_CONTEXT_BEGIN(lstMemContext(this->pub.backup)) { InfoBackupData infoBackupData = { .backupLabel = strDup(manData->backupLabel), .backrestFormat = manifestFormat(manifest), .backrestVersion = strDup(manData->backrestVersion), .backupInfoRepoSize = backupRepoSize, .backupInfoRepoSizeDelta = backupRepoSizeDelta, .backupInfoSize = backupSize, .backupInfoSizeDelta = backupSizeDelta, .backupPgId = manData->pgId, .backupTimestampStart = manData->backupTimestampStart, .backupTimestampStop = manData->backupTimestampStop, .backupType = manData->backupType, .backupError = varNewBool(backupError), .backupAnnotation = varDup(manData->annotation), .backupArchiveStart = strDup(manData->archiveStart), .backupArchiveStop = strDup(manData->archiveStop), .backupLsnStart = strDup(manData->lsnStart), .backupLsnStop = strDup(manData->lsnStop), .optionArchiveCheck = manData->backupOptionArchiveCheck, .optionArchiveCopy = manData->backupOptionArchiveCopy, .optionBackupStandby = manData->backupOptionStandby != NULL ? varBool(manData->backupOptionStandby) : false, .optionChecksumPage = manData->backupOptionChecksumPage != NULL ? varBool(manData->backupOptionChecksumPage) : false, .optionCompress = manData->backupOptionCompressType != compressTypeNone, .optionHardlink = manData->backupOptionHardLink, .optionOnline = manData->backupOptionOnline, }; // Add map sizes when block incr if (manData->blockIncr) { infoBackupData.backupInfoRepoSizeMap = varNewUInt64(backupRepoSizeMap); infoBackupData.backupInfoRepoSizeMapDelta = varNewUInt64(backupRepoSizeMapDelta); } if (manData->backupType != backupTypeFull) { // This list may not be sorted for manifests created before the reference list was added. Remove the last reference // since it will always be the current backup. Technically the current backup is always referenced but this is not // useful information for the user. infoBackupData.backupReference = strLstSort(strLstDup(manifestReferenceList(manifest)), sortOrderAsc); ASSERT( strEq( strLstGet(infoBackupData.backupReference, strLstSize(infoBackupData.backupReference) - 1), infoBackupData.backupLabel)); strLstRemoveIdx(infoBackupData.backupReference, strLstSize(infoBackupData.backupReference) - 1); infoBackupData.backupPrior = strDup(manData->backupLabelPrior); } // Add the backup data to the current backup list lstAdd(this->pub.backup, &infoBackupData); // Ensure the list is sorted ascending by the backupLabel lstSort(this->pub.backup, sortOrderAsc); } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_END(); this->pub.updated = true; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void infoBackupDataAnnotationSet(InfoBackup *const this, const String *const backupLabel, const KeyValue *const newAnnotationKv) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO_BACKUP, this); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_PARAM(KEY_VALUE, newAnnotationKv); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(infoBackupLabelExists(this, backupLabel)); ASSERT(newAnnotationKv != NULL); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.backup)) { // Get data for specified backup InfoBackupData *const infoBackupData = lstFind(this->pub.backup, &backupLabel); // Create annotation if it does not exist if (infoBackupData->backupAnnotation == NULL) infoBackupData->backupAnnotation = varNewKv(kvNew()); // Update annotations KeyValue *const annotationKv = varKv(infoBackupData->backupAnnotation); const VariantList *const newAnnotationKeyList = kvKeyList(newAnnotationKv); for (unsigned int keyIdx = 0; keyIdx < varLstSize(newAnnotationKeyList); keyIdx++) { const Variant *const newKey = varLstGet(newAnnotationKeyList, keyIdx); const Variant *const newValue = kvGet(newAnnotationKv, newKey); // If value is empty remove the key if (strEmpty(varStr(newValue))) { kvRemove(annotationKv, newKey); } // Else put new key/value (this will overwrite an existing value) else kvPut(annotationKv, newKey, newValue); } // Clean field if there are no annotations left if (varLstSize(kvKeyList(annotationKv)) == 0) infoBackupData->backupAnnotation = NULL; } MEM_CONTEXT_END(); this->pub.updated = true; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void infoBackupDataDelete(InfoBackup *const this, const String *const backupDeleteLabel) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(STRING, backupDeleteLabel); FUNCTION_LOG_END(); ASSERT(this != NULL); for (unsigned int idx = 0; idx < infoBackupDataTotal(this); idx++) { InfoBackupData backupData = infoBackupData(this, idx); if (strCmp(backupData.backupLabel, backupDeleteLabel) == 0) lstRemoveIdx(this->pub.backup, idx); } this->pub.updated = true; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN StringList * infoBackupDataLabelList(const InfoBackup *const this, const String *const expression) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(STRING, expression); FUNCTION_LOG_END(); ASSERT(this != NULL); // Return a 0 sized list if no current backups or none matching the filter StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Prepare regexp if an expression was passed RegExp *const regExp = (expression == NULL) ? NULL : regExpNew(expression); // For each backup label, compare it to the filter (if any) and sort it for return for (unsigned int backupLabelIdx = 0; backupLabelIdx < infoBackupDataTotal(this); backupLabelIdx++) { const InfoBackupData backupData = infoBackupData(this, backupLabelIdx); if (regExp == NULL || regExpMatch(regExp, backupData.backupLabel)) { strLstAdd(result, backupData.backupLabel); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN StringList * infoBackupDataDependentList(const InfoBackup *const this, const String *const backupLabel) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(STRING, backupLabel); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(backupLabel != NULL); // Return the given label as the only dependency or the given label and a list of labels that depend on it StringList *const result = strLstNew(); strLstAdd(result, backupLabel); MEM_CONTEXT_TEMP_BEGIN() { // For each backup label from oldest to newest in the current section, add each dependency to the list for (unsigned int backupLabelIdx = 0; backupLabelIdx < infoBackupDataTotal(this); backupLabelIdx++) { const InfoBackupData backupData = infoBackupData(this, backupLabelIdx); // If the backupPrior is in the dependency chain add the label to the list if (backupData.backupPrior != NULL && strLstExists(result, backupData.backupPrior)) strLstAdd(result, backupData.backupLabel); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ typedef struct InfoBackupLoadFileData { MemContext *memContext; // Mem context const Storage *storage; // Storage to load from const String *fileName; // Base filename const CipherSpec *cipherSpec; // Cipher spec InfoBackup *infoBackup; // Loaded infoBackup object } InfoBackupLoadFileData; static bool infoBackupLoadFileCallback(void *const data, const unsigned int try) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, data); FUNCTION_LOG_PARAM(UINT, try); FUNCTION_LOG_END(); ASSERT(data != NULL); InfoBackupLoadFileData *const loadData = data; bool result = false; if (try < 2) { MEM_CONTEXT_TEMP_BEGIN() { // Construct filename based on try const String *const fileName = try == 0 ? loadData->fileName : strNewFmt("%s" INFO_COPY_EXT, strZ(loadData->fileName)); // Attempt to load the file IoRead *const read = storageReadIo(storageNewReadP(loadData->storage, fileName)); cipherBlockFormatFilterGroupReadAdd(ioReadFilterGroup(read), loadData->cipherSpec); MEM_CONTEXT_BEGIN(loadData->memContext) { loadData->infoBackup = infoBackupNewLoad(read, loadData->cipherSpec); result = true; } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(BOOL, result); } FN_EXTERN InfoBackup * infoBackupLoadFile( const Storage *const storage, const String *const fileName, const CipherSpec *const cipherSpec) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(fileName != NULL); ASSERT(cipherSpec != NULL); InfoBackupLoadFileData data = { .memContext = memContextCurrent(), .storage = storage, .fileName = fileName, .cipherSpec = cipherSpec, }; MEM_CONTEXT_TEMP_BEGIN() { const char *const fileNamePath = strZ(storagePathP(storage, fileName)); TRY_BEGIN() { infoLoad( strNewFmt("unable to load info file '%s' or '%s" INFO_COPY_EXT "'", fileNamePath, fileNamePath), infoBackupLoadFileCallback, &data); } CATCH_ANY() { THROWP_FMT( errorType(), "%s\n" "HINT: backup.info cannot be opened and is required to perform a backup.\n" "HINT: has a stanza-create been performed?", errorMessage()); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INFO_BACKUP, data.infoBackup); } /**********************************************************************************************************************************/ // Helper to determine if any backup in the full backup set is in backup.info static bool infoBackupSetExists(const InfoBackup *const this, const String *const backupSet) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO_BACKUP, this); FUNCTION_TEST_PARAM(STRING, backupSet); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(backupSet != NULL); for (unsigned int backupIdx = 0; backupIdx < infoBackupDataTotal(this); backupIdx++) { if (strBeginsWith(infoBackupData(this, backupIdx).backupLabel, backupSet)) FUNCTION_TEST_RETURN(BOOL, true); } FUNCTION_TEST_RETURN(BOOL, false); } FN_EXTERN InfoBackup * infoBackupLoadFileReconstruct( const Storage *const storage, const String *const fileName, const CipherSpec *const cipherSpec) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(fileName != NULL); ASSERT(cipherSpec != NULL); InfoBackup *const infoBackup = infoBackupLoadFile(storage, fileName, cipherSpec); MEM_CONTEXT_TEMP_BEGIN() { // Get a list of backups in the repo const StringList *const backupList = strLstSort( storageListP( storage, STORAGE_REPO_BACKUP_STR, .expression = backupRegExpP(.full = true, .differential = true, .incremental = true)), sortOrderAsc); // Get the list of current backups and remove backups from current that are no longer in the repository const StringList *backupCurrentList = strLstSort(infoBackupDataLabelList(infoBackup, NULL), sortOrderAsc); for (unsigned int backupCurrIdx = 0; backupCurrIdx < strLstSize(backupCurrentList); backupCurrIdx++) { const String *const backupLabel = strLstGet(backupCurrentList, backupCurrIdx); // Only check the manifest when the backup is in the repository. When it is not there is no manifest either, so the // listing that has already been read is enough to know the backup is gone. const bool manifestExists = strLstExists(backupList, backupLabel) && storageExistsP(storage, strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel))); // If the manifest does not exist on disk and this backup has not already been deleted from the current list in the // infoBackup object, then remove it and its dependencies if (!manifestExists && infoBackupLabelExists(infoBackup, backupLabel)) { const StringList *const backupList = strLstSort( infoBackupDataDependentList(infoBackup, backupLabel), sortOrderDesc); for (unsigned int backupIdx = 0; backupIdx < strLstSize(backupList); backupIdx++) { const String *const removeBackup = strLstGet(backupList, backupIdx); LOG_WARN_FMT("backup '%s' missing manifest removed from " INFO_BACKUP_FILE, strZ(removeBackup)); infoBackupDataDelete(infoBackup, removeBackup); } } } // Get the updated current list of backups from backup.info backupCurrentList = strLstSort(infoBackupDataLabelList(infoBackup, NULL), sortOrderAsc); // For each backup in the repo, check if it exists in backup.info for (unsigned int backupIdx = 0; backupIdx < strLstSize(backupList); backupIdx++) { const String *const backupLabel = strLstGet(backupList, backupIdx); // If it does not exist in the list of current backups, then if it is valid, add it. A diff/incr backup is only added // when the backup it depends on is in backup.info, and since its label is generated from the label of that backup the // two are always in the same full backup set. When no backup from the set is in backup.info the manifest can never be // added no matter what it contains, so there is no reason to read it. Backups are processed oldest to newest, which // means a full backup added by this loop is in backup.info before the backups that depend on it are processed. if (!strLstExists(backupCurrentList, backupLabel) && (strSize(backupLabel) == DATE_TIME_LEN + 1 || infoBackupSetExists(infoBackup, strSubN(backupLabel, 0, DATE_TIME_LEN + 1)))) { const String *const manifestFileName = strNewFmt( STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)); // Check if a completed backup exists (backup.manifest only - ignore .copy) if (storageExistsP(storage, manifestFileName)) { bool found = false; const Manifest *const manifest = manifestLoadFile(storage, manifestFileName, infoBackupCipherSpec(infoBackup)); const ManifestData *const manData = manifestData(manifest); // If the pg data for the manifest exists in the history, then add it to current, but if something doesn't match // then warn that the backup is not valid for (unsigned int pgIdx = 0; pgIdx < infoPgDataTotal(infoBackupPg(infoBackup)); pgIdx++) { const InfoPgData pgHistory = infoPgData(infoBackupPg(infoBackup), pgIdx); // If there is an exact match with the history, system and version and there is no backup-prior dependency // or there is a backup-prior and it is in the list, then add this backup to the current backup list if (manData->pgId == pgHistory.id && manData->pgSystemId == pgHistory.systemId && manData->pgVersion == pgHistory.version && (manData->backupLabelPrior == NULL || infoBackupLabelExists(infoBackup, manData->backupLabelPrior))) { LOG_WARN_FMT("backup '%s' found in repository added to " INFO_BACKUP_FILE, strZ(backupLabel)); infoBackupDataAdd(infoBackup, manifest); found = true; break; } } if (!found) LOG_WARN_FMT("invalid backup '%s' cannot be added to current backups", strZ(manData->backupLabel)); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INFO_BACKUP, infoBackup); } /**********************************************************************************************************************************/ FN_EXTERN void infoBackupSaveFile( InfoBackup *const infoBackup, const Storage *const storage, const String *const fileName, const CipherSpec *const cipherSpec) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); ASSERT(storage != NULL); ASSERT(fileName != NULL); ASSERT(cipherSpec != NULL); if (infoBackup->pub.updated) { MEM_CONTEXT_TEMP_BEGIN() { // Write output into a buffer since it needs to be saved to storage twice Buffer *const buffer = bufNew(ioBufferSize()); IoWrite *const write = ioBufferWriteNew(buffer); cipherBlockFormatFilterGroupWriteAddP(ioWriteFilterGroup(write), cipherSpec, infoBackupFormat(infoBackup)); infoBackupSave(infoBackup, write); // Save the file and make a copy storagePutP(storageNewWriteP(storage, fileName), buffer); storagePutP(storageNewWriteP(storage, strNewFmt("%s" INFO_COPY_EXT, strZ(fileName))), buffer); } MEM_CONTEXT_TEMP_END(); infoBackup->pub.updated = false; } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void infoBackupDataToLog(const InfoBackupData *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{label: %s, pgId: %u}", strZ(this->backupLabel), this->backupPgId); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/infoBackup.h0000664000175100017510000002341115265101745017357 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Info Handler ***********************************************************************************************************************************/ #ifndef INFO_INFOBACKUP_H #define INFO_INFOBACKUP_H #include "common/type/stringId.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct InfoBackup InfoBackup; /*********************************************************************************************************************************** Backup type enum ***********************************************************************************************************************************/ typedef enum { backupTypeFull = STRID5("full", 0x632a60), backupTypeDiff = STRID5("diff", 0x319240), backupTypeIncr = STRID5("incr", 0x90dc90), } BackupType; #include "common/crypto/spec.h" #include "common/type/object.h" #include "common/type/string.h" #include "common/type/stringList.h" #include "info/infoPg.h" #include "info/manifest/manifest.h" #include "storage/storage.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define INFO_BACKUP_FILE "backup.info" #define INFO_BACKUP_PATH_FILE STORAGE_REPO_BACKUP "/" INFO_BACKUP_FILE STRING_DECLARE(INFO_BACKUP_PATH_FILE_STR); #define INFO_BACKUP_PATH_FILE_COPY INFO_BACKUP_PATH_FILE INFO_COPY_EXT STRING_DECLARE(INFO_BACKUP_PATH_FILE_COPY_STR); /*********************************************************************************************************************************** Information about an existing backup ***********************************************************************************************************************************/ typedef struct InfoBackupData { const String *backupLabel; // backupLabel must be first to allow for built-in list sorting unsigned int backrestFormat; const String *backrestVersion; Variant *backupAnnotation; // Backup annotations, if present const String *backupArchiveStart; const String *backupArchiveStop; uint64_t backupInfoRepoSize; uint64_t backupInfoRepoSizeDelta; const Variant *backupInfoRepoSizeMap; const Variant *backupInfoRepoSizeMapDelta; uint64_t backupInfoSize; uint64_t backupInfoSizeDelta; const String *backupLsnStart; const String *backupLsnStop; unsigned int backupPgId; const String *backupPrior; StringList *backupReference; time_t backupTimestampStart; time_t backupTimestampStop; BackupType backupType; const Variant *backupError; // Were errors detected during the backup? bool optionArchiveCheck; bool optionArchiveCopy; bool optionBackupStandby; bool optionChecksumPage; bool optionCompress; bool optionHardlink; bool optionOnline; } InfoBackupData; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN InfoBackup *infoBackupNew( unsigned int pgVersion, uint64_t pgSystemId, unsigned int pgCatalogVersion, unsigned int format, const CipherSpec *cipherSpecSub); // Create new object and load contents from IoRead FN_EXTERN InfoBackup *infoBackupNewLoad(IoRead *read, const CipherSpec *cipherSpec); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct InfoBackupPub { MemContext *memContext; // Mem context InfoPg *infoPg; // Contents of the DB data List *backup; // List of current backups and their associated data bool updated; // Has the info been updated since the last save? } InfoBackupPub; // PostgreSQL info FN_INLINE_ALWAYS InfoPg * infoBackupPg(const InfoBackup *const this) { return THIS_PUB(InfoBackup)->infoPg; } FN_EXTERN InfoBackup *infoBackupPgSet(InfoBackup *this, unsigned int pgVersion, uint64_t pgSystemId, unsigned int pgCatalogVersion); // Cipher spec for dependent files FN_INLINE_ALWAYS const CipherSpec * infoBackupCipherSpec(const InfoBackup *const this) { return infoPgCipherSpec(infoBackupPg(this)); } // Repository format FN_INLINE_ALWAYS unsigned int infoBackupFormat(const InfoBackup *const this) { return infoPgFormat(infoBackupPg(this)); } FN_EXTERN void infoBackupFormatSet(InfoBackup *this, unsigned int format); // Return a structure of the backup data from a specific index FN_EXTERN InfoBackupData infoBackupData(const InfoBackup *this, unsigned int backupDataIdx); // Does the specified backup label exist? FN_INLINE_ALWAYS bool infoBackupLabelExists(const InfoBackup *const this, const String *const backupLabel) { ASSERT_INLINE(backupLabel != NULL); return lstFind(THIS_PUB(InfoBackup)->backup, &backupLabel) != NULL; } // Return a pointer to a structure from the current backup data given a label, else NULL FN_INLINE_ALWAYS InfoBackupData * infoBackupDataByLabel(const InfoBackup *const this, const String *const backupLabel) { ASSERT_INLINE(infoBackupLabelExists(this, backupLabel)); return lstFind(THIS_PUB(InfoBackup)->backup, &backupLabel); } // Get total current backups FN_INLINE_ALWAYS unsigned int infoBackupDataTotal(const InfoBackup *const this) { return lstSize(THIS_PUB(InfoBackup)->backup); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add backup to the current list FN_EXTERN void infoBackupDataAdd(InfoBackup *this, const Manifest *manifest); // Set Annotation in the backup data for a specific backup label FN_EXTERN void infoBackupDataAnnotationSet(InfoBackup *this, const String *const backupLabel, const KeyValue *annotationKv); // Delete backup from the current backup list FN_EXTERN void infoBackupDataDelete(InfoBackup *this, const String *backupDeleteLabel); // Given a backup label, get the dependency list FN_EXTERN StringList *infoBackupDataDependentList(const InfoBackup *this, const String *backupLabel); // Return a list of current backup labels, applying a regex expression if provided FN_EXTERN StringList *infoBackupDataLabelList(const InfoBackup *this, const String *expression); // Move to a new parent mem context FN_INLINE_ALWAYS InfoBackup * infoBackupMove(InfoBackup *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void infoBackupFree(InfoBackup *const this) { objFree(this); } /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Load backup info FN_EXTERN InfoBackup *infoBackupLoadFile(const Storage *storage, const String *fileName, const CipherSpec *cipherSpec); // Load backup info and update it by adding valid backups from the repo or removing backups no longer in the repo FN_EXTERN InfoBackup *infoBackupLoadFileReconstruct( const Storage *storage, const String *fileName, const CipherSpec *cipherSpec); // Save backup info FN_EXTERN void infoBackupSaveFile( InfoBackup *infoBackup, const Storage *storage, const String *fileName, const CipherSpec *cipherSpec); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void infoBackupDataToLog(const InfoBackupData *this, StringStatic *debugLog); #define FUNCTION_LOG_INFO_BACKUP_TYPE \ InfoBackup * #define FUNCTION_LOG_INFO_BACKUP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoBackup", buffer, bufferSize) #define FUNCTION_LOG_INFO_BACKUP_DATA_TYPE \ InfoBackupData #define FUNCTION_LOG_INFO_BACKUP_DATA_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(&value, infoBackupDataToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/infoPg.c0000664000175100017510000004234315265101745016520 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Info Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include "common/debug.h" #include "common/ini.h" #include "common/log.h" #include "common/type/json.h" #include "common/type/list.h" #include "common/type/object.h" #include "info/info.h" #include "info/infoPg.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/helper.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct InfoPg { InfoPgPub pub; // Publicly accessible variables MemContext *memContext; // Mem context InfoPgType type; // Type of info file being loaded unsigned int historyCurrent; // Index of the current history item }; /*********************************************************************************************************************************** Internal constructor ***********************************************************************************************************************************/ static InfoPg * infoPgNewInternal(const InfoPgType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); InfoPg *this = OBJ_NEW_ALLOC(); *this = (InfoPg) { .pub = { .history = lstNewP(sizeof(InfoPgData)), }, .memContext = memContextCurrent(), .type = type, }; FUNCTION_TEST_RETURN(INFO_PG, this); } /**********************************************************************************************************************************/ FN_EXTERN InfoPg * infoPgNew(const InfoPgType type, const unsigned int format, const CipherSpec *const cipherSpecSub) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(UINT, format); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpecSub); FUNCTION_LOG_END(); InfoPg *this; OBJ_NEW_BASE_BEGIN(InfoPg, .childQty = MEM_CONTEXT_QTY_MAX) { this = infoPgNewInternal(type); this->pub.info = infoNew(format, cipherSpecSub); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO_PG, this); } /**********************************************************************************************************************************/ #define INFO_SECTION_DB "db" #define INFO_SECTION_DB_HISTORY "db:history" #define INFO_KEY_DB_CATALOG_VERSION "db-catalog-version" #define INFO_KEY_DB_CONTROL_VERSION "db-control-version" #define INFO_KEY_DB_SYSTEM_ID "db-system-id" #define INFO_KEY_DB_VERSION "db-version" typedef struct InfoPgLoadData { InfoLoadNewCallback *callbackFunction; // Callback function for child object void *callbackData; // Callback data for child object InfoPg *infoPg; // Pg info unsigned int currentId; // Current database id } InfoPgLoadData; static void infoPgLoadCallback(void *const data, const String *const section, const String *const key, JsonRead *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(JSON_READ, json); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(data != NULL); ASSERT(section != NULL); ASSERT(key != NULL); ASSERT(json != NULL); InfoPgLoadData *const loadData = data; // Process db section if (strEqZ(section, INFO_SECTION_DB)) { if (strEqZ(key, INFO_KEY_DB_ID)) loadData->currentId = jsonReadUInt(json); } // Process db:history section else if (strEqZ(section, INFO_SECTION_DB_HISTORY)) { InfoPgData infoPgData = {.id = cvtZToUInt(strZ(key))}; jsonReadObjectBegin(json); // Catalog version if (loadData->infoPg->type == infoPgBackup) { infoPgData.catalogVersion = jsonReadUInt(jsonReadKeyRequireZ(json, INFO_KEY_DB_CATALOG_VERSION)); infoPgData.controlVersion = jsonReadUInt(jsonReadKeyRequireZ(json, INFO_KEY_DB_CONTROL_VERSION)); } // System id infoPgData.systemId = jsonReadUInt64( jsonReadKeyRequireZ(json, loadData->infoPg->type == infoPgArchive ? INFO_KEY_DB_ID : INFO_KEY_DB_SYSTEM_ID)); // PostgreSQL version infoPgData.version = pgVersionFromStr(jsonReadStr(jsonReadKeyRequireZ(json, INFO_KEY_DB_VERSION))); // Insert at beginning of list so the history is reverse ordered lstInsert(loadData->infoPg->pub.history, 0, &infoPgData); } // Callback if set else if (loadData->callbackFunction != NULL) loadData->callbackFunction(loadData->callbackData, section, key, json); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN InfoPg * infoPgNewLoad( IoRead *const read, const InfoPgType type, const CipherSpec *const cipherSpec, InfoLoadNewCallback *const callbackFunction, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_PARAM(FUNCTIONP, callbackFunction); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); ASSERT(read != NULL); ASSERT(type == infoPgBackup || type == infoPgArchive); ASSERT((callbackFunction == NULL && callbackData == NULL) || (callbackFunction != NULL && callbackData != NULL)); InfoPg *this; OBJ_NEW_BASE_BEGIN(InfoPg, .childQty = MEM_CONTEXT_QTY_MAX) { this = infoPgNewInternal(type); // Set historyCurrent to UINT_MAX so we can detect if it was loaded correctly this->historyCurrent = UINT_MAX; // Load InfoPgLoadData loadData = { .callbackFunction = callbackFunction, .callbackData = callbackData, .infoPg = this, }; this->pub.info = infoNewLoad(read, cipherSpec, infoPgLoadCallback, &loadData); CHECK(FormatError, !lstEmpty(this->pub.history), "history is missing"); CHECK(FormatError, loadData.currentId > 0, "current id is missing"); // Find the current history item for (unsigned int historyIdx = 0; historyIdx < lstSize(this->pub.history); historyIdx++) { if (((InfoPgData *)lstGet(this->pub.history, historyIdx))->id == loadData.currentId) this->historyCurrent = historyIdx; } CHECK(FormatError, this->historyCurrent != UINT_MAX, "unable to find current id in history"); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO_PG, this); } /**********************************************************************************************************************************/ FN_EXTERN void infoPgAdd(InfoPg *const this, const InfoPgData *const infoPgData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_PARAM_P(INFO_PG_DATA, infoPgData); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(infoPgData != NULL); lstInsert(this->pub.history, 0, infoPgData); this->historyCurrent = 0; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN InfoPg * infoPgSet(InfoPg *const this, const unsigned int pgVersion, const uint64_t pgSystemId, const unsigned int pgCatalogVersion) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_PARAM(UINT, pgCatalogVersion); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { unsigned int pgDataId = 1; // If there is some history, then get the historyId of the most current and increment it if (infoPgDataTotal(this) > 0) pgDataId = infoPgCurrentDataId(this) + 1; // Set db values that are common to all info files InfoPgData infoPgData = { .id = pgDataId, .version = pgVersion, // This is different in archive.info due to a typo that can't be fixed without a format version bump .systemId = pgSystemId, }; // Catalog/control version required for backup info to preserve the repo format if (this->type == infoPgBackup) { infoPgData.catalogVersion = pgCatalogVersion; infoPgData.controlVersion = pgControlVersion(pgVersion); } // Add the pg data to the history list infoPgAdd(this, &infoPgData); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INFO_PG, this); } /**********************************************************************************************************************************/ typedef struct InfoPgSaveData { InfoSaveCallback *callbackFunction; // Callback function for child object void *callbackData; // Callback data for child object InfoPgType type; // Type of info file being loaded InfoPg *infoPg; // InfoPg object to be saved } InfoPgSaveData; static void infoPgSaveCallback(void *const data, const String *const sectionNext, InfoSave *const infoSaveData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(STRING, sectionNext); FUNCTION_TEST_PARAM(INFO_SAVE, infoSaveData); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(data != NULL); ASSERT(infoSaveData != NULL); InfoPgSaveData *const saveData = (InfoPgSaveData *)data; if (infoSaveSection(infoSaveData, INFO_SECTION_DB, sectionNext)) { if (saveData->callbackFunction != NULL) saveData->callbackFunction(saveData->callbackData, STRDEF(INFO_SECTION_DB), infoSaveData); const InfoPgData pgData = infoPgDataCurrent(saveData->infoPg); // These need to be saved because older pgBackRest versions expect them if (saveData->infoPg->type == infoPgBackup) { infoSaveValue(infoSaveData, INFO_SECTION_DB, INFO_KEY_DB_CATALOG_VERSION, jsonFromVar(VARUINT(pgData.catalogVersion))); infoSaveValue(infoSaveData, INFO_SECTION_DB, INFO_KEY_DB_CONTROL_VERSION, jsonFromVar(VARUINT(pgData.controlVersion))); } infoSaveValue(infoSaveData, INFO_SECTION_DB, INFO_KEY_DB_ID, jsonFromVar(VARUINT(pgData.id))); infoSaveValue(infoSaveData, INFO_SECTION_DB, INFO_KEY_DB_SYSTEM_ID, jsonFromVar(VARUINT64(pgData.systemId))); infoSaveValue(infoSaveData, INFO_SECTION_DB, INFO_KEY_DB_VERSION, jsonFromVar(VARSTR(pgVersionToStr(pgData.version)))); } if (infoSaveSection(infoSaveData, INFO_SECTION_DB_HISTORY, sectionNext)) { if (saveData->callbackFunction != NULL) saveData->callbackFunction(saveData->callbackData, STRDEF(INFO_SECTION_DB_HISTORY), infoSaveData); // Set the db history section in reverse so oldest history is first instead of last to be consistent with load for (unsigned int pgDataIdx = infoPgDataTotal(saveData->infoPg) - 1; (int)pgDataIdx >= 0; pgDataIdx--) { const InfoPgData pgData = infoPgData(saveData->infoPg, pgDataIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); // These need to be saved because older pgBackRest versions expect them if (saveData->infoPg->type == infoPgBackup) { jsonWriteUInt(jsonWriteKeyZ(json, INFO_KEY_DB_CATALOG_VERSION), pgData.catalogVersion); jsonWriteUInt(jsonWriteKeyZ(json, INFO_KEY_DB_CONTROL_VERSION), pgData.controlVersion); } if (saveData->infoPg->type == infoPgArchive) jsonWriteUInt64(jsonWriteKeyZ(json, INFO_KEY_DB_ID), pgData.systemId); if (saveData->infoPg->type == infoPgBackup) jsonWriteUInt64(jsonWriteKeyZ(json, INFO_KEY_DB_SYSTEM_ID), pgData.systemId); jsonWriteStr(jsonWriteKeyZ(json, INFO_KEY_DB_VERSION), pgVersionToStr(pgData.version)); infoSaveValue( infoSaveData, INFO_SECTION_DB_HISTORY, strZ(varStrForce(VARUINT(pgData.id))), jsonWriteResult(jsonWriteObjectEnd(json))); } } // Process the callback even if none of the sections above get executed if (saveData->callbackFunction != NULL) saveData->callbackFunction(saveData->callbackData, sectionNext, infoSaveData); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void infoPgSave(InfoPg *const this, IoWrite *const write, InfoSaveCallback *const callbackFunction, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_PARAM(FUNCTIONP, callbackFunction); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(write != NULL); ASSERT((callbackFunction == NULL && callbackData == NULL) || (callbackFunction != NULL && callbackData != NULL)); MEM_CONTEXT_TEMP_BEGIN() { InfoPgSaveData saveData = { .callbackFunction = callbackFunction, .callbackData = callbackData, .infoPg = this, }; infoSave(infoPgInfo(this), write, infoPgSaveCallback, &saveData); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * infoPgArchiveId(const InfoPg *const this, const unsigned int pgDataIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_PARAM(UINT, pgDataIdx); FUNCTION_LOG_END(); ASSERT(this != NULL); const InfoPgData pgData = infoPgData(this, pgDataIdx); String *const version = pgVersionToStr(pgData.version); String *const result = strNewFmt("%s-%u", strZ(version), pgData.id); strFree(version); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN InfoPgData infoPgData(const InfoPg *const this, const unsigned int pgDataIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_PARAM(UINT, pgDataIdx); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(INFO_PG_DATA, *(InfoPgData *)lstGet(this->pub.history, pgDataIdx)); } /**********************************************************************************************************************************/ FN_EXTERN InfoPgData infoPgDataCurrent(const InfoPg *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(INFO_PG_DATA, infoPgData(this, infoPgDataCurrentId(this))); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int infoPgDataCurrentId(const InfoPg *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(UINT, this->historyCurrent); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int infoPgCurrentDataId(const InfoPg *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_END(); ASSERT(this != NULL); InfoPgData currentData = infoPgDataCurrent(this); FUNCTION_LOG_RETURN(UINT, currentData.id); } /**********************************************************************************************************************************/ FN_EXTERN void infoPgDataToLog(const InfoPgData *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{id: %u, version: %u, systemId: %" PRIu64 ", catalogVersion: %u}", this->id, this->version, this->systemId, this->catalogVersion); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/infoPg.h0000664000175100017510000001416715265101745016530 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Info Handler ***********************************************************************************************************************************/ #ifndef INFO_INFOPG_H #define INFO_INFOPG_H #include /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct InfoPg InfoPg; #include "common/crypto/common.h" #include "common/ini.h" #include "info/info.h" #include "storage/storage.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define INFO_KEY_DB_ID "db-id" /*********************************************************************************************************************************** Information about the PostgreSQL cluster ***********************************************************************************************************************************/ typedef struct InfoPgData { unsigned int id; uint64_t systemId; unsigned int catalogVersion; unsigned int version; // Control version is required to maintain the file format for older versions of pgBackRest but should not be used elsewhere unsigned int controlVersion; } InfoPgData; /*********************************************************************************************************************************** Info types for determining data in DB section ***********************************************************************************************************************************/ typedef enum { infoPgArchive = STRID5("archive", 0x16c940e410), // archive info file infoPgBackup = STRID5("backup", 0x21558c220), // backup info file } InfoPgType; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN InfoPg *infoPgNew(InfoPgType type, unsigned int format, const CipherSpec *cipherSpecSub); // Create new object and load contents from a file FN_EXTERN InfoPg *infoPgNewLoad( IoRead *read, InfoPgType type, const CipherSpec *cipherSpec, InfoLoadNewCallback *callbackFunction, void *callbackData); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct InfoPgPub { Info *info; // Info contents List *history; // A list of InfoPgData } InfoPgPub; // Archive id FN_EXTERN String *infoPgArchiveId(const InfoPg *this, unsigned int pgDataIdx); // Base info FN_INLINE_ALWAYS Info * infoPgInfo(const InfoPg *const this) { return THIS_PUB(InfoPg)->info; } // Cipher spec for dependent files FN_INLINE_ALWAYS const CipherSpec * infoPgCipherSpec(const InfoPg *const this) { return infoCipherSpec(infoPgInfo(this)); } // Return current pgId from the history FN_EXTERN unsigned int infoPgCurrentDataId(const InfoPg *this); // PostgreSQL info FN_EXTERN InfoPgData infoPgData(const InfoPg *this, unsigned int pgDataIdx); // Current PostgreSQL data FN_EXTERN InfoPgData infoPgDataCurrent(const InfoPg *this); // Current history index FN_EXTERN unsigned int infoPgDataCurrentId(const InfoPg *this); // Repository format FN_INLINE_ALWAYS unsigned int infoPgFormat(const InfoPg *const this) { return infoFormat(infoPgInfo(this)); } // Total PostgreSQL data in the history FN_INLINE_ALWAYS unsigned int infoPgDataTotal(const InfoPg *const this) { return lstSize(THIS_PUB(InfoPg)->history); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add Postgres data to the history list at position 0 to ensure the latest history is always first in the list FN_EXTERN void infoPgAdd(InfoPg *this, const InfoPgData *infoPgData); // Save to IO FN_EXTERN void infoPgSave(InfoPg *this, IoWrite *write, InfoSaveCallback *callbackFunction, void *callbackData); // Set the InfoPg object data based on values passed FN_EXTERN InfoPg *infoPgSet( InfoPg *this, const unsigned int pgVersion, const uint64_t pgSystemId, const unsigned int pgCatalogVersion); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void infoPgDataToLog(const InfoPgData *this, StringStatic *debugLog); #define FUNCTION_LOG_INFO_PG_TYPE \ InfoPg * #define FUNCTION_LOG_INFO_PG_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoPg", buffer, bufferSize) #define FUNCTION_LOG_INFO_PG_DATA_TYPE \ InfoPgData #define FUNCTION_LOG_INFO_PG_DATA_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(&value, infoPgDataToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/manifest/0000775000175100017510000000000015265101745016732 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/manifest/build.c.inc0000664000175100017510000005673715265101745020767 0ustar00runnerrunner/*********************************************************************************************************************************** Build manifest from PGDATA ***********************************************************************************************************************************/ typedef struct ManifestBuildData { Manifest *manifest; // Manifest being build const Storage *storagePg; // PostgreSQL storage const String *tablespaceId; // Tablespace id if PostgreSQL version has one bool online; // Is this an online backup? bool checksumPage; // Are page checksums being checked? const String *manifestWalName; // Wal manifest name for this version of PostgreSQL RegExp *dbPathExp; // Identify paths containing relations RegExp *tempRelationExp; // Identify temp relations const Pack *tablespaceList; // List of tablespaces in the database ManifestLinkCheck *linkCheck; // List of links found during build (used for prefix check) StringList *excludeContent; // Exclude contents of directories StringList *excludeSingle; // Exclude a single file/link/path const ManifestBlockIncrMap *blockIncrMap; // Block incremental maps } ManifestBuildData; // Calculate block incremental size for a file. The block size is based on the size and age of the file. Larger files get larger // block sizes to reduce the cost of the map. Older files also get larger block sizes under the assumption that they are unlikely to // be modified if they have not been modified in a while. Very old and very small files skip block incremental entirely. // // Smaller blocks will be compressed/encrypted together in a larger super block for efficiency. static size_t manifestBuildBlockIncrSize(const ManifestBuildData *const buildData, const ManifestFile *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buildData); FUNCTION_TEST_PARAM(MANIFEST_FILE, file); FUNCTION_TEST_END(); size_t result = 0; // Search size map for the appropriate block size for (unsigned int sizeIdx = 0; sizeIdx < lstSize(buildData->blockIncrMap->sizeMap); sizeIdx++) { const ManifestBlockIncrSizeMap *const sizeMap = lstGet(buildData->blockIncrMap->sizeMap, sizeIdx); if (file->size >= sizeMap->fileSize) { result = sizeMap->blockSize; break; } } // If block size > 0 then search age map for a multiplier if (result != 0) { const time_t fileAge = buildData->manifest->pub.data.backupTimestampStart - file->timestamp; for (unsigned int ageIdx = 0; ageIdx < lstSize(buildData->blockIncrMap->ageMap); ageIdx++) { const ManifestBlockIncrAgeMap *const ageMap = lstGet(buildData->blockIncrMap->ageMap, ageIdx); if (fileAge >= (time_t)ageMap->fileAge) { result *= ageMap->blockMultiplier; break; } } } FUNCTION_TEST_RETURN(SIZE, result); } // Get checksum size for a block size. Since these checksums are used to determine if a block has changed (not for corruption) they // should be set larger than usual. For example, it is common to use 32-bit checksums to check for corruption in blocks up to 4MiB // but here we used more bits to ensure block changes are correctly detected, since valid changes may look a lot different than // corruption. static size_t manifestBuildBlockIncrChecksumSize(const ManifestBuildData *const buildData, const size_t blockSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buildData); FUNCTION_TEST_PARAM(SIZE, blockSize); FUNCTION_TEST_END(); size_t result = BLOCK_INCR_CHECKSUM_SIZE_MIN; // Search checksum size map for larger value for (unsigned int sizeIdx = 0; sizeIdx < lstSize(buildData->blockIncrMap->checksumSizeMap); sizeIdx++) { const ManifestBlockIncrChecksumSizeMap *const sizeMap = lstGet(buildData->blockIncrMap->checksumSizeMap, sizeIdx); if (blockSize >= sizeMap->blockSize) { result = sizeMap->checksumSize; break; } } FUNCTION_TEST_RETURN(SIZE, result); } // Process files/links/paths and add them to the manifest static void manifestBuildInfo( ManifestBuildData *const buildData, const String *manifestParentName, const String *pgPath, const bool dbPath, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buildData); FUNCTION_TEST_PARAM(STRING, manifestParentName); FUNCTION_TEST_PARAM(STRING, pgPath); FUNCTION_TEST_PARAM(BOOL, dbPath); FUNCTION_TEST_PARAM(STORAGE_INFO, *info); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(buildData != NULL); ASSERT(manifestParentName != NULL); ASSERT(pgPath != NULL); ASSERT(info != NULL); // Skip any path/file/link that begins with pgsql_tmp. The files are removed when the server is restarted and the directories // are recreated. if (strBeginsWithZ(info->name, PG_PREFIX_PGSQLTMP)) FUNCTION_TEST_RETURN_VOID(); // Get build data const unsigned int pgVersion = buildData->manifest->pub.data.pgVersion; // Construct the name used to identify this file/link/path in the manifest const String *manifestName = strNewFmt("%s/%s", strZ(manifestParentName), strZ(info->name)); // Skip excluded files/links/paths if (buildData->excludeSingle != NULL && strLstExists(buildData->excludeSingle, manifestName)) { LOG_INFO_FMT( "exclude '%s/%s' from backup using '%s' exclusion", strZ(pgPath), strZ(info->name), strZ(strSub(manifestName, sizeof(MANIFEST_TARGET_PGDATA)))); FUNCTION_TEST_RETURN_VOID(); } // Process storage types switch (info->type) { // Add paths // ------------------------------------------------------------------------------------------------------------------------- case storageTypePath: { // There should not be any paths in pg_tblspc if (strEqZ(manifestParentName, MANIFEST_TARGET_PGDATA "/" MANIFEST_TARGET_PGTBLSPC)) { THROW_FMT( LinkExpectedError, "'%s' is not a symlink - " MANIFEST_TARGET_PGTBLSPC " should contain only symlinks", strZ(manifestName)); } // Add path to manifest const ManifestPath path = { .name = manifestName, .mode = info->mode, .user = info->user, .group = info->group, }; manifestPathAdd(buildData->manifest, &path); // Skip excluded path content if (buildData->excludeContent != NULL && strLstExists(buildData->excludeContent, manifestName)) { LOG_INFO_FMT( "exclude contents of '%s/%s' from backup using '%s/' exclusion", strZ(pgPath), strZ(info->name), strZ(strSub(manifestName, sizeof(MANIFEST_TARGET_PGDATA)))); FUNCTION_TEST_RETURN_VOID(); } // Skip the contents of these paths if they exist in the base path since they won't be reused after recovery if (strEq(manifestParentName, MANIFEST_TARGET_PGDATA_STR)) { // Skip pg_dynshmem/* since these files cannot be reused on recovery if (strEqZ(info->name, PG_PATH_PGDYNSHMEM)) FUNCTION_TEST_RETURN_VOID(); // Skip pg_notify/* since these files cannot be reused on recovery if (strEqZ(info->name, PG_PATH_PGNOTIFY)) FUNCTION_TEST_RETURN_VOID(); // Skip pg_replslot/* since these files are generally not useful after a restore if (strEqZ(info->name, PG_PATH_PGREPLSLOT)) FUNCTION_TEST_RETURN_VOID(); // Skip pg_serial/* since these files are reset if (strEqZ(info->name, PG_PATH_PGSERIAL)) FUNCTION_TEST_RETURN_VOID(); // Skip pg_snapshots/* since these files cannot be reused on recovery if (strEqZ(info->name, PG_PATH_PGSNAPSHOTS)) FUNCTION_TEST_RETURN_VOID(); // Skip temporary statistics in pg_stat_tmp even when stats_temp_directory is set because PGSS_TEXT_FILE is always // created there in PostgreSQL < 15. PostgreSQL >= 15 no longer uses this directory, but it may be used by // extensions such as pg_stat_statements so it should still be excluded. if (strEqZ(info->name, PG_PATH_PGSTATTMP)) FUNCTION_TEST_RETURN_VOID(); // Skip pg_subtrans/* since these files are reset if (strEqZ(info->name, PG_PATH_PGSUBTRANS)) FUNCTION_TEST_RETURN_VOID(); } // Skip the contents of archive_status when online if (buildData->online && strEq(manifestParentName, buildData->manifestWalName) && strEqZ(info->name, PG_PATH_ARCHIVE_STATUS)) { FUNCTION_TEST_RETURN_VOID(); } // Recurse into the path const String *const pgPathSub = strNewFmt("%s/%s", strZ(pgPath), strZ(info->name)); const bool dbPathSub = regExpMatch(buildData->dbPathExp, manifestName); StorageIterator *const storageItr = storageNewItrP(buildData->storagePg, pgPathSub, .sortOrder = sortOrderAsc); MEM_CONTEXT_TEMP_RESET_BEGIN() { while (storageItrMore(storageItr)) { const StorageInfo info = storageItrNext(storageItr); manifestBuildInfo(buildData, manifestName, pgPathSub, dbPathSub, &info); // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); break; } // Add files // ------------------------------------------------------------------------------------------------------------------------- case storageTypeFile: { // There should not be any files in pg_tblspc if (strEqZ(manifestParentName, MANIFEST_TARGET_PGDATA "/" MANIFEST_TARGET_PGTBLSPC)) { THROW_FMT( LinkExpectedError, "'%s' is not a symlink - " MANIFEST_TARGET_PGTBLSPC " should contain only symlinks", strZ(manifestName)); } // Skip pg_internal.init since it is recreated on startup. It's also possible, (though unlikely) that a temp file with // the creating process id as the extension can exist so skip that as well. This seems to be a bug in PostgreSQL since // the temp file should be removed on startup. Use regExpMatchOne() here instead of preparing a regexp in advance since // the likelihood of needing the regexp should be very small. if (dbPath && strBeginsWithZ(info->name, PG_FILE_PGINTERNALINIT) && (strSize(info->name) == sizeof(PG_FILE_PGINTERNALINIT) - 1 || regExpMatchOne(STRDEF("\\.[0-9]+"), strSub(info->name, sizeof(PG_FILE_PGINTERNALINIT) - 1)))) { FUNCTION_TEST_RETURN_VOID(); } // Skip files in the root data path if (strEq(manifestParentName, MANIFEST_TARGET_PGDATA_STR)) { // Skip recovery files if (((strEqZ(info->name, PG_FILE_RECOVERYSIGNAL) || strEqZ(info->name, PG_FILE_STANDBYSIGNAL)) && pgVersion >= PG_VERSION_12) || ((strEqZ(info->name, PG_FILE_RECOVERYCONF) || strEqZ(info->name, PG_FILE_RECOVERYDONE)) && pgVersion < PG_VERSION_12) || // Skip temp file for safely writing postgresql.auto.conf (strEqZ(info->name, PG_FILE_POSTGRESQLAUTOCONFTMP)) || // Skip obsolete backup_label/backup_label.old strEqZ(info->name, PG_FILE_BACKUPLABEL) || strEqZ(info->name, PG_FILE_BACKUPLABELOLD) || // Skip backup_manifest/tmp in versions where it is created ((strEqZ(info->name, PG_FILE_BACKUPMANIFEST) || strEqZ(info->name, PG_FILE_BACKUPMANIFEST_TMP)) && pgVersion >= PG_VERSION_13) || // Skip running process options strEqZ(info->name, PG_FILE_POSTMTROPTS) || // Skip process id file to avoid confusing postgres after restore strEqZ(info->name, PG_FILE_POSTMTRPID)) { FUNCTION_TEST_RETURN_VOID(); } } // Skip the contents of the wal path when online. WAL will be restored from the archive or stored in the wal directory // at the end of the backup if the archive-copy option is set. if (buildData->online && strEq(manifestParentName, buildData->manifestWalName)) FUNCTION_TEST_RETURN_VOID(); // Skip temp relations in db paths if (dbPath && regExpMatch(buildData->tempRelationExp, info->name)) FUNCTION_TEST_RETURN_VOID(); // Add file to manifest ManifestFile file = { .name = manifestName, .copy = true, .mode = info->mode, .user = info->user, .group = info->group, .size = info->size, .sizeOriginal = info->size, .sizeRepo = info->size, .timestamp = info->timeModified, }; // When bundling zero-length files do not need to be copied if (info->size == 0 && buildData->manifest->pub.data.bundle) { file.copy = false; file.checksumSha1 = bufPtrConst(HASH_TYPE_SHA1_ZERO_BUF); } // Get block incremental size if (info->size != 0 && buildData->manifest->pub.data.blockIncr) { file.blockIncrSize = manifestBuildBlockIncrSize(buildData, &file); file.blockIncrChecksumSize = manifestBuildBlockIncrChecksumSize(buildData, file.blockIncrSize); } // Determine if this file should be page checksummed if (dbPath && buildData->checksumPage) { file.checksumPage = !strEndsWithZ(manifestName, "/" PG_FILE_PGFILENODEMAP) && !strEndsWithZ(manifestName, "/" PG_FILE_PGVERSION) && !strEqZ(manifestName, MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL); } manifestFileAdd(buildData->manifest, &file); break; } // Add links // ------------------------------------------------------------------------------------------------------------------------- case storageTypeLink: { // If the destination is another link then error. In the future we'll allow this by following the link chain to the // eventual destination but for now we are trying to maintain compatibility during the migration. To do this check we // need to read outside of the data directory but it is a read-only operation so is considered safe. const String *const linkDestinationAbsolute = strPathAbsolute(info->linkDestination, pgPath); const StorageInfo linkedCheck = storageInfoP( buildData->storagePg, linkDestinationAbsolute, .ignoreMissing = true, .noPathEnforce = true); if (linkedCheck.exists && linkedCheck.type == storageTypeLink) { THROW_FMT( LinkDestinationError, "link '%s/%s' cannot reference another link '%s'", strZ(pgPath), strZ(info->name), strZ(linkDestinationAbsolute)); } // Initialize link and target const ManifestLink link = { .name = manifestName, .user = info->user, .group = info->group, .destination = info->linkDestination, }; ManifestTarget target = { .name = manifestName, .type = manifestTargetTypeLink, }; // Make a copy of the link name because it will need to be modified when there are tablespace ids const String *linkName = info->name; // Is this a tablespace? if (strEq(manifestParentName, STRDEF(MANIFEST_TARGET_PGDATA "/" MANIFEST_TARGET_PGTBLSPC))) { // Strip pg_data off the manifest name so it begins with pg_tblspc instead. This reflects how the files are stored // in the backup directory. manifestName = strSub(manifestName, sizeof(MANIFEST_TARGET_PGDATA)); // Identify this target as a tablespace target.name = manifestName; target.tablespaceId = cvtZToUInt(strZ(info->name)); // Look for this tablespace in the provided list (list may be null for off-line backup) if (buildData->tablespaceList != NULL) { // Search list PackRead *const read = pckReadNew(buildData->tablespaceList); while (!pckReadNullP(read)) { pckReadArrayBeginP(read); if (target.tablespaceId == pckReadU32P(read)) target.tablespaceName = pckReadStrP(read); pckReadArrayEndP(read); } // Error if the tablespace could not be found. ??? This seems excessive, perhaps just warn here? if (target.tablespaceName == NULL) { THROW_FMT( AssertError, "tablespace with oid %u not found in tablespace map\n" "HINT: was a tablespace created or dropped during the backup?", target.tablespaceId); } } // If no tablespace name was found then create one if (target.tablespaceName == NULL) target.tablespaceName = strNewFmt("ts%s", strZ(info->name)); // Add a dummy pg_tblspc path entry if it does not already exist. This entry will be ignored by restore but it is // part of the original manifest format so we need to have it. lstSort(buildData->manifest->pub.pathList, sortOrderAsc); const ManifestPath *const pathBase = manifestPathFind(buildData->manifest, MANIFEST_TARGET_PGDATA_STR); if (manifestPathFindDefault(buildData->manifest, MANIFEST_TARGET_PGTBLSPC_STR, NULL) == NULL) { ManifestPath path = { .name = MANIFEST_TARGET_PGTBLSPC_STR, .mode = pathBase->mode, .user = pathBase->user, .group = pathBase->group, }; manifestPathAdd(buildData->manifest, &path); } // The tablespace link destination path is not the path where data will be stored so we can just store it as a dummy // path. This is because PostgreSQL creates a subpath with the version/catalog number so that multiple versions of // PostgreSQL can share a tablespace, which makes upgrades easier. const ManifestPath *const pathTblSpc = manifestPathFind( buildData->manifest, STRDEF(MANIFEST_TARGET_PGDATA "/" MANIFEST_TARGET_PGTBLSPC)); const ManifestPath path = { .name = manifestName, .mode = pathTblSpc->mode, .user = pathTblSpc->user, .group = pathTblSpc->group, }; manifestPathAdd(buildData->manifest, &path); // Update build structure to reflect the path added above and the tablespace id manifestParentName = manifestName; manifestName = strNewFmt("%s/%s", strZ(manifestName), strZ(buildData->tablespaceId)); pgPath = strNewFmt("%s/%s", strZ(pgPath), strZ(info->name)); linkName = buildData->tablespaceId; } // Add info about the linked file/path const String *const linkPgPath = strNewFmt("%s/%s", strZ(pgPath), strZ(linkName)); StorageInfo linkedInfo = storageInfoP( buildData->storagePg, linkPgPath, .followLink = true, .ignoreMissing = true); linkedInfo.name = linkName; // If the link destination exists then build the target if (linkedInfo.exists) { // If a path link then recurse if (linkedInfo.type == storageTypePath) { target.path = info->linkDestination; } // Else it must be a file or special (since we have already checked if it is a link) else { CHECK(FormatError, target.tablespaceId == 0, "tablespace links to a file"); // Identify target as a file target.path = strPath(info->linkDestination); target.file = strBase(info->linkDestination); } } // Else dummy up the target with a destination so manifestLinkCheck() can be run. This is so errors about links with // destinations in PGDATA will take precedence over missing a destination. We will probably simplify this once the // migration is done and it doesn't matter which error takes precedence. else target.path = info->linkDestination; // Add target and link manifestTargetAdd(buildData->manifest, &target); manifestLinkAdd(buildData->manifest, &link); // Make sure the link is valid manifestLinkCheckOne(buildData->manifest, buildData->linkCheck, manifestTargetTotal(buildData->manifest) - 1); // If the link check was successful but the destination does not exist then check it again to generate an error if (!linkedInfo.exists) storageInfoP(buildData->storagePg, linkPgPath, .followLink = true); // Recurse into the link destination manifestBuildInfo(buildData, manifestParentName, pgPath, dbPath, &linkedInfo); break; } // Skip special files // ------------------------------------------------------------------------------------------------------------------------- case storageTypeSpecial: LOG_WARN_FMT("exclude special file '%s/%s' from backup", strZ(pgPath), strZ(info->name)); break; } FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/manifest/link.c.inc0000664000175100017510000002265315265101745020613 0ustar00runnerrunner/*********************************************************************************************************************************** Ensure that symlinks do not point to the same file, directory, or subdirectory of another link There are two implementations: manifestLinkCheck(), which is externed, and manifestLinkCheckOne(), which is intended to be used internally during processing. manifestLinkCheck() works simply by calling manifestLinkCheckOne() for every link in the target list. manifestLinkCheckOne() is optimized to work quickly on a single link. ***********************************************************************************************************************************/ // Data needed in link list typedef struct ManifestLinkCheckItem { const String *path; // Link destination path terminated with / const String *file; // Link file if a file link unsigned int targetIdx; // Index of target used for error messages } ManifestLinkCheckItem; // Persistent data needed during processing of manifestLinkCheck/One() typedef struct ManifestLinkCheck { const String *basePath; // Base data path (initialized on first call) List *linkList; // Current list of link destination paths } ManifestLinkCheck; // Helper to initialize the link data static ManifestLinkCheck manifestLinkCheckInit(void) { FUNCTION_TEST_VOID(); FUNCTION_AUDIT_STRUCT(); FUNCTION_TEST_RETURN_TYPE( ManifestLinkCheck, (ManifestLinkCheck){.linkList = lstNewP(sizeof(ManifestLinkCheckItem), .comparator = lstComparatorStr)}); } // Helper to check a single link specified by targetIdx static void manifestLinkCheckOne(const Manifest *const this, ManifestLinkCheck *const linkCheck, const unsigned int targetIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM_P(VOID, linkCheck); FUNCTION_LOG_PARAM(UINT, targetIdx); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(linkCheck != NULL); ASSERT(linkCheck->linkList != NULL); ASSERT(targetIdx < manifestTargetTotal(this)); MEM_CONTEXT_TEMP_BEGIN() { const ManifestTarget *const target1 = manifestTarget(this, targetIdx); // Only check link targets if (target1->type == manifestTargetTypeLink) { // Create link destination path for comparison with other paths. It must end in / so subpaths can be detected without // matching valid partial path names at the end of the path. const String *path = NULL; MEM_CONTEXT_BEGIN(lstMemContext(linkCheck->linkList)) { path = strNewFmt("%s/", strZ(manifestTargetPath(this, target1))); // Get base bath if (linkCheck->basePath == NULL) { linkCheck->basePath = strNewFmt( "%s/", strZ(manifestTargetPath(this, manifestTargetFind(this, MANIFEST_TARGET_PGDATA_STR)))); } } MEM_CONTEXT_END(); // Check that link destination is not in base data path if (strBeginsWith(path, linkCheck->basePath)) { THROW_FMT( LinkDestinationError, "link '%s' destination '%s' is in PGDATA", strZ(manifestPathPg(target1->name)), strZ(manifestTargetPath(this, target1))); } // Check if the link destination path already exists const ManifestLinkCheckItem *const link = lstFind(linkCheck->linkList, &path); if (link != NULL) { // If both links are files make sure they don't link to the same file if (target1->file != NULL && link->file != NULL) { if (strEq(target1->file, link->file)) { const ManifestTarget *const target2 = manifestTarget(this, link->targetIdx); THROW_FMT( LinkDestinationError, "link '%s' (%s/%s) destination is the same file as link '%s' (%s/%s)", strZ(manifestPathPg(target1->name)), strZ(manifestTargetPath(this, target1)), strZ(target1->file), strZ(manifestPathPg(target2->name)), strZ(manifestTargetPath(this, target2)), strZ(target2->file)); } } // Else error because one of the links is a path and cannot link to the same path as another file/path link else { const ManifestTarget *const target2 = manifestTarget(this, link->targetIdx); THROW_FMT( LinkDestinationError, "link '%s' (%s) destination is the same directory as link '%s' (%s)", strZ(manifestPathPg(target1->name)), strZ(manifestTargetPath(this, target1)), strZ(manifestPathPg(target2->name)), strZ(manifestTargetPath(this, target2))); } } // Else add to the link list and check against other links else { // Add the link destination path and sort lstAdd(linkCheck->linkList, &(ManifestLinkCheckItem){.path = path, .file = target1->file, .targetIdx = targetIdx}); lstSort(linkCheck->linkList, sortOrderAsc); // Find the path in the sorted list const unsigned int linkIdx = lstFindIdx(linkCheck->linkList, &path); ASSERT(linkIdx != LIST_NOT_FOUND); // Check path links against other links (file links have already been checked) if (target1->file == NULL) { // Check the link destination path to be sure it is not a subpath of a prior link destination path for (unsigned int priorLinkIdx = linkIdx - 1; (int)priorLinkIdx >= 0; priorLinkIdx--) { const ManifestLinkCheckItem *const priorLink = lstGet(linkCheck->linkList, priorLinkIdx); // Skip file links since they are allowed to be in the same path with each other and in the parent path of a // linked destination path. if (priorLink->file != NULL) continue; if (strBeginsWith(path, priorLink->path)) { const ManifestTarget *const target2 = manifestTarget(this, priorLink->targetIdx); THROW_FMT( LinkDestinationError, "link '%s' (%s) destination is a subdirectory of link '%s' (%s)", strZ(manifestPathPg(target1->name)), strZ(manifestTargetPath(this, target1)), strZ(manifestPathPg(target2->name)), strZ(manifestTargetPath(this, target2))); } // Stop once the first prior path link has been checked since it must be a parent (if there is one) break; } // Check the link destination path to be sure it is not a parent path of a subsequent link destination path for (unsigned int nextLinkIdx = linkIdx + 1; nextLinkIdx < lstSize(linkCheck->linkList); nextLinkIdx++) { const ManifestLinkCheckItem *const nextLink = lstGet(linkCheck->linkList, nextLinkIdx); // Skip file links since they are allowed to be in the same path with each other and in the parent path of a // linked destination path. if (nextLink->file != NULL) continue; if (strBeginsWith(nextLink->path, path)) { const ManifestTarget *const target2 = manifestTarget(this, nextLink->targetIdx); THROW_FMT( LinkDestinationError, "link '%s' (%s) destination is a subdirectory of link '%s' (%s)", strZ(manifestPathPg(target2->name)), strZ(manifestTargetPath(this, target2)), strZ(manifestPathPg(target1->name)), strZ(manifestTargetPath(this, target1))); } // Stop once the first next path link has been checked since it must be a subpath (if there is one) break; } } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } FN_EXTERN void manifestLinkCheck(const Manifest *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Check all links ManifestLinkCheck linkCheck = manifestLinkCheckInit(); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(this); targetIdx++) manifestLinkCheckOne(this, &linkCheck, targetIdx); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/manifest/manifest.c0000664000175100017510000011322615265101745020711 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Manifest Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/log.h" #include "common/regExp.h" #include "common/type/json.h" #include "common/type/list.h" #include "info/manifest/manifest.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/storage.h" #include "version.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Manifest { ManifestPub pub; // Publicly accessible variables StringList *ownerList; // List of users/groups const String *fileUserDefault; // Default file user name const String *fileGroupDefault; // Default file group name mode_t fileModeDefault; // Default file mode }; // {uncrustify_off - order required for C includes} #include "store.c.inc" #include "link.c.inc" #include "build.c.inc" #include "serialize.c.inc" // {uncrustify_on} /*********************************************************************************************************************************** Internal constructor ***********************************************************************************************************************************/ static Manifest * manifestNewInternal(void) { FUNCTION_TEST_VOID(); FUNCTION_AUDIT_HELPER(); Manifest *this = OBJ_NEW_ALLOC(); *this = (Manifest) { .pub = { .memContext = memContextCurrent(), .dbList = lstNewP(sizeof(ManifestDb), .comparator = lstComparatorStr), .fileList = lstNewP(sizeof(ManifestFilePack *), .comparator = lstComparatorStr), .linkList = lstNewP(sizeof(ManifestLink), .comparator = lstComparatorStr), .pathList = lstNewP(sizeof(ManifestPath), .comparator = lstComparatorStr), .targetList = lstNewP(sizeof(ManifestTarget), .comparator = lstComparatorStr), .referenceList = strLstNew(), }, .ownerList = strLstNew(), }; FUNCTION_TEST_RETURN(MANIFEST, this); } // Regular expression constants #define RELATION_EXP "[0-9]+(_(fsm|vm)){0,1}(\\.[0-9]+){0,1}" #define DB_PATH_EXP \ "(" MANIFEST_TARGET_PGDATA "/(" PG_PATH_GLOBAL "|" PG_PATH_BASE "/[0-9]+)|" MANIFEST_TARGET_PGTBLSPC "/[0-9]+/%s/[0-9]+)" FN_EXTERN Manifest * manifestNewBuild( const Storage *const storagePg, const unsigned int pgVersion, const unsigned int pgCatalogVersion, const unsigned int format, const time_t timestampStart, const bool online, const bool checksumPage, const bool bundle, const bool blockIncr, const ManifestBlockIncrMap *blockIncrMap, const StringList *const excludeList, const Pack *const tablespaceList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storagePg); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT, pgCatalogVersion); FUNCTION_LOG_PARAM(UINT, format); FUNCTION_LOG_PARAM(TIME, timestampStart); FUNCTION_LOG_PARAM(BOOL, online); FUNCTION_LOG_PARAM(BOOL, checksumPage); FUNCTION_LOG_PARAM(BOOL, bundle); FUNCTION_LOG_PARAM(BOOL, blockIncr); FUNCTION_LOG_PARAM(VOID, blockIncrMap); FUNCTION_LOG_PARAM(STRING_LIST, excludeList); FUNCTION_LOG_PARAM(PACK, tablespaceList); FUNCTION_LOG_END(); ASSERT(storagePg != NULL); ASSERT(pgVersion != 0); Manifest *this; OBJ_NEW_BASE_BEGIN(Manifest, .childQty = MEM_CONTEXT_QTY_MAX) { this = manifestNewInternal(); this->pub.info = infoNew(format, NULL); this->pub.data.backrestVersion = strNewZ(PROJECT_VERSION); this->pub.data.pgVersion = pgVersion; this->pub.data.pgCatalogVersion = pgCatalogVersion; this->pub.data.backupTimestampStart = timestampStart; this->pub.data.backupType = backupTypeFull; this->pub.data.backupOptionOnline = online; this->pub.data.backupOptionChecksumPage = varNewBool(checksumPage); this->pub.data.bundle = bundle; this->pub.data.bundleRaw = blockIncr; this->pub.data.blockIncr = blockIncr; MEM_CONTEXT_TEMP_BEGIN() { // Data needed to build the manifest ManifestLinkCheck linkCheck = manifestLinkCheckInit(); ManifestBuildData buildData = { .manifest = this, .storagePg = storagePg, .tablespaceId = pgTablespaceId(pgVersion, pgCatalogVersion), .online = online, .checksumPage = checksumPage, .tablespaceList = tablespaceList, .linkCheck = &linkCheck, .manifestWalName = strNewFmt(MANIFEST_TARGET_PGDATA "/%s", strZ(pgWalPath(pgVersion))), .blockIncrMap = blockIncrMap, }; // Build expressions to identify databases paths and temp relations // --------------------------------------------------------------------------------------------------------------------- ASSERT(buildData.tablespaceId != NULL); // Expression to identify database paths buildData.dbPathExp = regExpNew(strNewFmt("^" DB_PATH_EXP "$", strZ(buildData.tablespaceId))); // Expression to find temp relations buildData.tempRelationExp = regExpNew(STRDEF("^t[0-9]+_" RELATION_EXP "$")); // Build list of exclusions // --------------------------------------------------------------------------------------------------------------------- if (excludeList != NULL) { for (unsigned int excludeIdx = 0; excludeIdx < strLstSize(excludeList); excludeIdx++) { const String *const exclude = strNewFmt(MANIFEST_TARGET_PGDATA "/%s", strZ(strLstGet(excludeList, excludeIdx))); // If the exclusions refers to the contents of a path if (strEndsWithZ(exclude, "/")) { if (buildData.excludeContent == NULL) buildData.excludeContent = strLstNew(); strLstAddSub(buildData.excludeContent, exclude, strSize(exclude) - 1); } // Otherwise exclude a single file/link/path else { if (buildData.excludeSingle == NULL) buildData.excludeSingle = strLstNew(); strLstAdd(buildData.excludeSingle, exclude); } } } // Build manifest // --------------------------------------------------------------------------------------------------------------------- const String *const pgPath = storagePathP(storagePg, NULL); const StorageInfo info = storageInfoP(storagePg, pgPath, .followLink = true); const ManifestPath path = { .name = MANIFEST_TARGET_PGDATA_STR, .mode = info.mode, .user = info.user, .group = info.group, }; manifestPathAdd(this, &path); // Generate file defaults from base path MEM_CONTEXT_BEGIN(this->pub.memContext) { this->fileUserDefault = strDup(path.user); this->fileGroupDefault = strDup(path.group); this->fileModeDefault = path.mode & (S_IRUSR | S_IWUSR | S_IRGRP); } MEM_CONTEXT_END(); const ManifestTarget target = { .name = MANIFEST_TARGET_PGDATA_STR, .path = pgPath, .type = manifestTargetTypePath, }; manifestTargetAdd(this, &target); // Gather info for the rest of the files/links/paths StorageIterator *const storageItr = storageNewItrP( storagePg, pgPath, .errorOnMissing = true, .sortOrder = sortOrderAsc); MEM_CONTEXT_TEMP_RESET_BEGIN() { while (storageItrMore(storageItr)) { const StorageInfo info = storageItrNext(storageItr); manifestBuildInfo(&buildData, MANIFEST_TARGET_PGDATA_STR, pgPath, false, &info); // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); // These may not be in order even if the incoming data was sorted lstSort(this->pub.fileList, sortOrderAsc); lstSort(this->pub.linkList, sortOrderAsc); lstSort(this->pub.pathList, sortOrderAsc); lstSort(this->pub.targetList, sortOrderAsc); // Remove unlogged relations from the manifest. This can't be done during the initial build because of the requirement // to check for _init files which will sort after the vast majority of the relation files. We could check storage for // each _init file but that would be expensive. // --------------------------------------------------------------------------------------------------------------------- RegExp *relationExp = regExpNew(strNewFmt("^" DB_PATH_EXP "/" RELATION_EXP "$", strZ(buildData.tablespaceId))); unsigned int fileIdx = 0; char lastRelationFileId[21] = ""; // Large enough for a 64-bit unsigned integer bool lastRelationFileIdUnlogged = false; #ifdef DEBUG_MEM // Record the temp context size before the loop begins const size_t sizeBegin = memContextSize(memContextCurrent()); #endif while (fileIdx < manifestFileTotal(this)) { // If this file looks like a relation. Note that this never matches on _init forks. const String *const filePathName = manifestFileNameGet(this, fileIdx); if (regExpMatch(relationExp, filePathName)) { // Get the filename (without path) const char *const fileName = strBaseZ(filePathName); const size_t fileNameSize = strlen(fileName); // Strip off the numeric part of the relation char relationFileId[sizeof(lastRelationFileId)]; unsigned int nameIdx = 0; for (; nameIdx < fileNameSize; nameIdx++) { if (!isdigit(fileName[nameIdx])) break; relationFileId[nameIdx] = fileName[nameIdx]; } relationFileId[nameIdx] = '\0'; // The filename must have characters ASSERT(relationFileId[0] != '\0'); // Store the last relation so it does not need to be found every time if (strcmp(lastRelationFileId, relationFileId) != 0) { // Determine if the relation is unlogged String *const relationInit = strNewFmt( "%.*s%s_init", (int)(strSize(filePathName) - fileNameSize), strZ(filePathName), relationFileId); lastRelationFileIdUnlogged = manifestFileExists(this, relationInit); strFree(relationInit); // Save the file id so we don't need to do the lookup next time if it doesn't change memcpy(lastRelationFileId, relationFileId, nameIdx + 1); } // If relation is unlogged then remove it if (lastRelationFileIdUnlogged) { manifestFileRemove(this, filePathName); continue; } } fileIdx++; } #ifdef DEBUG_MEM // Make sure that the temp context did not grow too much during the loop ASSERT(memContextSize(memContextCurrent()) - sizeBegin < 256); #endif } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(MANIFEST, this); } /**********************************************************************************************************************************/ FN_EXTERN void manifestBuildValidate(Manifest *const this, const bool delta, const time_t copyStart, const CompressType compressType) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM(BOOL, delta); FUNCTION_LOG_PARAM(TIME, copyStart); FUNCTION_LOG_PARAM(ENUM, compressType); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(copyStart > 0); MEM_CONTEXT_BEGIN(this->pub.memContext) { // Store the delta option. If true we can skip checks that automatically enable delta. this->pub.data.backupOptionDelta = varNewBool(delta); // If online then add one second to the copy start time to allow for database updates during the last second that the // manifest was being built. It's up to the caller to actually wait the remainder of the second, but for comparison purposes // we want the time when the waiting started. this->pub.data.backupTimestampCopyStart = copyStart + (this->pub.data.backupOptionOnline ? 1 : 0); // This value is not needed in this function, but it is needed for resumed manifests and this is last place to set it before // processing begins this->pub.data.backupOptionCompressType = compressType; } MEM_CONTEXT_END(); // Check the manifest for timestamp anomalies that require a delta backup (if delta is not already specified) if (!varBool(this->pub.data.backupOptionDelta)) { MEM_CONTEXT_TEMP_BEGIN() { for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(this); fileIdx++) { const ManifestFile file = manifestFile(this, fileIdx); // Check for timestamp in the future if (file.timestamp > copyStart) { LOG_WARN_FMT( "file '%s' has timestamp (%" PRId64 ") in the future (relative to copy start %" PRId64 "), enabling delta" " checksum", strZ(manifestPathPg(file.name)), (int64_t)file.timestamp, (int64_t)copyStart); this->pub.data.backupOptionDelta = BOOL_TRUE_VAR; break; } } } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static void manifestDeltaCheck(Manifest *const this, const Manifest *const manifestPrior) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM(MANIFEST, manifestPrior); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // Check for anomalies between manifests if delta is not already enabled if (!varBool(this->pub.data.backupOptionDelta)) { for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(this); fileIdx++) { const ManifestFile file = manifestFile(this, fileIdx); // If file was found in prior manifest then perform checks if (manifestFileExists(manifestPrior, file.name)) { const ManifestFile filePrior = manifestFileFind(manifestPrior, file.name); // Check for timestamp earlier than the prior backup if (file.timestamp < filePrior.timestamp) { LOG_WARN_FMT( "file '%s' has timestamp earlier than prior backup (prior %" PRId64 ", current %" PRId64 "), enabling" " delta checksum", strZ(manifestPathPg(file.name)), (int64_t)filePrior.timestamp, (int64_t)file.timestamp); this->pub.data.backupOptionDelta = BOOL_TRUE_VAR; break; } // Check for size change with no timestamp change if (file.sizeOriginal != filePrior.sizeOriginal && file.timestamp == filePrior.timestamp) { LOG_WARN_FMT( "file '%s' has same timestamp (%" PRId64 ") as prior but different size (prior %" PRIu64 ", current" " %" PRIu64 "), enabling delta checksum", strZ(manifestPathPg(file.name)), (int64_t)file.timestamp, filePrior.sizeOriginal, file.sizeOriginal); this->pub.data.backupOptionDelta = BOOL_TRUE_VAR; break; } } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void manifestBuildIncr( Manifest *const this, const Manifest *const manifestPrior, const BackupType type, const String *const archiveStart) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM(MANIFEST, manifestPrior); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(STRING, archiveStart); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(manifestPrior != NULL); ASSERT(type == backupTypeDiff || type == backupTypeIncr); ASSERT(type != backupTypeDiff || manifestPrior->pub.data.backupType == backupTypeFull); ASSERT(archiveStart == NULL || strSize(archiveStart) == 24); MEM_CONTEXT_BEGIN(this->pub.memContext) { // Set prior backup label this->pub.data.backupLabelPrior = strDup(manifestPrior->pub.data.backupLabel); // Copy reference list this->pub.referenceList = strLstDup(manifestPrior->pub.referenceList); // Set diff/incr backup type this->pub.data.backupType = type; // Bundle raw must not change in a backup set this->pub.data.bundleRaw = manifestPrior->pub.data.bundleRaw; } MEM_CONTEXT_END(); MEM_CONTEXT_TEMP_BEGIN() { // Enable delta if timelines differ if (archiveStart != NULL && manifestData(manifestPrior)->archiveStop != NULL && !strEq(strSubN(archiveStart, 0, 8), strSubN(manifestData(manifestPrior)->archiveStop, 0, 8))) { LOG_WARN_FMT( "a timeline switch has occurred since the %s backup, enabling delta checksum\n" "HINT: this is normal after restoring from backup or promoting a standby.", strZ(manifestData(manifestPrior)->backupLabel)); this->pub.data.backupOptionDelta = BOOL_TRUE_VAR; } // Else enable delta if online differs else if (manifestData(manifestPrior)->backupOptionOnline != this->pub.data.backupOptionOnline) { LOG_WARN_FMT( "the online option has changed since the %s backup, enabling delta checksum", strZ(manifestData(manifestPrior)->backupLabel)); this->pub.data.backupOptionDelta = BOOL_TRUE_VAR; } // Enable delta if/when there are timestamp anomalies manifestDeltaCheck(this, manifestPrior); // Find files to (possibly) reference in the prior manifest const bool delta = varBool(this->pub.data.backupOptionDelta); for (unsigned int fileIdx = 0; fileIdx < lstSize(this->pub.fileList); fileIdx++) { ManifestFile file = manifestFile(this, fileIdx); // Check if a prior file exists for files that will be copied (i.e. not zero-length files when bundling). If a prior // file does exist it may be possible to reference it instead of copying the file. Use the "find or NULL" variant so we // do one binary search instead of the manifestFileExists()/manifestFileFind() pair, which would do two. const ManifestFilePack *const filePackPrior = file.copy ? manifestFilePackFindDefault(manifestPrior, file.name) : NULL; if (filePackPrior != NULL) { const ManifestFile filePrior = manifestFileUnpack(manifestPrior, filePackPrior); // If file size is equal to prior size then the file can be referenced instead of copied if it has not changed (this // must be determined during the backup). const bool fileSizeEqual = file.size == filePrior.size; // If prior file was stored with block incremental and file size is at least prior file block size then preserve // prior values. If file size is less than prior file block size then the entire file will need to be stored and the // map will be useless as long as the file stays at this size. It is not possible to change the block size in a // backup set and the map info will be required to compare against the prior block incremental. const bool fileBlockIncrPreserve = filePrior.blockIncrMapSize > 0 && file.size >= filePrior.blockIncrSize; // Perform delta if enabled and file size is equal to prior but not zero. Files of unequal length are always // different while zero-length files are always the same, so it wastes time to check them. It is possible for a file // to be truncated down to equal the prior file during backup, but the overhead of checking for such an unlikely // event does not seem worth the possible space saved. file.delta = delta && fileSizeEqual && file.size != 0; // Do not copy if size and prior size are both zero. Zero-length files are always equal so the file can simply be // referenced to the prior file. Note that this is only for the case where zero-length files are being explicitly // written to the repo. Bundled zero-length files disable copy at manifest build time and never reference the prior // file, even if it is also zero-length. if (file.size == 0 && filePrior.size == 0) file.copy = false; // If delta is disabled and size/timestamp are equal then the file is not copied if (!file.delta && fileSizeEqual && file.timestamp == filePrior.timestamp) file.copy = false; ASSERT(file.copy || !file.delta); ASSERT(file.copy || fileSizeEqual); ASSERT(!file.delta || fileSizeEqual); // Preserve values if the file will not be copied, is possibly equal to the prior file, or will be stored with block // incremental and the prior file is also stored with block incremental. If the file will not be copied then the // file size must be equal to the prior file so there is no need to check that condition separately. if (fileSizeEqual || fileBlockIncrPreserve) { file.size = filePrior.size; file.sizeRepo = filePrior.sizeRepo; file.checksumSha1 = filePrior.checksumSha1; file.checksumRepoSha1 = filePrior.checksumRepoSha1; file.reference = filePrior.reference != NULL ? filePrior.reference : manifestPrior->pub.data.backupLabel; file.checksumPage = filePrior.checksumPage; file.checksumPageError = filePrior.checksumPageError; file.checksumPageErrorList = filePrior.checksumPageErrorList; file.bundleId = filePrior.bundleId; file.bundleOffset = filePrior.bundleOffset; file.blockIncrSize = filePrior.blockIncrSize; file.blockIncrChecksumSize = filePrior.blockIncrChecksumSize; file.blockIncrMapSize = filePrior.blockIncrMapSize; ASSERT(file.checksumSha1 != NULL); ASSERT( (!file.checksumPage && !file.checksumPageError && file.checksumPageErrorList == NULL) || (file.checksumPage && !file.checksumPageError && file.checksumPageErrorList == NULL) || (file.checksumPage && file.checksumPageError)); ASSERT(file.size == 0 || file.sizeRepo != 0); ASSERT(file.reference != NULL); ASSERT(file.bundleId != 0 || file.bundleOffset == 0); ASSERT( (file.blockIncrSize == 0 && file.blockIncrChecksumSize == 0 && file.blockIncrMapSize == 0) || (file.blockIncrSize > 0 && file.blockIncrChecksumSize > 0 && file.blockIncrMapSize > 0)); } // Update by index since we already have it from the enclosing loop. Avoids by-name search of manifestFileUpdate(). manifestFileUpdateByIdx(this, fileIdx, &file); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void manifestBuildComplete( Manifest *const this, const String *const lsnStart, const String *const archiveStart, const time_t timestampStop, const String *const lsnStop, const String *const archiveStop, const unsigned int pgId, const uint64_t pgSystemId, const Pack *const dbList, const bool optionArchiveCheck, const bool optionArchiveCopy, const size_t optionBufferSize, const unsigned int optionCompressLevel, const unsigned int optionCompressLevelNetwork, const bool optionHardLink, const unsigned int optionProcessMax, const bool optionStandby, const KeyValue *const annotation) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM(STRING, lsnStart); FUNCTION_LOG_PARAM(STRING, archiveStart); FUNCTION_LOG_PARAM(TIME, timestampStop); FUNCTION_LOG_PARAM(STRING, lsnStop); FUNCTION_LOG_PARAM(STRING, archiveStop); FUNCTION_LOG_PARAM(UINT, pgId); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_PARAM(PACK, dbList); FUNCTION_LOG_PARAM(BOOL, optionArchiveCheck); FUNCTION_LOG_PARAM(BOOL, optionArchiveCopy); FUNCTION_LOG_PARAM(SIZE, optionBufferSize); FUNCTION_LOG_PARAM(UINT, optionCompressLevel); FUNCTION_LOG_PARAM(UINT, optionCompressLevelNetwork); FUNCTION_LOG_PARAM(BOOL, optionHardLink); FUNCTION_LOG_PARAM(UINT, optionProcessMax); FUNCTION_LOG_PARAM(BOOL, optionStandby); FUNCTION_LOG_PARAM(KEY_VALUE, annotation); FUNCTION_LOG_END(); MEM_CONTEXT_BEGIN(this->pub.memContext) { // Save info this->pub.data.lsnStart = strDup(lsnStart); this->pub.data.archiveStart = strDup(archiveStart); this->pub.data.backupTimestampStop = timestampStop; this->pub.data.lsnStop = strDup(lsnStop); this->pub.data.archiveStop = strDup(archiveStop); this->pub.data.pgId = pgId; this->pub.data.pgSystemId = pgSystemId; // Save db list if (dbList != NULL) { PackRead *const read = pckReadNew(dbList); while (!pckReadNullP(read)) { pckReadArrayBeginP(read); const unsigned int id = pckReadU32P(read); const String *const name = pckReadStrP(read); const unsigned int lastSystemId = pckReadU32P(read); pckReadArrayEndP(read); manifestDbAdd(this, &(ManifestDb){.id = id, .name = name, .lastSystemId = lastSystemId}); } lstSort(this->pub.dbList, sortOrderAsc); } // Save annotations if (annotation != NULL) { this->pub.data.annotation = varNewKv(kvNew()); KeyValue *const manifestAnnotationKv = varKv(this->pub.data.annotation); const VariantList *const annotationKeyList = kvKeyList(annotation); for (unsigned int keyIdx = 0; keyIdx < varLstSize(annotationKeyList); keyIdx++) { const Variant *const key = varLstGet(annotationKeyList, keyIdx); const Variant *const value = kvGet(annotation, key); // Skip empty values if (!strEmpty(varStr(value))) kvPut(manifestAnnotationKv, key, value); } // Clean field if there are no annotations to save if (varLstSize(kvKeyList(manifestAnnotationKv)) == 0) this->pub.data.annotation = NULL; } // Save options this->pub.data.backupOptionArchiveCheck = optionArchiveCheck; this->pub.data.backupOptionArchiveCopy = optionArchiveCopy; this->pub.data.backupOptionBufferSize = varNewUInt64(optionBufferSize); this->pub.data.backupOptionCompressLevel = varNewUInt(optionCompressLevel); this->pub.data.backupOptionCompressLevelNetwork = varNewUInt(optionCompressLevelNetwork); this->pub.data.backupOptionHardLink = optionHardLink; this->pub.data.backupOptionProcessMax = varNewUInt(optionProcessMax); this->pub.data.backupOptionStandby = varNewBool(optionStandby); } MEM_CONTEXT_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Manifest * manifestNewLoad(IoRead *const read, const CipherSpec *const cipherSpec) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_END(); ASSERT(read != NULL); Manifest *this; OBJ_NEW_BASE_BEGIN(Manifest, .childQty = MEM_CONTEXT_QTY_MAX) { this = manifestNewInternal(); // Load the manifest ManifestLoadData loadData = { .memContext = memContextNewP("load", .childQty = MEM_CONTEXT_QTY_MAX), .manifest = this, }; // Set file defaults that will be updated when we know what the real defaults are. These need to be set to values that are // not valid for actual names or modes. this->fileUserDefault = STRDEF("@"); this->fileGroupDefault = this->fileUserDefault; this->fileModeDefault = (mode_t)-1; MEM_CONTEXT_BEGIN(loadData.memContext) { loadData.linkFoundList = lstNewP(sizeof(ManifestLoadFound)); loadData.pathFoundList = lstNewP(sizeof(ManifestLoadFound)); } MEM_CONTEXT_END(); this->pub.info = infoNewLoad(read, cipherSpec, manifestLoadCallback, &loadData); this->pub.data.backrestVersion = infoBackrestVersion(this->pub.info); // Add the label to the reference list in case the manifest was created before 2.42 when the explicit reference list was // added. Most references are added when the file list is loaded but the current backup will never be referenced from a file // (the reference is assumed) so it must be added here. if (!loadData.referenceListFound) strLstAddIfMissing(this->pub.referenceList, this->pub.data.backupLabel); // Process link defaults for (unsigned int linkIdx = 0; linkIdx < manifestLinkTotal(this); linkIdx++) { ManifestLink *link = lstGet(this->pub.linkList, linkIdx); ManifestLoadFound *found = lstGet(loadData.linkFoundList, linkIdx); if (!found->group) link->group = manifestOwnerCache(this, manifestOwnerGet(loadData.linkGroupDefault)); if (!found->user) link->user = manifestOwnerCache(this, manifestOwnerGet(loadData.linkUserDefault)); } // Process path defaults for (unsigned int pathIdx = 0; pathIdx < manifestPathTotal(this); pathIdx++) { ManifestPath *const path = lstGet(this->pub.pathList, pathIdx); const ManifestLoadFound *const found = lstGet(loadData.pathFoundList, pathIdx); if (!found->group) path->group = manifestOwnerCache(this, manifestOwnerGet(loadData.pathGroupDefault)); if (!found->mode) path->mode = loadData.pathModeDefault; if (!found->user) path->user = manifestOwnerCache(this, manifestOwnerGet(loadData.pathUserDefault)); } // Sort the lists. They should already be sorted in the file but it is possible that this system has a different collation // that renders that sort useless. // // This must happen *after* the default processing because found lists are in natural file order and it is not worth writing // comparator routines for them. lstSort(this->pub.dbList, sortOrderAsc); lstSort(this->pub.fileList, sortOrderAsc); lstSort(this->pub.linkList, sortOrderAsc); lstSort(this->pub.pathList, sortOrderAsc); lstSort(this->pub.targetList, sortOrderAsc); // Make sure the base path exists manifestTargetBase(this); // Discard the context holding temporary load data memContextDiscard(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(MANIFEST, this); } /**********************************************************************************************************************************/ FN_EXTERN void manifestValidate(Manifest *const this, const bool strict) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM(BOOL, strict); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { String *const error = strNew(); // Validate files for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(this); fileIdx++) { const ManifestFile file = manifestFile(this, fileIdx); // All files must have a checksum if (file.checksumSha1 == NULL) strCatFmt(error, "\nmissing checksum for file '%s'", strZ(file.name)); // These are strict checks to be performed only after a backup and before the final manifest save if (strict) { // Zero-length files must have a specific checksum if (file.size == 0 && !bufEq(HASH_TYPE_SHA1_ZERO_BUF, BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE))) { strCatFmt( error, "\ninvalid checksum '%s' for zero size file '%s'", strZ(strNewEncode(encodingHex, BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE))), strZ(file.name)); } // Non-zero size files must have non-zero repo size if (file.sizeRepo == 0 && file.size != 0) strCatFmt(error, "\nrepo size must be > 0 for file '%s'", strZ(file.name)); } } // Throw exception when there are errors if (strSize(error) > 0) THROW_FMT(FormatError, "manifest validation failed:%s", strZ(error)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * manifestPathPg(const String *const manifestPath) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, manifestPath); FUNCTION_TEST_END(); ASSERT(manifestPath != NULL); // If something in pg_data/ if (strBeginsWith(manifestPath, STRDEF(MANIFEST_TARGET_PGDATA "/"))) { FUNCTION_TEST_RETURN(STRING, strNewZ(strZ(manifestPath) + sizeof(MANIFEST_TARGET_PGDATA))); } // Else not pg_data (this is faster since the length of everything else will be different than pg_data) else if (!strEq(manifestPath, MANIFEST_TARGET_PGDATA_STR)) { // A tablespace target is the only valid option if not pg_data or pg_data/ ASSERT( strEq(manifestPath, MANIFEST_TARGET_PGTBLSPC_STR) || strBeginsWith(manifestPath, STRDEF(MANIFEST_TARGET_PGTBLSPC "/"))); FUNCTION_TEST_RETURN(STRING, strDup(manifestPath)); } FUNCTION_TEST_RETURN(STRING, NULL); } /**********************************************************************************************************************************/ FN_EXTERN void manifestBackupLabelSet(Manifest *const this, const String *const backupLabel) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->pub.data.backupLabel == NULL); MEM_CONTEXT_BEGIN(this->pub.memContext) { this->pub.data.backupLabel = strDup(backupLabel); strLstAdd(this->pub.referenceList, backupLabel); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/manifest/manifest.h0000664000175100017510000007222115265101745020715 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Manifest Handler The backup manifest stores a complete list of all files, links, and paths in a backup along with metadata such as checksums, sizes, timestamps, etc. A list of databases is also included for selective restore. The purpose of the manifest is to allow the restore command to confidently reconstruct the PostgreSQL data directory and ensure that nothing is missing or corrupt. It is also useful for reporting, e.g. size of backup, backup time, etc. ***********************************************************************************************************************************/ #ifndef INFO_MANIFEST_MANIFEST_H #define INFO_MANIFEST_MANIFEST_H #include "common/type/string.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define BACKUP_MANIFEST_EXT ".manifest" #define BACKUP_MANIFEST_FILE "backup" BACKUP_MANIFEST_EXT STRING_DECLARE(BACKUP_MANIFEST_FILE_STR); #define MANIFEST_PATH_BUNDLE "bundle" STRING_DECLARE(MANIFEST_PATH_BUNDLE_STR); #define MANIFEST_TARGET_PGDATA "pg_data" STRING_DECLARE(MANIFEST_TARGET_PGDATA_STR); #define MANIFEST_TARGET_PGTBLSPC "pg_tblspc" STRING_DECLARE(MANIFEST_TARGET_PGTBLSPC_STR); // Minimum size for the block incremental checksum #define BLOCK_INCR_CHECKSUM_SIZE_MIN 6 /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Manifest Manifest; #include "command/backup/common.h" #include "common/compress/helper.h" #include "common/crypto/hash.h" #include "common/crypto/spec.h" #include "common/type/object.h" #include "common/type/variant.h" #include "info/info.h" #include "info/infoBackup.h" #include "storage/storage.h" /*********************************************************************************************************************************** Manifest data ***********************************************************************************************************************************/ typedef struct ManifestData { const String *backrestVersion; // pgBackRest version const String *backupLabel; // Backup label (unique identifier for the backup) const String *backupLabelPrior; // Backup label for backup this diff/incr is based on time_t backupTimestampCopyStart; // When did the file copy start? time_t backupTimestampStart; // When did the backup start? time_t backupTimestampStop; // When did the backup stop? BackupType backupType; // Type of backup: full, diff, incr bool bundle; // Does the backup bundle files? bool bundleRaw; // Use raw compress/encrypt for bundling? bool blockIncr; // Does the backup perform block incremental? // ??? Note that these fields are redundant and verbose since storing the start/stop lsn as a uint64 would be sufficient. // However, we currently lack the functions to transform these values back and forth so this will do for now. const String *archiveStart; // First WAL file in the backup const String *archiveStop; // Last WAL file in the backup const String *lsnStart; // Start LSN for the backup const String *lsnStop; // Stop LSN for the backup unsigned int pgId; // PostgreSQL id in backup.info unsigned int pgVersion; // PostgreSQL version uint64_t pgSystemId; // PostgreSQL system identifier unsigned int pgCatalogVersion; // PostgreSQL catalog version const Variant *annotation; // Backup annotation(s) metadata bool backupOptionArchiveCheck; // Will WAL segments be checked at the end of the backup? bool backupOptionArchiveCopy; // Will WAL segments be copied to the backup? const Variant *backupOptionStandby; // Will the backup be performed from a standby? const Variant *backupOptionBufferSize; // Buffer size used for file/protocol operations const Variant *backupOptionChecksumPage; // Will page checksums be verified? CompressType backupOptionCompressType; // Compression type used for the backup const Variant *backupOptionCompressLevel; // Level used for compression (if type not none) const Variant *backupOptionCompressLevelNetwork; // Level used for network compression const Variant *backupOptionDelta; // Will a checksum delta be performed? bool backupOptionHardLink; // Will hardlinks be created in the backup? bool backupOptionOnline; // Will an online backup be performed? const Variant *backupOptionProcessMax; // How many processes will be used for backup? } ManifestData; /*********************************************************************************************************************************** Block incremental size maps ***********************************************************************************************************************************/ // Map file size to block size typedef struct ManifestBlockIncrSizeMap { unsigned int fileSize; // File size unsigned int blockSize; // Block size for files >= file size } ManifestBlockIncrSizeMap; // Map file age to block multiplier typedef struct ManifestBlockIncrAgeMap { uint32_t fileAge; // File age in seconds uint32_t blockMultiplier; // Block multiplier } ManifestBlockIncrAgeMap; // Map block size to checksum size typedef struct ManifestBlockIncrChecksumSizeMap { uint32_t blockSize; uint32_t checksumSize; } ManifestBlockIncrChecksumSizeMap; typedef struct ManifestBlockIncrMap { const List *sizeMap; // Block size map const List *ageMap; // File age map const List *checksumSizeMap; // Checksum size map } ManifestBlockIncrMap; /*********************************************************************************************************************************** Db type ***********************************************************************************************************************************/ typedef struct ManifestDb { const String *name; // Db name (must be first member in struct) unsigned int id; // Db oid unsigned int lastSystemId; // Highest oid used by system objects (deprecated - do not use) } ManifestDb; /*********************************************************************************************************************************** File type ***********************************************************************************************************************************/ typedef struct ManifestFile { const String *name; // File name (must be first member in struct) bool copy : 1; // Should the file be copied (backup only)? bool delta : 1; // Verify checksum in PGDATA before copying (backup only)? bool resume : 1; // Is the file being resumed (backup only)? bool checksumPage : 1; // Does this file have page checksums? bool checksumPageError : 1; // Is there an error in the page checksum? mode_t mode; // File mode const uint8_t *checksumSha1; // SHA1 checksum const uint8_t *checksumRepoSha1; // SHA1 checksum as stored in repo (including compression, etc.) const String *checksumPageErrorList; // List of page checksum errors if there are any const String *user; // User name const String *group; // Group name const String *reference; // Reference to a prior backup uint64_t bundleId; // Bundle id uint64_t bundleOffset; // Bundle offset size_t blockIncrSize; // Size of incremental blocks size_t blockIncrChecksumSize; // Size of incremental block checksum uint64_t blockIncrMapSize; // Block incremental map size // After manifest build size is either equal to sizeOriginal or it is copied from the prior file. After the file is backed up // the size will reflect the actual size found during backup. sizeOriginal is used to make sure all required bytes are copied // from a file even if it is referenced to a prior file. For example, an fsm file from a replica may be smaller than the // original on the primary, but during copy we need to be ready to read more bytes if in fact the size of the fsm has increased // on the replica. sizeRepo records the size of the file in the repository (even if it is part of a bundle). uint64_t size; // Final size (after copy) uint64_t sizeOriginal; // Original size (from manifest build) uint64_t sizeRepo; // Size in repo time_t timestamp; // Original timestamp } ManifestFile; /*********************************************************************************************************************************** Link type ***********************************************************************************************************************************/ typedef struct ManifestLink { const String *name; // Link name (must be first member in struct) const String *destination; // Link destination const String *user; // User name const String *group; // Group name } ManifestLink; /*********************************************************************************************************************************** Path type ***********************************************************************************************************************************/ typedef struct ManifestPath { const String *name; // Path name (must be first member in struct) mode_t mode; // Directory mode const String *user; // User name const String *group; // Group name } ManifestPath; /*********************************************************************************************************************************** Target type ***********************************************************************************************************************************/ typedef enum { manifestTargetTypePath, manifestTargetTypeLink, } ManifestTargetType; typedef struct ManifestTarget { const String *name; // Target name (must be first member in struct) ManifestTargetType type; // Target type const String *path; // Target path (if path or link) const String *file; // Target file (if file link) unsigned int tablespaceId; // Oid if this link is a tablespace const String *tablespaceName; // Name of the tablespace } ManifestTarget; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ // Build a new manifest for a PostgreSQL data directory FN_EXTERN Manifest *manifestNewBuild( const Storage *storagePg, unsigned int pgVersion, unsigned int pgCatalogVersion, unsigned int format, time_t timestampStart, bool online, bool checksumPage, bool bundle, bool blockIncr, const ManifestBlockIncrMap *blockIncrMap, const StringList *excludeList, const Pack *tablespaceList); // Load a manifest from IO FN_EXTERN Manifest *manifestNewLoad(IoRead *read, const CipherSpec *cipherSpec); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct ManifestPub { MemContext *memContext; // Mem context Info *info; // Base info object ManifestData data; // Manifest data and options List *dbList; // List of databases List *fileList; // List of files List *linkList; // List of links List *pathList; // List of paths List *targetList; // List of targets StringList *referenceList; // List of file references } ManifestPub; // Repository format FN_INLINE_ALWAYS unsigned int manifestFormat(const Manifest *const this) { return infoFormat(THIS_PUB(Manifest)->info); } // Get cipher spec for the files this manifest describes, set the pass they are encrypted with FN_INLINE_ALWAYS const CipherSpec * manifestCipherSpec(const Manifest *const this) { return infoCipherSpec(THIS_PUB(Manifest)->info); } FN_INLINE_ALWAYS void manifestCipherSpecSet(Manifest *const this, const CipherSpec *const cipherSpec) { infoCipherSpecSet(THIS_PUB(Manifest)->info, cipherSpec); } // Get manifest configuration and options FN_INLINE_ALWAYS const ManifestData * manifestData(const Manifest *const this) { return &(THIS_PUB(Manifest)->data); } // Get reference list FN_INLINE_ALWAYS const StringList * manifestReferenceList(const Manifest *const this) { return THIS_PUB(Manifest)->referenceList; } // Set backup label FN_EXTERN void manifestBackupLabelSet(Manifest *this, const String *backupLabel); /*********************************************************************************************************************************** Build functions ***********************************************************************************************************************************/ // Validate the timestamps in the manifest given a copy start time, i.e. all times should be <= the copy start time FN_EXTERN void manifestBuildValidate(Manifest *this, bool delta, time_t copyStart, CompressType compressType); // Create a diff/incr backup by comparing to a previous backup manifest FN_EXTERN void manifestBuildIncr(Manifest *this, const Manifest *prior, BackupType type, const String *archiveStart); // Set remaining values before the final save FN_EXTERN void manifestBuildComplete( Manifest *this, const String *lsnStart, const String *archiveStart, time_t timestampStop, const String *lsnStop, const String *archiveStop, unsigned int pgId, uint64_t pgSystemId, const Pack *dbList, bool optionArchiveCheck, bool optionArchiveCopy, size_t optionBufferSize, unsigned int optionCompressLevel, unsigned int optionCompressLevelNetwork, bool optionHardLink, unsigned int optionProcessMax, bool optionStandby, const KeyValue *annotation); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Ensure that symlinks do not point to the same file, directory, or subdirectory of another link FN_EXTERN void manifestLinkCheck(const Manifest *this); // Move to a new parent mem context FN_INLINE_ALWAYS Manifest * manifestMove(Manifest *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Manifest save FN_EXTERN void manifestSave(Manifest *this, IoWrite *write); // Validate a completed manifest. Use strict mode only when saving the manifest after a backup. FN_EXTERN void manifestValidate(Manifest *this, bool strict); /*********************************************************************************************************************************** Db functions and getters/setters ***********************************************************************************************************************************/ FN_INLINE_ALWAYS const ManifestDb * manifestDb(const Manifest *const this, const unsigned int dbIdx) { return lstGet(THIS_PUB(Manifest)->dbList, dbIdx); } // If the database requested is not found in the list, return the default passed rather than throw an error FN_INLINE_ALWAYS const ManifestDb * manifestDbFindDefault(const Manifest *const this, const String *const name, const ManifestDb *const dbDefault) { ASSERT_INLINE(name != NULL); return lstFindDefault(THIS_PUB(Manifest)->dbList, &name, (const void *)dbDefault); } FN_INLINE_ALWAYS unsigned int manifestDbTotal(const Manifest *const this) { return lstSize(THIS_PUB(Manifest)->dbList); } /*********************************************************************************************************************************** File functions and getters/setters ***********************************************************************************************************************************/ typedef struct ManifestFilePack ManifestFilePack; // Unpack file pack returned by manifestFilePackGet() FN_EXTERN ManifestFile manifestFileUnpack(const Manifest *manifest, const ManifestFilePack *filePack); // Get file in pack format by index FN_INLINE_ALWAYS const ManifestFilePack * manifestFilePackGet(const Manifest *const this, const unsigned int fileIdx) { return *(ManifestFilePack **)lstGet(THIS_PUB(Manifest)->fileList, fileIdx); } // Get file name FN_INLINE_ALWAYS const String * manifestFileNameGet(const Manifest *const this, const unsigned int fileIdx) { return (const String *)manifestFilePackGet(this, fileIdx); } // Get file by index FN_INLINE_ALWAYS ManifestFile manifestFile(const Manifest *const this, const unsigned int fileIdx) { return manifestFileUnpack(this, manifestFilePackGet(this, fileIdx)); } // Add a file FN_EXTERN void manifestFileAdd(Manifest *this, ManifestFile *file); // Find file in pack format by name FN_EXTERN const ManifestFilePack *manifestFilePackFind(const Manifest *this, const String *name); // Find file in pack format by name, returning NULL when missing. Cheaper than calling manifestFileExists() followed by // manifestFilePackFind() because it does only one binary search. FN_INLINE_ALWAYS const ManifestFilePack * manifestFilePackFindDefault(const Manifest *const this, const String *const name) { ASSERT_INLINE(name != NULL); const ManifestFilePack *const *const filePack = lstFindDefault(THIS_PUB(Manifest)->fileList, &name, NULL); return filePack == NULL ? NULL : *filePack; } // Find file by name FN_INLINE_ALWAYS ManifestFile manifestFileFind(const Manifest *const this, const String *const name) { ASSERT_INLINE(name != NULL); return manifestFileUnpack(this, manifestFilePackFind(this, name)); } // Does the file exist? FN_INLINE_ALWAYS bool manifestFileExists(const Manifest *const this, const String *const name) { ASSERT_INLINE(name != NULL); return lstFindDefault(THIS_PUB(Manifest)->fileList, &name, NULL) != NULL; } FN_EXTERN void manifestFileRemove(const Manifest *this, const String *name); FN_INLINE_ALWAYS unsigned int manifestFileTotal(const Manifest *const this) { return lstSize(THIS_PUB(Manifest)->fileList); } // Update a file with new data FN_EXTERN void manifestFileUpdate(Manifest *this, const ManifestFile *file); /*********************************************************************************************************************************** Link functions and getters/setters ***********************************************************************************************************************************/ FN_INLINE_ALWAYS const ManifestLink * manifestLink(const Manifest *const this, const unsigned int linkIdx) { return lstGet(THIS_PUB(Manifest)->linkList, linkIdx); } FN_EXTERN void manifestLinkAdd(Manifest *this, const ManifestLink *link); FN_EXTERN ManifestLink *manifestLinkFind(const Manifest *this, const String *name); // If the link requested is not found in the list, return the default passed rather than throw an error FN_INLINE_ALWAYS const ManifestLink * manifestLinkFindDefault(const Manifest *const this, const String *const name, const ManifestLink *const linkDefault) { ASSERT_INLINE(name != NULL); return lstFindDefault(THIS_PUB(Manifest)->linkList, &name, (const void *)linkDefault); } FN_EXTERN void manifestLinkRemove(const Manifest *this, const String *name); FN_INLINE_ALWAYS unsigned int manifestLinkTotal(const Manifest *const this) { return lstSize(THIS_PUB(Manifest)->linkList); } FN_EXTERN void manifestLinkUpdate(const Manifest *this, const String *name, const String *path); /*********************************************************************************************************************************** Path functions and getters/setters ***********************************************************************************************************************************/ FN_INLINE_ALWAYS const ManifestPath * manifestPath(const Manifest *const this, const unsigned int pathIdx) { return lstGet(THIS_PUB(Manifest)->pathList, pathIdx); } FN_EXTERN const ManifestPath *manifestPathFind(const Manifest *this, const String *name); // If the path requested is not found in the list, return the default passed rather than throw an error FN_INLINE_ALWAYS const ManifestPath * manifestPathFindDefault(const Manifest *const this, const String *const name, const ManifestPath *const pathDefault) { ASSERT_INLINE(name != NULL); return lstFindDefault(THIS_PUB(Manifest)->pathList, &name, (const void *)pathDefault); } // Data directory relative path for any manifest file/link/path/target name FN_EXTERN String *manifestPathPg(const String *manifestPath); FN_INLINE_ALWAYS unsigned int manifestPathTotal(const Manifest *const this) { return lstSize(THIS_PUB(Manifest)->pathList); } /*********************************************************************************************************************************** Target functions and getters/setters ***********************************************************************************************************************************/ FN_INLINE_ALWAYS const ManifestTarget * manifestTarget(const Manifest *const this, const unsigned int targetIdx) { return lstGet(THIS_PUB(Manifest)->targetList, targetIdx); } FN_EXTERN void manifestTargetAdd(Manifest *this, const ManifestTarget *target); FN_EXTERN ManifestTarget *manifestTargetFind(const Manifest *this, const String *name); // If the target requested is not found in the list, return the default passed rather than throw an error FN_INLINE_ALWAYS const ManifestTarget * manifestTargetFindDefault(const Manifest *const this, const String *const name, const ManifestTarget *const targetDefault) { ASSERT_INLINE(name != NULL); return lstFindDefault(THIS_PUB(Manifest)->targetList, &name, (const void *)targetDefault); } // Base target, i.e. the target that is the data directory FN_INLINE_ALWAYS const ManifestTarget * manifestTargetBase(const Manifest *const this) { return manifestTargetFind(this, MANIFEST_TARGET_PGDATA_STR); } // Absolute path to the target FN_EXTERN String *manifestTargetPath(const Manifest *this, const ManifestTarget *target); FN_EXTERN void manifestTargetRemove(const Manifest *this, const String *name); FN_INLINE_ALWAYS unsigned int manifestTargetTotal(const Manifest *const this) { return lstSize(THIS_PUB(Manifest)->targetList); } FN_EXTERN void manifestTargetUpdate(const Manifest *this, const String *name, const String *path, const String *file); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void manifestFree(Manifest *const this) { objFree(this); } /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Load backup manifest FN_EXTERN Manifest *manifestLoadFile(const Storage *storage, const String *fileName, const CipherSpec *cipherSpec); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_MANIFEST_TYPE \ Manifest * #define FUNCTION_LOG_MANIFEST_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Manifest", buffer, bufferSize) #define FUNCTION_LOG_MANIFEST_DB_TYPE \ ManifestDb * #define FUNCTION_LOG_MANIFEST_DB_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ManifestDb", buffer, bufferSize) #define FUNCTION_LOG_MANIFEST_FILE_TYPE \ ManifestFile * #define FUNCTION_LOG_MANIFEST_FILE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ManifestFile", buffer, bufferSize) #define FUNCTION_LOG_MANIFEST_LINK_TYPE \ ManifestLink * #define FUNCTION_LOG_MANIFEST_LINK_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ManifestLink", buffer, bufferSize) #define FUNCTION_LOG_MANIFEST_PATH_TYPE \ ManifestPath * #define FUNCTION_LOG_MANIFEST_PATH_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ManifestPath", buffer, bufferSize) #define FUNCTION_LOG_MANIFEST_TARGET_TYPE \ ManifestTarget * #define FUNCTION_LOG_MANIFEST_TARGET_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ManifestTarget", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/manifest/serialize.c.inc0000664000175100017510000014720715265101745021650 0ustar00runnerrunner/**********************************************************************************************************************************/ #define MANIFEST_TARGET_TYPE_LINK "link" #define MANIFEST_TARGET_TYPE_PATH "path" #define MANIFEST_SECTION_BACKUP "backup" #define MANIFEST_SECTION_BACKUP_DB "backup:db" #define MANIFEST_SECTION_BACKUP_OPTION "backup:option" #define MANIFEST_SECTION_BACKUP_TARGET "backup:target" #define MANIFEST_SECTION_DB "db" #define MANIFEST_SECTION_METADATA "metadata" #define MANIFEST_SECTION_TARGET_FILE "target:file" #define MANIFEST_SECTION_TARGET_FILE_DEFAULT "target:file:default" #define MANIFEST_SECTION_TARGET_LINK "target:link" #define MANIFEST_SECTION_TARGET_LINK_DEFAULT "target:link:default" #define MANIFEST_SECTION_TARGET_PATH "target:path" #define MANIFEST_SECTION_TARGET_PATH_DEFAULT "target:path:default" #define MANIFEST_KEY_ANNOTATION "annotation" #define MANIFEST_KEY_BACKUP_ARCHIVE_START "backup-archive-start" #define MANIFEST_KEY_BACKUP_ARCHIVE_STOP "backup-archive-stop" #define MANIFEST_KEY_BACKUP_BLOCK_INCR "backup-block-incr" #define MANIFEST_KEY_BACKUP_BUNDLE "backup-bundle" #define MANIFEST_KEY_BACKUP_BUNDLE_RAW "backup-bundle-raw" #define MANIFEST_KEY_BACKUP_LABEL "backup-label" #define MANIFEST_KEY_BACKUP_LSN_START "backup-lsn-start" #define MANIFEST_KEY_BACKUP_LSN_STOP "backup-lsn-stop" #define MANIFEST_KEY_BACKUP_PRIOR "backup-prior" #define MANIFEST_KEY_BACKUP_REFERENCE "backup-reference" #define MANIFEST_KEY_BACKUP_TIMESTAMP_COPY_START "backup-timestamp-copy-start" #define MANIFEST_KEY_BACKUP_TIMESTAMP_START "backup-timestamp-start" #define MANIFEST_KEY_BACKUP_TIMESTAMP_STOP "backup-timestamp-stop" #define MANIFEST_KEY_BACKUP_TYPE "backup-type" #define MANIFEST_KEY_BLOCK_INCR STRID5("bi", 0x1220) #define MANIFEST_KEY_BLOCK_INCR_CHECKSUM STRID5("bic", 0xd220) #define MANIFEST_KEY_BLOCK_INCR_MAP STRID5("bim", 0x35220) #define MANIFEST_KEY_BUNDLE_ID STRID5("bni", 0x25c20) #define MANIFEST_KEY_BUNDLE_OFFSET STRID5("bno", 0x3dc20) #define MANIFEST_KEY_CHECKSUM STRID5("checksum", 0x6d66b195030) #define MANIFEST_KEY_CHECKSUM_REPO STRID5("rck", 0x2c720) #define MANIFEST_KEY_CHECKSUM_PAGE "checksum-page" #define MANIFEST_KEY_CHECKSUM_PAGE_ERROR "checksum-page-error" #define MANIFEST_KEY_DB_CATALOG_VERSION "db-catalog-version" #define MANIFEST_KEY_DB_ID "db-id" #define MANIFEST_KEY_DB_LAST_SYSTEM_ID "db-last-system-id" #define MANIFEST_KEY_DB_SYSTEM_ID "db-system-id" #define MANIFEST_KEY_DB_VERSION "db-version" #define MANIFEST_KEY_DESTINATION STRID5("destination", 0x39e9a05c9a4ca40) #define MANIFEST_KEY_FILE STRID5("file", 0x2b1260) #define MANIFEST_KEY_GROUP "group" #define MANIFEST_KEY_MODE "mode" #define MANIFEST_KEY_PATH STRID5("path", 0x450300) #define MANIFEST_KEY_REFERENCE STRID5("reference", 0x51b8b2298b20) #define MANIFEST_KEY_SIZE STRID5("size", 0x2e9330) #define MANIFEST_KEY_SIZE_ORIGINAL STRID5("szo", 0x3f530) #define MANIFEST_KEY_SIZE_REPO STRID5("repo-size", 0x5d267b7c0b20) #define MANIFEST_KEY_TABLESPACE_ID "tablespace-id" #define MANIFEST_KEY_TABLESPACE_NAME "tablespace-name" #define MANIFEST_KEY_TIMESTAMP STRID5("timestamp", 0x10686932b5340) #define MANIFEST_KEY_TYPE STRID5("type", 0x2c3340) #define MANIFEST_KEY_USER "user" #define MANIFEST_KEY_OPTION_ARCHIVE_CHECK "option-archive-check" #define MANIFEST_KEY_OPTION_ARCHIVE_COPY "option-archive-copy" #define MANIFEST_KEY_OPTION_BACKUP_STANDBY "option-backup-standby" #define MANIFEST_KEY_OPTION_BUFFER_SIZE "option-buffer-size" #define MANIFEST_KEY_OPTION_CHECKSUM_PAGE "option-checksum-page" #define MANIFEST_KEY_OPTION_COMPRESS "option-compress" #define MANIFEST_KEY_OPTION_COMPRESS_TYPE "option-compress-type" #define MANIFEST_KEY_OPTION_COMPRESS_LEVEL "option-compress-level" #define MANIFEST_KEY_OPTION_COMPRESS_LEVEL_NETWORK "option-compress-level-network" #define MANIFEST_KEY_OPTION_DELTA "option-delta" #define MANIFEST_KEY_OPTION_HARDLINK "option-hardlink" #define MANIFEST_KEY_OPTION_ONLINE "option-online" #define MANIFEST_KEY_OPTION_PROCESS_MAX "option-process-max" // Keep track of which values were found during load and which need to be loaded from defaults. There is no point in having // multiple structs since most of the fields are the same and the size shouldn't be more than 4/8 bytes. typedef struct ManifestLoadFound { bool group : 1; bool mode : 1; bool user : 1; } ManifestLoadFound; typedef struct ManifestLoadData { MemContext *memContext; // Mem context for data needed only during load Manifest *manifest; // Manifest info bool referenceListFound; // Was a reference list found? List *linkFoundList; // Values found in links const Variant *linkGroupDefault; // Link default group const Variant *linkUserDefault; // Link default user List *pathFoundList; // Values found in paths const Variant *pathGroupDefault; // Path default group mode_t pathModeDefault; // Path default mode const Variant *pathUserDefault; // Path default user } ManifestLoadData; // Helper to transform a variant that could be boolean or string into a string. If the boolean is false return NULL else return the // string. The boolean cannot be true. static const String * manifestOwnerGet(const Variant *const owner) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, owner); FUNCTION_TEST_END(); ASSERT(owner != NULL); // If bool then it should be false. This indicates that the owner could not be mapped to a name during the backup. if (varType(owner) == varTypeBool) { CHECK(FormatError, !varBool(owner), "owner bool must be false"); FUNCTION_TEST_RETURN_CONST(STRING, NULL); } FUNCTION_TEST_RETURN_CONST(STRING, varStr(owner)); } // Helper to check the variant type of owner and duplicate (call in the containing context) static const Variant * manifestOwnerDefaultGet(const Variant *const ownerDefault) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, ownerDefault); FUNCTION_TEST_END(); ASSERT(ownerDefault != NULL); // Bool = false means the owner was not mapped to a name if (varType(ownerDefault) == varTypeBool) { // Value must be false CHECK(FormatError, !varBool(ownerDefault), "owner bool must be false"); FUNCTION_TEST_RETURN_CONST(VARIANT, BOOL_FALSE_VAR); } // Return a duplicate of the owner passed in FUNCTION_TEST_RETURN_CONST(VARIANT, varDup(ownerDefault)); } static void manifestLoadCallback(void *const callbackData, const String *const section, const String *const key, JsonRead *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(JSON_READ, json); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(callbackData != NULL); ASSERT(section != NULL); ASSERT(key != NULL); ASSERT(json != NULL); ManifestLoadData *const loadData = (ManifestLoadData *)callbackData; Manifest *const manifest = loadData->manifest; // ----------------------------------------------------------------------------------------------------------------------------- if (strEqZ(section, MANIFEST_SECTION_TARGET_FILE)) { ManifestFile file = {.name = key}; jsonReadObjectBegin(json); // Block incremental info if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_BLOCK_INCR)) { file.blockIncrSize = (size_t)jsonReadUInt64(json) * BLOCK_INCR_SIZE_FACTOR; if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_BLOCK_INCR_CHECKSUM)) file.blockIncrChecksumSize = (size_t)jsonReadUInt64(json); else file.blockIncrChecksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN; if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_BLOCK_INCR_MAP)) file.blockIncrMapSize = jsonReadUInt64(json); } // Bundle info if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_BUNDLE_ID)) { file.bundleId = jsonReadUInt64(json); if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_BUNDLE_OFFSET)) file.bundleOffset = jsonReadUInt64(json); } // The checksum might not exist if this is a partial save that was done during the backup to preserve checksums for already // backed up files if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_CHECKSUM)) file.checksumSha1 = bufPtr(bufNewDecode(encodingHex, jsonReadStr(json))); // Page checksum errors if (jsonReadKeyExpectZ(json, MANIFEST_KEY_CHECKSUM_PAGE)) { file.checksumPage = true; file.checksumPageError = !jsonReadBool(json); if (jsonReadKeyExpectZ(json, MANIFEST_KEY_CHECKSUM_PAGE_ERROR)) file.checksumPageErrorList = jsonFromVar(jsonReadVar(json)); } // Group if (jsonReadKeyExpectZ(json, MANIFEST_KEY_GROUP)) file.group = manifestOwnerGet(jsonReadVar(json)); else file.group = manifest->fileGroupDefault; // Mode if (jsonReadKeyExpectZ(json, MANIFEST_KEY_MODE)) file.mode = cvtZToMode(strZ(jsonReadStr(json))); else file.mode = manifest->fileModeDefault; // The repo checksum might not exist if this is a partial save that was done during the backup to preserve checksums for // already backed up files or if this is an older manifest if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_CHECKSUM_REPO)) file.checksumRepoSha1 = bufPtr(bufNewDecode(encodingHex, jsonReadStr(json))); // Reference if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_REFERENCE)) { file.reference = jsonReadStr(json); if (!loadData->referenceListFound) file.reference = strLstAddIfMissing(manifest->pub.referenceList, file.reference); } // If "repo-size" is not present in the manifest file, then it is the same as size (i.e. uncompressed) - to save space, // the repo-size is only stored in the manifest file if it is different than size. const bool sizeRepoExists = jsonReadKeyExpectStrId(json, MANIFEST_KEY_SIZE_REPO); if (sizeRepoExists) file.sizeRepo = jsonReadUInt64(json); // Size is required so error if it is not present. Older versions removed the size before the backup to ensure that the // manifest was updated during the backup, so size can be missing in partial manifests. This error will prevent older // partials from being resumed. if (!jsonReadKeyExpectStrId(json, MANIFEST_KEY_SIZE)) THROW_FMT(FormatError, "missing size for file '%s'", strZ(key)); file.size = jsonReadUInt64(json); // If repo size did not exist then if (!sizeRepoExists) file.sizeRepo = file.size; // If file size is zero then assign the static zero hash if (file.size == 0) file.checksumSha1 = bufPtrConst(HASH_TYPE_SHA1_ZERO_BUF); // If original is not present in the manifest file then it is the same as size (i.e. the file did not change size during // copy) -- to save space the original size is only stored in the manifest file if it is different than size. if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_SIZE_ORIGINAL)) file.sizeOriginal = jsonReadUInt64(json); else file.sizeOriginal = file.size; // Timestamp is required so error if it is not present if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_TIMESTAMP)) file.timestamp = (time_t)jsonReadInt64(json); else THROW_FMT(FormatError, "missing timestamp for file '%s'", strZ(key)); // User if (jsonReadKeyExpectZ(json, MANIFEST_KEY_USER)) file.user = manifestOwnerGet(jsonReadVar(json)); else file.user = manifest->fileUserDefault; manifestFileAdd(manifest, &file); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_TARGET_PATH)) { ManifestPath path = {.name = key}; ManifestLoadFound valueFound = {0}; jsonReadObjectBegin(json); if (jsonReadKeyExpectZ(json, MANIFEST_KEY_GROUP)) { valueFound.group = true; path.group = manifestOwnerGet(jsonReadVar(json)); } if (jsonReadKeyExpectZ(json, MANIFEST_KEY_MODE)) { valueFound.mode = true; path.mode = cvtZToMode(strZ(jsonReadStr(json))); } if (jsonReadKeyExpectZ(json, MANIFEST_KEY_USER)) { valueFound.user = true; path.user = manifestOwnerGet(jsonReadVar(json)); } lstAdd(loadData->pathFoundList, &valueFound); manifestPathAdd(manifest, &path); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_TARGET_LINK)) { ManifestLink link = {.name = key}; ManifestLoadFound valueFound = {0}; jsonReadObjectBegin(json); // Link destination link.destination = jsonReadStr(jsonReadKeyRequireStrId(json, MANIFEST_KEY_DESTINATION)); // Group if (jsonReadKeyExpectZ(json, MANIFEST_KEY_GROUP)) { valueFound.group = true; link.group = manifestOwnerGet(jsonReadVar(json)); } // User if (jsonReadKeyExpectZ(json, MANIFEST_KEY_USER)) { valueFound.user = true; link.user = manifestOwnerGet(jsonReadVar(json)); } lstAdd(loadData->linkFoundList, &valueFound); manifestLinkAdd(manifest, &link); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_TARGET_FILE_DEFAULT)) { if (strEqZ(key, MANIFEST_KEY_MODE)) manifest->fileModeDefault = cvtZToMode(strZ(jsonReadStr(json))); else { const Variant *const valueVar = jsonReadVar(json); MEM_CONTEXT_BEGIN(manifest->pub.memContext) { if (strEqZ(key, MANIFEST_KEY_GROUP)) manifest->fileGroupDefault = strDup(manifestOwnerGet(valueVar)); else if (strEqZ(key, MANIFEST_KEY_USER)) manifest->fileUserDefault = strDup(manifestOwnerGet(valueVar)); } MEM_CONTEXT_END(); } } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_TARGET_PATH_DEFAULT)) { if (strEqZ(key, MANIFEST_KEY_MODE)) loadData->pathModeDefault = cvtZToMode(strZ(jsonReadStr(json))); else { const Variant *const valueVar = jsonReadVar(json); MEM_CONTEXT_BEGIN(loadData->memContext) { if (strEqZ(key, MANIFEST_KEY_GROUP)) loadData->pathGroupDefault = manifestOwnerDefaultGet(valueVar); else if (strEqZ(key, MANIFEST_KEY_USER)) loadData->pathUserDefault = manifestOwnerDefaultGet(valueVar); } MEM_CONTEXT_END(); } } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_TARGET_LINK_DEFAULT)) { const Variant *const valueVar = jsonReadVar(json); MEM_CONTEXT_BEGIN(loadData->memContext) { if (strEqZ(key, MANIFEST_KEY_GROUP)) loadData->linkGroupDefault = manifestOwnerDefaultGet(valueVar); else if (strEqZ(key, MANIFEST_KEY_USER)) loadData->linkUserDefault = manifestOwnerDefaultGet(valueVar); } MEM_CONTEXT_END(); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_BACKUP_TARGET)) { ManifestTarget target = {.name = key}; jsonReadObjectBegin(json); // File if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_FILE)) target.file = jsonReadStr(json); // Path target.path = jsonReadStr(jsonReadKeyRequireStrId(json, MANIFEST_KEY_PATH)); // Tablespace oid if (jsonReadKeyExpectZ(json, MANIFEST_KEY_TABLESPACE_ID)) { target.tablespaceId = cvtZToUInt(strZ(jsonReadStr(json))); target.tablespaceName = jsonReadStr(jsonReadKeyRequireZ(json, MANIFEST_KEY_TABLESPACE_NAME)); } // Tablespace type const String *const targetType = jsonReadStr(jsonReadKeyRequireStrId(json, MANIFEST_KEY_TYPE)); ASSERT(strEqZ(targetType, MANIFEST_TARGET_TYPE_LINK) || strEqZ(targetType, MANIFEST_TARGET_TYPE_PATH)); target.type = strEqZ(targetType, MANIFEST_TARGET_TYPE_PATH) ? manifestTargetTypePath : manifestTargetTypeLink; manifestTargetAdd(manifest, &target); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_DB)) { ManifestDb db = {.name = key}; jsonReadObjectBegin(json); // Database oid db.id = jsonReadUInt(jsonReadKeyRequireZ(json, MANIFEST_KEY_DB_ID)); // Last system oid db.lastSystemId = jsonReadUInt(jsonReadKeyRequireZ(json, MANIFEST_KEY_DB_LAST_SYSTEM_ID)); manifestDbAdd(manifest, &db); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_METADATA)) { MEM_CONTEXT_BEGIN(manifest->pub.memContext) { if (strEqZ(key, MANIFEST_KEY_ANNOTATION)) manifest->pub.data.annotation = jsonReadVar(json); } MEM_CONTEXT_END(); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_BACKUP)) { if (strEqZ(key, MANIFEST_KEY_BACKUP_BLOCK_INCR)) manifest->pub.data.blockIncr = jsonReadBool(json); else if (strEqZ(key, MANIFEST_KEY_BACKUP_BUNDLE)) manifest->pub.data.bundle = jsonReadBool(json); else if (strEqZ(key, MANIFEST_KEY_BACKUP_BUNDLE_RAW)) manifest->pub.data.bundleRaw = jsonReadBool(json); else if (strEqZ(key, MANIFEST_KEY_BACKUP_TIMESTAMP_COPY_START)) manifest->pub.data.backupTimestampCopyStart = (time_t)jsonReadUInt64(json); else if (strEqZ(key, MANIFEST_KEY_BACKUP_TIMESTAMP_START)) manifest->pub.data.backupTimestampStart = (time_t)jsonReadUInt64(json); else if (strEqZ(key, MANIFEST_KEY_BACKUP_TIMESTAMP_STOP)) manifest->pub.data.backupTimestampStop = (time_t)jsonReadUInt64(json); else if (strEqZ(key, MANIFEST_KEY_BACKUP_TYPE)) { manifest->pub.data.backupType = (BackupType)jsonReadStrId(json); ASSERT( manifest->pub.data.backupType == backupTypeFull || manifest->pub.data.backupType == backupTypeDiff || manifest->pub.data.backupType == backupTypeIncr); } else { MEM_CONTEXT_BEGIN(manifest->pub.memContext) { String *const valueStr = jsonReadStr(json); if (strEqZ(key, MANIFEST_KEY_BACKUP_ARCHIVE_START)) manifest->pub.data.archiveStart = valueStr; else if (strEqZ(key, MANIFEST_KEY_BACKUP_ARCHIVE_STOP)) manifest->pub.data.archiveStop = valueStr; else if (strEqZ(key, MANIFEST_KEY_BACKUP_LABEL)) manifest->pub.data.backupLabel = valueStr; else if (strEqZ(key, MANIFEST_KEY_BACKUP_LSN_START)) manifest->pub.data.lsnStart = valueStr; else if (strEqZ(key, MANIFEST_KEY_BACKUP_LSN_STOP)) manifest->pub.data.lsnStop = valueStr; else if (strEqZ(key, MANIFEST_KEY_BACKUP_PRIOR)) manifest->pub.data.backupLabelPrior = valueStr; else if (strEqZ(key, MANIFEST_KEY_BACKUP_REFERENCE)) { manifest->pub.referenceList = strLstNewSplitZ(valueStr, ","); loadData->referenceListFound = true; strFree(valueStr); } } MEM_CONTEXT_END(); } } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_BACKUP_DB)) { if (strEqZ(key, MANIFEST_KEY_DB_ID)) manifest->pub.data.pgId = jsonReadUInt(json); else if (strEqZ(key, MANIFEST_KEY_DB_SYSTEM_ID)) manifest->pub.data.pgSystemId = jsonReadUInt64(json); else if (strEqZ(key, MANIFEST_KEY_DB_CATALOG_VERSION)) manifest->pub.data.pgCatalogVersion = jsonReadUInt(json); else if (strEqZ(key, MANIFEST_KEY_DB_VERSION)) manifest->pub.data.pgVersion = pgVersionFromStr(jsonReadStr(json)); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_BACKUP_OPTION)) { // Required options if (strEqZ(key, MANIFEST_KEY_OPTION_ARCHIVE_CHECK)) manifest->pub.data.backupOptionArchiveCheck = jsonReadBool(json); else if (strEqZ(key, MANIFEST_KEY_OPTION_ARCHIVE_COPY)) manifest->pub.data.backupOptionArchiveCopy = jsonReadBool(json); // Historically this option meant to add gz compression else if (strEqZ(key, MANIFEST_KEY_OPTION_COMPRESS)) manifest->pub.data.backupOptionCompressType = jsonReadBool(json) ? compressTypeGz : compressTypeNone; // This new option allows any type of compression to be specified. It must be parsed after the option above so the value // does not get overwritten. Since options are stored in alpha order this should always be true. else if (strEqZ(key, MANIFEST_KEY_OPTION_COMPRESS_TYPE)) manifest->pub.data.backupOptionCompressType = compressTypeEnum(strStrId(jsonReadStr(json))); else if (strEqZ(key, MANIFEST_KEY_OPTION_HARDLINK)) manifest->pub.data.backupOptionHardLink = jsonReadBool(json); else if (strEqZ(key, MANIFEST_KEY_OPTION_ONLINE)) manifest->pub.data.backupOptionOnline = jsonReadBool(json); else { // Options that were added after v1.00 and may not be present in every manifest MEM_CONTEXT_BEGIN(manifest->pub.memContext) { if (strEqZ(key, MANIFEST_KEY_OPTION_BUFFER_SIZE)) manifest->pub.data.backupOptionBufferSize = varNewUInt(jsonReadUInt(json)); else if (strEqZ(key, MANIFEST_KEY_OPTION_COMPRESS_LEVEL)) manifest->pub.data.backupOptionCompressLevel = varNewUInt(jsonReadUInt(json)); else if (strEqZ(key, MANIFEST_KEY_OPTION_COMPRESS_LEVEL_NETWORK)) manifest->pub.data.backupOptionCompressLevelNetwork = varNewUInt(jsonReadUInt(json)); else if (strEqZ(key, MANIFEST_KEY_OPTION_PROCESS_MAX)) manifest->pub.data.backupOptionProcessMax = varNewUInt(jsonReadUInt(json)); else if (strEqZ(key, MANIFEST_KEY_OPTION_BACKUP_STANDBY)) manifest->pub.data.backupOptionStandby = varNewBool(jsonReadBool(json)); else if (strEqZ(key, MANIFEST_KEY_OPTION_CHECKSUM_PAGE)) manifest->pub.data.backupOptionChecksumPage = jsonReadVar(json); else if (strEqZ(key, MANIFEST_KEY_OPTION_DELTA)) manifest->pub.data.backupOptionDelta = jsonReadVar(json); } MEM_CONTEXT_END(); } } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ typedef struct ManifestSaveData { Manifest *manifest; // Manifest object to be saved const Variant *userDefault; // Default user const Variant *groupDefault; // Default group mode_t fileModeDefault; // File default mode mode_t pathModeDefault; // Path default mode } ManifestSaveData; // Helper to convert the owner MCV to a default. If the input is NULL boolean false should be returned, else the owner string. static const Variant * manifestOwnerVar(const String *const ownerDefault) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, ownerDefault); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(VARIANT, ownerDefault == NULL ? BOOL_FALSE_VAR : varNewStr(ownerDefault)); } static void manifestSaveCallback(void *const callbackData, const String *const sectionNext, InfoSave *const infoSaveData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STRING, sectionNext); FUNCTION_TEST_PARAM(INFO_SAVE, infoSaveData); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(callbackData != NULL); ASSERT(infoSaveData != NULL); ManifestSaveData *const saveData = (ManifestSaveData *)callbackData; Manifest *const manifest = saveData->manifest; // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_BACKUP, sectionNext)) { if (manifest->pub.data.archiveStart != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_ARCHIVE_START, jsonFromVar(VARSTR(manifest->pub.data.archiveStart))); } if (manifest->pub.data.archiveStop != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_ARCHIVE_STOP, jsonFromVar(VARSTR(manifest->pub.data.archiveStop))); } if (manifest->pub.data.blockIncr) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_BLOCK_INCR, jsonFromVar(VARBOOL(manifest->pub.data.blockIncr))); } if (manifest->pub.data.bundle) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_BUNDLE, jsonFromVar(VARBOOL(manifest->pub.data.bundle))); } if (manifest->pub.data.bundleRaw) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_BUNDLE_RAW, jsonFromVar(VARBOOL(manifest->pub.data.bundleRaw))); } infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_LABEL, jsonFromVar(VARSTR(manifest->pub.data.backupLabel))); if (manifest->pub.data.lsnStart != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_LSN_START, jsonFromVar(VARSTR(manifest->pub.data.lsnStart))); } if (manifest->pub.data.lsnStop != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_LSN_STOP, jsonFromVar(VARSTR(manifest->pub.data.lsnStop))); } if (manifest->pub.data.backupLabelPrior != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_PRIOR, jsonFromVar(VARSTR(manifest->pub.data.backupLabelPrior))); } infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_REFERENCE, jsonFromVar(VARSTR(strLstJoin(manifest->pub.referenceList, ",")))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_TIMESTAMP_COPY_START, jsonFromVar(VARINT64(manifest->pub.data.backupTimestampCopyStart))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_TIMESTAMP_START, jsonFromVar(VARINT64(manifest->pub.data.backupTimestampStart))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_TIMESTAMP_STOP, jsonFromVar(VARINT64(manifest->pub.data.backupTimestampStop))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_TYPE, jsonFromVar(VARSTR(strNewStrId(manifest->pub.data.backupType)))); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_BACKUP_DB, sectionNext)) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_DB, MANIFEST_KEY_DB_CATALOG_VERSION, jsonFromVar(VARUINT(manifest->pub.data.pgCatalogVersion))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_DB, "db-control-version", jsonFromVar(VARUINT(pgControlVersion(manifest->pub.data.pgVersion)))); infoSaveValue(infoSaveData, MANIFEST_SECTION_BACKUP_DB, MANIFEST_KEY_DB_ID, jsonFromVar(VARUINT(manifest->pub.data.pgId))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_DB, MANIFEST_KEY_DB_SYSTEM_ID, jsonFromVar(VARUINT64(manifest->pub.data.pgSystemId))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_DB, MANIFEST_KEY_DB_VERSION, jsonFromVar(VARSTR(pgVersionToStr(manifest->pub.data.pgVersion)))); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, sectionNext)) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_ARCHIVE_CHECK, jsonFromVar(VARBOOL(manifest->pub.data.backupOptionArchiveCheck))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_ARCHIVE_COPY, jsonFromVar(VARBOOL(manifest->pub.data.backupOptionArchiveCopy))); if (manifest->pub.data.backupOptionStandby != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_BACKUP_STANDBY, jsonFromVar(manifest->pub.data.backupOptionStandby)); } if (manifest->pub.data.backupOptionBufferSize != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_BUFFER_SIZE, jsonFromVar(manifest->pub.data.backupOptionBufferSize)); } if (manifest->pub.data.backupOptionChecksumPage != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_CHECKSUM_PAGE, jsonFromVar(manifest->pub.data.backupOptionChecksumPage)); } // Set the option when compression is turned on. In older versions this also implied gz compression but in newer versions // the type option must also be set if compression is not gz. infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_COMPRESS, jsonFromVar(VARBOOL(manifest->pub.data.backupOptionCompressType != compressTypeNone))); if (manifest->pub.data.backupOptionCompressLevel != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_COMPRESS_LEVEL, jsonFromVar(manifest->pub.data.backupOptionCompressLevel)); } if (manifest->pub.data.backupOptionCompressLevelNetwork != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_COMPRESS_LEVEL_NETWORK, jsonFromVar(manifest->pub.data.backupOptionCompressLevelNetwork)); } // Set the compression type. Older versions will ignore this and assume gz compression if the compress option is set. infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_COMPRESS_TYPE, jsonFromVar(VARSTR(compressTypeStr(manifest->pub.data.backupOptionCompressType)))); if (manifest->pub.data.backupOptionDelta != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_DELTA, jsonFromVar(manifest->pub.data.backupOptionDelta)); } infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_HARDLINK, jsonFromVar(VARBOOL(manifest->pub.data.backupOptionHardLink))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_ONLINE, jsonFromVar(VARBOOL(manifest->pub.data.backupOptionOnline))); if (manifest->pub.data.backupOptionProcessMax != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_PROCESS_MAX, jsonFromVar(manifest->pub.data.backupOptionProcessMax)); } } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_BACKUP_TARGET, sectionNext)) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); if (target->file != NULL) jsonWriteStr(jsonWriteKeyStrId(json, MANIFEST_KEY_FILE), target->file); jsonWriteStr(jsonWriteKeyStrId(json, MANIFEST_KEY_PATH), target->path); if (target->tablespaceId != 0) jsonWriteStrFmt(jsonWriteKeyZ(json, MANIFEST_KEY_TABLESPACE_ID), "%u", target->tablespaceId); if (target->tablespaceName != NULL) jsonWriteStr(jsonWriteKeyZ(json, MANIFEST_KEY_TABLESPACE_NAME), target->tablespaceName); jsonWriteZ( jsonWriteKeyStrId(json, MANIFEST_KEY_TYPE), target->type == manifestTargetTypePath ? MANIFEST_TARGET_TYPE_PATH : MANIFEST_TARGET_TYPE_LINK); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_TARGET, strZ(target->name), jsonWriteResult(jsonWriteObjectEnd(json))); MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_DB, sectionNext)) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int dbIdx = 0; dbIdx < manifestDbTotal(manifest); dbIdx++) { const ManifestDb *const db = manifestDb(manifest, dbIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); jsonWriteUInt(jsonWriteKeyZ(json, MANIFEST_KEY_DB_ID), db->id); jsonWriteUInt(jsonWriteKeyZ(json, MANIFEST_KEY_DB_LAST_SYSTEM_ID), db->lastSystemId); infoSaveValue(infoSaveData, MANIFEST_SECTION_DB, strZ(db->name), jsonWriteResult(jsonWriteObjectEnd(json))); MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_METADATA, sectionNext)) { if (manifest->pub.data.annotation != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_METADATA, MANIFEST_KEY_ANNOTATION, jsonFromVar(manifest->pub.data.annotation)); } } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_TARGET_FILE, sectionNext)) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFile file = manifestFile(manifest, fileIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); // Block incremental info if (file.blockIncrSize != 0) { jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_BLOCK_INCR), file.blockIncrSize / BLOCK_INCR_SIZE_FACTOR); if (file.blockIncrChecksumSize != BLOCK_INCR_CHECKSUM_SIZE_MIN) jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_BLOCK_INCR_CHECKSUM), file.blockIncrChecksumSize); if (file.blockIncrMapSize != 0) jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_BLOCK_INCR_MAP), file.blockIncrMapSize); } // Bundle info if (file.bundleId != 0) { jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_BUNDLE_ID), file.bundleId); if (file.bundleOffset != 0) jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_BUNDLE_OFFSET), file.bundleOffset); } // Save if the file size is not zero and the checksum exists. The checksum might not exist if this is a partial save // performed during a backup. if (file.size != 0 && file.checksumSha1 != NULL) { jsonWriteStr( jsonWriteKeyStrId(json, MANIFEST_KEY_CHECKSUM), strNewEncode(encodingHex, BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE))); } if (file.checksumPage) { jsonWriteBool(jsonWriteKeyZ(json, MANIFEST_KEY_CHECKSUM_PAGE), !file.checksumPageError); if (file.checksumPageErrorList != NULL) jsonWriteJson(jsonWriteKeyZ(json, MANIFEST_KEY_CHECKSUM_PAGE_ERROR), file.checksumPageErrorList); } if (!varEq(manifestOwnerVar(file.group), saveData->groupDefault)) jsonWriteVar(jsonWriteKeyZ(json, MANIFEST_KEY_GROUP), manifestOwnerVar(file.group)); if (file.mode != saveData->fileModeDefault) jsonWriteStrFmt(jsonWriteKeyZ(json, MANIFEST_KEY_MODE), "%04o", file.mode); // Save if the repo checksum is not null. The repo checksum for zero-length files may vary depending on compression // and encryption applied. if (file.checksumRepoSha1 != NULL) { jsonWriteStr( jsonWriteKeyStrId(json, MANIFEST_KEY_CHECKSUM_REPO), strNewEncode(encodingHex, BUF(file.checksumRepoSha1, HASH_TYPE_SHA1_SIZE))); } if (file.reference != NULL) jsonWriteStr(jsonWriteKeyStrId(json, MANIFEST_KEY_REFERENCE), file.reference); if (file.sizeRepo != file.size) jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_SIZE_REPO), file.sizeRepo); jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_SIZE), file.size); if (file.sizeOriginal != file.size) jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_SIZE_ORIGINAL), file.sizeOriginal); jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_TIMESTAMP), (uint64_t)file.timestamp); if (!varEq(manifestOwnerVar(file.user), saveData->userDefault)) jsonWriteVar(jsonWriteKeyZ(json, MANIFEST_KEY_USER), manifestOwnerVar(file.user)); infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_FILE, strZ(file.name), jsonWriteResult(jsonWriteObjectEnd(json))); MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_TARGET_FILE_DEFAULT, sectionNext)) { infoSaveValue(infoSaveData, MANIFEST_SECTION_TARGET_FILE_DEFAULT, MANIFEST_KEY_GROUP, jsonFromVar(saveData->groupDefault)); infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_FILE_DEFAULT, MANIFEST_KEY_MODE, jsonFromVar(VARSTR(strNewFmt("%04o", saveData->fileModeDefault)))); infoSaveValue(infoSaveData, MANIFEST_SECTION_TARGET_FILE_DEFAULT, MANIFEST_KEY_USER, jsonFromVar(saveData->userDefault)); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_TARGET_LINK, sectionNext)) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int linkIdx = 0; linkIdx < manifestLinkTotal(manifest); linkIdx++) { const ManifestLink *const link = manifestLink(manifest, linkIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); jsonWriteStr(jsonWriteKeyStrId(json, MANIFEST_KEY_DESTINATION), link->destination); if (!varEq(manifestOwnerVar(link->group), saveData->groupDefault)) jsonWriteVar(jsonWriteKeyZ(json, MANIFEST_KEY_GROUP), manifestOwnerVar(link->group)); if (!varEq(manifestOwnerVar(link->user), saveData->userDefault)) jsonWriteVar(jsonWriteKeyZ(json, MANIFEST_KEY_USER), manifestOwnerVar(link->user)); infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_LINK, strZ(link->name), jsonWriteResult(jsonWriteObjectEnd(json))); MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_TARGET_LINK_DEFAULT, sectionNext)) { if (manifestLinkTotal(manifest) > 0) { infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_LINK_DEFAULT, MANIFEST_KEY_GROUP, jsonFromVar(saveData->groupDefault)); infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_LINK_DEFAULT, MANIFEST_KEY_USER, jsonFromVar(saveData->userDefault)); } } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_TARGET_PATH, sectionNext)) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int pathIdx = 0; pathIdx < manifestPathTotal(manifest); pathIdx++) { const ManifestPath *const path = manifestPath(manifest, pathIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); if (!varEq(manifestOwnerVar(path->group), saveData->groupDefault)) jsonWriteVar(jsonWriteKeyZ(json, MANIFEST_KEY_GROUP), manifestOwnerVar(path->group)); if (path->mode != saveData->pathModeDefault) jsonWriteStrFmt(jsonWriteKeyZ(json, MANIFEST_KEY_MODE), "%04o", path->mode); if (!varEq(manifestOwnerVar(path->user), saveData->userDefault)) jsonWriteVar(jsonWriteKeyZ(json, MANIFEST_KEY_USER), manifestOwnerVar(path->user)); infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_PATH, strZ(path->name), jsonWriteResult(jsonWriteObjectEnd(json))); MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_TARGET_PATH_DEFAULT, sectionNext)) { infoSaveValue(infoSaveData, MANIFEST_SECTION_TARGET_PATH_DEFAULT, MANIFEST_KEY_GROUP, jsonFromVar(saveData->groupDefault)); infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_PATH_DEFAULT, MANIFEST_KEY_MODE, jsonFromVar(VARSTR(strNewFmt("%04o", saveData->pathModeDefault)))); infoSaveValue(infoSaveData, MANIFEST_SECTION_TARGET_PATH_DEFAULT, MANIFEST_KEY_USER, jsonFromVar(saveData->userDefault)); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ typedef struct ManifestLoadFileData { MemContext *memContext; // Mem context const Storage *storage; // Storage to load from const String *fileName; // Base filename const CipherSpec *cipherSpec; // Cipher spec Manifest *manifest; // Loaded manifest object } ManifestLoadFileData; static bool manifestLoadFileCallback(void *const data, const unsigned int try) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, data); FUNCTION_LOG_PARAM(UINT, try); FUNCTION_LOG_END(); ASSERT(data != NULL); ManifestLoadFileData *const loadData = data; bool result = false; if (try < 2) { MEM_CONTEXT_TEMP_BEGIN() { // Construct filename based on try const String *const fileName = try == 0 ? loadData->fileName : strNewFmt("%s" INFO_COPY_EXT, strZ(loadData->fileName)); // Attempt to load the file. A manifest has no header of its own, so it is decrypted with the spec as it was given. IoRead *const read = storageReadIo(storageNewReadP(loadData->storage, fileName)); cipherBlockFilterGroupAdd(ioReadFilterGroup(read), cipherModeDecrypt, loadData->cipherSpec); MEM_CONTEXT_BEGIN(loadData->memContext) { loadData->manifest = manifestNewLoad(read, loadData->cipherSpec); result = true; } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(BOOL, result); } FN_EXTERN Manifest * manifestLoadFile( const Storage *const storage, const String *const fileName, const CipherSpec *const cipherSpec) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_PARAM(CIPHER_SPEC, cipherSpec); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(fileName != NULL); ASSERT(cipherSpec != NULL); ManifestLoadFileData data = { .memContext = memContextCurrent(), .storage = storage, .fileName = fileName, .cipherSpec = cipherSpec, }; MEM_CONTEXT_TEMP_BEGIN() { const char *const fileNamePath = strZ(storagePathP(storage, fileName)); infoLoad( strNewFmt("unable to load backup manifest file '%s' or '%s" INFO_COPY_EXT "'", fileNamePath, fileNamePath), manifestLoadFileCallback, &data); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(MANIFEST, data.manifest); } FN_EXTERN void manifestSave(Manifest *const this, IoWrite *const write) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(write != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Files can be added from outside the manifest so make sure they are sorted lstSort(this->pub.fileList, sortOrderAsc); // Set default values based on the base path const ManifestPath *const pathBase = manifestPathFind(this, MANIFEST_TARGET_PGDATA_STR); ManifestSaveData saveData = { .manifest = this, .userDefault = manifestOwnerVar(pathBase->user), .groupDefault = manifestOwnerVar(pathBase->group), .fileModeDefault = pathBase->mode & (S_IRUSR | S_IWUSR | S_IRGRP), .pathModeDefault = pathBase->mode, }; // Save manifest infoSave(this->pub.info, write, manifestSaveCallback, &saveData); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/info/manifest/store.c.inc0000664000175100017510000006504215265101745021011 0ustar00runnerrunner/*********************************************************************************************************************************** Store manifest data in memory ***********************************************************************************************************************************/ /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ STRING_EXTERN(BACKUP_MANIFEST_FILE_STR, BACKUP_MANIFEST_FILE); STRING_EXTERN(MANIFEST_TARGET_PGDATA_STR, MANIFEST_TARGET_PGDATA); STRING_EXTERN(MANIFEST_TARGET_PGTBLSPC_STR, MANIFEST_TARGET_PGTBLSPC); // All block incremental sizes must be divisible by this factor #define BLOCK_INCR_SIZE_FACTOR 8192 /*********************************************************************************************************************************** // Add owner to the owner list if it is not there already and return the pointer. This saves a lot of space. ***********************************************************************************************************************************/ static const String * manifestOwnerCache(Manifest *const this, const String *const owner) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, owner); FUNCTION_TEST_END(); ASSERT(this != NULL); if (owner != NULL) FUNCTION_TEST_RETURN_CONST(STRING, strLstAddIfMissing(this->ownerList, owner)); FUNCTION_TEST_RETURN_CONST(STRING, NULL); } /*********************************************************************************************************************************** Db functions ***********************************************************************************************************************************/ static void manifestDbAdd(Manifest *const this, const ManifestDb *const db) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_DB, db); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(db != NULL); ASSERT(db->name != NULL); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.dbList)) { const ManifestDb dbAdd = { .id = db->id, .lastSystemId = db->lastSystemId, .name = strDup(db->name), }; lstAdd(this->pub.dbList, &dbAdd); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** File functions ***********************************************************************************************************************************/ // Base time used as a delta to reduce the size of packed timestamps. This will be set on the first call to manifestFilePack(). static time_t manifestPackBaseTime = -1; // Flags used to reduce the size of packed data. They should be ordered from most to least likely and can be reordered at will. typedef enum { manifestFilePackFlagChecksum, manifestFilePackFlagChecksumRepo, manifestFilePackFlagReference, manifestFilePackFlagBundle, manifestFilePackFlagBlockIncr, manifestFilePackFlagCopy, manifestFilePackFlagDelta, manifestFilePackFlagResume, manifestFilePackFlagChecksumPage, manifestFilePackFlagChecksumPageError, manifestFilePackFlagChecksumPageErrorList, manifestFilePackFlagSizeOriginal, manifestFilePackFlagMode, manifestFilePackFlagUser, manifestFilePackFlagUserNull, manifestFilePackFlagGroup, manifestFilePackFlagGroupNull, } ManifestFilePackFlag; // Pack file into a compact format to save memory static ManifestFilePack * manifestFilePack(const Manifest *const manifest, const ManifestFile *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, manifest); FUNCTION_TEST_PARAM(MANIFEST_FILE, file); FUNCTION_TEST_END(); ASSERT(manifest != NULL); ASSERT(file != NULL); uint8_t buffer[512]; size_t bufferPos = 0; // Flags uint64_t flag = 0; if (file->checksumSha1 != NULL) flag |= 1 << manifestFilePackFlagChecksum; if (file->checksumRepoSha1 != NULL) flag |= 1 << manifestFilePackFlagChecksumRepo; if (file->copy) flag |= 1 << manifestFilePackFlagCopy; if (file->delta) flag |= 1 << manifestFilePackFlagDelta; if (file->resume) flag |= 1 << manifestFilePackFlagResume; if (file->checksumPage) flag |= 1 << manifestFilePackFlagChecksumPage; if (file->checksumPageError) flag |= 1 << manifestFilePackFlagChecksumPageError; if (file->checksumPageErrorList != NULL) flag |= 1 << manifestFilePackFlagChecksumPageErrorList; if (file->reference != NULL) flag |= 1 << manifestFilePackFlagReference; if (file->bundleId != 0) flag |= 1 << manifestFilePackFlagBundle; if (file->blockIncrSize != 0) flag |= 1 << manifestFilePackFlagBlockIncr; if (file->sizeOriginal != file->size) flag |= 1 << manifestFilePackFlagSizeOriginal; if (file->mode != manifest->fileModeDefault) flag |= 1 << manifestFilePackFlagMode; if (file->user == NULL) flag |= 1 << manifestFilePackFlagUserNull; else if (!strEq(file->user, manifest->fileUserDefault)) flag |= 1 << manifestFilePackFlagUser; if (file->group == NULL) flag |= 1 << manifestFilePackFlagGroupNull; else if (!strEq(file->group, manifest->fileGroupDefault)) flag |= 1 << manifestFilePackFlagGroup; cvtUInt64ToVarInt128(flag, buffer, &bufferPos, sizeof(buffer)); // Size cvtUInt64ToVarInt128(file->size, buffer, &bufferPos, sizeof(buffer)); // Original size if (flag & (1 << manifestFilePackFlagSizeOriginal)) cvtUInt64ToVarInt128(file->sizeOriginal, buffer, &bufferPos, sizeof(buffer)); // Use the first timestamp that appears as the base for all other timestamps. Ideally we would like a timestamp as close to the // middle as possible but it doesn't seem worth doing the calculation. if (manifestPackBaseTime == -1) manifestPackBaseTime = file->timestamp; // Timestamp cvtUInt64ToVarInt128(cvtInt64ToZigZag(manifestPackBaseTime - file->timestamp), buffer, &bufferPos, sizeof(buffer)); // SHA1 checksum if (file->checksumSha1 != NULL) { memcpy((uint8_t *)buffer + bufferPos, file->checksumSha1, HASH_TYPE_SHA1_SIZE); bufferPos += HASH_TYPE_SHA1_SIZE; } // SHA1 repo checksum if (file->checksumRepoSha1 != NULL) { memcpy((uint8_t *)buffer + bufferPos, file->checksumRepoSha1, HASH_TYPE_SHA1_SIZE); bufferPos += HASH_TYPE_SHA1_SIZE; } // Reference if (file->reference != NULL) { cvtUInt64ToVarInt128( strLstFindIdxP(manifest->pub.referenceList, file->reference, .required = true), buffer, &bufferPos, sizeof(buffer)); } // Mode if (flag & (1 << manifestFilePackFlagMode)) cvtUInt64ToVarInt128(file->mode, buffer, &bufferPos, sizeof(buffer)); // User/group if (flag & (1 << manifestFilePackFlagUser)) { cvtUInt64ToVarInt128( strLstFindIdxP(manifest->ownerList, file->user, .required = true), buffer, &bufferPos, sizeof(buffer)); } if (flag & (1 << manifestFilePackFlagGroup)) { cvtUInt64ToVarInt128( strLstFindIdxP(manifest->ownerList, file->group, .required = true), buffer, &bufferPos, sizeof(buffer)); } // Repo size cvtUInt64ToVarInt128(file->sizeRepo, buffer, &bufferPos, sizeof(buffer)); // Bundle if (flag & (1 << manifestFilePackFlagBundle)) { cvtUInt64ToVarInt128(file->bundleId, buffer, &bufferPos, sizeof(buffer)); cvtUInt64ToVarInt128(file->bundleOffset, buffer, &bufferPos, sizeof(buffer)); } // Block incremental if (flag & (1 << manifestFilePackFlagBlockIncr)) { ASSERT(file->blockIncrSize % BLOCK_INCR_SIZE_FACTOR == 0); cvtUInt64ToVarInt128(file->blockIncrSize / BLOCK_INCR_SIZE_FACTOR, buffer, &bufferPos, sizeof(buffer)); cvtUInt64ToVarInt128(file->blockIncrChecksumSize, buffer, &bufferPos, sizeof(buffer)); cvtUInt64ToVarInt128(file->blockIncrMapSize, buffer, &bufferPos, sizeof(buffer)); } // Allocate memory for the file pack const size_t nameSize = strSize(file->name) + 1; uint8_t *const result = memNew( sizeof(StringPub) + nameSize + bufferPos + (file->checksumPageErrorList != NULL ? ALIGN_OFFSET(StringPub, nameSize + bufferPos) + sizeof(StringPub) + strSize(file->checksumPageErrorList) + 1 : 0)); // Create string object for the file name *(StringPub *)result = (StringPub){.size = (unsigned int)strSize(file->name), .buffer = (char *)result + sizeof(StringPub)}; size_t resultPos = sizeof(StringPub); memcpy(result + resultPos, (const uint8_t *)strZ(file->name), nameSize); resultPos += nameSize; // Copy pack data memcpy(result + resultPos, buffer, bufferPos); // Create string object for the checksum error list if (file->checksumPageErrorList != NULL) { resultPos += bufferPos + ALIGN_OFFSET(StringPub, nameSize + bufferPos); *(StringPub *)(result + resultPos) = (StringPub) { .size = (unsigned int)strSize(file->checksumPageErrorList), .buffer = (char *)result + resultPos + sizeof(StringPub), }; resultPos += sizeof(StringPub); memcpy(result + resultPos, (const uint8_t *)strZ(file->checksumPageErrorList), strSize(file->checksumPageErrorList) + 1); } FUNCTION_TEST_RETURN_TYPE_P(ManifestFilePack, (ManifestFilePack *)result); } FN_EXTERN ManifestFile manifestFileUnpack(const Manifest *const manifest, const ManifestFilePack *const filePack) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, manifest); FUNCTION_TEST_PARAM_P(VOID, filePack); FUNCTION_TEST_END(); ASSERT(filePack != NULL); ASSERT(manifestPackBaseTime != -1); ManifestFile result = {0}; size_t bufferPos = 0; // Name result.name = (const String *)filePack; bufferPos += sizeof(StringPub) + strSize(result.name) + 1; // Flags const uint64_t flag = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); result.copy = (flag >> manifestFilePackFlagCopy) & 1; result.delta = (flag >> manifestFilePackFlagDelta) & 1; result.resume = (flag >> manifestFilePackFlagResume) & 1; // Size result.size = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); // Original size if (flag & (1 << manifestFilePackFlagSizeOriginal)) result.sizeOriginal = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); else result.sizeOriginal = result.size; // Timestamp result.timestamp = manifestPackBaseTime - (time_t)cvtInt64FromZigZag(cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX)); // Checksum page result.checksumPage = (flag >> manifestFilePackFlagChecksumPage) & 1; // SHA1 checksum if (flag & (1 << manifestFilePackFlagChecksum)) { result.checksumSha1 = (const uint8_t *)filePack + bufferPos; bufferPos += HASH_TYPE_SHA1_SIZE; } // SHA1 repo checksum if (flag & (1 << manifestFilePackFlagChecksumRepo)) { result.checksumRepoSha1 = (const uint8_t *)filePack + bufferPos; bufferPos += HASH_TYPE_SHA1_SIZE; } // Reference if (flag & (1 << manifestFilePackFlagReference)) { result.reference = strLstGet( manifest->pub.referenceList, (unsigned int)cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX)); } // Mode if (flag & (1 << manifestFilePackFlagMode)) result.mode = (mode_t)cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); else result.mode = manifest->fileModeDefault; // User/group if (flag & (1 << manifestFilePackFlagUser)) { result.user = strLstGet( manifest->ownerList, (unsigned int)cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX)); } else if (!(flag & (1 << manifestFilePackFlagUserNull))) result.user = manifest->fileUserDefault; if (flag & (1 << manifestFilePackFlagGroup)) { result.group = strLstGet( manifest->ownerList, (unsigned int)cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX)); } else if (!(flag & (1 << manifestFilePackFlagGroupNull))) result.group = manifest->fileGroupDefault; // Repo size result.sizeRepo = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); // Bundle if (flag & (1 << manifestFilePackFlagBundle)) { result.bundleId = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); result.bundleOffset = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); } // Block incremental if (flag & (1 << manifestFilePackFlagBlockIncr)) { result.blockIncrSize = (size_t)cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX) * BLOCK_INCR_SIZE_FACTOR; result.blockIncrChecksumSize = (size_t)cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); result.blockIncrMapSize = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); } // Checksum page error result.checksumPageError = flag & (1 << manifestFilePackFlagChecksumPageError) ? true : false; if (flag & (1 << manifestFilePackFlagChecksumPageErrorList)) result.checksumPageErrorList = (const String *)((const uint8_t *)filePack + bufferPos + ALIGN_OFFSET(StringPub, bufferPos)); FUNCTION_TEST_RETURN_TYPE(ManifestFile, result); } FN_EXTERN void manifestFileAdd(Manifest *const this, ManifestFile *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_FILE, file); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(file->name != NULL); file->user = manifestOwnerCache(this, file->user); file->group = manifestOwnerCache(this, file->group); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.fileList)) { const ManifestFilePack *const filePack = manifestFilePack(this, file); lstAdd(this->pub.fileList, &filePack); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } // Update file pack by creating a new one and then freeing the old one static void manifestFilePackUpdate(Manifest *const this, ManifestFilePack **const filePack, const ManifestFile *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM_P(VOID, filePack); FUNCTION_TEST_PARAM(MANIFEST_FILE, file); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(filePack != NULL); ASSERT(file != NULL); ASSERT( (!file->checksumPage && !file->checksumPageError && file->checksumPageErrorList == NULL) || (file->checksumPage && !file->checksumPageError && file->checksumPageErrorList == NULL) || (file->checksumPage && file->checksumPageError)); ASSERT(file->size != 0 || (file->bundleId == 0 && file->bundleOffset == 0)); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.fileList)) { ManifestFilePack *const filePackOld = *filePack; *filePack = manifestFilePack(this, file); memFree(filePackOld); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } static ManifestFilePack ** manifestFilePackFindInternal(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); ManifestFilePack **const filePack = lstFind(this->pub.fileList, &name); if (filePack == NULL) THROW_FMT(AssertError, "unable to find '%s' in manifest file list", strZ(name)); FUNCTION_TEST_RETURN_TYPE_PP(ManifestFilePack, filePack); } const ManifestFilePack * manifestFilePackFind(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); FUNCTION_TEST_RETURN_TYPE_P(ManifestFilePack, *manifestFilePackFindInternal(this, name)); } FN_EXTERN void manifestFileRemove(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); if (!lstRemove(this->pub.fileList, &name)) THROW_FMT(AssertError, "unable to remove '%s' from manifest file list", strZ(name)); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void manifestFileUpdate(Manifest *const this, const ManifestFile *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_FILE, file); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(file != NULL); ManifestFilePack **const filePack = manifestFilePackFindInternal(this, file->name); manifestFilePackUpdate(this, filePack, file); FUNCTION_TEST_RETURN_VOID(); } // Update a file by index. Equivalent to manifestFileUpdate() but skips the by-name binary search since the caller already knows // the file's position in the list. Used by hot loops that iterate the file list (e.g. manifestBuildIncr). static void manifestFileUpdateByIdx(Manifest *const this, const unsigned int fileIdx, const ManifestFile *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(UINT, fileIdx); FUNCTION_TEST_PARAM(MANIFEST_FILE, file); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(fileIdx < lstSize(this->pub.fileList)); manifestFilePackUpdate(this, lstGet(this->pub.fileList, fileIdx), file); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Link functions ***********************************************************************************************************************************/ FN_EXTERN void manifestLinkAdd(Manifest *const this, const ManifestLink *const link) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_LINK, link); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(link != NULL); ASSERT(link->name != NULL); ASSERT(link->destination != NULL); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.linkList)) { const ManifestLink linkAdd = { .destination = strDup(link->destination), .name = strDup(link->name), .group = manifestOwnerCache(this, link->group), .user = manifestOwnerCache(this, link->user), }; lstAdd(this->pub.linkList, &linkAdd); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN ManifestLink * manifestLinkFind(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); ManifestLink *const result = lstFind(this->pub.linkList, &name); if (result == NULL) THROW_FMT(AssertError, "unable to find '%s' in manifest link list", strZ(name)); FUNCTION_TEST_RETURN(MANIFEST_LINK, result); } FN_EXTERN void manifestLinkRemove(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); if (!lstRemove(this->pub.linkList, &name)) THROW_FMT(AssertError, "unable to remove '%s' from manifest link list", strZ(name)); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void manifestLinkUpdate(const Manifest *const this, const String *const name, const String *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_PARAM(STRING, destination); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); ASSERT(destination != NULL); ManifestLink *const link = (ManifestLink *const)manifestLinkFind(this, name); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.linkList)) { if (!strEq(link->destination, destination)) link->destination = strDup(destination); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Path functions ***********************************************************************************************************************************/ static void manifestPathAdd(Manifest *const this, const ManifestPath *const path) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_PATH, path); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(path != NULL); ASSERT(path->name != NULL); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.pathList)) { const ManifestPath pathAdd = { .mode = path->mode, .name = strDup(path->name), .group = manifestOwnerCache(this, path->group), .user = manifestOwnerCache(this, path->user), }; lstAdd(this->pub.pathList, &pathAdd); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN const ManifestPath * manifestPathFind(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); const ManifestPath *const result = lstFind(this->pub.pathList, &name); if (result == NULL) THROW_FMT(AssertError, "unable to find '%s' in manifest path list", strZ(name)); FUNCTION_TEST_RETURN_CONST(MANIFEST_PATH, result); } /*********************************************************************************************************************************** Manifest functions ***********************************************************************************************************************************/ FN_EXTERN void manifestTargetAdd(Manifest *const this, const ManifestTarget *const target) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_TARGET, target); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(target != NULL); ASSERT(target->path != NULL); ASSERT(target->name != NULL); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.targetList)) { const ManifestTarget targetAdd = { .file = strDup(target->file), .name = strDup(target->name), .path = strDup(target->path), .tablespaceId = target->tablespaceId, .tablespaceName = strDup(target->tablespaceName), .type = target->type, }; lstAdd(this->pub.targetList, &targetAdd); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN ManifestTarget * manifestTargetFind(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); ManifestTarget *const result = lstFind(this->pub.targetList, &name); if (result == NULL) THROW_FMT(AssertError, "unable to find '%s' in manifest target list", strZ(name)); FUNCTION_TEST_RETURN(MANIFEST_TARGET, result); } FN_EXTERN String * manifestTargetPath(const Manifest *const this, const ManifestTarget *const target) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_TARGET, target); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(target != NULL); // If the target path is already absolute then just return it if (strBeginsWith(target->path, FSLASH_STR)) FUNCTION_TEST_RETURN(STRING, strDup(target->path)); // Construct it from the base pg path and a relative path String *result; MEM_CONTEXT_TEMP_BEGIN() { String *const pgPath = strCat(strNew(), strPath(manifestPathPg(target->name))); if (strSize(pgPath) != 0) strCatZ(pgPath, "/"); strCat(pgPath, target->path); MEM_CONTEXT_PRIOR_BEGIN() { result = strPathAbsolute(pgPath, manifestTargetBase(this)->path); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN void manifestTargetRemove(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); if (!lstRemove(this->pub.targetList, &name)) THROW_FMT(AssertError, "unable to remove '%s' from manifest target list", strZ(name)); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void manifestTargetUpdate(const Manifest *const this, const String *const name, const String *const path, const String *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(STRING, file); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); ASSERT(path != NULL); ManifestTarget *const target = (ManifestTarget *)manifestTargetFind(this, name); ASSERT((target->file == NULL && file == NULL) || (target->file != NULL && file != NULL)); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.targetList)) { if (!strEq(target->path, path)) target->path = strDup(path); if (!strEq(target->file, file)) target->file = strDup(file); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/main.c0000664000175100017510000002637215265101745015273 0ustar00runnerrunner/*********************************************************************************************************************************** Main ***********************************************************************************************************************************/ #include #include #include #include #include "command/annotate/annotate.h" #include "command/archive/get/get.h" #include "command/archive/push/push.h" #include "command/backup/backup.h" #include "command/check/check.h" #include "command/command.h" #include "command/control/start.h" #include "command/control/stop.h" #include "command/exit.h" #include "command/expire/expire.h" #include "command/help/help.h" #include "command/info/info.h" #include "command/local/local.h" #include "command/lock.h" #include "command/manifest/manifest.h" #include "command/remote/remote.h" #include "command/repo/get.h" #include "command/repo/ls.h" #include "command/repo/rm.h" #include "command/restore/restore.h" #include "command/server/ping.h" #include "command/server/server.h" #include "command/stanza/create.h" #include "command/stanza/delete.h" #include "command/stanza/upgrade.h" #include "command/verify/verify.h" #include "common/debug.h" #include "common/io/fdRead.h" #include "common/io/fdWrite.h" #include "common/io/io.h" #include "common/stat.h" #include "config/config.h" #include "config/load.h" #include "postgres/interface.h" #include "protocol/helper.h" #include "storage/azure/helper.h" #include "storage/cifs/helper.h" #include "storage/gcs/helper.h" #include "storage/helper.h" #include "storage/s3/helper.h" #include "storage/sftp/helper.h" #include "version.h" int main(int argListSize, const char *argList[]) { // Set stack trace and mem context error cleanup handlers static const ErrorHandlerFunction errorHandlerList[] = {stackTraceClean, memContextClean}; errorHandlerSet(errorHandlerList, LENGTH_OF(errorHandlerList)); // Set storage helpers static const StorageHelper storageHelperList[] = { STORAGE_AZURE_HELPER, STORAGE_CIFS_HELPER, STORAGE_GCS_HELPER, STORAGE_S3_HELPER, #ifdef HAVE_LIBSSH2 STORAGE_SFTP_HELPER, #endif STORAGE_END_HELPER }; storageHelperInit(storageHelperList); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INT, argListSize); FUNCTION_LOG_PARAM(CHARPY, argList); FUNCTION_LOG_END(); // Initialize command with the start time cmdInit(); // Initialize statistics collector statInit(); // Initialize exit handler exitInit(); // Process commands volatile int result = 0; volatile bool error = false; TRY_BEGIN() { // Load the configuration // ------------------------------------------------------------------------------------------------------------------------- cfgLoad((unsigned int)argListSize, argList); const ConfigCommandRole commandRole = cfgCommandRole(); // Reserve the first free list bucket for the I/O buffer size, the allocation that churns most while processing files memContextFreeListReserve(ioBufferSize()); // Main/async commands // ------------------------------------------------------------------------------------------------------------------------- if (commandRole == cfgCmdRoleMain || commandRole == cfgCmdRoleAsync) { const ConfigCommand command = cfgCommandHelp() ? cfgCmdHelp : cfgCommand(); switch (command) { // Annotate command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdAnnotate: cmdAnnotate(); break; // Archive get command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdArchiveGet: { if (commandRole == cfgCmdRoleAsync) cmdArchiveGetAsync(); else result = cmdArchiveGet(); break; } // Archive push command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdArchivePush: { if (commandRole == cfgCmdRoleAsync) cmdArchivePushAsync(); else cmdArchivePush(); break; } // Backup command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdBackup: { // Run backup cmdBackup(); if (cfgOptionBool(cfgOptExpireAuto)) { // Switch to expire command cmdEnd(NULL); cfgCommandSet(cfgCmdExpire, cfgCmdRoleMain); cfgLoadLogFile(); cmdBegin(); // Null out any backup percent complete value in the backup lock file cmdLockWriteP(); // Run expire cmdExpire(); } break; } // Check command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdCheck: cmdCheck(); break; // Expire command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdExpire: cmdExpire(); break; // Info command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdInfo: cmdInfo(); break; // Manifest command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdManifest: cmdManifest(); break; // Repository get file command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdRepoGet: result = cmdStorageGet(); break; // Repository list paths/files command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdRepoLs: cmdStorageList(); break; // Repository remove paths/files command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdRepoRm: cmdStorageRemove(); break; // Server command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdServer: cmdServer((unsigned int)argListSize, argList); break; // Server ping command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdServerPing: cmdServerPing(); break; // Restore command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdRestore: cmdRestore(); break; // Stanza create command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdStanzaCreate: cmdStanzaCreate(); break; // Stanza delete command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdStanzaDelete: cmdStanzaDelete(); break; // Stanza upgrade command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdStanzaUpgrade: cmdStanzaUpgrade(); break; // Start command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdStart: cmdStart(); break; // Stop command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdStop: cmdStop(); break; // Verify command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdVerify: cmdVerify(); break; // Help/version commands // ----------------------------------------------------------------------------------------------------------------- case cfgCmdHelp: case cfgCmdVersion: cmdHelp(); break; } } // Local/remote commands // ------------------------------------------------------------------------------------------------------------------------- else { ASSERT(commandRole == cfgCmdRoleLocal || commandRole == cfgCmdRoleRemote); const String *const service = commandRole == cfgCmdRoleLocal ? PROTOCOL_SERVICE_LOCAL_STR : PROTOCOL_SERVICE_REMOTE_STR; const String *const name = strNewFmt("%s-%s", strZ(service), strZ(cfgOptionDisplay(cfgOptProcess))); const TimeMSec timeout = cfgOptionUInt64(cfgOptProtocolTimeout); ProtocolServer *const server = protocolServerNew( name, service, ioFdReadNewOpen(name, STDIN_FILENO, timeout), ioFdWriteNewOpen(name, STDOUT_FILENO, timeout)); if (commandRole == cfgCmdRoleLocal) cmdLocal(server); else cmdRemote(server); } } CATCH_FATAL() { error = true; result = exitSafe(result, true); } TRY_END(); // Free protocol objects protocolFree(); FUNCTION_LOG_RETURN(INT, error ? result : exitSafe(result, false)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/meson.build0000664000175100017510000002231115265101745016332 0ustar00runnerrunner#################################################################################################################################### # Make externed functions/variables static when unity is enabled #################################################################################################################################### if get_option('unity') == 'on' configuration.set('FN_EXTERN', 'static', description: 'Function is static for unity build') configuration.set('VR_EXTERN_DECLARE', 'static', description: 'Variable declaration is static for unity build') configuration.set('VR_EXTERN_DEFINE', 'static', description: 'Variable definition is static for unity build') else configuration.set( 'FN_EXTERN', 'extern __attribute__((visibility ("internal")))', description: 'Extern function required by other compilation units') configuration.set( 'VR_EXTERN_DECLARE', 'extern __attribute__((visibility ("internal")))', description: 'Extern variable declaration required by other compilation units') configuration.set('VR_EXTERN_DEFINE', '', description: 'Extern variable definition required by other compilation units') endif #################################################################################################################################### # Write configuration #################################################################################################################################### configure_file(output: 'build.auto.h', configuration: configuration) #################################################################################################################################### # Code generation #################################################################################################################################### # Generator that writes the code built from the declarations in the source. A distribution tarball ships the generated files and # excludes the generator, and a unit test build comments out this subdir, so neither looks for it. if not dist build_code = find_program('../build/build.py') # The library the generator is built from, which the generated code depends on just as much as the declarations it is generated # from, since how something is rendered decides the result as much as what is declared. Whichever generator runs loads all of # the library, so this is the whole of it rather than the part that writes a particular file. Meson has no glob, so the list is # built here and a module added to the library is picked up the next time meson configures. build_code_lib = run_command( [ python, '-c', '''import glob, sys; print("\n".join(sorted(glob.glob(sys.argv[1] + "/**/*.py", recursive=True))))''', meson.current_source_dir() / '../build/lib', ], check: true, ).stdout().split() endif # Build help.auto.c.inc (or use the copy shipped in the distribution tarball) subdir('command/help') # Build interface.auto.c.inc (or use the copy shipped in the distribution tarball) subdir('postgres') #################################################################################################################################### # pgBackRest target #################################################################################################################################### src_pgbackrest = [ 'command/annotate/annotate.c', 'command/archive/common.c', 'command/archive/find.c', 'command/archive/get/file.c', 'command/archive/get/get.c', 'command/archive/get/protocol.c', 'command/archive/push/file.c', 'command/archive/push/protocol.c', 'command/archive/push/push.c', 'command/backup/backup.c', 'command/backup/blockIncr.c', 'command/backup/blockMap.c', 'command/backup/common.c', 'command/backup/pageChecksum.c', 'command/backup/protocol.c', 'command/check/check.c', 'command/check/common.c', 'command/check/report.c', 'command/command.c', 'command/control/common.c', 'command/control/start.c', 'command/control/stop.c', 'command/exit.c', 'command/expire/expire.c', 'command/help/help.c', 'command/info/info.c', 'command/local/local.c', 'command/lock.c', 'command/manifest/manifest.c', 'command/remote/remote.c', 'command/repo/common.c', 'command/repo/get.c', 'command/repo/ls.c', 'command/repo/rm.c', 'command/restore/blockChecksum.c', 'command/restore/blockDelta.c', 'command/restore/protocol.c', 'command/restore/restore.c', 'command/restore/timeline.c', 'command/server/ping.c', 'command/server/server.c', 'command/stanza/common.c', 'command/stanza/create.c', 'command/stanza/delete.c', 'command/stanza/upgrade.c', 'command/verify/file.c', 'command/verify/protocol.c', 'command/verify/verify.c', 'common/compress/bz2/common.c', 'common/compress/bz2/compress.c', 'common/compress/bz2/decompress.c', 'common/compress/common.c', 'common/compress/gz/common.c', 'common/compress/gz/compress.c', 'common/compress/gz/decompress.c', 'common/compress/helper.c', 'common/compress/lz4/common.c', 'common/compress/lz4/compress.c', 'common/compress/lz4/decompress.c', 'common/compress/zst/common.c', 'common/compress/zst/compress.c', 'common/compress/zst/decompress.c', 'common/crypto/cipherBlock.c', 'common/crypto/common.c', 'common/crypto/hash.c', 'common/crypto/spec.c', 'common/crypto/xxhash.c', 'common/debug.c', 'common/encode.c', 'common/error/error.c', 'common/error/retry.c', 'common/exec.c', 'common/fork.c', 'common/format/cipherBlockFormat.c', 'common/format/cipherSpecMap.c', 'common/format/format.c', 'common/ini.c', 'common/io/bufferRead.c', 'common/io/bufferWrite.c', 'common/io/client.c', 'common/io/fd.c', 'common/io/fdRead.c', 'common/io/fdWrite.c', 'common/io/filter/buffer.c', 'common/io/filter/filter.c', 'common/io/filter/group.c', 'common/io/filter/sink.c', 'common/io/filter/size.c', 'common/io/http/client.c', 'common/io/http/common.c', 'common/io/http/header.c', 'common/io/http/query.c', 'common/io/http/request.c', 'common/io/http/response.c', 'common/io/http/session.c', 'common/io/http/url.c', 'common/io/io.c', 'common/io/limitRead.c', 'common/io/read.c', 'common/io/server.c', 'common/io/session.c', 'common/io/socket/address.c', 'common/io/socket/client.c', 'common/io/socket/common.c', 'common/io/socket/server.c', 'common/io/socket/session.c', 'common/io/tls/client.c', 'common/io/tls/common.c', 'common/io/tls/server.c', 'common/io/tls/session.c', 'common/io/write.c', 'common/lock.c', 'common/log.c', 'common/memContext.c', 'common/regExp.c', 'common/stackTrace.c', 'common/stat.c', 'common/time.c', 'common/type/blob.c', 'common/type/buffer.c', 'common/type/convert.c', 'common/type/json.c', 'common/type/keyValue.c', 'common/type/list.c', 'common/type/object.c', 'common/type/pack.c', 'common/type/string.c', 'common/type/stringId.c', 'common/type/stringList.c', 'common/type/stringStatic.c', 'common/type/stringZ.c', 'common/type/variant.c', 'common/type/variantList.c', 'common/type/xml.c', 'common/user.c', 'common/wait.c', 'config/common.c', 'config/config.c', 'config/exec.c', 'config/load.c', 'config/parse.c', 'config/protocol.c', 'db/db.c', 'db/helper.c', 'db/protocol.c', 'info/info.c', 'info/infoArchive.c', 'info/infoBackup.c', 'info/infoPg.c', 'info/manifest/manifest.c', 'postgres/client.c', 'postgres/interface.c', 'postgres/interface/crc32.c', 'postgres/interface/page.c', 'protocol/client.c', 'protocol/helper.c', 'protocol/parallel.c', 'protocol/parallelJob.c', 'protocol/server.c', 'storage/azure/helper.c', 'storage/azure/read.c', 'storage/azure/storage.c', 'storage/azure/write.c', 'storage/cifs/helper.c', 'storage/cifs/storage.c', 'storage/gcs/helper.c', 'storage/gcs/read.c', 'storage/gcs/storage.c', 'storage/gcs/write.c', 'storage/helper.c', 'storage/iterator.c', 'storage/list.c', 'storage/posix/read.c', 'storage/posix/storage.c', 'storage/posix/write.c', 'storage/read.c', 'storage/readMulti.c', 'storage/remote/protocol.c', 'storage/remote/read.c', 'storage/remote/readMulti.c', 'storage/remote/storage.c', 'storage/remote/write.c', 'storage/s3/helper.c', 'storage/s3/read.c', 'storage/s3/storage.c', 'storage/s3/write.c', 'storage/sftp/helper.c', 'storage/sftp/read.c', 'storage/sftp/storage.c', 'storage/sftp/write.c', 'storage/storage.c', 'storage/write.c', 'main.c', ] pgbackrest_exe = executable( 'pgbackrest', src_pgbackrest, help_auto_c_inc, interface_auto_c_inc, dependencies: [ lib_backtrace, lib_bz2, lib_openssl, lib_lz4, lib_systemd, lib_pq, lib_socket, lib_ssh2, lib_xml, lib_z, lib_zstd, ], install: true, ) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.7680757 pgbackrest-2.60.0/src/postgres/0000775000175100017510000000000015265102133016027 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/client.c0000664000175100017510000003613515265101745017471 0ustar00runnerrunner/*********************************************************************************************************************************** Postgres Client ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/fd.h" #include "common/log.h" #include "common/wait.h" #include "postgres/client.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct PgClient { PgClientPub pub; // Publicly accessible variables PGconn *connection; // Pg connection }; /*********************************************************************************************************************************** Close protocol connection ***********************************************************************************************************************************/ static void pgClientFreeResource(THIS_VOID) { THIS(PgClient); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PG_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); PQfinish(this->connection); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN PgClient * pgClientNew( const String *const host, const unsigned int port, const String *const database, const String *const user, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(STRING, database); FUNCTION_LOG_PARAM(STRING, user); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(port >= 1 && port <= 65535); ASSERT(database != NULL); OBJ_NEW_BEGIN(PgClient, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (PgClient) { .pub = { .host = strDup(host), .port = port, .database = strDup(database), .user = strDup(user), .timeout = timeout, }, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(PG_CLIENT, this); } /*********************************************************************************************************************************** Just ignore notices and warnings ***********************************************************************************************************************************/ static void pgClientNoticeProcessor(void *const arg, const char *const message) { (void)arg; (void)message; } /*********************************************************************************************************************************** Encode string to escape ' and \ ***********************************************************************************************************************************/ static String * pgClientEscape(const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); ASSERT(string != NULL); String *const result = strCatZ(strNew(), "'"); // Iterate all characters in the string for (unsigned stringIdx = 0; stringIdx < strSize(string); stringIdx++) { char stringChar = strZ(string)[stringIdx]; // These characters are escaped if (stringChar == '\'' || stringChar == '\\') strCatChr(result, '\\'); strCatChr(result, stringChar); } strCatChr(result, '\''); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN PgClient * pgClientOpen(PgClient *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PG_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); CHECK(AssertError, this->connection == NULL, "invalid connection"); MEM_CONTEXT_TEMP_BEGIN() { // Base connection string String *const connInfo = strCatFmt( strNew(), "dbname=%s port=%u", strZ(pgClientEscape(pgClientDatabase(this))), pgClientPort(this)); // Add user if specified if (pgClientUser(this) != NULL) strCatFmt(connInfo, " user=%s", strZ(pgClientEscape(pgClientUser(this)))); // Add host if specified if (pgClientHost(this) != NULL) strCatFmt(connInfo, " host=%s", strZ(pgClientEscape(pgClientHost(this)))); // Make the connection this->connection = PQconnectdb(strZ(connInfo)); // Set a callback to shutdown the connection memContextCallbackSet(objMemContext(this), pgClientFreeResource, this); // Handle errors if (PQstatus(this->connection) != CONNECTION_OK) { THROW_FMT( DbConnectError, "unable to connect to '%s': %s", strZ(connInfo), strZ(strTrim(strNewZ(PQerrorMessage(this->connection))))); } // Set notice and warning processor PQsetNoticeProcessor(this->connection, pgClientNoticeProcessor, NULL); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PG_CLIENT, this); } /**********************************************************************************************************************************/ FN_EXTERN Pack * pgClientQuery(PgClient *const this, const String *const query, const PgClientQueryResult resultType) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PG_CLIENT, this); FUNCTION_LOG_PARAM(STRING, query); FUNCTION_LOG_PARAM(STRING_ID, resultType); FUNCTION_LOG_END(); ASSERT(this != NULL); CHECK(AssertError, this->connection != NULL, "invalid connection"); ASSERT(query != NULL); ASSERT(resultType != 0); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Send the query without waiting for results so we can timeout if needed if (!PQsendQuery(this->connection, strZ(query))) { THROW_FMT( DbQueryError, "unable to send query '%s': %s", strZ(query), strZ(strTrim(strNewZ(PQerrorMessage(this->connection))))); } // Wait for a result Wait *const wait = waitNew(pgClientTimeout(this)); bool busy = false; do { PQconsumeInput(this->connection); busy = PQisBusy(this->connection); } while (busy && fdReadyRead(PQsocket(this->connection), waitRemains(wait))); // If the query is still busy after the timeout attempt to cancel if (busy) { PGcancel *const cancel = PQgetCancel(this->connection); // If cancel is NULL then more than likely the server process crashed or disconnected if (cancel == NULL) THROW_FMT(DbQueryError, "unable to cancel query '%s': connection was lost", strZ(query)); TRY_BEGIN() { char error[256]; if (!PQcancel(cancel, error, sizeof(error))) THROW_FMT(DbQueryError, "unable to cancel query '%s': %s", strZ(query), strZ(strTrim(strNewZ(error)))); } FINALLY() { PQfreeCancel(cancel); } TRY_END(); } // Get the result (even if query was cancelled -- to prevent the connection being left in a bad state) PGresult *const pgResult = PQgetResult(this->connection); TRY_BEGIN() { // Throw timeout error if cancelled if (busy) THROW_FMT(DbQueryError, "query '%s' timed out after %" PRIu64 "ms", strZ(query), pgClientTimeout(this)); // If this was a command that returned no results then we are done const ExecStatusType resultStatus = PQresultStatus(pgResult); if (resultStatus == PGRES_COMMAND_OK) { if (resultType != pgClientQueryResultNone && resultType != pgClientQueryResultAny) THROW_FMT(DbQueryError, "result expected from '%s'", strZ(query)); } else { // If no tuples then the result is an error if (resultStatus != PGRES_TUPLES_OK) { THROW_FMT( DbQueryError, "unable to execute query '%s': %s", strZ(query), strZ(strTrim(strNewZ(PQresultErrorMessage(pgResult))))); } // Expect some rows to be returned if (resultType == pgClientQueryResultNone) THROW_FMT(DbQueryError, "no result expected from '%s'", strZ(query)); // Fetch row and column values PackWrite *const pack = pckWriteNewP(); const int rowTotal = PQntuples(pgResult); const int columnTotal = PQnfields(pgResult); if (resultType != pgClientQueryResultAny) { if (rowTotal != 1) THROW_FMT(DbQueryError, "expected one row from '%s'", strZ(query)); if (resultType == pgClientQueryResultColumn && columnTotal != 1) THROW_FMT(DbQueryError, "expected one column from '%s'", strZ(query)); } // Get column types Oid *const columnType = memNew(sizeof(int) * (size_t)columnTotal); for (int columnIdx = 0; columnIdx < columnTotal; columnIdx++) columnType[columnIdx] = PQftype(pgResult, columnIdx); // Get values for (int rowIdx = 0; rowIdx < rowTotal; rowIdx++) { if (resultType == pgClientQueryResultAny) pckWriteArrayBeginP(pack); for (int columnIdx = 0; columnIdx < columnTotal; columnIdx++) { const char *const value = PQgetvalue(pgResult, rowIdx, columnIdx); // If value is zero-length then check if it is null if (value[0] == '\0' && PQgetisnull(pgResult, rowIdx, columnIdx)) { pckWriteNullP(pack); } // Else convert the value to a variant else { // Convert column type. Not all PostgreSQL types are supported but these should suffice. switch (columnType[columnIdx]) { // Boolean type case 16: // bool pckWriteBoolP(pack, varBoolForce(varNewStrZ(value)), .defaultWrite = true); break; // Text/char types case 18: // char case 19: // name case 25: // text pckWriteStrP(pack, STR(value), .defaultWrite = true); break; // 64-bit integer type case 20: // int8 pckWriteI64P(pack, cvtZToInt64(value), .defaultWrite = true); break; // 32-bit integer type case 23: // int4 pckWriteI32P(pack, cvtZToInt(value), .defaultWrite = true); break; // 32-bit unsigned integer type case 26: // oid pckWriteU32P(pack, cvtZToUInt(value), .defaultWrite = true); break; default: THROW_FMT( FormatError, "unable to parse type %u in column %d for query '%s'", columnType[columnIdx], columnIdx, strZ(query)); } } } if (resultType == pgClientQueryResultAny) pckWriteArrayEndP(pack); } // End pack and return result pckWriteEndP(pack); result = pckMove(pckWriteResult(pack), memContextPrior()); } } FINALLY() { // Free the result PQclear(pgResult); // The command is only complete once a NULL result is returned. A non-NULL result here means an extra result was // received, which almost always means the connection was lost while the query was running. PGresult *const pgResultExtra = PQgetResult(this->connection); if (pgResultExtra != NULL) { // Attempt to retrieve the error so the actual cause is reported const String *const error = strTrim(strNewZ(PQresultErrorMessage(pgResultExtra))); PQclear(pgResultExtra); THROW_FMT( DbQueryError, "unable to complete query '%s'%s%s\n" "HINT: this is usually caused by the connection to PostgreSQL being lost while the query was running.\n" "HINT: check idle_session_timeout and idle_in_transaction_session_timeout for aggressive settings.\n" "HINT: check the PostgreSQL log and network for a crash, restart, or timeout.", strZ(query), strEmpty(error) ? "" : ": ", strZ(error)); } } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN void pgClientToLog(const PgClient *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{host: "); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT(pgClientHost(this), strToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", database: "); strToLog(pgClientDatabase(this), debugLog); strStcCat(debugLog, ", user: "); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT(pgClientUser(this), strToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt( debugLog, ", port: %u, queryTimeout %" PRIu64 "}", pgClientPort(this), pgClientTimeout(this)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/client.h0000664000175100017510000001241215265101745017466 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Client Connect to a PostgreSQL database and run queries. This is not intended to be a general purpose client but is suitable for pgBackRest's limited needs. In particular, data type support is limited to text, int, and bool types so it may be necessary to add casts to queries to output one of these types. ***********************************************************************************************************************************/ #ifndef POSTGRES_QUERY_H #define POSTGRES_QUERY_H #include "common/time.h" #include "common/type/object.h" #include "common/type/pack.h" #include "common/type/string.h" /*********************************************************************************************************************************** Query result types ***********************************************************************************************************************************/ typedef enum { pgClientQueryResultAny = STRID5("any", 0x65c10), // Any number of rows/columns expected (even none) pgClientQueryResultRow = STRID5("row", 0x5df20), // One row expected pgClientQueryResultColumn = STRID5("column", 0x1cdab1e30), // One row/column expected pgClientQueryResultNone = STRID5("none", 0x2b9ee0), // No rows expected } PgClientQueryResult; /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct PgClient PgClient; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN PgClient *pgClientNew( const String *host, const unsigned int port, const String *database, const String *user, const TimeMSec timeout); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct PgClientPub { const String *host; // Pg host unsigned int port; // Pg port const String *database; // Pg database const String *user; // Pg user TimeMSec timeout; // Timeout for statements/queries } PgClientPub; // Pg host FN_INLINE_ALWAYS const String * pgClientHost(const PgClient *const this) { return THIS_PUB(PgClient)->host; } // Pg port FN_INLINE_ALWAYS unsigned int pgClientPort(const PgClient *const this) { return THIS_PUB(PgClient)->port; } // Pg database FN_INLINE_ALWAYS const String * pgClientDatabase(const PgClient *const this) { return THIS_PUB(PgClient)->database; } // Pg user FN_INLINE_ALWAYS const String * pgClientUser(const PgClient *const this) { return THIS_PUB(PgClient)->user; } // Timeout for statements/queries FN_INLINE_ALWAYS TimeMSec pgClientTimeout(const PgClient *const this) { return THIS_PUB(PgClient)->timeout; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Open connection to PostgreSQL FN_EXTERN PgClient *pgClientOpen(PgClient *this); // Move to a new parent mem context FN_INLINE_ALWAYS PgClient * pgClientMove(PgClient *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Execute a query and return results FN_EXTERN Pack *pgClientQuery(PgClient *this, const String *query, PgClientQueryResult resultType); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void pgClientFree(PgClient *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void pgClientToLog(const PgClient *this, StringStatic *debugLog); #define FUNCTION_LOG_PG_CLIENT_TYPE \ PgClient * #define FUNCTION_LOG_PG_CLIENT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, pgClientToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/interface/0000775000175100017510000000000015265101745017777 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/interface/crc32.c0000664000175100017510000000756315265101745021072 0ustar00runnerrunner/*********************************************************************************************************************************** CRC-32 Calculation ***********************************************************************************************************************************/ #include #include "postgres/interface/crc32.h" /**********************************************************************************************************************************/ static const uint32_t crc32c_lookup[256] = { 0x00000000, 0xF26B8303, 0xE13B70F7, 0x1350F3F4, 0xC79A971F, 0x35F1141C, 0x26A1E7E8, 0xD4CA64EB, 0x8AD958CF, 0x78B2DBCC, 0x6BE22838, 0x9989AB3B, 0x4D43CFD0, 0xBF284CD3, 0xAC78BF27, 0x5E133C24, 0x105EC76F, 0xE235446C, 0xF165B798, 0x030E349B, 0xD7C45070, 0x25AFD373, 0x36FF2087, 0xC494A384, 0x9A879FA0, 0x68EC1CA3, 0x7BBCEF57, 0x89D76C54, 0x5D1D08BF, 0xAF768BBC, 0xBC267848, 0x4E4DFB4B, 0x20BD8EDE, 0xD2D60DDD, 0xC186FE29, 0x33ED7D2A, 0xE72719C1, 0x154C9AC2, 0x061C6936, 0xF477EA35, 0xAA64D611, 0x580F5512, 0x4B5FA6E6, 0xB93425E5, 0x6DFE410E, 0x9F95C20D, 0x8CC531F9, 0x7EAEB2FA, 0x30E349B1, 0xC288CAB2, 0xD1D83946, 0x23B3BA45, 0xF779DEAE, 0x05125DAD, 0x1642AE59, 0xE4292D5A, 0xBA3A117E, 0x4851927D, 0x5B016189, 0xA96AE28A, 0x7DA08661, 0x8FCB0562, 0x9C9BF696, 0x6EF07595, 0x417B1DBC, 0xB3109EBF, 0xA0406D4B, 0x522BEE48, 0x86E18AA3, 0x748A09A0, 0x67DAFA54, 0x95B17957, 0xCBA24573, 0x39C9C670, 0x2A993584, 0xD8F2B687, 0x0C38D26C, 0xFE53516F, 0xED03A29B, 0x1F682198, 0x5125DAD3, 0xA34E59D0, 0xB01EAA24, 0x42752927, 0x96BF4DCC, 0x64D4CECF, 0x77843D3B, 0x85EFBE38, 0xDBFC821C, 0x2997011F, 0x3AC7F2EB, 0xC8AC71E8, 0x1C661503, 0xEE0D9600, 0xFD5D65F4, 0x0F36E6F7, 0x61C69362, 0x93AD1061, 0x80FDE395, 0x72966096, 0xA65C047D, 0x5437877E, 0x4767748A, 0xB50CF789, 0xEB1FCBAD, 0x197448AE, 0x0A24BB5A, 0xF84F3859, 0x2C855CB2, 0xDEEEDFB1, 0xCDBE2C45, 0x3FD5AF46, 0x7198540D, 0x83F3D70E, 0x90A324FA, 0x62C8A7F9, 0xB602C312, 0x44694011, 0x5739B3E5, 0xA55230E6, 0xFB410CC2, 0x092A8FC1, 0x1A7A7C35, 0xE811FF36, 0x3CDB9BDD, 0xCEB018DE, 0xDDE0EB2A, 0x2F8B6829, 0x82F63B78, 0x709DB87B, 0x63CD4B8F, 0x91A6C88C, 0x456CAC67, 0xB7072F64, 0xA457DC90, 0x563C5F93, 0x082F63B7, 0xFA44E0B4, 0xE9141340, 0x1B7F9043, 0xCFB5F4A8, 0x3DDE77AB, 0x2E8E845F, 0xDCE5075C, 0x92A8FC17, 0x60C37F14, 0x73938CE0, 0x81F80FE3, 0x55326B08, 0xA759E80B, 0xB4091BFF, 0x466298FC, 0x1871A4D8, 0xEA1A27DB, 0xF94AD42F, 0x0B21572C, 0xDFEB33C7, 0x2D80B0C4, 0x3ED04330, 0xCCBBC033, 0xA24BB5A6, 0x502036A5, 0x4370C551, 0xB11B4652, 0x65D122B9, 0x97BAA1BA, 0x84EA524E, 0x7681D14D, 0x2892ED69, 0xDAF96E6A, 0xC9A99D9E, 0x3BC21E9D, 0xEF087A76, 0x1D63F975, 0x0E330A81, 0xFC588982, 0xB21572C9, 0x407EF1CA, 0x532E023E, 0xA145813D, 0x758FE5D6, 0x87E466D5, 0x94B49521, 0x66DF1622, 0x38CC2A06, 0xCAA7A905, 0xD9F75AF1, 0x2B9CD9F2, 0xFF56BD19, 0x0D3D3E1A, 0x1E6DCDEE, 0xEC064EED, 0xC38D26C4, 0x31E6A5C7, 0x22B65633, 0xD0DDD530, 0x0417B1DB, 0xF67C32D8, 0xE52CC12C, 0x1747422F, 0x49547E0B, 0xBB3FFD08, 0xA86F0EFC, 0x5A048DFF, 0x8ECEE914, 0x7CA56A17, 0x6FF599E3, 0x9D9E1AE0, 0xD3D3E1AB, 0x21B862A8, 0x32E8915C, 0xC083125F, 0x144976B4, 0xE622F5B7, 0xF5720643, 0x07198540, 0x590AB964, 0xAB613A67, 0xB831C993, 0x4A5A4A90, 0x9E902E7B, 0x6CFBAD78, 0x7FAB5E8C, 0x8DC0DD8F, 0xE330A81A, 0x115B2B19, 0x020BD8ED, 0xF0605BEE, 0x24AA3F05, 0xD6C1BC06, 0xC5914FF2, 0x37FACCF1, 0x69E9F0D5, 0x9B8273D6, 0x88D28022, 0x7AB90321, 0xAE7367CA, 0x5C18E4C9, 0x4F48173D, 0xBD23943E, 0xF36E6F75, 0x0105EC76, 0x12551F82, 0xE03E9C81, 0x34F4F86A, 0xC69F7B69, 0xD5CF889D, 0x27A40B9E, 0x79B737BA, 0x8BDCB4B9, 0x988C474D, 0x6AE7C44E, 0xBE2DA0A5, 0x4C4623A6, 0x5F16D052, 0xAD7D5351, }; FN_EXTERN uint32_t crc32cOne(const uint8_t *data, size_t size) { uint32_t result = 0xffffffff; while (size--) result = crc32c_lookup[(result ^ *data++) & 0xFF] ^ (result >> 8); return result ^ 0xffffffff; } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/interface/crc32.h0000664000175100017510000000153315265101745021066 0ustar00runnerrunner/*********************************************************************************************************************************** CRC-32 Calculation CRC-32 and CRC-32C calculations required to validate the integrity of pg_control. ***********************************************************************************************************************************/ #ifndef POSTGRES_INTERFACE_CRC32_H #define POSTGRES_INTERFACE_CRC32_H #include #include /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Generate CRC-32C checksum FN_EXTERN uint32_t crc32cOne(const uint8_t *data, size_t size); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/interface/page.c0000664000175100017510000001633715265101745021071 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Page Checksum Adapted from PostgreSQL src/include/storage/checksum_impl.h. ***********************************************************************************************************************************/ #include #include #include "postgres/interface/static.vendor.h" /*********************************************************************************************************************************** Page checksum calculation ***********************************************************************************************************************************/ // Number of checksums to calculate in parallel #define PARALLEL_SUM 32 // Prime multiplier of FNV-1a hash #define FNV_PRIME 16777619 // Calculate one round of the checksum #define CHECKSUM_ROUND(checksum, value) \ do \ { \ const uint32_t tmp = (checksum) ^ (value); \ checksum = tmp * FNV_PRIME ^ (tmp >> 17); \ } while (0) // Main calculation loop #define CHECKSUM_CASE(pageSize) \ case pageSize: \ for (uint32_t i = 0; i < (uint32) (pageSize / (sizeof(uint32) * PARALLEL_SUM)); i++) \ for (uint32_t j = 0; j < PARALLEL_SUM; j++) \ CHECKSUM_ROUND(sums[j], ((PgPageChecksum##pageSize *)page)->data[i][j]); \ \ break; /*********************************************************************************************************************************** Define unions that will make the code valid under strict aliasing for each page size ***********************************************************************************************************************************/ #define CHECKSUM_UNION(pageSize) \ typedef union \ { \ PageHeaderData phdr; \ uint32_t data[pageSize / (sizeof(uint32_t) * PARALLEL_SUM)][PARALLEL_SUM]; \ } PgPageChecksum##pageSize; CHECKSUM_UNION(pgPageSize1); CHECKSUM_UNION(pgPageSize2); CHECKSUM_UNION(pgPageSize4); CHECKSUM_UNION(pgPageSize8); CHECKSUM_UNION(pgPageSize16); CHECKSUM_UNION(pgPageSize32); /**********************************************************************************************************************************/ FN_EXTERN uint16_t pgPageChecksum(uint8_t *const page, const uint32_t blockNo, const PgPageSize pageSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(BYTEDATA, page); FUNCTION_TEST_PARAM(UINT, blockNo); FUNCTION_TEST_PARAM(ENUM, pageSize); FUNCTION_TEST_END(); // Save pd_checksum and temporarily set it to zero, so that the checksum calculation isn't affected by the old checksum stored // on the page. Restore it after, because actually updating the checksum is NOT part of the API of this function. const uint16_t checksumPrior = ((PageHeaderData *)page)->pd_checksum; ((PageHeaderData *)page)->pd_checksum = 0; // Initialize partial checksums to their corresponding offsets uint32_t sums[PARALLEL_SUM] = { 0x5b1f36e9, 0xb8525960, 0x02ab50aa, 0x1de66d2a, 0x79ff467a, 0x9bb9f8a3, 0x217e7cd2, 0x83e13d2c, 0xf8d4474f, 0xe39eb970, 0x42c6ae16, 0x993216fa, 0x7b093b5d, 0x98daff3c, 0xf718902a, 0x0b1c9cdb, 0xe58f764b, 0x187636bc, 0x5d7b3bb1, 0xe73de7de, 0x92bec979, 0xcca6c0b2, 0x304a0979, 0x85aa43d4, 0x783125bb, 0x6ca8eaa2, 0xe407eac6, 0x4b5cfc3e, 0x9fbf8c76, 0x15ca20be, 0xf2ca9fd3, 0x959bd756, }; // Main checksum calculation switch (pageSize) { CHECKSUM_CASE(pgPageSize8); // Default page size should be checked first CHECKSUM_CASE(pgPageSize1); CHECKSUM_CASE(pgPageSize2); CHECKSUM_CASE(pgPageSize4); CHECKSUM_CASE(pgPageSize16); CHECKSUM_CASE(pgPageSize32); default: pgPageSizeCheck(pageSize); } // Add in two rounds of zeroes for additional mixing for (uint32_t i = 0; i < 2; i++) for (uint32_t j = 0; j < PARALLEL_SUM; j++) CHECKSUM_ROUND(sums[j], 0); // Xor fold partial checksums together uint32 result = 0; for (uint32_t i = 0; i < PARALLEL_SUM; i++) result ^= sums[i]; // Restore prior checksum ((PageHeaderData *)page)->pd_checksum = checksumPrior; // Mix in the block number to detect transposed pages result ^= blockNo; // Reduce to a uint16 (to fit in the pd_checksum field) with an offset of one. That avoids checksums of zero, which seems like a // good idea. FUNCTION_TEST_RETURN(UINT16, (uint16_t)((result % 65535) + 1)); } /**********************************************************************************************************************************/ FN_EXTERN bool pgPageSizeValid(const PgPageSize pageSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, pageSize); FUNCTION_TEST_END(); switch (pageSize) { case pgPageSize1: case pgPageSize2: case pgPageSize4: case pgPageSize8: case pgPageSize16: case pgPageSize32: FUNCTION_TEST_RETURN(BOOL, true); } FUNCTION_TEST_RETURN(BOOL, false); } /**********************************************************************************************************************************/ FN_EXTERN void pgPageSizeCheck(const PgPageSize pageSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, pageSize); FUNCTION_TEST_END(); if (!pgPageSizeValid(pageSize)) { THROW_FMT( FormatError, "page size is %u but only %i, %i, %i, %i, %i, and %i are supported", pageSize, pgPageSize1, pgPageSize2, pgPageSize4, pgPageSize8, pgPageSize16, pgPageSize32); } FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/interface/static.vendor.h0000664000175100017510000003147215265101745022742 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Types That Do Not Vary By Version Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group Portions Copyright (c) 1994, Regents of the University of California For each supported release of PostgreSQL check the types in this file to see if they have changed. The easiest way to do this is to copy and paste in place and check git to see if there are any diffs. Tabs should be copied as is to make this process easy even though the pgBackRest project does not use tabs elsewhere. Comments should be copied with the types they apply to, even if the comment has not changed. This does get repetitive, but has no runtime cost and makes the rules a bit easier to follow. If a comment is changed then the newer comment should be copied. If the *type* has changed then it must be moved to version.auto.c which could have a large impact on dependencies. Hopefully that won't happen often. Note when adding new types it is safer to add them to version.auto.c unless they are needed for code that must be compatible across all versions of PostgreSQL supported by pgBackRest. ***********************************************************************************************************************************/ #ifndef POSTGRES_INTERFACE_STATICVENDOR_H #define POSTGRES_INTERFACE_STATICVENDOR_H #include "common/assert.h" #include "postgres/interface.h" /*********************************************************************************************************************************** Types from src/include/c.h ***********************************************************************************************************************************/ // uint16 type // --------------------------------------------------------------------------------------------------------------------------------- typedef uint16_t uint16; // uint32 type // --------------------------------------------------------------------------------------------------------------------------------- typedef uint32_t uint32; // uint64 type // --------------------------------------------------------------------------------------------------------------------------------- typedef uint64_t uint64; // TransactionId type // --------------------------------------------------------------------------------------------------------------------------------- typedef uint32 TransactionId; // FLEXIBLE_ARRAY_MEMBER macro // --------------------------------------------------------------------------------------------------------------------------------- /* * We require C99, hence the compiler should understand flexible array * members. However, for documentation purposes we still consider it to be * project style to write "field[FLEXIBLE_ARRAY_MEMBER]" not just "field[]". * When computing the size of such an object, use "offsetof(struct s, f)" * for portability. Don't use "offsetof(struct s, f[0])", as this doesn't * work with MSVC and with C++ compilers. */ #define FLEXIBLE_ARRAY_MEMBER /* empty */ /*********************************************************************************************************************************** Types from src/include/storage/checksum.h ***********************************************************************************************************************************/ // ChecksumStateType enum // --------------------------------------------------------------------------------------------------------------------------------- /* * Checksum state 0 is used for when data checksums are disabled (OFF), and * PG_DATA_CHECKSUM_VERSION defines that data checksums are enabled. New * states must be added at the end. */ typedef enum ChecksumStateType { PG_DATA_CHECKSUM_OFF = 0, PG_DATA_CHECKSUM_VERSION = 1, } ChecksumStateType; /*********************************************************************************************************************************** Types from src/include/storage/itemid.h ***********************************************************************************************************************************/ // ItemIdData type // --------------------------------------------------------------------------------------------------------------------------------- /* * A line pointer on a buffer page. See buffer page definitions and comments * for an explanation of how line pointers are used. * * In some cases a line pointer is "in use" but does not have any associated * storage on the page. By convention, lp_len == 0 in every line pointer * that does not have storage, independently of its lp_flags state. */ typedef struct ItemIdData { unsigned lp_off:15, /* offset to tuple (from start of page) */ lp_flags:2, /* state of line pointer, see below */ lp_len:15; /* byte length of tuple */ } ItemIdData; /*********************************************************************************************************************************** Types from src/include/storage/bufpage.h ***********************************************************************************************************************************/ // LocationIndex type // --------------------------------------------------------------------------------------------------------------------------------- /* * location (byte offset) within a page. * * note that this is actually limited to 2^15 because we have limited * ItemIdData.lp_off and ItemIdData.lp_len to 15 bits (see itemid.h). */ typedef uint16 LocationIndex; // PageXLogRecPtr type // --------------------------------------------------------------------------------------------------------------------------------- /* * Store the LSN as a single 64-bit value, to allow atomic loads/stores. * * For historical reasons, the storage of 64-bit LSN values depends on CPU * endianness; PageXLogRecPtr used to be a struct consisting of two 32-bit * values. When reading (and writing) the pd_lsn field from page headers, the * caller must convert from (and convert to) the platform's native endianness. */ typedef struct { uint64 lsn; } PageXLogRecPtr; // PageHeaderData type // --------------------------------------------------------------------------------------------------------------------------------- /* * disk page organization * * space management information generic to any page * * pd_lsn - identifies xlog record for last change to this page. * pd_checksum - page checksum, if set. * pd_flags - flag bits. * pd_lower - offset to start of free space. * pd_upper - offset to end of free space. * pd_special - offset to start of special space. * pd_pagesize_version - size in bytes and page layout version number. * pd_prune_xid - oldest XID among potentially prunable tuples on page. * * The LSN is used by the buffer manager to enforce the basic rule of WAL: * "thou shalt write xlog before data". A dirty buffer cannot be dumped * to disk until xlog has been flushed at least as far as the page's LSN. * * pd_checksum stores the page checksum, if it has been set for this page; * zero is a valid value for a checksum. If a checksum is not in use then * we leave the field unset. This will typically mean the field is zero * though non-zero values may also be present if databases have been * pg_upgraded from releases prior to 9.3, when the same byte offset was * used to store the current timelineid when the page was last updated. * Note that there is no indication on a page as to whether the checksum * is valid or not, a deliberate design choice which avoids the problem * of relying on the page contents to decide whether to verify it. Hence * there are no flag bits relating to checksums. * * pd_prune_xid is a hint field that helps determine whether pruning will be * useful. It is currently unused in index pages. * * The page version number and page size are packed together into a single * uint16 field. This is for historical reasons: before PostgreSQL 7.3, * there was no concept of a page version number, and doing it this way * lets us pretend that pre-7.3 databases have page version number zero. * We constrain page sizes to be multiples of 256, leaving the low eight * bits available for a version number. * * Minimum possible page size is perhaps 64B to fit page header, opaque space * and a minimal tuple; of course, in reality you want it much bigger, so * the constraint on pagesize mod 256 is not an important restriction. * On the high end, we can only support pages up to 32KB because lp_off/lp_len * are 15 bits. */ typedef struct PageHeaderData { /* XXX LSN is member of *any* block, not only page-organized ones */ PageXLogRecPtr pd_lsn; /* LSN: next byte after last byte of xlog * record for last change to this page */ uint16 pd_checksum; /* checksum */ uint16 pd_flags; /* flag bits, see below */ LocationIndex pd_lower; /* offset to start of free space */ LocationIndex pd_upper; /* offset to end of free space */ LocationIndex pd_special; /* offset to start of special space */ uint16 pd_pagesize_version; TransactionId pd_prune_xid; /* oldest prunable XID, or zero if none */ ItemIdData pd_linp[FLEXIBLE_ARRAY_MEMBER]; /* line pointer array */ } PageHeaderData; // PageHeader type // --------------------------------------------------------------------------------------------------------------------------------- typedef PageHeaderData *PageHeader; /*********************************************************************************************************************************** Types from src/include/access/transam.h ***********************************************************************************************************************************/ // FirstNormalObjectId define // --------------------------------------------------------------------------------------------------------------------------------- /* * Object ID (OID) zero is InvalidOid. * * OIDs 1-9999 are reserved for manual assignment (see .dat files in * src/include/catalog/). Of these, 8000-9999 are reserved for * development purposes (such as in-progress patches and forks); * they should not appear in released versions. * * OIDs 10000-11999 are reserved for assignment by genbki.pl, for use * when the .dat files in src/include/catalog/ do not specify an OID * for a catalog entry that requires one. Note that genbki.pl assigns * these OIDs independently in each catalog, so they're not guaranteed * to be globally unique. Furthermore, the bootstrap backend and * initdb's post-bootstrap processing can also assign OIDs in this range. * The normal OID-generation logic takes care of any OID conflicts that * might arise from that. * * OIDs 12000-16383 are reserved for unpinned objects created by initdb's * post-bootstrap processing. initdb forces the OID generator up to * 12000 as soon as it's made the pinned objects it's responsible for. * * OIDs beginning at 16384 are assigned from the OID generator * during normal multiuser operation. (We force the generator up to * 16384 as soon as we are in normal operation.) * * The choices of 8000, 10000 and 12000 are completely arbitrary, and can be * moved if we run low on OIDs in any category. Changing the macros below, * and updating relevant documentation (see bki.sgml and RELEASE_CHANGES), * should be sufficient to do this. Moving the 16384 boundary between * initdb-assigned OIDs and user-defined objects would be substantially * more painful, however, since some user-defined OIDs will appear in * on-disk data; such a change would probably break pg_upgrade. * * NOTE: if the OID generator wraps around, we skip over OIDs 0-16383 * and resume with 16384. This minimizes the odds of OID conflict, by not * reassigning OIDs that might have been assigned during initdb. Critically, * it also ensures that no user-created object will be considered pinned. */ #define FirstNormalObjectId 16384 /*********************************************************************************************************************************** Types from src/include/access/xlog_internal.h ***********************************************************************************************************************************/ // WalSegMinSize/WalSegMinSize macros // --------------------------------------------------------------------------------------------------------------------------------- /* wal_segment_size can range from 1MB to 1GB */ #define WalSegMinSize 1024 * 1024 #define WalSegMaxSize 1024 * 1024 * 1024 // IsPowerOf2/IsValidWalSegSize macros // --------------------------------------------------------------------------------------------------------------------------------- /* check that the given size is a valid wal_segment_size */ #define IsPowerOf2(x) (x > 0 && ((x) & ((x)-1)) == 0) #define IsValidWalSegSize(size) \ (IsPowerOf2(size) && \ ((size) >= WalSegMinSize && (size) <= WalSegMaxSize)) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/interface/version.intern.h0000664000175100017510000002616715265101745023147 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Version Interface Macros for building version-specific functions that interface with the types in version.vendor.h. Due to the way PostgreSQL types evolve over time, this seems to be the easiest way to extract information from them. These macros should be kept as simple as possible, with most of the logic contained in postgres/interface.c. ***********************************************************************************************************************************/ #include "postgres/interface/version.vendor.h" #include "postgres/version.h" /*********************************************************************************************************************************** Values that vary by version Each is captured as a constant named for the version so that the interface struct, which is built after every version has undefined what it defined, can still name it. A define cannot do this because it is expanded where it is used rather than where it is made, by which time the value it was given is gone. One macro per value so that a value which does not vary between two versions is captured once for both, the same way a function that does not vary is rendered once for both. ***********************************************************************************************************************************/ // Control version that pg_control matches for this version of PostgreSQL #define PG_INTERFACE_VALUE_CONTROL_VERSION(version) \ enum {pgInterfaceControlVersion##version = PG_CONTROL_VERSION} // Catalog version that pg_control matches for this version of PostgreSQL #define PG_INTERFACE_VALUE_CATALOG_VERSION(version) \ enum {pgInterfaceCatalogVersion##version = CATALOG_VERSION_NO} // Magic that the WAL header matches for this version of PostgreSQL #define PG_INTERFACE_VALUE_WAL_MAGIC(version) \ enum {pgInterfaceWalMagic##version = XLOG_PAGE_MAGIC} // Offset of the crc in pg_control, which is also the length of the part the crc is calculated over since the crc is last #define PG_INTERFACE_VALUE_CONTROL_CRC_OFFSET(version) \ enum {pgInterfaceControlCrcOffset##version = offsetof(ControlFileData, crc)} /*********************************************************************************************************************************** Determine whether pg_control matches an interface Rendered once after the versions, in whichever form fits the versions that were rendered. The range form is rendered when one of them has not been released, since only then is there an interface accepting more than a single catalog version. Rendering the form that is used means there is never a comparison that no supported version can fail. An unreleased version accepts any catalog version until the end of the year the one it was built with encodes. ***********************************************************************************************************************************/ #define PG_INTERFACE_CONTROL_MATCH() \ static bool \ pgInterfaceControlMatch(const PgInterface *const interface, const PgControlCommon *const control) \ { \ return control->controlVersion == interface->controlVersion && control->catalogVersion == interface->catalogVersion; \ } #define PG_INTERFACE_CONTROL_MATCH_RANGE() \ static bool \ pgInterfaceControlMatch(const PgInterface *const interface, const PgControlCommon *const control) \ { \ if (control->controlVersion != interface->controlVersion) \ return false; \ \ if (interface->unreleased) \ { \ return \ control->catalogVersion >= interface->catalogVersion && \ control->catalogVersion < (interface->catalogVersion / 100000 + 1) * 100000; \ } \ \ return control->catalogVersion == interface->catalogVersion; \ } /*********************************************************************************************************************************** Read the version specific pg_control into a general data structure ***********************************************************************************************************************************/ #define PG_INTERFACE_CONTROL(version) \ static PgControl \ pgInterfaceControl##version(const uint8_t *controlFile) \ { \ ASSERT(controlFile != NULL); \ \ return (PgControl) \ { \ .systemId = ((const ControlFileData *)controlFile)->system_identifier, \ .catalogVersion = ((const ControlFileData *)controlFile)->catalog_version_no, \ .checkpoint = ((const ControlFileData *)controlFile)->checkPoint, \ .timeline = ((const ControlFileData *)controlFile)->checkPointCopy.ThisTimeLineID, \ .pageSize = ((const ControlFileData *)controlFile)->blcksz, \ .walSegmentSize = ((const ControlFileData *)controlFile)->xlog_seg_size, \ .pageChecksumVersion = ((const ControlFileData *)controlFile)->data_checksum_version, \ }; \ } /*********************************************************************************************************************************** Invalidate control checkpoint. PostgreSQL skips the first segment so any LSN in that segment is invalid. ***********************************************************************************************************************************/ #define PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(version) \ static void \ pgInterfaceControlCheckpointInvalidate##version(uint8_t *const controlFile) \ { \ ((ControlFileData *)controlFile)->checkPoint = PG_CONTROL_CHECKPOINT_INVALID; \ } /*********************************************************************************************************************************** Read the version specific WAL header into a general data structure ***********************************************************************************************************************************/ #define PG_INTERFACE_WAL(version) \ static PgWal \ pgInterfaceWal##version(const uint8_t *walFile) \ { \ ASSERT(walFile != NULL); \ \ return (PgWal) \ { \ .systemId = ((const XLogLongPageHeaderData *)walFile)->xlp_sysid, \ .size = ((const XLogLongPageHeaderData *)walFile)->xlp_seg_size, \ }; \ } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/interface/version.vendor.h0000664000175100017510000021141615265101745023136 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Types That Vary By Version Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group Portions Copyright (c) 1994, Regents of the University of California For each supported release of PostgreSQL check the types in this file to see if they have changed. The easiest way to do this is to copy and paste in place and check git to see if there are any diffs. Tabs should be copied as is to make this process easy even though the pgBackRest project does not use tabs elsewhere. New versions should always be added to the top of each type's #if block, underneath `PG_VERSION > PG_VERSION_MAX` to cause as little churn as possible. This also ensures that new versions will not work until PG_VERSION_MAX and this file have been updated. New data structures do not need to add #elif branches for old versions. See FullTransactionId as an example. Comments should be copied with the types they apply to, even if the comment has not changed. This does get repetitive, but has no runtime cost and makes the rules a bit easier to follow. If a comment has syntax only changes, then the new version of the comment can be applied to older versions of the type. If a comment has changed in a way that implies a difference in the way the type is used, then a new version of the comment and type should be created. Unversioned types that are only used by versioned types are also included in this file. ***********************************************************************************************************************************/ #include "postgres/interface/static.vendor.h" #include "postgres/version.h" /*********************************************************************************************************************************** Types from src/include/c.h ***********************************************************************************************************************************/ // int64 type. The definition in c.h is more complicated but here we can rely on stdint.h for the correct type. // --------------------------------------------------------------------------------------------------------------------------------- typedef int64_t int64; // MultiXactId type // --------------------------------------------------------------------------------------------------------------------------------- /* MultiXactId must be equivalent to TransactionId, to fit in t_xmax */ typedef TransactionId MultiXactId; // MultiXactOffset // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_19 typedef uint64 MultiXactOffset; #elif PG_VERSION >= PG_VERSION_96 typedef uint32 MultiXactOffset; #endif /*********************************************************************************************************************************** Types from src/include/pgtime.h ***********************************************************************************************************************************/ // pg_time_t type // --------------------------------------------------------------------------------------------------------------------------------- /* * The API of this library is generally similar to the corresponding * C library functions, except that we use pg_time_t which (we hope) is * 64 bits wide, and which is most definitely signed not unsigned. */ typedef int64 pg_time_t; /*********************************************************************************************************************************** Types from src/include/postgres_ext.h ***********************************************************************************************************************************/ // Oid Type // --------------------------------------------------------------------------------------------------------------------------------- /* * Object ID is a fundamental type in Postgres. */ typedef unsigned int Oid; /*********************************************************************************************************************************** Types from src/include/port/pg_crc32c.h ***********************************************************************************************************************************/ // pg_crc32/c type // --------------------------------------------------------------------------------------------------------------------------------- typedef uint32 pg_crc32c; /*********************************************************************************************************************************** Types from src/include/access/xlogdefs.h ***********************************************************************************************************************************/ // XLogRecPtr type // --------------------------------------------------------------------------------------------------------------------------------- /* * Pointer to a location in the XLOG. These pointers are 64 bits wide, * because we don't want them ever to overflow. */ typedef uint64 XLogRecPtr; // TimeLineID type // --------------------------------------------------------------------------------------------------------------------------------- /* * TimeLineID (TLI) - identifies different database histories to prevent * confusion after restoring a prior state of a database installation. * TLI does not change in a normal stop/restart of the database (including * crash-and-recover cases); but we must assign a new TLI after doing * a recovery to a prior state, a/k/a point-in-time recovery. This makes * the new WAL logfile sequence we generate distinguishable from the * sequence that was generated in the previous incarnation. */ typedef uint32 TimeLineID; /*********************************************************************************************************************************** Types from src/include/catalog/catversion.h ***********************************************************************************************************************************/ // CATALOG_VERSION_NO define // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_19 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202605131 #elif PG_VERSION >= PG_VERSION_18 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202506291 #elif PG_VERSION >= PG_VERSION_17 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202406281 #elif PG_VERSION >= PG_VERSION_16 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202307071 #elif PG_VERSION >= PG_VERSION_15 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202209061 #elif PG_VERSION >= PG_VERSION_14 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202107181 #elif PG_VERSION >= PG_VERSION_13 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202007201 #elif PG_VERSION >= PG_VERSION_12 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 201909212 #elif PG_VERSION >= PG_VERSION_11 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 201809051 #elif PG_VERSION >= PG_VERSION_10 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 201707211 #elif PG_VERSION >= PG_VERSION_96 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 201608131 #endif /*********************************************************************************************************************************** Types from src/include/access/transam.h ***********************************************************************************************************************************/ // FullTransactionId type // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_12 /* * A 64 bit value that contains an epoch and a TransactionId. This is * wrapped in a struct to prevent implicit conversion to/from TransactionId. * Not all values represent valid normal XIDs. */ typedef struct FullTransactionId { uint64 value; } FullTransactionId; #endif /*********************************************************************************************************************************** Types from src/include/catalog/pg_control.h ***********************************************************************************************************************************/ // PG_CONTROL_VERSION define // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_19 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1905 #elif PG_VERSION >= PG_VERSION_18 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1800 #elif PG_VERSION >= PG_VERSION_17 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1700 #elif PG_VERSION >= PG_VERSION_13 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1300 #elif PG_VERSION >= PG_VERSION_12 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1201 #elif PG_VERSION >= PG_VERSION_11 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1100 #elif PG_VERSION >= PG_VERSION_10 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1002 #elif PG_VERSION >= PG_VERSION_96 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 960 #endif // MOCK_AUTH_NONCE_LEN define // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_10 /* Nonce key length, see below */ #define MOCK_AUTH_NONCE_LEN 32 #endif // CheckPoint Type // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_19 /* * Body of CheckPoint XLOG records. This is declared here because we keep * a copy of the latest one in pg_control for possible disaster recovery. * Changing this struct requires a PG_CONTROL_VERSION bump. */ typedef struct CheckPoint { XLogRecPtr redo; /* next RecPtr available when we began to * create CheckPoint (i.e. REDO start point) */ TimeLineID ThisTimeLineID; /* current TLI */ TimeLineID PrevTimeLineID; /* previous TLI, if this record begins a new * timeline (equals ThisTimeLineID otherwise) */ bool fullPageWrites; /* current full_page_writes */ int wal_level; /* current wal_level */ bool logicalDecodingEnabled; /* current logical decoding status */ FullTransactionId nextXid; /* next free transaction ID */ Oid nextOid; /* next free OID */ MultiXactId nextMulti; /* next free MultiXactId */ MultiXactOffset nextMultiOffset; /* next free MultiXact offset */ TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */ Oid oldestXidDB; /* database with minimum datfrozenxid */ MultiXactId oldestMulti; /* cluster-wide minimum datminmxid */ Oid oldestMultiDB; /* database with minimum datminmxid */ pg_time_t time; /* time stamp of checkpoint */ TransactionId oldestCommitTsXid; /* oldest Xid with valid commit * timestamp */ TransactionId newestCommitTsXid; /* newest Xid with valid commit * timestamp */ /* * Oldest XID still running. This is only needed to initialize hot standby * mode from an online checkpoint, so we only bother calculating this for * online checkpoints and only when wal_level is replica. Otherwise it's * set to InvalidTransactionId. */ TransactionId oldestActiveXid; } CheckPoint; #elif PG_VERSION >= PG_VERSION_17 /* * Body of CheckPoint XLOG records. This is declared here because we keep * a copy of the latest one in pg_control for possible disaster recovery. * Changing this struct requires a PG_CONTROL_VERSION bump. */ typedef struct CheckPoint { XLogRecPtr redo; /* next RecPtr available when we began to * create CheckPoint (i.e. REDO start point) */ TimeLineID ThisTimeLineID; /* current TLI */ TimeLineID PrevTimeLineID; /* previous TLI, if this record begins a new * timeline (equals ThisTimeLineID otherwise) */ bool fullPageWrites; /* current full_page_writes */ int wal_level; /* current wal_level */ FullTransactionId nextXid; /* next free transaction ID */ Oid nextOid; /* next free OID */ MultiXactId nextMulti; /* next free MultiXactId */ MultiXactOffset nextMultiOffset; /* next free MultiXact offset */ TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */ Oid oldestXidDB; /* database with minimum datfrozenxid */ MultiXactId oldestMulti; /* cluster-wide minimum datminmxid */ Oid oldestMultiDB; /* database with minimum datminmxid */ pg_time_t time; /* time stamp of checkpoint */ TransactionId oldestCommitTsXid; /* oldest Xid with valid commit * timestamp */ TransactionId newestCommitTsXid; /* newest Xid with valid commit * timestamp */ /* * Oldest XID still running. This is only needed to initialize hot standby * mode from an online checkpoint, so we only bother calculating this for * online checkpoints and only when wal_level is replica. Otherwise it's * set to InvalidTransactionId. */ TransactionId oldestActiveXid; } CheckPoint; #elif PG_VERSION >= PG_VERSION_14 /* * Body of CheckPoint XLOG records. This is declared here because we keep * a copy of the latest one in pg_control for possible disaster recovery. * Changing this struct requires a PG_CONTROL_VERSION bump. */ typedef struct CheckPoint { XLogRecPtr redo; /* next RecPtr available when we began to * create CheckPoint (i.e. REDO start point) */ TimeLineID ThisTimeLineID; /* current TLI */ TimeLineID PrevTimeLineID; /* previous TLI, if this record begins a new * timeline (equals ThisTimeLineID otherwise) */ bool fullPageWrites; /* current full_page_writes */ FullTransactionId nextXid; /* next free transaction ID */ Oid nextOid; /* next free OID */ MultiXactId nextMulti; /* next free MultiXactId */ MultiXactOffset nextMultiOffset; /* next free MultiXact offset */ TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */ Oid oldestXidDB; /* database with minimum datfrozenxid */ MultiXactId oldestMulti; /* cluster-wide minimum datminmxid */ Oid oldestMultiDB; /* database with minimum datminmxid */ pg_time_t time; /* time stamp of checkpoint */ TransactionId oldestCommitTsXid; /* oldest Xid with valid commit * timestamp */ TransactionId newestCommitTsXid; /* newest Xid with valid commit * timestamp */ /* * Oldest XID still running. This is only needed to initialize hot standby * mode from an online checkpoint, so we only bother calculating this for * online checkpoints and only when wal_level is replica. Otherwise it's * set to InvalidTransactionId. */ TransactionId oldestActiveXid; } CheckPoint; #elif PG_VERSION >= PG_VERSION_12 /* * Body of CheckPoint XLOG records. This is declared here because we keep * a copy of the latest one in pg_control for possible disaster recovery. * Changing this struct requires a PG_CONTROL_VERSION bump. */ typedef struct CheckPoint { XLogRecPtr redo; /* next RecPtr available when we began to * create CheckPoint (i.e. REDO start point) */ TimeLineID ThisTimeLineID; /* current TLI */ TimeLineID PrevTimeLineID; /* previous TLI, if this record begins a new * timeline (equals ThisTimeLineID otherwise) */ bool fullPageWrites; /* current full_page_writes */ FullTransactionId nextFullXid; /* next free full transaction ID */ Oid nextOid; /* next free OID */ MultiXactId nextMulti; /* next free MultiXactId */ MultiXactOffset nextMultiOffset; /* next free MultiXact offset */ TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */ Oid oldestXidDB; /* database with minimum datfrozenxid */ MultiXactId oldestMulti; /* cluster-wide minimum datminmxid */ Oid oldestMultiDB; /* database with minimum datminmxid */ pg_time_t time; /* time stamp of checkpoint */ TransactionId oldestCommitTsXid; /* oldest Xid with valid commit * timestamp */ TransactionId newestCommitTsXid; /* newest Xid with valid commit * timestamp */ /* * Oldest XID still running. This is only needed to initialize hot standby * mode from an online checkpoint, so we only bother calculating this for * online checkpoints and only when wal_level is replica. Otherwise it's * set to InvalidTransactionId. */ TransactionId oldestActiveXid; } CheckPoint; #elif PG_VERSION >= PG_VERSION_96 /* * Body of CheckPoint XLOG records. This is declared here because we keep * a copy of the latest one in pg_control for possible disaster recovery. * Changing this struct requires a PG_CONTROL_VERSION bump. */ typedef struct CheckPoint { XLogRecPtr redo; /* next RecPtr available when we began to * create CheckPoint (i.e. REDO start point) */ TimeLineID ThisTimeLineID; /* current TLI */ TimeLineID PrevTimeLineID; /* previous TLI, if this record begins a new * timeline (equals ThisTimeLineID otherwise) */ bool fullPageWrites; /* current full_page_writes */ uint32 nextXidEpoch; /* higher-order bits of nextXid */ TransactionId nextXid; /* next free XID */ Oid nextOid; /* next free OID */ MultiXactId nextMulti; /* next free MultiXactId */ MultiXactOffset nextMultiOffset; /* next free MultiXact offset */ TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */ Oid oldestXidDB; /* database with minimum datfrozenxid */ MultiXactId oldestMulti; /* cluster-wide minimum datminmxid */ Oid oldestMultiDB; /* database with minimum datminmxid */ pg_time_t time; /* time stamp of checkpoint */ TransactionId oldestCommitTsXid; /* oldest Xid with valid commit * timestamp */ TransactionId newestCommitTsXid; /* newest Xid with valid commit * timestamp */ /* * Oldest XID still running. This is only needed to initialize hot standby * mode from an online checkpoint, so we only bother calculating this for * online checkpoints and only when wal_level is replica. Otherwise it's * set to InvalidTransactionId. */ TransactionId oldestActiveXid; } CheckPoint; #endif // DBState enum // --------------------------------------------------------------------------------------------------------------------------------- /* * System status indicator. Note this is stored in pg_control; if you change * it, you must bump PG_CONTROL_VERSION */ typedef enum DBState { DB_STARTUP = 0, DB_SHUTDOWNED, DB_SHUTDOWNED_IN_RECOVERY, DB_SHUTDOWNING, DB_IN_CRASH_RECOVERY, DB_IN_ARCHIVE_RECOVERY, DB_IN_PRODUCTION, } DBState; // ControlFileData type // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_19 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_wal_senders; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 slru_pages_per_segment; /* size of each SLRU segment */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* * Data checksum state at cluster initialization. Since the state can be * changed during runtime, we need to store the initial value for system * functions which report initdb settings. */ uint32 data_checksum_version_init; /* Current data checksums state */ uint32 data_checksum_version; /* * True if the default signedness of char is "signed" on a platform where * the cluster is initialized. */ bool default_char_signedness; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_18 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_wal_senders; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* * True if the default signedness of char is "signed" on a platform where * the cluster is initialized. */ bool default_char_signedness; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_15 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_wal_senders; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_13 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. If it's false, but backupStartPoint is set, a backup_label * file was found at startup but it may have been a leftover from a stray * pg_start_backup() call, not accompanied by pg_stop_backup(). */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_wal_senders; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_12 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. If it's false, but backupStartPoint is set, a backup_label * file was found at startup but it may have been a leftover from a stray * pg_start_backup() call, not accompanied by pg_stop_backup(). */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_wal_senders; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ /* flags indicating pass-by-value status of various types */ bool float4ByVal; /* float4 pass-by-value? */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_11 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. If it's false, but backupStartPoint is set, a backup_label * file was found at startup but it may have been a leftover from a stray * pg_start_backup() call, not accompanied by pg_stop_backup(). */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ /* flags indicating pass-by-value status of various types */ bool float4ByVal; /* float4 pass-by-value? */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_10 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ XLogRecPtr prevCheckPoint; /* previous check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. If it's false, but backupStartPoint is set, a backup_label * file was found at startup but it may have been a leftover from a stray * pg_start_backup() call, not accompanied by pg_stop_backup(). */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ /* flags indicating pass-by-value status of various types */ bool float4ByVal; /* float4 pass-by-value? */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_96 /* * Contents of pg_control. * * NOTE: try to keep this under 512 bytes so that it will fit on one physical * sector of typical disk drives. This reduces the odds of corruption due to * power failure midway through a write. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ XLogRecPtr prevCheckPoint; /* previous check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. If it's false, but backupStartPoint is set, a backup_label * file was found at startup but it may have been a leftover from a stray * pg_start_backup() call, not accompanied by pg_stop_backup(). */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ /* flag indicating internal format of timestamp, interval, time */ bool enableIntTimes; /* int64 storage enabled? */ /* flags indicating pass-by-value status of various types */ bool float4ByVal; /* float4 pass-by-value? */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #endif /*********************************************************************************************************************************** Types from src/include/access/xlog_internal.h ***********************************************************************************************************************************/ // XLOG_PAGE_MAGIC define // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_19 #define XLOG_PAGE_MAGIC 0xD122 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_18 #define XLOG_PAGE_MAGIC 0xD118 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_17 #define XLOG_PAGE_MAGIC 0xD116 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_16 #define XLOG_PAGE_MAGIC 0xD113 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_15 #define XLOG_PAGE_MAGIC 0xD110 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_14 #define XLOG_PAGE_MAGIC 0xD10D /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_13 #define XLOG_PAGE_MAGIC 0xD106 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_12 #define XLOG_PAGE_MAGIC 0xD101 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_11 #define XLOG_PAGE_MAGIC 0xD098 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_10 #define XLOG_PAGE_MAGIC 0xD097 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_96 #define XLOG_PAGE_MAGIC 0xD093 /* can be used as WAL version indicator */ #endif // XLogPageHeaderData type // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_14 /* * Each page of XLOG file has a header like this: */ typedef struct XLogPageHeaderData { uint16 xlp_magic; /* magic value for correctness checks */ uint16 xlp_info; /* flag bits, see below */ TimeLineID xlp_tli; /* TimeLineID of first record on page */ XLogRecPtr xlp_pageaddr; /* XLOG address of this page */ /* * When there is not enough space on current page for whole record, we * continue on the next page. xlp_rem_len is the number of bytes * remaining from a previous page; it tracks xl_tot_len in the initial * header. Note that the continuation data isn't necessarily aligned. */ uint32 xlp_rem_len; /* total len of remaining data for record */ } XLogPageHeaderData; #elif PG_VERSION >= PG_VERSION_96 /* * Each page of XLOG file has a header like this: */ typedef struct XLogPageHeaderData { uint16 xlp_magic; /* magic value for correctness checks */ uint16 xlp_info; /* flag bits, see below */ TimeLineID xlp_tli; /* TimeLineID of first record on page */ XLogRecPtr xlp_pageaddr; /* XLOG address of this page */ /* * When there is not enough space on current page for whole record, we * continue on the next page. xlp_rem_len is the number of bytes * remaining from a previous page. * * Note that xlp_rem_len includes backup-block data; that is, it tracks * xl_tot_len not xl_len in the initial header. Also note that the * continuation data isn't necessarily aligned. */ uint32 xlp_rem_len; /* total len of remaining data for record */ } XLogPageHeaderData; #endif // XLogLongPageHeaderData type // --------------------------------------------------------------------------------------------------------------------------------- /* * When the XLP_LONG_HEADER flag is set, we store additional fields in the * page header. (This is ordinarily done just in the first page of an * XLOG file.) The additional fields serve to identify the file accurately. */ typedef struct XLogLongPageHeaderData { XLogPageHeaderData std; /* standard header fields */ uint64 xlp_sysid; /* system identifier from pg_control */ uint32 xlp_seg_size; /* just as a cross-check */ uint32 xlp_xlog_blcksz; /* just as a cross-check */ } XLogLongPageHeaderData; // XLP_LONG_HEADER define // --------------------------------------------------------------------------------------------------------------------------------- /* This flag indicates a "long" page header */ #define XLP_LONG_HEADER 0x0002 ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1792312410.769613 pgbackrest-2.60.0/src/postgres/interface.auto.c.inc0000644000175100017510000012042215265102133021651 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Interface Automatically generated by 'build.py postgres' -- do not modify directly. ***********************************************************************************************************************************/ /*********************************************************************************************************************************** PostgreSQL 19 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_19 #define CheckPoint CheckPoint_19 #define ControlFileData ControlFileData_19 #define DBState DBState_19 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_19 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_19 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_19 #define DB_SHUTDOWNED DB_SHUTDOWNED_19 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_19 #define DB_SHUTDOWNING DB_SHUTDOWNING_19 #define DB_STARTUP DB_STARTUP_19 #define FullTransactionId FullTransactionId_19 #define MultiXactId MultiXactId_19 #define MultiXactOffset MultiXactOffset_19 #define Oid Oid_19 #define TimeLineID TimeLineID_19 #define XLogLongPageHeaderData XLogLongPageHeaderData_19 #define XLogPageHeaderData XLogPageHeaderData_19 #define XLogRecPtr XLogRecPtr_19 #define int64 int64_19 #define pg_crc32c pg_crc32c_19 #define pg_time_t pg_time_t_19 #define CATALOG_VERSION_NO_MAX #include "postgres/interface/version.intern.h" PG_INTERFACE_VALUE_CONTROL_VERSION(19); PG_INTERFACE_VALUE_CATALOG_VERSION(19); PG_INTERFACE_VALUE_WAL_MAGIC(19); PG_INTERFACE_VALUE_CONTROL_CRC_OFFSET(19); PG_INTERFACE_CONTROL(19); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(19); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 18 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_18 #define CheckPoint CheckPoint_18 #define ControlFileData ControlFileData_18 #define DBState DBState_18 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_18 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_18 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_18 #define DB_SHUTDOWNED DB_SHUTDOWNED_18 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_18 #define DB_SHUTDOWNING DB_SHUTDOWNING_18 #define DB_STARTUP DB_STARTUP_18 #define FullTransactionId FullTransactionId_18 #define MultiXactId MultiXactId_18 #define MultiXactOffset MultiXactOffset_18 #define Oid Oid_18 #define TimeLineID TimeLineID_18 #define XLogLongPageHeaderData XLogLongPageHeaderData_18 #define XLogPageHeaderData XLogPageHeaderData_18 #define XLogRecPtr XLogRecPtr_18 #define int64 int64_18 #define pg_crc32c pg_crc32c_18 #define pg_time_t pg_time_t_18 #include "postgres/interface/version.intern.h" PG_INTERFACE_VALUE_CONTROL_VERSION(18); PG_INTERFACE_VALUE_CATALOG_VERSION(18); PG_INTERFACE_VALUE_WAL_MAGIC(18); PG_INTERFACE_VALUE_CONTROL_CRC_OFFSET(18); PG_INTERFACE_CONTROL(18); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(18); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 17 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_17 #define CheckPoint CheckPoint_17 #define ControlFileData ControlFileData_17 #define DBState DBState_17 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_17 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_17 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_17 #define DB_SHUTDOWNED DB_SHUTDOWNED_17 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_17 #define DB_SHUTDOWNING DB_SHUTDOWNING_17 #define DB_STARTUP DB_STARTUP_17 #define FullTransactionId FullTransactionId_17 #define MultiXactId MultiXactId_17 #define MultiXactOffset MultiXactOffset_17 #define Oid Oid_17 #define TimeLineID TimeLineID_17 #define XLogLongPageHeaderData XLogLongPageHeaderData_17 #define XLogPageHeaderData XLogPageHeaderData_17 #define XLogRecPtr XLogRecPtr_17 #define int64 int64_17 #define pg_crc32c pg_crc32c_17 #define pg_time_t pg_time_t_17 #include "postgres/interface/version.intern.h" PG_INTERFACE_VALUE_CONTROL_VERSION(17); PG_INTERFACE_VALUE_CATALOG_VERSION(17); PG_INTERFACE_VALUE_WAL_MAGIC(17); PG_INTERFACE_VALUE_CONTROL_CRC_OFFSET(17); PG_INTERFACE_CONTROL(17); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(17); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 16 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_16 #define CheckPoint CheckPoint_16 #define ControlFileData ControlFileData_16 #define DBState DBState_16 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_16 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_16 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_16 #define DB_SHUTDOWNED DB_SHUTDOWNED_16 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_16 #define DB_SHUTDOWNING DB_SHUTDOWNING_16 #define DB_STARTUP DB_STARTUP_16 #define FullTransactionId FullTransactionId_16 #define MultiXactId MultiXactId_16 #define MultiXactOffset MultiXactOffset_16 #define Oid Oid_16 #define TimeLineID TimeLineID_16 #define XLogLongPageHeaderData XLogLongPageHeaderData_16 #define XLogPageHeaderData XLogPageHeaderData_16 #define XLogRecPtr XLogRecPtr_16 #define int64 int64_16 #define pg_crc32c pg_crc32c_16 #define pg_time_t pg_time_t_16 #include "postgres/interface/version.intern.h" PG_INTERFACE_VALUE_CATALOG_VERSION(16); PG_INTERFACE_VALUE_WAL_MAGIC(16); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 15 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_15 #define CheckPoint CheckPoint_15 #define ControlFileData ControlFileData_15 #define DBState DBState_15 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_15 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_15 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_15 #define DB_SHUTDOWNED DB_SHUTDOWNED_15 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_15 #define DB_SHUTDOWNING DB_SHUTDOWNING_15 #define DB_STARTUP DB_STARTUP_15 #define FullTransactionId FullTransactionId_15 #define MultiXactId MultiXactId_15 #define MultiXactOffset MultiXactOffset_15 #define Oid Oid_15 #define TimeLineID TimeLineID_15 #define XLogLongPageHeaderData XLogLongPageHeaderData_15 #define XLogPageHeaderData XLogPageHeaderData_15 #define XLogRecPtr XLogRecPtr_15 #define int64 int64_15 #define pg_crc32c pg_crc32c_15 #define pg_time_t pg_time_t_15 #include "postgres/interface/version.intern.h" PG_INTERFACE_VALUE_CATALOG_VERSION(15); PG_INTERFACE_VALUE_WAL_MAGIC(15); PG_INTERFACE_VALUE_CONTROL_CRC_OFFSET(15_16); // Shared with 16 PG_INTERFACE_CONTROL(15_16); // Shared with 16 PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(15_16); // Shared with 16 #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 14 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_14 #define CheckPoint CheckPoint_14 #define ControlFileData ControlFileData_14 #define DBState DBState_14 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_14 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_14 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_14 #define DB_SHUTDOWNED DB_SHUTDOWNED_14 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_14 #define DB_SHUTDOWNING DB_SHUTDOWNING_14 #define DB_STARTUP DB_STARTUP_14 #define FullTransactionId FullTransactionId_14 #define MultiXactId MultiXactId_14 #define MultiXactOffset MultiXactOffset_14 #define Oid Oid_14 #define TimeLineID TimeLineID_14 #define XLogLongPageHeaderData XLogLongPageHeaderData_14 #define XLogPageHeaderData XLogPageHeaderData_14 #define XLogRecPtr XLogRecPtr_14 #define int64 int64_14 #define pg_crc32c pg_crc32c_14 #define pg_time_t pg_time_t_14 #include "postgres/interface/version.intern.h" PG_INTERFACE_VALUE_CATALOG_VERSION(14); PG_INTERFACE_VALUE_WAL_MAGIC(14); PG_INTERFACE_VALUE_CONTROL_CRC_OFFSET(14); PG_INTERFACE_CONTROL(14); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(14); PG_INTERFACE_WAL(14_19); // Shared with 15, 16, 17, 18, 19 #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 13 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_13 #define CheckPoint CheckPoint_13 #define ControlFileData ControlFileData_13 #define DBState DBState_13 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_13 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_13 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_13 #define DB_SHUTDOWNED DB_SHUTDOWNED_13 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_13 #define DB_SHUTDOWNING DB_SHUTDOWNING_13 #define DB_STARTUP DB_STARTUP_13 #define FullTransactionId FullTransactionId_13 #define MultiXactId MultiXactId_13 #define MultiXactOffset MultiXactOffset_13 #define Oid Oid_13 #define TimeLineID TimeLineID_13 #define XLogLongPageHeaderData XLogLongPageHeaderData_13 #define XLogPageHeaderData XLogPageHeaderData_13 #define XLogRecPtr XLogRecPtr_13 #define int64 int64_13 #define pg_crc32c pg_crc32c_13 #define pg_time_t pg_time_t_13 #include "postgres/interface/version.intern.h" PG_INTERFACE_VALUE_CONTROL_VERSION(13_16); // Shared with 14, 15, 16 PG_INTERFACE_VALUE_CATALOG_VERSION(13); PG_INTERFACE_VALUE_WAL_MAGIC(13); PG_INTERFACE_VALUE_CONTROL_CRC_OFFSET(13); PG_INTERFACE_CONTROL(13); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(13); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 12 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_12 #define CheckPoint CheckPoint_12 #define ControlFileData ControlFileData_12 #define DBState DBState_12 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_12 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_12 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_12 #define DB_SHUTDOWNED DB_SHUTDOWNED_12 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_12 #define DB_SHUTDOWNING DB_SHUTDOWNING_12 #define DB_STARTUP DB_STARTUP_12 #define FullTransactionId FullTransactionId_12 #define MultiXactId MultiXactId_12 #define MultiXactOffset MultiXactOffset_12 #define Oid Oid_12 #define TimeLineID TimeLineID_12 #define XLogLongPageHeaderData XLogLongPageHeaderData_12 #define XLogPageHeaderData XLogPageHeaderData_12 #define XLogRecPtr XLogRecPtr_12 #define int64 int64_12 #define pg_crc32c pg_crc32c_12 #define pg_time_t pg_time_t_12 #include "postgres/interface/version.intern.h" PG_INTERFACE_VALUE_CONTROL_VERSION(12); PG_INTERFACE_VALUE_CATALOG_VERSION(12); PG_INTERFACE_VALUE_WAL_MAGIC(12); PG_INTERFACE_VALUE_CONTROL_CRC_OFFSET(12); PG_INTERFACE_CONTROL(12); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(12); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 11 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_11 #define CheckPoint CheckPoint_11 #define ControlFileData ControlFileData_11 #define DBState DBState_11 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_11 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_11 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_11 #define DB_SHUTDOWNED DB_SHUTDOWNED_11 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_11 #define DB_SHUTDOWNING DB_SHUTDOWNING_11 #define DB_STARTUP DB_STARTUP_11 #define FullTransactionId FullTransactionId_11 #define MultiXactId MultiXactId_11 #define MultiXactOffset MultiXactOffset_11 #define Oid Oid_11 #define TimeLineID TimeLineID_11 #define XLogLongPageHeaderData XLogLongPageHeaderData_11 #define XLogPageHeaderData XLogPageHeaderData_11 #define XLogRecPtr XLogRecPtr_11 #define int64 int64_11 #define pg_crc32c pg_crc32c_11 #define pg_time_t pg_time_t_11 #include "postgres/interface/version.intern.h" PG_INTERFACE_VALUE_CONTROL_VERSION(11); PG_INTERFACE_VALUE_CATALOG_VERSION(11); PG_INTERFACE_VALUE_WAL_MAGIC(11); PG_INTERFACE_VALUE_CONTROL_CRC_OFFSET(11); PG_INTERFACE_CONTROL(11); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(11); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 10 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_10 #define CheckPoint CheckPoint_10 #define ControlFileData ControlFileData_10 #define DBState DBState_10 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_10 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_10 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_10 #define DB_SHUTDOWNED DB_SHUTDOWNED_10 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_10 #define DB_SHUTDOWNING DB_SHUTDOWNING_10 #define DB_STARTUP DB_STARTUP_10 #define FullTransactionId FullTransactionId_10 #define MultiXactId MultiXactId_10 #define MultiXactOffset MultiXactOffset_10 #define Oid Oid_10 #define TimeLineID TimeLineID_10 #define XLogLongPageHeaderData XLogLongPageHeaderData_10 #define XLogPageHeaderData XLogPageHeaderData_10 #define XLogRecPtr XLogRecPtr_10 #define int64 int64_10 #define pg_crc32c pg_crc32c_10 #define pg_time_t pg_time_t_10 #include "postgres/interface/version.intern.h" PG_INTERFACE_VALUE_CONTROL_VERSION(10); PG_INTERFACE_VALUE_CATALOG_VERSION(10); PG_INTERFACE_VALUE_WAL_MAGIC(10); PG_INTERFACE_VALUE_CONTROL_CRC_OFFSET(10); PG_INTERFACE_CONTROL(10); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(10); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 9.6 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_96 #define CheckPoint CheckPoint_96 #define ControlFileData ControlFileData_96 #define DBState DBState_96 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_96 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_96 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_96 #define DB_SHUTDOWNED DB_SHUTDOWNED_96 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_96 #define DB_SHUTDOWNING DB_SHUTDOWNING_96 #define DB_STARTUP DB_STARTUP_96 #define FullTransactionId FullTransactionId_96 #define MultiXactId MultiXactId_96 #define MultiXactOffset MultiXactOffset_96 #define Oid Oid_96 #define TimeLineID TimeLineID_96 #define XLogLongPageHeaderData XLogLongPageHeaderData_96 #define XLogPageHeaderData XLogPageHeaderData_96 #define XLogRecPtr XLogRecPtr_96 #define int64 int64_96 #define pg_crc32c pg_crc32c_96 #define pg_time_t pg_time_t_96 #include "postgres/interface/version.intern.h" PG_INTERFACE_VALUE_CONTROL_VERSION(96); PG_INTERFACE_VALUE_CATALOG_VERSION(96); PG_INTERFACE_VALUE_WAL_MAGIC(96); PG_INTERFACE_VALUE_CONTROL_CRC_OFFSET(96); PG_INTERFACE_CONTROL(96); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(96); PG_INTERFACE_WAL(96_13); // Shared with 10, 11, 12, 13 #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL control match ***********************************************************************************************************************************/ PG_INTERFACE_CONTROL_MATCH_RANGE(); /*********************************************************************************************************************************** PostgreSQL interface struct A function shared by more than one version is named for the range it covers, so a version may name a function whose name does not begin with it. ***********************************************************************************************************************************/ static const PgInterface pgInterface[] = { { .version = PG_VERSION_19, .unreleased = true, .controlVersion = pgInterfaceControlVersion19, .catalogVersion = pgInterfaceCatalogVersion19, .walMagic = pgInterfaceWalMagic19, .controlCrcOffset = pgInterfaceControlCrcOffset19, .control = pgInterfaceControl19, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate19, .wal = pgInterfaceWal14_19, }, { .version = PG_VERSION_18, .controlVersion = pgInterfaceControlVersion18, .catalogVersion = pgInterfaceCatalogVersion18, .walMagic = pgInterfaceWalMagic18, .controlCrcOffset = pgInterfaceControlCrcOffset18, .control = pgInterfaceControl18, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate18, .wal = pgInterfaceWal14_19, }, { .version = PG_VERSION_17, .controlVersion = pgInterfaceControlVersion17, .catalogVersion = pgInterfaceCatalogVersion17, .walMagic = pgInterfaceWalMagic17, .controlCrcOffset = pgInterfaceControlCrcOffset17, .control = pgInterfaceControl17, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate17, .wal = pgInterfaceWal14_19, }, { .version = PG_VERSION_16, .controlVersion = pgInterfaceControlVersion13_16, .catalogVersion = pgInterfaceCatalogVersion16, .walMagic = pgInterfaceWalMagic16, .controlCrcOffset = pgInterfaceControlCrcOffset15_16, .control = pgInterfaceControl15_16, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate15_16, .wal = pgInterfaceWal14_19, }, { .version = PG_VERSION_15, .controlVersion = pgInterfaceControlVersion13_16, .catalogVersion = pgInterfaceCatalogVersion15, .walMagic = pgInterfaceWalMagic15, .controlCrcOffset = pgInterfaceControlCrcOffset15_16, .control = pgInterfaceControl15_16, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate15_16, .wal = pgInterfaceWal14_19, }, { .version = PG_VERSION_14, .controlVersion = pgInterfaceControlVersion13_16, .catalogVersion = pgInterfaceCatalogVersion14, .walMagic = pgInterfaceWalMagic14, .controlCrcOffset = pgInterfaceControlCrcOffset14, .control = pgInterfaceControl14, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate14, .wal = pgInterfaceWal14_19, }, { .version = PG_VERSION_13, .controlVersion = pgInterfaceControlVersion13_16, .catalogVersion = pgInterfaceCatalogVersion13, .walMagic = pgInterfaceWalMagic13, .controlCrcOffset = pgInterfaceControlCrcOffset13, .control = pgInterfaceControl13, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate13, .wal = pgInterfaceWal96_13, }, { .version = PG_VERSION_12, .controlVersion = pgInterfaceControlVersion12, .catalogVersion = pgInterfaceCatalogVersion12, .walMagic = pgInterfaceWalMagic12, .controlCrcOffset = pgInterfaceControlCrcOffset12, .control = pgInterfaceControl12, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate12, .wal = pgInterfaceWal96_13, }, { .version = PG_VERSION_11, .controlVersion = pgInterfaceControlVersion11, .catalogVersion = pgInterfaceCatalogVersion11, .walMagic = pgInterfaceWalMagic11, .controlCrcOffset = pgInterfaceControlCrcOffset11, .control = pgInterfaceControl11, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate11, .wal = pgInterfaceWal96_13, }, { .version = PG_VERSION_10, .controlVersion = pgInterfaceControlVersion10, .catalogVersion = pgInterfaceCatalogVersion10, .walMagic = pgInterfaceWalMagic10, .controlCrcOffset = pgInterfaceControlCrcOffset10, .control = pgInterfaceControl10, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate10, .wal = pgInterfaceWal96_13, }, { .version = PG_VERSION_96, .controlVersion = pgInterfaceControlVersion96, .catalogVersion = pgInterfaceCatalogVersion96, .walMagic = pgInterfaceWalMagic96, .controlCrcOffset = pgInterfaceControlCrcOffset96, .control = pgInterfaceControl96, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate96, .wal = pgInterfaceWal96_13, }, }; ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/interface.c0000664000175100017510000007155615265101745020161 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Interface ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/time.h" #include "postgres/interface.h" #include "postgres/interface/crc32.h" #include "postgres/interface/static.vendor.h" #include "postgres/version.h" #include "storage/helper.h" /*********************************************************************************************************************************** Defines for various Postgres paths and files ***********************************************************************************************************************************/ STRING_EXTERN(PG_FILE_PGVERSION_STR, PG_FILE_PGVERSION); STRING_EXTERN(PG_FILE_POSTGRESQLAUTOCONF_STR, PG_FILE_POSTGRESQLAUTOCONF); STRING_EXTERN(PG_FILE_POSTMTRPID_STR, PG_FILE_POSTMTRPID); STRING_EXTERN(PG_FILE_RECOVERYCONF_STR, PG_FILE_RECOVERYCONF); STRING_EXTERN(PG_FILE_RECOVERYDONE_STR, PG_FILE_RECOVERYDONE); STRING_EXTERN(PG_FILE_RECOVERYSIGNAL_STR, PG_FILE_RECOVERYSIGNAL); STRING_EXTERN(PG_FILE_STANDBYSIGNAL_STR, PG_FILE_STANDBYSIGNAL); STRING_EXTERN(PG_PATH_GLOBAL_STR, PG_PATH_GLOBAL); STRING_EXTERN(PG_NAME_WAL_STR, PG_NAME_WAL); STRING_EXTERN(PG_NAME_XLOG_STR, PG_NAME_XLOG); // Wal path names depending on version STRING_STATIC(PG_PATH_PGWAL_STR, "pg_wal"); STRING_STATIC(PG_PATH_PGXLOG_STR, "pg_xlog"); // Transaction commit log path names depending on version STRING_STATIC(PG_PATH_PGCLOG_STR, "pg_clog"); STRING_STATIC(PG_PATH_PGXACT_STR, "pg_xact"); // Lsn name used in functions depending on version STRING_STATIC(PG_NAME_LSN_STR, "lsn"); STRING_STATIC(PG_NAME_LOCATION_STR, "location"); /*********************************************************************************************************************************** The control file is 8192 bytes but only the first 512 bytes are used to prevent torn pages even on really old storage with 512-byte sectors. This is true across all versions of PostgreSQL. Unfortunately, this is not sufficient to prevent torn pages during concurrent reads and writes so retries are required. ***********************************************************************************************************************************/ #define PG_CONTROL_SIZE 8192 #define PG_CONTROL_DATA_SIZE 512 /*********************************************************************************************************************************** PostgreSQL interface definitions The interface functions are documented here rather than the auto-generated file. ***********************************************************************************************************************************/ typedef struct PgInterface { // Version of PostgreSQL supported by this interface unsigned int version; // Control version and catalog version that pg_control matches for this version of PostgreSQL. A version that has not been // released has no catalog version of its own yet, so it accepts a range of them rather than only its own. uint32_t controlVersion; uint32_t catalogVersion; bool unreleased; // Magic that the WAL header matches for this version of PostgreSQL uint16_t walMagic; // Offset of the crc in pg_control size_t controlCrcOffset; // Convert pg_control to a common data structure PgControl (*control)(const uint8_t *); // Invalidate control checkpoint void (*controlCheckpointInvalidate)(uint8_t *); // Convert WAL header to a common data structure PgWal (*wal)(const uint8_t *); } PgInterface; /*********************************************************************************************************************************** These pg_control fields are common to all versions of PostgreSQL, so we can use them to match an interface and to generate error messages when the pg_control version cannot be found. ***********************************************************************************************************************************/ typedef struct PgControlCommon { uint64_t systemId; uint32_t controlVersion; uint32_t catalogVersion; } PgControlCommon; // Include auto-generated interfaces #include "postgres/interface.auto.c.inc" /*********************************************************************************************************************************** Get the interface for a PostgreSQL version ***********************************************************************************************************************************/ static const PgInterface * pgInterfaceVersion(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); const PgInterface *result = NULL; for (unsigned int interfaceIdx = 0; interfaceIdx < LENGTH_OF(pgInterface); interfaceIdx++) { if (pgInterface[interfaceIdx].version == pgVersion) { result = &pgInterface[interfaceIdx]; break; } } // If the version was not found then error if (result == NULL) { THROW_FMT( VersionNotSupportedError, "invalid " PG_NAME " version %s\n" "HINT: is this version of " PG_NAME " supported?\n" "HINT: is " PROJECT_NAME " up to date on all hosts?", strZ(pgVersionToStr(pgVersion))); } FUNCTION_TEST_RETURN_TYPE_CONST_P(PgInterface, result); } /**********************************************************************************************************************************/ FN_EXTERN bool pgDbIsTemplate(const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, strEqZ(name, "template0") || strEqZ(name, "template1")); } /**********************************************************************************************************************************/ FN_EXTERN bool pgDbIsSystem(const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, strEqZ(name, "postgres") || pgDbIsTemplate(name)); } /**********************************************************************************************************************************/ FN_EXTERN bool pgDbIsSystemId(const unsigned int id) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, id); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, id < FirstNormalObjectId); } /*********************************************************************************************************************************** Check expected WAL segment size for older PostgreSQL versions ***********************************************************************************************************************************/ static void pgWalSegmentSizeCheck(const unsigned int walSegmentSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, walSegmentSize); FUNCTION_TEST_END(); // Check that the WAL segment size is valid if (!IsValidWalSegSize(walSegmentSize)) { THROW_FMT( FormatError, "wal segment size is %u but must be a power of two between %d and %d inclusive", walSegmentSize, WalSegMinSize, WalSegMaxSize); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Validate the CRC in pg_control. If the version has been forced this may required scanning pg_control for the offset. Return the offset found so it can be used to update the CRC. ***********************************************************************************************************************************/ static size_t pgControlCrcValidate(const Buffer *const controlFile, const PgInterface *const interface, const bool pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, controlFile); FUNCTION_LOG_PARAM_P(VOID, interface); FUNCTION_LOG_PARAM(BOOL, pgVersionForce); FUNCTION_LOG_END(); ASSERT(controlFile != NULL); ASSERT(interface != NULL); size_t result = interface->controlCrcOffset; do { // Calculate CRC and retrieve expected CRC const uint32_t crcCalculated = crc32cOne(bufPtrConst(controlFile), result); const uint32_t crcExpected = *((const uint32_t *)(bufPtrConst(controlFile) + result)); // If CRC does not match if (crcCalculated != crcExpected) { // If version is forced then the CRC might be later in the file (assuming the fork added extra fields to pg_control). // Increment the offset by CRC data size and continue to try again. if (pgVersionForce) { result += sizeof(uint32_t); if (result <= bufUsed(controlFile) - sizeof(uint32_t)) continue; } // If no retry then error THROW_FMT( ChecksumError, "calculated " PG_FILE_PGCONTROL " checksum does not match expected value\n" "HINT: calculated 0x%x but expected value is 0x%x\n" "%s" "HINT: is " PG_FILE_PGCONTROL " corrupt?\n" "HINT: does " PG_FILE_PGCONTROL " have a different layout than expected?", crcCalculated, crcExpected, pgVersionForce ? "HINT: checksum values may be misleading due to forced version scan\n" : ""); } // Do not retry if the CRC is valid break; } while (true); FUNCTION_LOG_RETURN(SIZE, result); } /*********************************************************************************************************************************** Update the CRC in pg_control ***********************************************************************************************************************************/ static void pgControlCrcUpdate(Buffer *const controlFile, const size_t crcOffset) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, controlFile); FUNCTION_LOG_PARAM(SIZE, crcOffset); FUNCTION_LOG_END(); ASSERT(controlFile != NULL); ASSERT(crcOffset != 0); *((uint32_t *)(bufPtr(controlFile) + crcOffset)) = crc32cOne(bufPtrConst(controlFile), crcOffset); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static PgControl pgControlFromBuffer(const Buffer *const controlFile, const String *const pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, controlFile); FUNCTION_LOG_PARAM(STRING, pgVersionForce); FUNCTION_LOG_END(); ASSERT(controlFile != NULL); // Search for the version of PostgreSQL that uses this control file const PgInterface *interface = NULL; if (pgVersionForce != NULL) interface = pgInterfaceVersion(pgVersionFromStr(pgVersionForce)); else { const PgControlCommon *const controlCommon = (const PgControlCommon *)bufPtrConst(controlFile); for (unsigned int interfaceIdx = 0; interfaceIdx < LENGTH_OF(pgInterface); interfaceIdx++) { if (pgInterfaceControlMatch(&pgInterface[interfaceIdx], controlCommon)) { interface = &pgInterface[interfaceIdx]; break; } } // If the version was not found then error with the control and catalog version that were found if (interface == NULL) { THROW_FMT( VersionNotSupportedError, "unexpected control version = %u and catalog version = %u\n" "HINT: is this version of " PG_NAME " supported?\n" "HINT: is " PROJECT_NAME " up to date on all hosts?", controlCommon->controlVersion, controlCommon->catalogVersion); } } // Get info from the control file PgControl result = interface->control(bufPtrConst(controlFile)); result.version = interface->version; // Validate the CRC pgControlCrcValidate(controlFile, interface, pgVersionForce != NULL); // Check the segment size pgWalSegmentSizeCheck(result.walSegmentSize); // Check the page size pgPageSizeCheck(result.pageSize); // Check the checksum version if (result.pageChecksumVersion > PG_DATA_CHECKSUM_VERSION) { THROW_FMT( FormatError, "page checksum version is %u but must be <= %d", result.pageChecksumVersion, PG_DATA_CHECKSUM_VERSION); } FUNCTION_LOG_RETURN(PG_CONTROL, result); } // Helper to compare control data to last read static bool pgControlBufferEq(const Buffer *const last, const Buffer *const current) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, last); FUNCTION_TEST_PARAM(BUFFER, current); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, last != NULL && bufEq(last, current)); } FN_EXTERN Buffer * pgControlBufferFromFile(const Storage *const storage, const String *const pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, pgVersionForce); FUNCTION_LOG_END(); ASSERT(storage != NULL); Buffer *result; MEM_CONTEXT_TEMP_BEGIN() { // On filesystems that do not implement atomicity of concurrent reads and writes, we might get garbage if the server is // writing the pg_control file at the same time as we try to read it. Keep trying until success or we read the same data // twice in a row. Do not use a timeout here because there are plenty of other errors that can happen and we don't want to // wait for them. Buffer *controlFileLast = NULL; bool done = false; do { // Read control file Buffer *const controlFile = storageGetP( storageNewReadP(storage, STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)), .exactSize = PG_CONTROL_DATA_SIZE); TRY_BEGIN() { // Check that control data is valid pgControlFromBuffer(controlFile, pgVersionForce); // Create a buffer of the correct size to hold pg_control and zero out the remaining portion result = bufNew(PG_CONTROL_SIZE); bufCat(result, controlFile); memset(bufPtr(result) + bufUsed(controlFile), 0, bufSize(result) - bufUsed(controlFile)); bufUsedSet(result, bufSize(result)); bufMove(result, memContextPrior()); done = true; } CATCH_ANY() { // If we get the same bad data twice in a row then error if (pgControlBufferEq(controlFileLast, controlFile)) RETHROW(); // Copy data to last bufFree(controlFileLast); controlFileLast = controlFile; // Sleep to let data stabilize sleepMSec(50); } TRY_END(); } while (!done); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BUFFER, result); } FN_EXTERN PgControl pgControlFromFile(const Storage *const storage, const String *const pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, pgVersionForce); FUNCTION_LOG_END(); ASSERT(storage != NULL); Buffer *const buffer = pgControlBufferFromFile(storage, pgVersionForce); const PgControl result = pgControlFromBuffer(buffer, pgVersionForce); bufFree(buffer); FUNCTION_LOG_RETURN(PG_CONTROL, result); } /**********************************************************************************************************************************/ FN_EXTERN void pgControlCheckpointInvalidate(Buffer *const buffer, const String *const pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(STRING, pgVersionForce); FUNCTION_LOG_END(); ASSERT(buffer != NULL); const PgControl pgControl = pgControlFromBuffer(buffer, pgVersionForce); const PgInterface *const interface = pgInterfaceVersion(pgControl.version); const size_t crcOffset = pgControlCrcValidate(buffer, interface, pgVersionForce); pgInterfaceVersion(pgControl.version)->controlCheckpointInvalidate(bufPtr(buffer)); pgControlCrcUpdate(buffer, crcOffset); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN uint32_t pgControlVersion(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(UINT32, pgInterfaceVersion(pgVersion)->controlVersion); } /*********************************************************************************************************************************** These WAL header fields are common to all versions of PostgreSQL, so we can use them to generate error messages when the WAL magic cannot be found. ***********************************************************************************************************************************/ typedef struct PgWalCommon { uint16_t magic; uint16_t flag; } PgWalCommon; #define PG_WAL_LONG_HEADER 0x0002 /**********************************************************************************************************************************/ FN_EXTERN PgWal pgWalFromBuffer(const Buffer *const walBuffer, const String *const pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, walBuffer); FUNCTION_LOG_PARAM(STRING, pgVersionForce); FUNCTION_LOG_END(); ASSERT(walBuffer != NULL); // Check that this is a long format WAL header if (!(((const PgWalCommon *)bufPtrConst(walBuffer))->flag & PG_WAL_LONG_HEADER)) THROW_FMT(FormatError, "first page header in WAL file is expected to be in long format"); // Search for the version of PostgreSQL that uses this WAL magic const PgInterface *interface = NULL; if (pgVersionForce != NULL) interface = pgInterfaceVersion(pgVersionFromStr(pgVersionForce)); else { for (unsigned int interfaceIdx = 0; interfaceIdx < LENGTH_OF(pgInterface); interfaceIdx++) { if (((const PgWalCommon *)bufPtrConst(walBuffer))->magic == pgInterface[interfaceIdx].walMagic) { interface = &pgInterface[interfaceIdx]; break; } } // If the version was not found then error with the magic that was found if (interface == NULL) { THROW_FMT( VersionNotSupportedError, "unexpected WAL magic %u\n" "HINT: is this version of " PG_NAME " supported?\n" "HINT: is " PROJECT_NAME " up to date on all hosts?", ((const PgWalCommon *)bufPtrConst(walBuffer))->magic); } } // Get info from the control file PgWal result = interface->wal(bufPtrConst(walBuffer)); result.version = interface->version; // Check the segment size pgWalSegmentSizeCheck(result.size); FUNCTION_LOG_RETURN(PG_WAL, result); } FN_EXTERN PgWal pgWalFromFile(const String *const walFile, const Storage *const storage, const String *const pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walFile); FUNCTION_LOG_PARAM(STRING, pgVersionForce); FUNCTION_LOG_END(); ASSERT(walFile != NULL); PgWal result; MEM_CONTEXT_TEMP_BEGIN() { // Read WAL segment header const Buffer *const walBuffer = storageGetP(storageNewReadP(storage, walFile), .exactSize = PG_WAL_HEADER_SIZE); result = pgWalFromBuffer(walBuffer, pgVersionForce); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PG_WAL, result); } /**********************************************************************************************************************************/ FN_EXTERN String * pgTablespaceId(const unsigned int pgVersion, const unsigned int pgCatalogVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_PARAM(UINT, pgCatalogVersion); FUNCTION_TEST_END(); String *result; MEM_CONTEXT_TEMP_BEGIN() { const String *const pgVersionStr = pgVersionToStr(pgVersion); MEM_CONTEXT_PRIOR_BEGIN() { result = strNewFmt("PG_%s_%u", strZ(pgVersionStr), pgCatalogVersion); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN uint64_t pgLsnFromStr(const String *lsn) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, lsn); FUNCTION_TEST_END(); uint64_t result = 0; MEM_CONTEXT_TEMP_BEGIN() { const StringList *const lsnPart = strLstNewSplit(lsn, FSLASH_STR); CHECK(FormatError, strLstSize(lsnPart) == 2, "lsn requires two parts"); result = (cvtZToUInt64Base(strZ(strLstGet(lsnPart, 0)), 16) << 32) + cvtZToUInt64Base(strZ(strLstGet(lsnPart, 1)), 16); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(UINT64, result); } FN_EXTERN String * pgLsnToStr(const uint64_t lsn) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, lsn); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(STRING, strNewFmt("%x/%x", (unsigned int)(lsn >> 32), (unsigned int)(lsn & 0xFFFFFFFF))); } /**********************************************************************************************************************************/ FN_EXTERN String * pgLsnToWalSegment(const uint32_t timeline, const uint64_t lsn, const unsigned int walSegmentSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, timeline); FUNCTION_TEST_PARAM(UINT64, lsn); FUNCTION_TEST_PARAM(UINT, walSegmentSize); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN( STRING, strNewFmt("%08X%08X%08X", timeline, (unsigned int)(lsn >> 32), (unsigned int)(lsn & 0xFFFFFFFF) / walSegmentSize)); } /**********************************************************************************************************************************/ FN_EXTERN uint32_t pgTimelineFromWalSegment(const String *const walSegment) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, walSegment); FUNCTION_TEST_END(); ASSERT(walSegment != NULL); ASSERT(strSize(walSegment) == 24); FUNCTION_TEST_RETURN(UINT32, cvtZSubNToUIntBase(strZ(walSegment), 0, 8, 16)); } /**********************************************************************************************************************************/ FN_EXTERN StringList * pgLsnRangeToWalSegmentList( const uint32_t timeline, const uint64_t lsnStart, const uint64_t lsnStop, const unsigned int walSegmentSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, timeline); FUNCTION_TEST_PARAM(UINT64, lsnStart); FUNCTION_TEST_PARAM(UINT64, lsnStop); FUNCTION_TEST_PARAM(UINT, walSegmentSize); FUNCTION_TEST_END(); ASSERT(timeline != 0); ASSERT(lsnStart <= lsnStop); ASSERT(walSegmentSize != 0); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Calculate the start and stop segments unsigned int startMajor = (unsigned int)(lsnStart >> 32); unsigned int startMinor = (unsigned int)(lsnStart & 0xFFFFFFFF) / walSegmentSize; const unsigned int stopMajor = (unsigned int)(lsnStop >> 32); const unsigned int stopMinor = (unsigned int)(lsnStop & 0xFFFFFFFF) / walSegmentSize; const unsigned int minorPerMajor = 0xFFFFFFFF / walSegmentSize; // Create list strLstAddFmt(result, "%08X%08X%08X", timeline, startMajor, startMinor); while (!(startMajor == stopMajor && startMinor == stopMinor)) { startMinor++; if (startMinor > minorPerMajor) { startMajor++; startMinor = 0; } strLstAddFmt(result, "%08X%08X%08X", timeline, startMajor, startMinor); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN const String * pgLsnName(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, pgVersion >= PG_VERSION_WAL_RENAME ? PG_NAME_LSN_STR : PG_NAME_LOCATION_STR); } /*********************************************************************************************************************************** Get WAL name (wal/xlog) for a PostgreSQL version ***********************************************************************************************************************************/ FN_EXTERN const String * pgWalName(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, pgVersion >= PG_VERSION_WAL_RENAME ? PG_NAME_WAL_STR : PG_NAME_XLOG_STR); } /**********************************************************************************************************************************/ FN_EXTERN const String * pgWalPath(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, pgVersion >= PG_VERSION_WAL_RENAME ? PG_PATH_PGWAL_STR : PG_PATH_PGXLOG_STR); } /**********************************************************************************************************************************/ FN_EXTERN const String * pgXactPath(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, pgVersion >= PG_VERSION_WAL_RENAME ? PG_PATH_PGXACT_STR : PG_PATH_PGCLOG_STR); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int pgVersionFromStr(const String *const version) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, version); FUNCTION_TEST_END(); ASSERT(version != NULL); // If format not number.number (9.6) or number only (10) then error. No check for valid/supported PG version is on purpose. if (!regExpMatchOne(STRDEF("^[0-9]+[.]*[0-9]+$"), version)) THROW_FMT(AssertError, "version %s format is invalid", strZ(version)); // If there is no dot then only the major version is needed const int idxStart = strChr(version, '.'); if (idxStart == -1) FUNCTION_TEST_RETURN(UINT, cvtZToUInt(strZ(version)) * 10000); // Major and minor version are needed FUNCTION_TEST_RETURN( UINT, cvtZSubNToUInt(strZ(version), 0, (size_t)idxStart) * 10000 + cvtZToUInt(strZ(version) + (size_t)idxStart + 1) * 100); } FN_EXTERN String * pgVersionToStr(const unsigned int version) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, version); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN( STRING, version >= PG_VERSION_10 ? strNewFmt("%u", version / 10000) : strNewFmt("%u.%u", version / 10000, version % 10000 / 100)); } /**********************************************************************************************************************************/ FN_EXTERN void pgControlToLog(const PgControl *const pgControl, StringStatic *const debugLog) { strStcFmt( debugLog, "{version: %u, systemId: %" PRIu64 ", walSegmentSize: %u, pageChecksumVersion: %u}", pgControl->version, pgControl->systemId, pgControl->walSegmentSize, pgControl->pageChecksumVersion); } FN_EXTERN void pgWalToLog(const PgWal *const pgWal, StringStatic *const debugLog) { strStcFmt(debugLog, "{version: %u, systemId: %" PRIu64 "}", pgWal->version, pgWal->systemId); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/interface.h0000664000175100017510000002645515265101745020164 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Interface ***********************************************************************************************************************************/ #ifndef POSTGRES_INTERFACE_H #define POSTGRES_INTERFACE_H #include #include #include "common/debug.h" #include "common/type/string.h" #include "storage/storage.h" /*********************************************************************************************************************************** Defines for various Postgres paths and files ***********************************************************************************************************************************/ #define PG_FILE_BACKUPLABEL "backup_label" #define PG_FILE_BACKUPLABELOLD "backup_label.old" #define PG_FILE_BACKUPMANIFEST "backup_manifest" #define PG_FILE_BACKUPMANIFEST_TMP "backup_manifest.tmp" #define PG_FILE_PGCONTROL "pg_control" #define PG_FILE_PGFILENODEMAP "pg_filenode.map" #define PG_FILE_PGINTERNALINIT "pg_internal.init" #define PG_FILE_PGVERSION "PG_VERSION" STRING_DECLARE(PG_FILE_PGVERSION_STR); #define PG_FILE_POSTGRESQLAUTOCONF "postgresql.auto.conf" STRING_DECLARE(PG_FILE_POSTGRESQLAUTOCONF_STR); #define PG_FILE_POSTGRESQLAUTOCONFTMP "postgresql.auto.conf.tmp" #define PG_FILE_POSTMTROPTS "postmas""ter.opts" #define PG_FILE_POSTMTRPID "postmas""ter.pid" STRING_DECLARE(PG_FILE_POSTMTRPID_STR); #define PG_FILE_RECOVERYCONF "recovery.conf" STRING_DECLARE(PG_FILE_RECOVERYCONF_STR); #define PG_FILE_RECOVERYDONE "recovery.done" STRING_DECLARE(PG_FILE_RECOVERYDONE_STR); #define PG_FILE_RECOVERYSIGNAL "recovery.signal" STRING_DECLARE(PG_FILE_RECOVERYSIGNAL_STR); #define PG_FILE_STANDBYSIGNAL "standby.signal" STRING_DECLARE(PG_FILE_STANDBYSIGNAL_STR); #define PG_FILE_TABLESPACEMAP "tablespace_map" #define PG_PATH_ARCHIVE_STATUS "archive_status" #define PG_PATH_BASE "base" #define PG_PATH_GLOBAL "global" STRING_DECLARE(PG_PATH_GLOBAL_STR); #define PG_PATH_PGMULTIXACT "pg_multixact" #define PG_PATH_PGDYNSHMEM "pg_dynshmem" #define PG_PATH_PGNOTIFY "pg_notify" #define PG_PATH_PGREPLSLOT "pg_replslot" #define PG_PATH_PGSERIAL "pg_serial" #define PG_PATH_PGSNAPSHOTS "pg_snapshots" #define PG_PATH_PGSTATTMP "pg_stat_tmp" #define PG_PATH_PGSUBTRANS "pg_subtrans" #define PG_PATH_PGTBLSPC "pg_tblspc" #define PG_PREFIX_PGSQLTMP "pgsql_tmp" #define PG_NAME_WAL "wal" STRING_DECLARE(PG_NAME_WAL_STR); #define PG_NAME_XLOG "xlog" STRING_DECLARE(PG_NAME_XLOG_STR); /*********************************************************************************************************************************** Define allowed page sizes ***********************************************************************************************************************************/ typedef enum { pgPageSize1 = 1 * 1024, pgPageSize2 = 2 * 1024, pgPageSize4 = 4 * 1024, pgPageSize8 = 8 * 1024, pgPageSize16 = 16 * 1024, pgPageSize32 = 32 * 1024, } PgPageSize; /*********************************************************************************************************************************** Define default segment size and pages per segment Segment size can only be changed at compile time and is not known to be well-tested, so only the default segment size is supported. ***********************************************************************************************************************************/ #define PG_SEGMENT_SIZE_DEFAULT ((unsigned int)(1 * 1024 * 1024 * 1024)) /*********************************************************************************************************************************** WAL header size. It doesn't seem worth tracking the exact size of the WAL header across versions of PostgreSQL so just set it to something far larger needed but <= the minimum read size on just about any system. ***********************************************************************************************************************************/ #define PG_WAL_HEADER_SIZE ((unsigned int)(512)) /*********************************************************************************************************************************** Checkpoint written into pg_control on restore. This will prevent PostgreSQL from starting if backup_label is not present. ***********************************************************************************************************************************/ #define PG_CONTROL_CHECKPOINT_INVALID 0xDEAD /*********************************************************************************************************************************** PostgreSQL Control File Info ***********************************************************************************************************************************/ typedef struct PgControl { unsigned int version; uint64_t systemId; unsigned int catalogVersion; uint64_t checkpoint; // Last checkpoint LSN uint32_t timeline; // Current timeline PgPageSize pageSize; unsigned int walSegmentSize; unsigned int pageChecksumVersion; // Page checksum version (0 if no checksum, 1 if checksum) } PgControl; /*********************************************************************************************************************************** PostgreSQL WAL Info ***********************************************************************************************************************************/ typedef struct PgWal { unsigned int version; unsigned int size; uint64_t systemId; } PgWal; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Is this a template database? FN_EXTERN bool pgDbIsTemplate(const String *name); // Is this a system database, i.e. template or postgres? FN_EXTERN bool pgDbIsSystem(const String *name); // Does this database have a system id, i.e. less than the minimum assignable user id? FN_EXTERN bool pgDbIsSystemId(unsigned int id); // Get info from pg_control FN_EXTERN Buffer *pgControlBufferFromFile(const Storage *storage, const String *pgVersionForce); FN_EXTERN PgControl pgControlFromFile(const Storage *storage, const String *pgVersionForce); // Invalidate checkpoint record in pg_control FN_EXTERN void pgControlCheckpointInvalidate(Buffer *buffer, const String *pgVersionForce); // Get the control version for a PostgreSQL version FN_EXTERN uint32_t pgControlVersion(unsigned int pgVersion); // Convert version string to version number and vice versa FN_EXTERN unsigned int pgVersionFromStr(const String *version); FN_EXTERN String *pgVersionToStr(unsigned int version); // Get info from WAL header FN_EXTERN PgWal pgWalFromFile(const String *walFile, const Storage *storage, const String *pgVersionForce); FN_EXTERN PgWal pgWalFromBuffer(const Buffer *walBuffer, const String *pgVersionForce); // Get the tablespace identifier used to distinguish versions in a tablespace directory, e.g. PG_15_202209061 FN_EXTERN String *pgTablespaceId(unsigned int pgVersion, unsigned int pgCatalogVersion); // Convert a string to an lsn and vice versa FN_EXTERN uint64_t pgLsnFromStr(const String *lsn); FN_EXTERN String *pgLsnToStr(uint64_t lsn); // Convert a timeline and lsn to a wal segment and vice versa FN_EXTERN String *pgLsnToWalSegment(uint32_t timeline, uint64_t lsn, unsigned int walSegmentSize); // Get timeline from WAL segment name FN_EXTERN uint32_t pgTimelineFromWalSegment(const String *walSegment); // Convert a timeline and lsn range to a list of wal segments FN_EXTERN StringList *pgLsnRangeToWalSegmentList( uint32_t timeline, uint64_t lsnStart, uint64_t lsnStop, unsigned int walSegmentSize); // Get name used for lsn in functions (this was changed in PostgreSQL 10 for consistency since lots of names were changing) FN_EXTERN const String *pgLsnName(unsigned int pgVersion); // Calculate the checksum for a page. Page cannot be const because the page header is temporarily modified during processing. FN_EXTERN uint16_t pgPageChecksum(uint8_t *page, uint32_t blockNo, PgPageSize pageSize); // Returns true if page size is valid, false otherwise FN_EXTERN bool pgPageSizeValid(PgPageSize pageSize); // Throws an error if page size is not valid FN_EXTERN void pgPageSizeCheck(PgPageSize pageSize); FN_EXTERN const String *pgWalName(unsigned int pgVersion); // Get wal path (this was changed in PostgreSQL 10 to avoid including "log" in the name) FN_EXTERN const String *pgWalPath(unsigned int pgVersion); // Get transaction commit log path (this was changed in PostgreSQL 10 to avoid including "log" in the name) FN_EXTERN const String *pgXactPath(unsigned int pgVersion); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void pgControlToLog(const PgControl *pgControl, StringStatic *debugLog); FN_EXTERN void pgWalToLog(const PgWal *pgWal, StringStatic *debugLog); #define FUNCTION_LOG_PG_CONTROL_TYPE \ PgControl #define FUNCTION_LOG_PG_CONTROL_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(&value, pgControlToLog, buffer, bufferSize) #define FUNCTION_LOG_PG_WAL_TYPE \ PgWal #define FUNCTION_LOG_PG_WAL_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(&value, pgWalToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/meson.build0000664000175100017510000000230015265101745020174 0ustar00runnerrunner#################################################################################################################################### # Generate PostgreSQL interfaces #################################################################################################################################### # Generate interface.auto.c.inc, unless this is a distribution tarball that ships the file. The generator writes into the build path # so the file lands where the output is declared, else it would be missing there and ninja would run the generator on every build. if not dist interface_auto_c_inc = custom_target( 'interface.auto.c.inc', output: 'interface.auto.c.inc', depend_files: [ '../../build/postgres.yaml', 'interface/version.intern.h', 'interface/version.vendor.h', build_code_lib, ], command: [ build_code, 'postgres', '--repo-path=@CURRENT_SOURCE_DIR@/../..', '--build-path=@BUILD_ROOT@', ], ) else # The shipped file is included by interface.c via the implicit source include path, so it is not a compiled source interface_auto_c_inc = [] endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/version.auto.h0000664000175100017510000000533015265101745020645 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Version Automatically generated by 'build.py postgres-version' -- do not modify directly. ***********************************************************************************************************************************/ #ifndef POSTGRES_VERSION_AUTO_H #define POSTGRES_VERSION_AUTO_H /*********************************************************************************************************************************** PostgreSQL version constants ***********************************************************************************************************************************/ #define PG_VERSION_96 90600 #define PG_VERSION_10 100000 #define PG_VERSION_11 110000 #define PG_VERSION_12 120000 #define PG_VERSION_13 130000 #define PG_VERSION_14 140000 #define PG_VERSION_15 150000 #define PG_VERSION_16 160000 #define PG_VERSION_17 170000 #define PG_VERSION_18 180000 #define PG_VERSION_19 190000 #define PG_VERSION_MAX PG_VERSION_19 /*********************************************************************************************************************************** PostgreSQL version string constants for use in error messages ***********************************************************************************************************************************/ #define PG_VERSION_96_Z "9.6" #define PG_VERSION_10_Z "10" #define PG_VERSION_11_Z "11" #define PG_VERSION_12_Z "12" #define PG_VERSION_13_Z "13" #define PG_VERSION_14_Z "14" #define PG_VERSION_15_Z "15" #define PG_VERSION_16_Z "16" #define PG_VERSION_17_Z "17" #define PG_VERSION_18_Z "18" #define PG_VERSION_19_Z "19" #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/postgres/version.h0000664000175100017510000000251615265101745017701 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Version Constants ***********************************************************************************************************************************/ #ifndef POSTGRES_VERSION_H #define POSTGRES_VERSION_H #include "postgres/version.auto.h" /*********************************************************************************************************************************** PostgreSQL name ***********************************************************************************************************************************/ #define PG_NAME "PostgreSQL" /*********************************************************************************************************************************** Version where various PostgreSQL capabilities were introduced ***********************************************************************************************************************************/ // xlog was renamed to wal #define PG_VERSION_WAL_RENAME PG_VERSION_10 // recovery settings are implemented as GUCs (recovery.conf is no longer valid) #define PG_VERSION_RECOVERY_GUC PG_VERSION_12 #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/protocol/0000775000175100017510000000000015265101745016032 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/protocol/client.c0000664000175100017510000006202215265101745017456 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Client ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/time.h" #include "common/type/json.h" #include "common/type/keyValue.h" #include "protocol/client.h" #include "protocol/server.h" #include "version.h" /*********************************************************************************************************************************** Minimum time between keep-alive noops ***********************************************************************************************************************************/ #define PROTOCOL_KEEP_ALIVE_TIME ((TimeMSec)30000) /*********************************************************************************************************************************** Client state enum ***********************************************************************************************************************************/ typedef enum { // Client is waiting for a response / ready to send a request protocolClientStateIdle = STRID5("idle", 0x2b0890), // Request is in progress protocolClientStateRequest = STRID5("request", 0x5265ac4b20), // Response is in progress protocolClientStateResponse = STRID5("response", 0x2cdcf84cb20), } ProtocolClientState; /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct ProtocolClient { ProtocolClientPub pub; // Publicly accessible variables ProtocolClientState state; // Current client state IoWrite *write; // Write interface const String *name; // Name displayed in logging const String *errorPrefix; // Prefix used when throwing error TimeMSec keepAliveTime; // Last time data was put to the server uint64_t sessionTotal; // Total sessions (used to generate session ids) List *sessionList; // Session list }; struct ProtocolClientSession { ProtocolClientSessionPub pub; // Publicly accessible variables ProtocolClient *client; // Protocol client StringId command; // Command uint64_t sessionId; // Session id bool async; // Async requests allowed? // Stored message (read by another session and stored for later retrieval) bool stored; // Is a message currently stored? bool close; // Should the session be closed? ProtocolMessageType type; // Type of last message received PackRead *packRead; // Last message received (if any) }; /*********************************************************************************************************************************** Check protocol state ***********************************************************************************************************************************/ static void protocolClientStateExpectIdle(const ProtocolClient *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PROTOCOL_CLIENT, this); FUNCTION_TEST_END(); if (this->state != protocolClientStateIdle) { THROW_FMT( ProtocolError, "client state is '%s' but expected '%s'", zNewStrId(this->state), zNewStrId(protocolClientStateIdle)); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Find a client session ***********************************************************************************************************************************/ static unsigned int protocolClientSessionFindIdx(const ProtocolClient *const this, const uint64_t sessionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PROTOCOL_CLIENT, this); FUNCTION_TEST_PARAM(UINT64, sessionId); FUNCTION_TEST_END(); unsigned int result = 0; for (; result < lstSize(this->sessionList); result++) { if ((*(ProtocolClientSession **)lstGet(this->sessionList, result))->sessionId == sessionId) break; } CHECK_FMT(ProtocolError, result < lstSize(this->sessionList), "unable to find protocol client session %" PRIu64, sessionId); FUNCTION_TEST_RETURN(UINT, result); } /**********************************************************************************************************************************/ static void protocolClientRequestInternal(ProtocolClientSession *const this, const ProtocolCommandType type, PackWrite *const param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(PACK_WRITE, param); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!protocolClientSessionQueued(this)); // Expect idle state before request protocolClientStateExpectIdle(this->client); MEM_CONTEXT_TEMP_BEGIN() { // Switch state to request this->client->state = protocolClientStateRequest; // Write request PackWrite *const request = pckWriteNewIo(this->client->write); pckWriteU32P(request, protocolMessageTypeRequest, .defaultWrite = true); pckWriteStrIdP(request, this->command); pckWriteStrIdP(request, type); pckWriteU64P(request, this->sessionId); pckWriteBoolP(request, this->pub.open); // Write request parameters if (param != NULL) { pckWriteEndP(param); pckWritePackP(request, pckWriteResult(param)); } pckWriteEndP(request); // Flush to send request immediately ioWriteFlush(this->client->write); // Reset the keep alive time since data was just put to the server this->client->keepAliveTime = timeMSec(); this->pub.queued = true; // Switch state back to idle after successful request this->client->state = protocolClientStateIdle; } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Process errors ***********************************************************************************************************************************/ static void protocolClientError(ProtocolClient *const this, const ProtocolMessageType type, PackRead *const error) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, this); FUNCTION_LOG_PARAM(ENUM, type); FUNCTION_LOG_PARAM(PACK_READ, error); FUNCTION_LOG_END(); ASSERT(this != NULL); if (type == protocolMessageTypeError) { ASSERT(error != NULL); MEM_CONTEXT_TEMP_BEGIN() { const ErrorType *const type = errorTypeFromCode(pckReadI32P(error)); const String *const message = strNewFmt("%s: %s", strZ(this->errorPrefix), strZ(pckReadStrP(error))); const String *const stack = pckReadStrP(error); pckReadEndP(error); CHECK(FormatError, message != NULL && stack != NULL, "invalid error data"); errorInternalThrow(type, __FILE__, __func__, __LINE__, strZ(message), strZ(stack)); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static PackRead * protocolClientResponseInternal(ProtocolClientSession *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Expect data-get state before data get protocolClientStateExpectIdle(this->client); PackRead *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Check the session for a stored response bool found = false; ProtocolMessageType type = protocolMessageTypeResponse; bool close = false; PackRead *packRead = NULL; if (this->stored) { found = true; type = this->type; close = this->close; packRead = pckReadMove(this->packRead, memContextCurrent()); this->stored = false; } // If not stored then read from protocol while (!found) { // Switch state to response this->client->state = protocolClientStateResponse; // Read response PackRead *const response = pckReadNewIo(this->client->pub.read); const uint64_t sessionId = pckReadU64P(response); type = (ProtocolMessageType)pckReadU32P(response); close = pckReadBoolP(response); packRead = pckReadPackReadP(response); pckReadEndP(response); // If this response is for another session then store it with that session. Session id 0 indicates a fatal error on the // server that should be reported by the first session that sees it. ASSERT(sessionId != 0 || type == protocolMessageTypeError); if (sessionId != 0 && sessionId != this->sessionId) { ProtocolClientSession *const sessionOther = *(ProtocolClientSession **)lstGet( this->client->sessionList, protocolClientSessionFindIdx(this->client, sessionId)); CHECK_FMT( ProtocolError, !sessionOther->stored, "protocol client session %" PRIu64 " already has a stored response", sessionOther->sessionId); sessionOther->stored = true; sessionOther->type = type; sessionOther->close = close; sessionOther->packRead = objMove(packRead, objMemContext(sessionOther)); } else found = true; // Switch state back to idle after successful response this->client->state = protocolClientStateIdle; } // The result is no longer queued -- either it will generate an error or be returned to the user this->pub.queued = false; // Close the session if requested if (close) this->pub.open = false; // Check if this result is an error and if so throw it protocolClientError(this->client, type, packRead); CHECK(FormatError, type == protocolMessageTypeResponse, "expected response message"); // Return result to the caller result = objMove(packRead, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK_READ, result); } /*********************************************************************************************************************************** Close protocol connection ***********************************************************************************************************************************/ static void protocolClientFreeResource(THIS_VOID) { THIS(ProtocolClient); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Stop client sessions from sending cancel requests for (unsigned int sessionIdx = 0; sessionIdx < lstSize(this->sessionList); sessionIdx++) memContextCallbackClear(objMemContext(*(ProtocolClientSession **)lstGet(this->sessionList, sessionIdx))); // Send an exit request but don't wait to see if it succeeds MEM_CONTEXT_TEMP_BEGIN() { // Switch state to idle so the request is sent no matter the current state this->state = protocolClientStateIdle; // Send exit request protocolClientRequestInternal(protocolClientSessionNewP(this, PROTOCOL_COMMAND_EXIT), protocolCommandTypeProcess, NULL); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolClient * protocolClientNew(const String *const name, const String *const service, IoRead *const read, IoWrite *const write) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(STRING, service); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_END(); ASSERT(name != NULL); ASSERT(read != NULL); ASSERT(write != NULL); OBJ_NEW_BEGIN(ProtocolClient, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (ProtocolClient) { .pub = { .read = read, }, .state = protocolClientStateIdle, .write = write, .name = strDup(name), .errorPrefix = strNewFmt("raised from %s", strZ(name)), .keepAliveTime = timeMSec(), .sessionList = lstNewP(sizeof(ProtocolClientSession *)), }; // Read, parse, and check the protocol greeting MEM_CONTEXT_TEMP_BEGIN() { JsonRead *const greeting = jsonReadNew(ioReadLine(this->pub.read)); jsonReadObjectBegin(greeting); const struct { const StringId key; const char *const value; } expected[] = { {.key = PROTOCOL_GREETING_NAME, .value = PROJECT_NAME}, {.key = PROTOCOL_GREETING_SERVICE, .value = strZ(service)}, {.key = PROTOCOL_GREETING_VERSION, .value = PROJECT_VERSION}, }; for (unsigned int expectedIdx = 0; expectedIdx < LENGTH_OF(expected); expectedIdx++) { if (!jsonReadKeyExpectStrId(greeting, expected[expectedIdx].key)) THROW_FMT(ProtocolError, "unable to find greeting key '%s'", zNewStrId(expected[expectedIdx].key)); if (jsonReadTypeNext(greeting) != jsonTypeString) THROW_FMT(ProtocolError, "greeting key '%s' must be string type", zNewStrId(expected[expectedIdx].key)); const String *const actualValue = jsonReadStr(greeting); if (!strEqZ(actualValue, expected[expectedIdx].value)) { THROW_FMT( ProtocolError, "expected value '%s' for greeting key '%s' but got '%s'\n" "HINT: is the same version of " PROJECT_NAME " installed on the local and remote host?", expected[expectedIdx].value, zNewStrId(expected[expectedIdx].key), strZ(actualValue)); } } jsonReadObjectEnd(greeting); } MEM_CONTEXT_TEMP_END(); // Set a callback to shutdown the protocol memContextCallbackSet(objMemContext(this), protocolClientFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(PROTOCOL_CLIENT, this); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * protocolClientRequest(ProtocolClient *const this, const StringId command, const ProtocolClientRequestParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, this); FUNCTION_LOG_PARAM(STRING_ID, command); FUNCTION_LOG_PARAM(PACK_WRITE, param.param); FUNCTION_LOG_END(); ASSERT(this != NULL); PackRead *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { ProtocolClientSession *const session = protocolClientSessionNewP(this, command); protocolClientRequestInternal(session, protocolCommandTypeProcess, param.param); MEM_CONTEXT_PRIOR_BEGIN() { result = protocolClientResponseInternal(session); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK_READ, result); } /**********************************************************************************************************************************/ FN_EXTERN void protocolClientNoOp(ProtocolClient *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); protocolClientRequestP(this, PROTOCOL_COMMAND_NOOP); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolClientKeepAlive(ProtocolClient *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // A request is data put to the server, so a client that is busy needs no noop to keep the server from timing out if (timeMSec() - this->keepAliveTime >= PROTOCOL_KEEP_ALIVE_TIME) protocolClientNoOp(this); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Free client session ***********************************************************************************************************************************/ static void protocolClientSessionFreeResource(THIS_VOID) { THIS(ProtocolClientSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // If a result is queued then read it before sending cancel if (protocolClientSessionQueued(this)) { // Free stored result if (this->stored) { pckReadFree(this->packRead); } // Else read and free result else pckReadFree(protocolClientResponseInternal(this)); } // If open then cancel if (this->pub.open) { protocolClientRequestInternal(this, protocolCommandTypeCancel, NULL); protocolClientResponseInternal(this); ASSERT(!this->pub.open); } // Remove from session list lstRemoveIdx(this->client->sessionList, protocolClientSessionFindIdx(this->client, this->sessionId)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolClientSession * protocolClientSessionNew(ProtocolClient *const client, const StringId command, const ProtocolClientSessionNewParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client); FUNCTION_LOG_PARAM(STRING_ID, command); FUNCTION_LOG_PARAM(BOOL, param.async); FUNCTION_LOG_END(); ASSERT(client != NULL); OBJ_NEW_BEGIN(ProtocolClientSession, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (ProtocolClientSession) { .client = client, .command = command, .sessionId = ++client->sessionTotal, .async = param.async, }; } OBJ_NEW_END(); // If async then the session will need to be tracked in case another session gets a result for this session if (this->async) { lstAdd(this->client->sessionList, &this); // Set a callback to cleanup the session memContextCallbackSet(objMemContext(this), protocolClientSessionFreeResource, this); } FUNCTION_LOG_RETURN(PROTOCOL_CLIENT_SESSION, this); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * protocolClientSessionOpen(ProtocolClientSession *const this, const ProtocolClientSessionOpenParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_PARAM(PACK_WRITE, param.param); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->pub.open); protocolClientRequestInternal(this, protocolCommandTypeOpen, param.param); this->pub.open = true; // Set a callback to cleanup the session if it was not already done for async if (!this->async) { lstAdd(this->client->sessionList, &this); memContextCallbackSet(objMemContext(this), protocolClientSessionFreeResource, this); } FUNCTION_LOG_RETURN(PACK_READ, protocolClientResponseInternal(this)); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * protocolClientSessionRequest(ProtocolClientSession *const this, const ProtocolClientRequestParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_PARAM(PACK_WRITE, param.param); FUNCTION_LOG_END(); ASSERT(this != NULL); protocolClientRequestInternal(this, protocolCommandTypeProcess, param.param); FUNCTION_LOG_RETURN(PACK_READ, protocolClientResponseInternal(this)); } /**********************************************************************************************************************************/ FN_EXTERN void protocolClientSessionRequestAsync(ProtocolClientSession *const this, const ProtocolClientRequestParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_PARAM(PACK_WRITE, param.param); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->async); protocolClientRequestInternal(this, protocolCommandTypeProcess, param.param); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * protocolClientSessionResponse(ProtocolClientSession *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->async); FUNCTION_LOG_RETURN(PACK_READ, protocolClientResponseInternal(this)); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * protocolClientSessionClose(ProtocolClientSession *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.open); protocolClientRequestInternal(this, protocolCommandTypeClose, NULL); PackRead *const result = protocolClientResponseInternal(this); ASSERT(!this->pub.open); memContextCallbackClear(objMemContext(this)); protocolClientSessionFreeResource(this); FUNCTION_LOG_RETURN(PACK_READ, result); } /**********************************************************************************************************************************/ FN_EXTERN void protocolClientSessionCancel(ProtocolClientSession *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); memContextCallbackClear(objMemContext(this)); protocolClientSessionFreeResource(this); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolClientToLog(const ProtocolClient *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{name: %s, state: ", strZ(this->name)); strStcResultSizeInc(debugLog, strIdToLog(this->state, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCatChr(debugLog, '}'); } /**********************************************************************************************************************************/ FN_EXTERN void protocolClientSessionToLog(const ProtocolClientSession *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{client: "); protocolClientToLog(this->client, debugLog); strStcCat(debugLog, ", command: "); strStcResultSizeInc(debugLog, strIdToLog(this->command, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt( debugLog, ", sessionId: %" PRIu64 ", queued %s, stored: %s", this->sessionId, cvtBoolToConstZ(protocolClientSessionQueued(this)), cvtBoolToConstZ(this->stored)); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/protocol/client.h0000664000175100017510000003230115265101745017460 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Client There are three ways to make a request using the client: 1. Synchronous This is the most common way to make a request. Simply call protocolClientRequestP() and process the result. 2. Synchronous with Session In some cases it is useful to keep session state between requests. An example of this is queries against a database where it is more efficient to maintain the connection to the database between queries. The session is created with protocolClientSessionNewP(), opened with protocolClientSessionOpenP(), and closed with protocolClientSessionClose(). Requests are made with protocolClientSessionRequest(). 3. Asynchronous with Session Asynchronous requests provide better performance by allowing the client and server to process at the same time. An example of this is reading a file where a asynchronous request for the next block of data can be sent before the current block is being processed. The next block should be waiting when processing of the current block is complete. The same session functions are used for creation, open, and close, except that .async = true is passed to protocolClientSessionNewP(). Asynchronous requests are made with protocolClientSessionRequestAsync() and the responses are read with protocolClientSessionResponse(). Note that there can only be one outstanding asynchronous request, therefore protocolClientSessionResponse() must be called before protocolClientSessionRequestAsync() can be called again. ***********************************************************************************************************************************/ #ifndef PROTOCOL_CLIENT_H #define PROTOCOL_CLIENT_H /*********************************************************************************************************************************** Message types used by the protocol ***********************************************************************************************************************************/ typedef enum { // Data passed from server to client protocolMessageTypeResponse = 0, // Request sent from the client to the server protocolMessageTypeRequest = 1, // An error occurred on the server and the request ended abnormally. protocolMessageTypeResponse will not be sent to the client. protocolMessageTypeError = 2, } ProtocolMessageType; /*********************************************************************************************************************************** Command types ***********************************************************************************************************************************/ typedef enum { protocolCommandTypeOpen = STRID5("opn", 0x3a0f0), // Open command for processing protocolCommandTypeProcess = STRID5("prc", 0xe500), // Process command protocolCommandTypeClose = STRID5("cls", 0x4d830), // Close command protocolCommandTypeCancel = STRID5("cnc", 0xdc30), // Cancel command } ProtocolCommandType; /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ProtocolClient ProtocolClient; typedef struct ProtocolClientSession ProtocolClientSession; #include "common/io/read.h" #include "common/io/write.h" #include "common/type/object.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define PROTOCOL_GREETING_NAME STRID5("name", 0x2b42e0) #define PROTOCOL_GREETING_SERVICE STRID5("service", 0x1469b48b30) #define PROTOCOL_GREETING_VERSION STRID5("version", 0x39e99c8b60) #define PROTOCOL_COMMAND_CONFIG STRID5("config", 0xe9339e30) #define PROTOCOL_COMMAND_EXIT STRID5("exit", 0xa27050) #define PROTOCOL_COMMAND_NOOP STRID5("noop", 0x83dee0) /*********************************************************************************************************************************** This size should be safe for most pack data without wasting a lot of space. If binary data is being transferred then this size can be added to the expected binary size to account for overhead. ***********************************************************************************************************************************/ #define PROTOCOL_PACK_DEFAULT_SIZE 1024 // Pack large enough for standard data. Note that the buffer will automatically resize when required. FN_INLINE_ALWAYS PackWrite * protocolPackNew(void) { return pckWriteNewP(.size = PROTOCOL_PACK_DEFAULT_SIZE); } /*********************************************************************************************************************************** Client Constructors ***********************************************************************************************************************************/ FN_EXTERN ProtocolClient *protocolClientNew(const String *name, const String *service, IoRead *read, IoWrite *write); /*********************************************************************************************************************************** Client Getters/Setters ***********************************************************************************************************************************/ typedef struct ProtocolClientPub { IoRead *read; // Read interface } ProtocolClientPub; // Read file descriptor FN_INLINE_ALWAYS int protocolClientIoReadFd(ProtocolClient *const this) { return ioReadFd(THIS_PUB(ProtocolClient)->read); } /*********************************************************************************************************************************** Client Functions ***********************************************************************************************************************************/ // Move to a new parent mem context FN_INLINE_ALWAYS ProtocolClient * protocolClientMove(ProtocolClient *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Do not send exit command to the server when the client is freed FN_INLINE_ALWAYS void protocolClientNoExit(ProtocolClient *const this) { memContextCallbackClear(objMemContext(this)); } // Send noop to test connection or keep it alive FN_EXTERN void protocolClientNoOp(ProtocolClient *this); // Send noop only when the client has been idle long enough for the server to be at risk of timing out. Callers send a keep-alive // once per unit of work, which is far more often than the server needs one, so most calls do nothing. FN_EXTERN void protocolClientKeepAlive(ProtocolClient *this); // Simple request that does not require a session or async typedef struct ProtocolClientRequestParam { VAR_PARAM_HEADER; PackWrite *param; } ProtocolClientRequestParam; #define protocolClientRequestP(this, command, ...) \ protocolClientRequest(this, command, (ProtocolClientRequestParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackRead *protocolClientRequest(ProtocolClient *this, StringId command, ProtocolClientRequestParam param); /*********************************************************************************************************************************** Client Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void protocolClientFree(ProtocolClient *const this) { objFree(this); } /*********************************************************************************************************************************** Session Constructors ***********************************************************************************************************************************/ // New session typedef struct ProtocolClientSessionNewParam { VAR_PARAM_HEADER; bool async; // Async requests allowed? } ProtocolClientSessionNewParam; #define protocolClientSessionNewP(client, command, ...) \ protocolClientSessionNew(client, command, (ProtocolClientSessionNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN ProtocolClientSession *protocolClientSessionNew( ProtocolClient *client, StringId command, ProtocolClientSessionNewParam param); /*********************************************************************************************************************************** Session Getters/Setters ***********************************************************************************************************************************/ typedef struct ProtocolClientSessionPub { bool open; // Is the session open? bool queued; // Is a response currently queued? } ProtocolClientSessionPub; // Is a response currently queued? FN_INLINE_ALWAYS bool protocolClientSessionQueued(const ProtocolClientSession *const this) { return THIS_PUB(ProtocolClientSession)->queued; } // Is the session closed? FN_INLINE_ALWAYS bool protocolClientSessionClosed(const ProtocolClientSession *const this) { return !THIS_PUB(ProtocolClientSession)->open; } /*********************************************************************************************************************************** Session Functions ***********************************************************************************************************************************/ // Session open typedef struct ProtocolClientSessionOpenParam { VAR_PARAM_HEADER; PackWrite *param; } ProtocolClientSessionOpenParam; #define protocolClientSessionOpenP(this, ...) \ protocolClientSessionOpen(this, (ProtocolClientSessionOpenParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackRead *protocolClientSessionOpen(ProtocolClientSession *const this, ProtocolClientSessionOpenParam param); // Session request #define protocolClientSessionRequestP(this, ...) \ protocolClientSessionRequest(this, (ProtocolClientRequestParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackRead *protocolClientSessionRequest(ProtocolClientSession *const this, ProtocolClientRequestParam param); // Session async request #define protocolClientSessionRequestAsyncP(this, ...) \ protocolClientSessionRequestAsync(this, (ProtocolClientRequestParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void protocolClientSessionRequestAsync(ProtocolClientSession *const this, ProtocolClientRequestParam param); // Session response after a call to protocolClientSessionRequestAsyncP() FN_EXTERN PackRead *protocolClientSessionResponse(ProtocolClientSession *const this); // Session close FN_EXTERN PackRead *protocolClientSessionClose(ProtocolClientSession *const this); // Session cancel FN_EXTERN void protocolClientSessionCancel(ProtocolClientSession *const this); /*********************************************************************************************************************************** Session Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void protocolClientSessionFree(ProtocolClientSession *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void protocolClientToLog(const ProtocolClient *this, StringStatic *debugLog); #define FUNCTION_LOG_PROTOCOL_CLIENT_TYPE \ ProtocolClient * #define FUNCTION_LOG_PROTOCOL_CLIENT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, protocolClientToLog, buffer, bufferSize) FN_EXTERN void protocolClientSessionToLog(const ProtocolClientSession *this, StringStatic *debugLog); #define FUNCTION_LOG_PROTOCOL_CLIENT_SESSION_TYPE \ ProtocolClientSession * #define FUNCTION_LOG_PROTOCOL_CLIENT_SESSION_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, protocolClientSessionToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/protocol/helper.c0000664000175100017510000010747615265101745017474 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Helper ***********************************************************************************************************************************/ #include #include #include "command/lock.h" #include "common/crypto/common.h" #include "common/debug.h" #include "common/exec.h" #include "common/io/client.h" #include "common/io/socket/client.h" #include "common/io/socket/server.h" #include "common/io/tls/client.h" #include "common/io/tls/server.h" #include "common/memContext.h" #include "config/config.intern.h" #include "config/exec.h" #include "config/load.h" #include "config/parse.h" #include "config/protocol.h" #include "postgres/version.h" #include "protocol/helper.h" #include "version.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ STRING_EXTERN(PROTOCOL_SERVICE_LOCAL_STR, PROTOCOL_SERVICE_LOCAL); STRING_EXTERN(PROTOCOL_SERVICE_REMOTE_STR, PROTOCOL_SERVICE_REMOTE); /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ typedef enum { protocolClientLocal = STRID5("local", 0xc08dec0), protocolClientRemote = STRID5("remote", 0xb47b4b20), } ProtocolClientType; typedef struct ProtocolHelperClient { ProtocolClientType type; // Client type ProtocolStorageType storageType; // Storage type unsigned int hostIdx; // Host index unsigned int processId; // Process id displayed in logs Exec *exec; // Executed client IoClient *ioClient; // Io client, e.g. TlsClient IoSession *ioSession; // Io session, e.g. TlsSession ProtocolClient *client; // Protocol client } ProtocolHelperClient; static struct { MemContext *memContext; // Mem context for protocol helper List *clientList; // Client List } protocolHelper; /*********************************************************************************************************************************** Init local mem context and data structure ***********************************************************************************************************************************/ static void protocolHelperInit(void) { // In the protocol helper has not been initialized if (protocolHelper.memContext == NULL) { // Create a mem context to store protocol objects MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN("ProtocolHelper", .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX) { protocolHelper.memContext = MEM_CONTEXT_NEW(); protocolHelper.clientList = lstNewP(sizeof(ProtocolHelperClient)); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); } } /*********************************************************************************************************************************** Init local mem context and data structure ***********************************************************************************************************************************/ static ProtocolHelperClient * protocolHelperClientGet(ProtocolClientType type, ProtocolStorageType storageType, unsigned int hostIdx, unsigned int processId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_PARAM(STRING_ID, storageType); FUNCTION_TEST_PARAM(UINT, hostIdx); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_END(); ASSERT(protocolHelper.clientList != NULL); for (unsigned int clientIdx = 0; clientIdx < lstSize(protocolHelper.clientList); clientIdx++) { ProtocolHelperClient *const match = lstGet(protocolHelper.clientList, clientIdx); if (match->type == type && match->storageType == storageType && match->hostIdx == hostIdx && match->processId == processId) FUNCTION_TEST_RETURN_P(VOID, match); } FUNCTION_TEST_RETURN_P(VOID, NULL); } /**********************************************************************************************************************************/ FN_EXTERN bool repoIsLocal(const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN(BOOL, !cfgOptionIdxTest(cfgOptRepoHost, repoIdx)); } /**********************************************************************************************************************************/ FN_EXTERN bool pgIsLocal(const unsigned int pgIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, pgIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN(BOOL, !cfgOptionIdxTest(cfgOptPgHost, pgIdx)); } /**********************************************************************************************************************************/ FN_EXTERN void pgIsLocalVerify(void) { FUNCTION_TEST_VOID(); if (!pgIsLocal(cfgOptionGroupIdxDefault(cfgOptGrpPg))) THROW_FMT(HostInvalidError, "%s command must be run on the " PG_NAME " host", cfgCommandName()); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Get the command line required for local protocol execution ***********************************************************************************************************************************/ static StringList * protocolLocalParam(const ProtocolStorageType protocolStorageType, const unsigned int hostIdx, const unsigned int processId) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType); FUNCTION_LOG_PARAM(UINT, hostIdx); FUNCTION_LOG_PARAM(UINT, processId); FUNCTION_LOG_END(); StringList *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Option replacements KeyValue *const optionReplace = kvNew(); // Add the process id -- used when more than one process will be called kvPut(optionReplace, VARSTRDEF(CFGOPT_PROCESS), VARUINT(processId)); // Add the pg default. Don't do this for repos because the repo default should come from the user or the local should // handle all the repos equally. Repos don't get special handling like pg primaries or standbys. if (protocolStorageType == protocolStorageTypePg) kvPut(optionReplace, VARSTRDEF(CFGOPT_PG), VARUINT(cfgOptionGroupIdxToKey(cfgOptGrpPg, hostIdx))); // Add the remote type kvPut(optionReplace, VARSTRDEF(CFGOPT_REMOTE_TYPE), VARSTR(strNewStrId(protocolStorageType))); // Only enable file logging on the local when requested kvPut( optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_FILE), VARUINT64( cfgOptionBool(cfgOptLogSubprocess) ? cfgOptionStrId(cfgOptLogLevelFile) : CFGOPTVAL_LOG_LEVEL_FILE_OFF_STRID)); // Always output errors on stderr for debugging purposes kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_STDERR), VARUINT64(CFGOPTVAL_LOG_LEVEL_STDERR_ERROR_STRID)); // Disable output to stdout since it is used by the protocol kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_CONSOLE), VARUINT64(CFGOPTVAL_LOG_LEVEL_FILE_OFF_STRID)); MEM_CONTEXT_PRIOR_BEGIN() { result = cfgExecParam(cfgCommand(), cfgCmdRoleLocal, optionReplace, true, false); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ // Helper to execute the local process. This is a separate function solely so that it can be shimmed during testing. static void protocolLocalExec( ProtocolHelperClient *const helper, const ProtocolStorageType protocolStorageType, const unsigned int hostIdx, const unsigned int processId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, helper); FUNCTION_TEST_PARAM(ENUM, protocolStorageType); FUNCTION_TEST_PARAM(UINT, hostIdx); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(helper != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Execute the protocol command const String *name = strNewFmt(PROTOCOL_SERVICE_LOCAL "-%u process", processId); const StringList *const param = protocolLocalParam(protocolStorageType, hostIdx, processId); MEM_CONTEXT_PRIOR_BEGIN() { helper->exec = execNew(cfgBin(), param, name, cfgOptionUInt64(cfgOptProtocolTimeout)); } MEM_CONTEXT_PRIOR_END(); execOpen(helper->exec); // Create protocol object name = strNewFmt(PROTOCOL_SERVICE_LOCAL "-%u protocol", processId); MEM_CONTEXT_PRIOR_BEGIN() { helper->client = protocolClientNew( name, PROTOCOL_SERVICE_LOCAL_STR, execIoRead(helper->exec), execIoWrite(helper->exec)); } MEM_CONTEXT_PRIOR_END(); // Move client to exec context so they are freed together protocolClientMove(helper->client, execMemContext(helper->exec)); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN ProtocolClient * protocolLocalGet(const ProtocolStorageType protocolStorageType, const unsigned int hostIdx, const unsigned int processId) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType); FUNCTION_LOG_PARAM(UINT, hostIdx); FUNCTION_LOG_PARAM(UINT, processId); FUNCTION_LOG_END(); protocolHelperInit(); // Create protocol object ProtocolHelperClient *protocolHelperClient = protocolHelperClientGet( protocolClientLocal, protocolStorageType, hostIdx, processId); if (protocolHelperClient == NULL) { MEM_CONTEXT_BEGIN(protocolHelper.memContext) { ProtocolHelperClient protocolHelperClientAdd = { .type = protocolClientLocal, .storageType = protocolStorageType, .hostIdx = hostIdx, .processId = processId, }; protocolLocalExec(&protocolHelperClientAdd, protocolStorageType, hostIdx, processId); protocolHelperClient = lstAdd(protocolHelper.clientList, &protocolHelperClientAdd); } MEM_CONTEXT_END(); // Send noop to catch initialization errors protocolClientNoOp(protocolHelperClient->client); } FUNCTION_LOG_RETURN(PROTOCOL_CLIENT, protocolHelperClient->client); } /*********************************************************************************************************************************** Free the protocol client and underlying exec'd process. Log any errors as warnings since it is not worth terminating the process while closing a local/remote that has already completed its work. The warning will be an indication that something is not right. ***********************************************************************************************************************************/ static void protocolHelperClientFree(ProtocolHelperClient *const protocolHelperClient) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, protocolHelperClient); FUNCTION_LOG_END(); // Try to shutdown the protocol but only warn on error TRY_BEGIN() { protocolClientFree(protocolHelperClient->client); } CATCH_ANY() { LOG_WARN(errorMessage()); } TRY_END(); // Try to end the child process but only warn on error TRY_BEGIN() { execFree(protocolHelperClient->exec); } CATCH_ANY() { LOG_WARN(errorMessage()); } TRY_END(); // Free the io client/session (there should be no errors) ioSessionFree(protocolHelperClient->ioSession); ioClientFree(protocolHelperClient->ioClient); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ // Helper to check if client is authorized for a stanza static bool protocolServerAuthorize(const String *authListStr, const String *const stanza) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, authListStr); FUNCTION_TEST_PARAM(STRING, stanza); FUNCTION_TEST_END(); ASSERT(authListStr != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { // Empty list is not valid. ??? It would be better if this were done during config parsing. authListStr = strTrim(strDup(authListStr)); if (strEmpty(authListStr)) THROW(OptionInvalidValueError, "'" CFGOPT_TLS_SERVER_AUTH "' option must have a value"); // If * then all stanzas are authorized if (strEqZ(authListStr, "*")) { result = true; } // Else check the stanza list for a match with the specified stanza. Each entry will need to be trimmed before comparing. else if (stanza != NULL) { const StringList *const authList = strLstNewSplitZ(authListStr, ","); for (unsigned int authListIdx = 0; authListIdx < strLstSize(authList); authListIdx++) { if (strEq(strTrim(strLstGet(authList, authListIdx)), stanza)) { result = true; break; } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(BOOL, result); } FN_EXTERN ProtocolServer * protocolServer(IoServer *const tlsServer, IoSession *const socketSession) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_SERVER, tlsServer); FUNCTION_LOG_PARAM(IO_SESSION, socketSession); FUNCTION_LOG_END(); ProtocolServer *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Start TLS IoSession *const tlsSession = ioServerAccept(tlsServer, socketSession); result = protocolServerNew( PROTOCOL_SERVICE_REMOTE_STR, PROTOCOL_SERVICE_REMOTE_STR, ioSessionIoReadP(tlsSession), ioSessionIoWrite(tlsSession)); // If session is authenticated if (ioSessionAuthenticated(tlsSession)) { TRY_BEGIN() { // Get parameter list from the client. This needs to happen even if we cannot authorize the client so that the // session id gets set for the error. const ProtocolServerRequestResult command = protocolServerRequest(result); CHECK(FormatError, command.id == PROTOCOL_COMMAND_CONFIG, "expected config command"); // Get list of authorized stanzas for this client CHECK(AssertError, cfgOptionTest(cfgOptTlsServerAuth), "missing auth data"); const String *const clientAuthList = strDup( varStr(kvGet(cfgOptionKv(cfgOptTlsServerAuth), VARSTR(ioSessionPeerName(tlsSession))))); // Error if the client is not authorized for anything if (clientAuthList == NULL) THROW(AccessError, "access denied"); // Load parameter list from the client StringList *const paramList = pckReadStrLstP(pckReadNew(command.param)); strLstInsert(paramList, 0, cfgBin()); cfgLoad(strLstSize(paramList), strLstPtr(paramList)); // Error if the client is not authorized for the requested stanza if (!protocolServerAuthorize(clientAuthList, cfgOptionStrNull(cfgOptStanza))) THROW(AccessError, "access denied"); } CATCH_ANY() { protocolServerError(result, errorCode(), STR(errorMessage()), STR(errorStackTrace())); RETHROW(); } TRY_END(); // Ack the config command protocolServerResponseP(result); // Move result to prior context and move session into result so there is only one return value protocolServerMove(result, memContextPrior()); ioSessionMove(tlsSession, objMemContext(result)); } // Else the client can only detect that the server is alive else { // A noop command should have been received const ProtocolServerRequestResult command = protocolServerRequest(result); CHECK(FormatError, command.id == PROTOCOL_COMMAND_NOOP, "expected config command"); // Send a data end message and return a NULL server protocolServerResponseP(result); // Set result to NULL so there is no server for the caller to use. The TLS session will be freed when the temp mem // context ends. result = NULL; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER, result); } /*********************************************************************************************************************************** Get the command line required for remote protocol execution ***********************************************************************************************************************************/ static StringList * protocolRemoteParam(const ProtocolStorageType protocolStorageType, const unsigned int hostIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType); FUNCTION_LOG_PARAM(UINT, hostIdx); FUNCTION_LOG_END(); StringList *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Is this a repo remote? const bool isRepo = protocolStorageType == protocolStorageTypeRepo; // Option replacements KeyValue *const optionReplace = kvNew(); // Replace config options with the host versions const unsigned int optConfig = isRepo ? cfgOptRepoHostConfig : cfgOptPgHostConfig; kvPut( optionReplace, VARSTRDEF(CFGOPT_CONFIG), cfgOptionIdxSource(optConfig, hostIdx) != cfgSourceDefault ? VARSTR(cfgOptionIdxStr(optConfig, hostIdx)) : NULL); const unsigned int optConfigIncludePath = isRepo ? cfgOptRepoHostConfigIncludePath : cfgOptPgHostConfigIncludePath; kvPut( optionReplace, VARSTRDEF(CFGOPT_CONFIG_INCLUDE_PATH), cfgOptionIdxSource(optConfigIncludePath, hostIdx) != cfgSourceDefault ? VARSTR(cfgOptionIdxStr(optConfigIncludePath, hostIdx)) : NULL); const unsigned int optConfigPath = isRepo ? cfgOptRepoHostConfigPath : cfgOptPgHostConfigPath; kvPut( optionReplace, VARSTRDEF(CFGOPT_CONFIG_PATH), cfgOptionIdxSource(optConfigPath, hostIdx) != cfgSourceDefault ? VARSTR(cfgOptionIdxStr(optConfigPath, hostIdx)) : NULL); // Update/remove repo/pg options that are sent to the remote for (ConfigOption optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++) { // Skip options that are not part of a group if (!cfgOptionGroup(optionId)) continue; bool remove = false; bool skipHostZero = false; // Remove repo options that are not needed on the remote if (cfgOptionGroupId(optionId) == cfgOptGrpRepo) { // If remote type is pg then remove all repo options since they will not be used if (protocolStorageType == protocolStorageTypePg) { remove = true; } // Else remove repo options for indexes that are not being passed to the repo. This prevents the remote from getting // partial info about a repo, which could cause an error during validation. else { for (unsigned int optionIdx = 0; optionIdx < cfgOptionIdxTotal(optionId); optionIdx++) { if (cfgOptionIdxTest(optionId, optionIdx) && optionIdx != hostIdx) kvPut(optionReplace, VARSTRZ(cfgOptionIdxName(optionId, optionIdx)), NULL); } } } // Remove pg options that are not needed on the remote else { ASSERT(cfgOptionGroupId(optionId) == cfgOptGrpPg); // Move pg options to host index 0 (key 1) so they will be in the default index on the remote host if (protocolStorageType != protocolStorageTypeRepo) { if (hostIdx != 0) { kvPut( optionReplace, VARSTRZ(cfgOptionIdxName(optionId, 0)), cfgOptionIdxSource(optionId, hostIdx) != cfgSourceDefault ? cfgOptionIdxVar(optionId, hostIdx) : NULL); } remove = true; skipHostZero = true; } } // Remove options that have been marked for removal if they are not already null or invalid. This is more efficient // because cfgExecParam() won't have to search through as large a list looking for overrides. if (remove) { // Loop through option indexes for (unsigned int optionIdx = 0; optionIdx < cfgOptionIdxTotal(optionId); optionIdx++) { if (cfgOptionIdxTest(optionId, optionIdx) && !(skipHostZero && optionIdx == 0)) kvPut(optionReplace, VARSTRZ(cfgOptionIdxName(optionId, optionIdx)), NULL); } } } // Set repo default so the remote only operates on a single repo if (protocolStorageType == protocolStorageTypeRepo) { kvPut(optionReplace, VARSTRDEF(CFGOPT_REPO), VARUINT(cfgOptionGroupIdxToKey(cfgOptGrpRepo, hostIdx))); } // Else remove repo selection when the remote is pg else kvPut(optionReplace, VARSTRDEF(CFGOPT_REPO), NULL); // Add the process id if not set. This means that the remote is being started from the main process and should always get a // process id of 0. if (!cfgOptionTest(cfgOptProcess)) kvPut(optionReplace, VARSTRDEF(CFGOPT_PROCESS), VARINT(0)); // Don't pass log-path or lock-path since these are host specific kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_PATH), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOCK_PATH), NULL); // Only enable file logging on the remote when requested kvPut( optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_FILE), VARUINT64( cfgOptionBool(cfgOptLogSubprocess) ? cfgOptionStrId(cfgOptLogLevelFile) : CFGOPTVAL_LOG_LEVEL_FILE_OFF_STRID)); // Always output errors on stderr for debugging purposes kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_STDERR), VARUINT64(CFGOPTVAL_LOG_LEVEL_STDERR_ERROR_STRID)); // Disable output to stdout since it is used by the protocol kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_CONSOLE), VARUINT64(CFGOPTVAL_LOG_LEVEL_FILE_OFF_STRID)); // Add the remote type kvPut(optionReplace, VARSTRDEF(CFGOPT_REMOTE_TYPE), VARSTR(strNewStrId(protocolStorageType))); // Add lock required on the remote if (cfgLockRemoteRequired()) kvPut(optionReplace, VARSTRDEF(CFGOPT_LOCK), varNewVarLst(varLstNewStrLst(cmdLockList()))); MEM_CONTEXT_PRIOR_BEGIN() { result = cfgExecParam(cfgCommand(), cfgCmdRoleRemote, optionReplace, false, true); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } // Helper to add SSH parameters when executing the remote via SSH static StringList * protocolRemoteParamSsh(const ProtocolStorageType protocolStorageType, const unsigned int hostIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType); FUNCTION_LOG_PARAM(UINT, hostIdx); FUNCTION_LOG_END(); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Is this a repo remote? const bool isRepo = protocolStorageType == protocolStorageTypeRepo; // Fixed parameters for ssh command strLstAddZ(result, "-o"); strLstAddZ(result, "LogLevel=error"); strLstAddZ(result, "-o"); strLstAddZ(result, "Compression=no"); strLstAddZ(result, "-o"); strLstAddZ(result, "PasswordAuthentication=no"); // Append port if specified const ConfigOption optHostPort = isRepo ? cfgOptRepoHostPort : cfgOptPgHostPort; if (cfgOptionIdxTest(optHostPort, hostIdx)) { strLstAddZ(result, "-p"); strLstAddFmt(result, "%u", cfgOptionIdxUInt(optHostPort, hostIdx)); } // Append user/host strLstAddFmt( result, "%s@%s", strZ(cfgOptionIdxStr(isRepo ? cfgOptRepoHostUser : cfgOptPgHostUser, hostIdx)), strZ(cfgOptionIdxStr(isRepo ? cfgOptRepoHost : cfgOptPgHost, hostIdx))); // Add remote command and parameters StringList *const paramList = protocolRemoteParam(protocolStorageType, hostIdx); strLstInsert(paramList, 0, cfgOptionIdxStr(isRepo ? cfgOptRepoHostCmd : cfgOptPgHostCmd, hostIdx)); strLstAdd(result, strLstJoin(paramList, " ")); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ // Helper to execute the local process. This is a separate function solely so that it can be shimmed during testing. static void protocolRemoteExec( ProtocolHelperClient *const helper, const ProtocolStorageType protocolStorageType, const unsigned int hostIdx, const unsigned int processId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, helper); FUNCTION_TEST_PARAM(ENUM, protocolStorageType); FUNCTION_TEST_PARAM(UINT, hostIdx); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(helper != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Get remote info const bool isRepo = protocolStorageType == protocolStorageTypeRepo; const StringId remoteType = cfgOptionIdxStrId(isRepo ? cfgOptRepoHostType : cfgOptPgHostType, hostIdx); const String *const host = cfgOptionIdxStr(isRepo ? cfgOptRepoHost : cfgOptPgHost, hostIdx); // Handle remote types IoRead *read; IoWrite *write; switch (remoteType) { // SSH remote case CFGOPTVAL_REPO_HOST_TYPE_SSH: { // Exec SSH const String *const name = strNewFmt(PROTOCOL_SERVICE_REMOTE "-%u process on '%s'", processId, strZ(host)); const StringList *const param = protocolRemoteParamSsh(protocolStorageType, hostIdx); MEM_CONTEXT_PRIOR_BEGIN() { helper->exec = execNew(cfgOptionStr(cfgOptCmdSsh), param, name, cfgOptionUInt64(cfgOptProtocolTimeout)); } MEM_CONTEXT_PRIOR_END(); execOpen(helper->exec); read = execIoRead(helper->exec); write = execIoWrite(helper->exec); break; } // TLS remote default: { ASSERT(remoteType == CFGOPTVAL_REPO_HOST_TYPE_TLS); // Negotiate TLS MEM_CONTEXT_PRIOR_BEGIN() { helper->ioClient = tlsClientNewP( sckClientNew( host, cfgOptionIdxUInt(isRepo ? cfgOptRepoHostPort : cfgOptPgHostPort, hostIdx), cfgOptionUInt64(cfgOptIoTimeout), cfgOptionUInt64(cfgOptProtocolTimeout)), host, cfgOptionUInt64(cfgOptIoTimeout), cfgOptionUInt64(cfgOptProtocolTimeout), true, .caFile = cfgOptionIdxStrNull(isRepo ? cfgOptRepoHostCaFile : cfgOptPgHostCaFile, hostIdx), .caPath = cfgOptionIdxStrNull(isRepo ? cfgOptRepoHostCaPath : cfgOptPgHostCaPath, hostIdx), .certFile = cfgOptionIdxStr(isRepo ? cfgOptRepoHostCertFile : cfgOptPgHostCertFile, hostIdx), .keyFile = cfgOptionIdxStr(isRepo ? cfgOptRepoHostKeyFile : cfgOptPgHostKeyFile, hostIdx)); helper->ioSession = ioClientOpen(helper->ioClient); } MEM_CONTEXT_PRIOR_END(); read = ioSessionIoReadP(helper->ioSession); write = ioSessionIoWrite(helper->ioSession); break; } } // Create protocol object const String *const name = strNewFmt( PROTOCOL_SERVICE_REMOTE "-%u %s protocol on '%s'", processId, zNewStrId(remoteType), strZ(host)); MEM_CONTEXT_PRIOR_BEGIN() { helper->client = protocolClientNew(name, PROTOCOL_SERVICE_REMOTE_STR, read, write); } MEM_CONTEXT_PRIOR_END(); // Remote initialization switch (remoteType) { // SSH remote case CFGOPTVAL_REPO_HOST_TYPE_SSH: // Move client to exec context so they are freed together protocolClientMove(helper->client, execMemContext(helper->exec)); break; // TLS remote default: { ASSERT(remoteType == CFGOPTVAL_REPO_HOST_TYPE_TLS); TRY_BEGIN() { // Pass parameters to server PackWrite *const param = protocolPackNew(); pckWriteStrLstP(param, protocolRemoteParam(protocolStorageType, hostIdx)); protocolClientRequestP(helper->client, PROTOCOL_COMMAND_CONFIG, .param = param); } CATCH_ANY() { // Clear the callback so the client does not try to shut down the connection. Attempting to shut down the // connection will fail since the server has already disconnected and a new error will be thrown, masking the // original error. memContextCallbackClear(objMemContext(helper->client)); RETHROW(); } TRY_END(); break; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN ProtocolClient * protocolRemoteGet(const ProtocolStorageType protocolStorageType, const unsigned int hostIdx, const bool create) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType); FUNCTION_LOG_PARAM(UINT, hostIdx); FUNCTION_LOG_PARAM(BOOL, create); FUNCTION_LOG_END(); // Is this a repo remote? const bool isRepo = protocolStorageType == protocolStorageTypeRepo; protocolHelperInit(); // Determine protocol id for the remote. If the process option is set then use that since we want the remote protocol id to // match the local protocol id. Otherwise set to 0 since the remote is being started from a main process and there should only // be one remote per host. const unsigned int processId = cfgOptionTest(cfgOptProcess) ? cfgOptionUInt(cfgOptProcess) : 0; // Create protocol object ProtocolHelperClient *protocolHelperClient = protocolHelperClientGet( protocolClientRemote, protocolStorageType, hostIdx, processId); if (protocolHelperClient == NULL && create) { MEM_CONTEXT_BEGIN(protocolHelper.memContext) { ProtocolHelperClient protocolHelperClientAdd = { .type = protocolClientRemote, .storageType = protocolStorageType, .hostIdx = hostIdx, .processId = processId, }; protocolRemoteExec(&protocolHelperClientAdd, protocolStorageType, hostIdx, processId); protocolHelperClient = lstAdd(protocolHelper.clientList, &protocolHelperClientAdd); // Send noop to catch initialization errors protocolClientNoOp(protocolHelperClient->client); // Get cipher options from the remote if none are locally configured if (isRepo && cfgOptionIdxStrId(cfgOptRepoCipherType, hostIdx) == cipherTypeNone) { // Options to query VariantList *const param = varLstNew(); varLstAdd(param, varNewStrZ(cfgOptionIdxName(cfgOptRepoCipherType, hostIdx))); varLstAdd(param, varNewStrZ(cfgOptionIdxName(cfgOptRepoCipherPass, hostIdx))); const VariantList *const optionList = configOptionRemote(protocolHelperClient->client, param); if (varUInt64(varLstGet(optionList, 0)) != cipherTypeNone) { cfgOptionIdxSet(cfgOptRepoCipherType, hostIdx, cfgSourceConfig, varLstGet(optionList, 0)); cfgOptionIdxSet(cfgOptRepoCipherPass, hostIdx, cfgSourceConfig, varLstGet(optionList, 1)); } } } MEM_CONTEXT_END(); } FUNCTION_LOG_RETURN(PROTOCOL_CLIENT, protocolHelperClient != NULL ? protocolHelperClient->client : NULL); } /**********************************************************************************************************************************/ FN_EXTERN void protocolHelperFree(ProtocolClient *const client) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client); FUNCTION_LOG_END(); if (client != NULL && protocolHelper.clientList != NULL) { for (unsigned int clientIdx = 0; clientIdx < lstSize(protocolHelper.clientList); clientIdx++) { ProtocolHelperClient *const match = lstGet(protocolHelper.clientList, clientIdx); if (match->client == client) { protocolHelperClientFree(match); lstRemoveIdx(protocolHelper.clientList, clientIdx); break; } } } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolKeepAlive(void) { FUNCTION_LOG_VOID(logLevelTrace); if (protocolHelper.clientList) { for (unsigned int clientIdx = 0; clientIdx < lstSize(protocolHelper.clientList); clientIdx++) { ProtocolHelperClient *const match = lstGet(protocolHelper.clientList, clientIdx); if (match->type == protocolClientRemote) protocolClientKeepAlive(match->client); } } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolFree(void) { FUNCTION_LOG_VOID(logLevelTrace); if (protocolHelper.clientList != NULL) { while (lstSize(protocolHelper.clientList) > 0) { protocolHelperClientFree(lstGet(protocolHelper.clientList, 0)); lstRemoveIdx(protocolHelper.clientList, 0); } } FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/protocol/helper.h0000664000175100017510000000604315265101745017465 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Helper ***********************************************************************************************************************************/ #ifndef PROTOCOL_HELPER_H #define PROTOCOL_HELPER_H #include "common/type/stringId.h" /*********************************************************************************************************************************** Protocol storage type enum ***********************************************************************************************************************************/ typedef enum { protocolStorageTypePg = STRID5("pg", 0xf00), protocolStorageTypeRepo = STRID5("repo", 0x7c0b20), } ProtocolStorageType; #include "common/io/server.h" #include "protocol/server.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define PROTOCOL_SERVICE_LOCAL "local" STRING_DECLARE(PROTOCOL_SERVICE_LOCAL_STR); #define PROTOCOL_SERVICE_REMOTE "remote" STRING_DECLARE(PROTOCOL_SERVICE_REMOTE_STR); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Is pg local? FN_EXTERN bool pgIsLocal(unsigned int pgIdx); // Error if PostgreSQL is not local, i.e. pg-host is set FN_EXTERN void pgIsLocalVerify(void); // Is the repository local? FN_EXTERN bool repoIsLocal(unsigned int repoIdx); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Send keepalives to all remotes FN_EXTERN void protocolKeepAlive(void); // Local protocol client FN_EXTERN ProtocolClient *protocolLocalGet(ProtocolStorageType protocolStorageType, unsigned int hostId, unsigned int protocolId); // Remote protocol client FN_EXTERN ProtocolClient *protocolRemoteGet(ProtocolStorageType protocolStorageType, unsigned int hostId, bool create); // Free (shutdown) a local/remote client FN_EXTERN void protocolHelperFree(ProtocolClient *client); // Initialize a server FN_EXTERN ProtocolServer *protocolServer(IoServer *const tlsServer, IoSession *const socketSession); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_EXTERN void protocolFree(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/protocol/parallel.c0000664000175100017510000002741315265101745020001 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Parallel Executor ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/log.h" #include "common/macro.h" #include "common/type/keyValue.h" #include "common/type/list.h" #include "protocol/helper.h" #include "protocol/parallel.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ProtocolParallelJobData { ProtocolParallelJob *job; // Job ProtocolClientSession *session; // Protocol session for the job } ProtocolParallelJobData; struct ProtocolParallel { TimeMSec timeout; // Max time to wait for jobs before returning ParallelJobCallback *callbackFunction; // Function to get new jobs void *callbackData; // Data to pass to callback function List *clientList; // List of clients to process jobs List *jobList; // List of jobs to be processed ProtocolParallelJobData *clientJobList; // Jobs being processing by each client ProtocolParallelJobState state; // Overall state of job processing }; /**********************************************************************************************************************************/ FN_EXTERN ProtocolParallel * protocolParallelNew(const TimeMSec timeout, ParallelJobCallback *const callbackFunction, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(UINT64, timeout); FUNCTION_LOG_PARAM(FUNCTIONP, callbackFunction); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); ASSERT(callbackFunction != NULL); ASSERT(callbackData != NULL); OBJ_NEW_BEGIN(ProtocolParallel, .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX) { *this = (ProtocolParallel) { .timeout = timeout, .callbackFunction = callbackFunction, .callbackData = callbackData, .clientList = lstNewP(sizeof(ProtocolClient *)), .jobList = lstNewP(sizeof(ProtocolParallelJob *)), .state = protocolParallelJobStatePending, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(PROTOCOL_PARALLEL, this); } /**********************************************************************************************************************************/ FN_EXTERN void protocolParallelClientAdd(ProtocolParallel *const this, ProtocolClient *const client) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL, this); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(client != NULL); ASSERT(this->state == protocolParallelJobStatePending); if (protocolClientIoReadFd(client) == -1) THROW(AssertError, "client with read fd is required"); lstAdd(this->clientList, &client); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int protocolParallelProcess(ProtocolParallel *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->state != protocolParallelJobStateDone); unsigned int result = 0; MEM_CONTEXT_TEMP_BEGIN() { // If called for the first time, initialize processing if (this->state == protocolParallelJobStatePending) { MEM_CONTEXT_OBJ_BEGIN(this) { this->clientJobList = memNew(lstSize(this->clientList) * sizeof(ProtocolParallelJobData)); for (unsigned int jobIdx = 0; jobIdx < lstSize(this->clientList); jobIdx++) this->clientJobList[jobIdx] = (ProtocolParallelJobData){0}; } MEM_CONTEXT_OBJ_END(); this->state = protocolParallelJobStateRunning; } // Initialize the file descriptor set used for select fd_set selectSet; FD_ZERO(&selectSet); int fdMax = -1; // Find clients that are running jobs unsigned int clientRunningTotal = 0; for (unsigned int clientIdx = 0; clientIdx < lstSize(this->clientList); clientIdx++) { if (this->clientJobList[clientIdx].job != NULL) { int fd = protocolClientIoReadFd(*(ProtocolClient **)lstGet(this->clientList, clientIdx)); FD_SET(fd, &selectSet); // Find the max file descriptor needed for select() MAX_ASSIGN(fdMax, fd); clientRunningTotal++; } } // If clients are running then wait for one to finish if (clientRunningTotal > 0) { // Initialize timeout struct used for select. Recreate this structure each time since Linux (at least) will modify it. struct timeval timeoutSelect; timeoutSelect.tv_sec = (time_t)(this->timeout / MSEC_PER_SEC); timeoutSelect.tv_usec = (suseconds_t)(this->timeout % MSEC_PER_SEC * 1000); // Determine if there is data to be read const int completed = select(fdMax + 1, &selectSet, NULL, NULL, &timeoutSelect); THROW_ON_SYS_ERROR(completed == -1, AssertError, "unable to select from parallel client(s)"); // If any jobs have completed then get the results if (completed > 0) { for (unsigned int clientIdx = 0; clientIdx < lstSize(this->clientList); clientIdx++) { ProtocolParallelJob *const job = this->clientJobList[clientIdx].job; if (job != NULL && FD_ISSET( protocolClientIoReadFd(*(ProtocolClient **)lstGet(this->clientList, clientIdx)), &selectSet)) { MEM_CONTEXT_TEMP_BEGIN() { TRY_BEGIN() { protocolParallelJobResultSet( job, protocolClientSessionResponse(this->clientJobList[clientIdx].session)); } CATCH_ANY() { protocolParallelJobErrorSet(job, errorCode(), STR(errorMessage())); } TRY_END(); protocolParallelJobStateSet(job, protocolParallelJobStateDone); this->clientJobList[clientIdx].job = NULL; protocolClientSessionFree(this->clientJobList[clientIdx].session); } MEM_CONTEXT_TEMP_END(); } } result = (unsigned int)completed; } } // Find new jobs to be run for (unsigned int clientIdx = 0; clientIdx < lstSize(this->clientList); clientIdx++) { ProtocolClient **const clientRef = lstGet(this->clientList, clientIdx); // If nothing is running for this client and it has not already been freed if (this->clientJobList[clientIdx].job == NULL && *clientRef != NULL) { MEM_CONTEXT_BEGIN(lstMemContext(this->jobList)) { // Get a new job ProtocolParallelJob *const job = this->callbackFunction(this->callbackData, clientIdx); ProtocolClient *const client = *clientRef; // If a new job was found if (job != NULL) { // Add to the job list lstAdd(this->jobList, &job); // Put command ProtocolClientSession *const session = protocolClientSessionNewP( client, protocolParallelJobCommand(job), .async = true); protocolClientSessionRequestAsyncP(session, .param = protocolParallelJobParam(job)); // Set client id and running state protocolParallelJobProcessIdSet(job, clientIdx + 1); protocolParallelJobStateSet(job, protocolParallelJobStateRunning); this->clientJobList[clientIdx].job = job; this->clientJobList[clientIdx].session = session; } // Else no more jobs for this client so free it. Clear the client reference so it is not freed again (and its // freed memory read while logging) on a subsequent call. else { protocolHelperFree(client); *clientRef = NULL; } } MEM_CONTEXT_END(); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolParallelJob * protocolParallelResult(ProtocolParallel *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->state == protocolParallelJobStateRunning); ProtocolParallelJob *result = NULL; // Find the next completed job for (unsigned int jobIdx = 0; jobIdx < lstSize(this->jobList); jobIdx++) { ProtocolParallelJob *job = *(ProtocolParallelJob **)lstGet(this->jobList, jobIdx); if (protocolParallelJobState(job) == protocolParallelJobStateDone) { result = protocolParallelJobMove(job, memContextCurrent()); lstRemoveIdx(this->jobList, jobIdx); break; } } FUNCTION_LOG_RETURN(PROTOCOL_PARALLEL_JOB, result); } /**********************************************************************************************************************************/ FN_EXTERN bool protocolParallelDone(ProtocolParallel *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->state != protocolParallelJobStatePending); // If there are no jobs left then we are done if (this->state != protocolParallelJobStateDone && lstEmpty(this->jobList)) this->state = protocolParallelJobStateDone; FUNCTION_LOG_RETURN(BOOL, this->state == protocolParallelJobStateDone); } /**********************************************************************************************************************************/ FN_EXTERN void protocolParallelToLog(const ProtocolParallel *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{state: "); strStcResultSizeInc(debugLog, strIdToLog(this->state, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt(debugLog, ", clientTotal: %u, jobTotal: %u}", lstSize(this->clientList), lstSize(this->jobList)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/protocol/parallel.h0000664000175100017510000000742715265101745020011 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Parallel Executor ***********************************************************************************************************************************/ #ifndef PROTOCOL_PARALLEL_H #define PROTOCOL_PARALLEL_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ProtocolParallel ProtocolParallel; #include "common/time.h" #include "common/type/object.h" #include "protocol/client.h" #include "protocol/parallelJob.h" /*********************************************************************************************************************************** Job request callback Called whenever a new job is required for processing. If no more jobs are available then NULL is returned. Note that NULL must be returned to each clientIdx in case job distribution varies by clientIdx. ***********************************************************************************************************************************/ typedef ProtocolParallelJob *ParallelJobCallback(void *data, unsigned int clientIdx); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN ProtocolParallel *protocolParallelNew(TimeMSec timeout, ParallelJobCallback *callbackFunction, void *callbackData); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Are all jobs done? FN_EXTERN bool protocolParallelDone(ProtocolParallel *this); // Completed job result FN_EXTERN ProtocolParallelJob *protocolParallelResult(ProtocolParallel *this); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add client FN_EXTERN void protocolParallelClientAdd(ProtocolParallel *this, ProtocolClient *client); // Process jobs FN_EXTERN unsigned int protocolParallelProcess(ProtocolParallel *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void protocolParallelFree(ProtocolParallel *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void protocolParallelToLog(const ProtocolParallel *this, StringStatic *debugLog); #define FUNCTION_LOG_PROTOCOL_PARALLEL_TYPE \ ProtocolParallel * #define FUNCTION_LOG_PROTOCOL_PARALLEL_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, protocolParallelToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/protocol/parallelJob.c0000664000175100017510000001304415265101745020427 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Parallel Job ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "protocol/parallelJob.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct ProtocolParallelJob { ProtocolParallelJobPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN ProtocolParallelJob * protocolParallelJobNew(const Variant *const key, const StringId command, PackWrite *const param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(VARIANT, key); FUNCTION_LOG_PARAM(STRING_ID, command); FUNCTION_LOG_PARAM(PACK_WRITE, param); FUNCTION_LOG_END(); OBJ_NEW_BEGIN(ProtocolParallelJob, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (ProtocolParallelJob) { .pub = { .state = protocolParallelJobStatePending, .key = varDup(key), .command = command, .param = pckWriteMove(param, objMemContext(this)), }, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(PROTOCOL_PARALLEL_JOB, this); } /**********************************************************************************************************************************/ FN_EXTERN void protocolParallelJobErrorSet(ProtocolParallelJob *const this, const int code, const String *const message) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, this); FUNCTION_LOG_PARAM(INT, code); FUNCTION_LOG_PARAM(STRING, message); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(code != 0); ASSERT(message != NULL); MEM_CONTEXT_OBJ_BEGIN(this) { this->pub.code = code; this->pub.message = strDup(message); } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolParallelJobProcessIdSet(ProtocolParallelJob *const this, const unsigned int processId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, this); FUNCTION_LOG_PARAM(UINT, processId); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(processId > 0); this->pub.processId = processId; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolParallelJobResultSet(ProtocolParallelJob *const this, PackRead *const result) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, this); FUNCTION_LOG_PARAM(PACK_READ, result); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(protocolParallelJobErrorCode(this) == 0); this->pub.result = pckReadMove(result, objMemContext(this)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolParallelJobStateSet(ProtocolParallelJob *const this, const ProtocolParallelJobState state) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, this); FUNCTION_LOG_PARAM(STRING_ID, state); FUNCTION_LOG_END(); ASSERT(this != NULL); if (this->pub.state == protocolParallelJobStatePending && state == protocolParallelJobStateRunning) this->pub.state = protocolParallelJobStateRunning; else if (this->pub.state == protocolParallelJobStateRunning && state == protocolParallelJobStateDone) this->pub.state = protocolParallelJobStateDone; else { THROW_FMT( AssertError, "invalid state transition from '%s' to '%s'", zNewStrId(protocolParallelJobState(this)), zNewStrId(state)); } FUNCTION_LOG_RETURN_VOID(); } FN_EXTERN void protocolParallelJobToLog(const ProtocolParallelJob *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{state: "); strStcResultSizeInc(debugLog, strIdToLog(protocolParallelJobState(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", key: "); varToLog(protocolParallelJobKey(this), debugLog); strStcCat(debugLog, ", command: "); strStcResultSizeInc( debugLog, strIdToLog(protocolParallelJobCommand(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", result: "); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT( protocolParallelJobResult(this), pckReadToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt(debugLog, ", code: %d, message: ", protocolParallelJobErrorCode(this)); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT( protocolParallelJobErrorMessage(this), strToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/protocol/parallelJob.h0000664000175100017510000001416615265101745020442 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Parallel Job ***********************************************************************************************************************************/ #ifndef PROTOCOL_PARALLEL_JOB_H #define PROTOCOL_PARALLEL_JOB_H #include "common/type/stringId.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ProtocolParallelJob ProtocolParallelJob; /*********************************************************************************************************************************** Job state enum ***********************************************************************************************************************************/ typedef enum { protocolParallelJobStatePending = STRID5("pending", 0x1dc9238b00), protocolParallelJobStateRunning = STRID5("running", 0x1dc973ab20), protocolParallelJobStateDone = STRID5("done", 0x2b9e40), } ProtocolParallelJobState; #include "common/time.h" #include "common/type/object.h" #include "common/type/pack.h" #include "protocol/client.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN ProtocolParallelJob *protocolParallelJobNew(const Variant *key, StringId command, PackWrite *param); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct ProtocolParallelJobPub { const Variant *key; // Unique key used to identify the job StringId command; // Command to be executed PackWrite *param; // Command parameters unsigned int processId; // Process that executed this job ProtocolParallelJobState state; // Current state of the job int code; // Non-zero result indicates an error String *message; // Message if there was a error PackRead *result; // Result if job was successful } ProtocolParallelJobPub; // Job command FN_INLINE_ALWAYS StringId protocolParallelJobCommand(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->command; } // Job command parameters FN_INLINE_ALWAYS PackWrite * protocolParallelJobParam(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->param; } // Job error FN_INLINE_ALWAYS int protocolParallelJobErrorCode(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->code; } FN_INLINE_ALWAYS const String * protocolParallelJobErrorMessage(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->message; } FN_EXTERN void protocolParallelJobErrorSet(ProtocolParallelJob *this, int code, const String *message); // Job key FN_INLINE_ALWAYS const Variant * protocolParallelJobKey(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->key; } // Process Id FN_INLINE_ALWAYS unsigned int protocolParallelJobProcessId(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->processId; } FN_EXTERN void protocolParallelJobProcessIdSet(ProtocolParallelJob *this, unsigned int processId); // Job result FN_INLINE_ALWAYS PackRead * protocolParallelJobResult(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->result; } FN_EXTERN void protocolParallelJobResultSet(ProtocolParallelJob *const this, PackRead *const result); // Job state FN_INLINE_ALWAYS ProtocolParallelJobState protocolParallelJobState(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->state; } FN_EXTERN void protocolParallelJobStateSet(ProtocolParallelJob *this, ProtocolParallelJobState state); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Move to new parent mem context FN_INLINE_ALWAYS ProtocolParallelJob * protocolParallelJobMove(ProtocolParallelJob *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void protocolParallelJobFree(ProtocolParallelJob *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void protocolParallelJobToLog(const ProtocolParallelJob *this, StringStatic *debugLog); #define FUNCTION_LOG_PROTOCOL_PARALLEL_JOB_TYPE \ ProtocolParallelJob * #define FUNCTION_LOG_PROTOCOL_PARALLEL_JOB_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, protocolParallelJobToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/protocol/server.c0000664000175100017510000006165515265101745017521 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Server ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/error/retry.h" #include "common/log.h" #include "common/time.h" #include "common/type/json.h" #include "common/type/keyValue.h" #include "common/type/list.h" #include "protocol/helper.h" #include "protocol/server.h" #include "version.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct ProtocolServer { IoRead *read; // Read interface IoWrite *write; // Write interface const String *name; // Name displayed in logging List *sessionList; // List of active sessions uint64_t sessionId; // Current session being processed }; struct ProtocolServerResult { void *sessionData; // Session data PackWrite *data; // Result data size_t extra; // Extra bytes for initializing data bool close; // Close session? }; // Track server sessions typedef struct ProtocolServerSession { uint64_t id; // Session id void *data; // Data for the session } ProtocolServerSession; /**********************************************************************************************************************************/ FN_EXTERN ProtocolServer * protocolServerNew(const String *const name, const String *const service, IoRead *const read, IoWrite *const write) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(STRING, service); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_END(); ASSERT(name != NULL); ASSERT(read != NULL); ASSERT(write != NULL); OBJ_NEW_BEGIN(ProtocolServer, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (ProtocolServer) { .read = read, .write = write, .name = strDup(name), .sessionList = lstNewP(sizeof(ProtocolServerSession)), }; // Send the protocol greeting MEM_CONTEXT_TEMP_BEGIN() { JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); jsonWriteZ(jsonWriteKeyStrId(json, PROTOCOL_GREETING_NAME), PROJECT_NAME); jsonWriteStr(jsonWriteKeyStrId(json, PROTOCOL_GREETING_SERVICE), service); jsonWriteZ(jsonWriteKeyStrId(json, PROTOCOL_GREETING_VERSION), PROJECT_VERSION); ioWriteStrLine(this->write, jsonWriteResult(jsonWriteObjectEnd(json))); ioWriteFlush(this->write); } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER, this); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerResponse(ProtocolServer *const this, const ProtocolServerResponseParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PROTOCOL_SERVER, this); FUNCTION_TEST_PARAM(ENUM, param.type); FUNCTION_TEST_PARAM(BOOL, param.close); FUNCTION_TEST_PARAM(PACK_WRITE, param.data); FUNCTION_TEST_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { // End the pack if (param.data != NULL) pckWriteEndP(param.data); // Write the result PackWrite *const resultMessage = pckWriteNewIo(this->write); pckWriteU64P(resultMessage, this->sessionId); pckWriteU32P(resultMessage, param.type, .defaultWrite = true); pckWriteBoolP(resultMessage, param.close); pckWritePackP(resultMessage, pckWriteResult(param.data)); pckWriteEndP(resultMessage); ioWriteFlush(this->write); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerError(ProtocolServer *this, const int code, const String *const message, const String *const stack) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PROTOCOL_SERVER, this); FUNCTION_LOG_PARAM(INT, code); FUNCTION_LOG_PARAM(STRING, message); FUNCTION_LOG_PARAM(STRING, stack); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(code != 0); ASSERT(message != NULL); ASSERT(stack != NULL); MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = protocolPackNew(); pckWriteI32P(packWrite, code); pckWriteStrP(packWrite, message); pckWriteStrP(packWrite, stack); protocolServerResponseP(this, .type = protocolMessageTypeError, .data = packWrite); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerRequestResult protocolServerRequest(ProtocolServer *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_SERVER, this); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ProtocolServerRequestResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { PackRead *const request = pckReadNewIo(this->read); const ProtocolMessageType type = (ProtocolMessageType)pckReadU32P(request); CHECK(FormatError, type == protocolMessageTypeRequest, "expected request message"); MEM_CONTEXT_PRIOR_BEGIN() { result.id = pckReadStrIdP(request); result.type = (ProtocolCommandType)pckReadStrIdP(request); this->sessionId = pckReadU64P(request); result.sessionRequired = pckReadBoolP(request); result.param = pckReadPackP(request); ASSERT(this->sessionId != 0); } MEM_CONTEXT_PRIOR_END(); pckReadEndP(request); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerProcess(ProtocolServer *const this, const VariantList *const retryInterval, const List *const handlerList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PROTOCOL_SERVER, this); FUNCTION_LOG_PARAM(VARIANT_LIST, retryInterval); FUNCTION_LOG_PARAM(LIST, handlerList); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(handlerList != NULL); ASSERT(!lstEmpty(handlerList)); // Loop until exit request is received bool exit = false; do { TRY_BEGIN() { MEM_CONTEXT_TEMP_BEGIN() { // Get request ProtocolServerRequestResult request = protocolServerRequest(this); // Find the handler const ProtocolServerHandler *handler = NULL; for (unsigned int handlerIdx = 0; handlerIdx < lstSize(handlerList); handlerIdx++) { const ProtocolServerHandler *const handlerMatch = lstGet(handlerList, handlerIdx); if (request.id == handlerMatch->command) { handler = handlerMatch; break; } } // If handler was found then process if (handler != NULL) { // Send the request to the handler MEM_CONTEXT_TEMP_BEGIN() { // Variables to store first error message and retry messages ErrorRetry *const errRetry = errRetryNew(); const String *errStackTrace = NULL; // Initialize retries in case of request failure bool retry = false; unsigned int retryRemaining = retryInterval != NULL ? varLstSize(retryInterval) : 0; TimeMSec retrySleepMs = 0; // Handler retry loop do { retry = false; TRY_BEGIN() { // Process request type switch (request.type) { // Open a protocol session case protocolCommandTypeOpen: { ASSERT(handler->open != NULL); // Call open handler ProtocolServerResult *const openResult = handler->open(pckReadNew(request.param)); ASSERT(openResult != NULL); ASSERT(!openResult->close); ProtocolServerSession session = {.id = this->sessionId}; // Send data protocolServerResponseP( this, .data = openResult->data, .close = openResult->sessionData == NULL); // Create session if open returned session data if (openResult->sessionData != NULL) { session.data = objMove(openResult->sessionData, objMemContext(this->sessionList)); lstAdd(this->sessionList, &session); } break; } // Process or close/cancel protocol session default: { // Find session data void *sessionData = NULL; unsigned int sessionListIdx = 0; if (request.sessionRequired) { ASSERT(handler->processSession != NULL); ASSERT(handler->process == NULL); for (; sessionListIdx < lstSize(this->sessionList); sessionListIdx++) { ProtocolServerSession *const session = lstGet(this->sessionList, sessionListIdx); if (session->id == this->sessionId) { sessionData = session->data; break; } } // Error when session not found if (sessionData == NULL && request.type != protocolCommandTypeCancel) { THROW_FMT( ProtocolError, "unable to find session id %" PRIu64 " for request %s:%s", this->sessionId, zNewStrId(request.id), zNewStrId(request.type)); } } // Process request type switch (request.type) { // Process request case protocolCommandTypeProcess: { // Process session ProtocolServerResult *processResult; if (handler->processSession != NULL) { ASSERT(handler->process == NULL); CHECK_FMT( ProtocolError, this->sessionId != 0, "no session id for request %s:%s", zNewStrId(request.id), zNewStrId(request.type)); processResult = handler->processSession(pckReadNew(request.param), sessionData); } // Standalone process else { ASSERT(handler->process != NULL); ASSERT(!request.sessionRequired); processResult = handler->process(pckReadNew(request.param)); } if (processResult == NULL) protocolServerResponseP(this); else { ASSERT(processResult->sessionData == NULL); ASSERT(handler->processSession != NULL || !processResult->close); protocolServerResponseP( this, .data = processResult->data, .close = processResult->close); // Free session when close is true. This optimization allows an explicit // close to be skipped. if (processResult->close) { objFree(sessionData); lstRemoveIdx(this->sessionList, sessionListIdx); } } break; } // Close protocol session case protocolCommandTypeClose: { // If there is a close handler then call it PackWrite *data = NULL; if (handler->close != NULL) { ProtocolServerResult *const closeResult = handler->close( pckReadNew(request.param), sessionData); ASSERT(closeResult != NULL); ASSERT(closeResult->sessionData == NULL); ASSERT(!closeResult->close); data = closeResult->data; } // Send data protocolServerResponseP(this, .data = data, .close = true); // Free the session objFree(sessionData); lstRemoveIdx(this->sessionList, sessionListIdx); break; } // Cancel protocol session default: { CHECK_FMT( ProtocolError, request.type == protocolCommandTypeCancel, "unknown request type '%s'", zNewStrId(request.type)); // Send NULL data protocolServerResponseP(this, .close = true); // Free the session if (sessionData != NULL) { objFree(sessionData); lstRemoveIdx(this->sessionList, sessionListIdx); } break; } } } } } CATCH_ANY() { // Add the error retry info errRetryAddP(errRetry); // On first error record the stack trace. Only the first error will contain a stack trace since // the first error is most likely to contain valuable information. if (errStackTrace == NULL) errStackTrace = strNewZ(errorStackTrace()); // Are there retries remaining? if (retryRemaining > 0) { // Get the sleep interval for this retry retrySleepMs = varUInt64(varLstGet(retryInterval, varLstSize(retryInterval) - retryRemaining)); // Log the retry LOG_DEBUG_FMT( "retry %s after %" PRIu64 "ms: %s", errorTypeName(errorType()), retrySleepMs, errorMessage()); // Sleep for interval sleepMSec(retrySleepMs); // Decrement retries remaining and retry retryRemaining--; retry = true; // Send keep-alive to remotes. A retry means the request is taking longer than usual so make // sure the remote does not timeout. protocolKeepAlive(); } // Else report error to the client else { protocolServerError( this, errorTypeCode(errRetryType(errRetry)), errRetryMessage(errRetry), errStackTrace); } } TRY_END(); } while (retry); } MEM_CONTEXT_TEMP_END(); } // Else check built-in requests else { switch (request.id) { case PROTOCOL_COMMAND_EXIT: exit = true; break; case PROTOCOL_COMMAND_NOOP: protocolServerResponseP(this); break; default: THROW_FMT( ProtocolError, "invalid request '%s' (0x%" PRIx64 ")", zNewStrId(request.id), request.id); } } // Send keep-alive to remotes. When a local process is doing work that does not involve the remote it is important // that the remote does not timeout. This will send a keep alive once per unit of work that is performed by the // local process. protocolKeepAlive(); } MEM_CONTEXT_TEMP_END(); } CATCH_FATAL() { // Zero session id so a fatal error will be handled by the first client that sees it this->sessionId = 0; // Report error to the client protocolServerError(this, errorCode(), STR(errorMessage()), STR(errorStackTrace())); // Rethrow so the process exits with an error RETHROW(); } TRY_END(); } while (!exit); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * protocolServerResultNew(const ProtocolServerResultNewParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, param.extra); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(ProtocolServerResult, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (ProtocolServerResult) { .extra = param.extra, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(PROTOCOL_SERVER_RESULT, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * protocolServerResultData(ProtocolServerResult *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PROTOCOL_SERVER_RESULT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->data == NULL); MEM_CONTEXT_OBJ_BEGIN(this) { this->data = pckWriteNewP(.size = PROTOCOL_PACK_DEFAULT_SIZE + this->extra); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(PACK_WRITE, this->data); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerResultSessionDataSet(ProtocolServerResult *const this, void *const sessionData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PROTOCOL_SERVER_RESULT, this); FUNCTION_TEST_PARAM_P(VOID, sessionData); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(sessionData != NULL); this->sessionData = objMove(sessionData, objMemContext(this)); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerResultCloseSet(ProtocolServerResult *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PROTOCOL_SERVER_RESULT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); this->close = true; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerToLog(const ProtocolServer *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{name: %s}", strZ(this->name)); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerResultToLog(const ProtocolServerResult *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{data: %s, sessionData: %s, close: %s}", cvtBoolToConstZ(this->data != NULL), cvtBoolToConstZ(this->sessionData != NULL), cvtBoolToConstZ(this->close)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/protocol/server.h0000664000175100017510000001763215265101745017522 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Server ***********************************************************************************************************************************/ #ifndef PROTOCOL_SERVER_H #define PROTOCOL_SERVER_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ProtocolServer ProtocolServer; typedef struct ProtocolServerResult ProtocolServerResult; #include "common/io/read.h" #include "common/io/write.h" #include "common/type/object.h" #include "common/type/pack.h" #include "common/type/stringId.h" #include "protocol/client.h" /*********************************************************************************************************************************** Protocol command handler type and structure An array of this struct must be passed to protocolServerProcess() for the server to process commands. Each command handler should implement a single command, as defined by the command string. ***********************************************************************************************************************************/ typedef ProtocolServerResult *(*ProtocolServerCommandOpenHandler)(PackRead *param); typedef ProtocolServerResult *(*ProtocolServerCommandProcessHandler)(PackRead *param); typedef ProtocolServerResult *(*ProtocolServerCommandProcessSessionHandler)(PackRead *param, void *sessionData); typedef ProtocolServerResult *(*ProtocolServerCommandCloseHandler)(PackRead *param, void *sessionData); typedef struct ProtocolServerHandler { StringId command; // 5-bit StringId that identifies the protocol command ProtocolServerCommandOpenHandler open; // Function that opens the protocol session ProtocolServerCommandProcessHandler process; // Function that processes the protocol command ProtocolServerCommandProcessSessionHandler processSession; // Function that processes the protocol command for a session ProtocolServerCommandCloseHandler close; // Function that closes the protocol session } ProtocolServerHandler; /*********************************************************************************************************************************** Server Constructors ***********************************************************************************************************************************/ FN_EXTERN ProtocolServer *protocolServerNew(const String *name, const String *service, IoRead *read, IoWrite *write); /*********************************************************************************************************************************** Server Functions ***********************************************************************************************************************************/ // Get command from the client. Outside ProtocolServer, this is used when the first noop command needs to be processed before // running protocolServerProcess(), which allows an error to be returned to the client if initialization fails. typedef struct ProtocolServerRequestResult { StringId id; // Command identifier ProtocolCommandType type; // Command type bool sessionRequired; // Session with more than one command Pack *param; // Parameter pack } ProtocolServerRequestResult; FN_EXTERN ProtocolServerRequestResult protocolServerRequest(ProtocolServer *this); // Send response to client typedef struct ProtocolServerResponseParam { VAR_PARAM_HEADER; bool close; // Has the session been closed? ProtocolMessageType type; // Message type PackWrite *data; // Response data } ProtocolServerResponseParam; #define protocolServerResponseP(this, ...) \ protocolServerResponse(this, (ProtocolServerResponseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void protocolServerResponse(ProtocolServer *const this, ProtocolServerResponseParam param); // Send an error to client FN_EXTERN void protocolServerError(ProtocolServer *this, int code, const String *message, const String *stack); // Process requests FN_EXTERN void protocolServerProcess(ProtocolServer *this, const VariantList *retryInterval, const List *handlerList); // Move to a new parent mem context FN_INLINE_ALWAYS ProtocolServer * protocolServerMove(ProtocolServer *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Server Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void protocolServerFree(ProtocolServer *const this) { objFree(this); } /*********************************************************************************************************************************** Result Constructors ***********************************************************************************************************************************/ typedef struct ProtocolServerResultNewParam { VAR_PARAM_HEADER; size_t extra; } ProtocolServerResultNewParam; #define protocolServerResultNewP(...) \ protocolServerResultNew((ProtocolServerResultNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN ProtocolServerResult *protocolServerResultNew(ProtocolServerResultNewParam param); /*********************************************************************************************************************************** Result Getters/Setters ***********************************************************************************************************************************/ // Create PackWrite object required to send data to the client FN_EXTERN PackWrite *protocolServerResultData(ProtocolServerResult *this); // Set session data FN_EXTERN void protocolServerResultSessionDataSet(ProtocolServerResult *const this, void *const sessionData); // Close session FN_EXTERN void protocolServerResultCloseSet(ProtocolServerResult *const this); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void protocolServerToLog(const ProtocolServer *this, StringStatic *debugLog); #define FUNCTION_LOG_PROTOCOL_SERVER_TYPE \ ProtocolServer * #define FUNCTION_LOG_PROTOCOL_SERVER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, protocolServerToLog, buffer, bufferSize) FN_EXTERN void protocolServerResultToLog(const ProtocolServerResult *this, StringStatic *debugLog); #define FUNCTION_LOG_PROTOCOL_SERVER_RESULT_TYPE \ ProtocolServerResult * #define FUNCTION_LOG_PROTOCOL_SERVER_RESULT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, protocolServerResultToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/0000775000175100017510000000000015265101745015635 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/azure/0000775000175100017510000000000015265101745016763 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/azure/helper.c0000664000175100017510000001101115265101745020400 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage Helper ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/url.h" #include "common/io/io.h" #include "common/log.h" #include "config/config.h" #include "storage/azure/helper.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN Storage * storageAzureHelper(const unsigned int repoIdx, const bool write, StoragePathExpressionCallback pathExpressionCallback) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM_P(VOID, pathExpressionCallback); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxStrId(cfgOptRepoType, repoIdx) == STORAGE_AZURE_TYPE); Storage *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Parse the endpoint url const HttpUrl *const url = httpUrlNewParseP( cfgOptionIdxStr(cfgOptRepoAzureEndpoint, repoIdx), .type = httpProtocolTypeAny, .defaultType = httpProtocolTypeHttps); const String *endpoint = httpUrlHost(url); unsigned int port = httpUrlPort(url); HttpProtocolType protocolType = httpUrlProtocolType(url); StorageAzureUriStyle uriStyle = (StorageAzureUriStyle)cfgOptionIdxSeq(cfgOptRepoAzureUriStyle, repoIdx); // If the host is set then set it as the endpoint. The host option is used to set path-style URIs when working with Azurite. // This was ill-advised, so the uri-style option was added to allow the user to select the URI style used by the server. // Preserve the old behavior when uri-style is defaulted. if (cfgOptionIdxStrNull(cfgOptRepoStorageHost, repoIdx) != NULL) { const HttpUrl *const url = httpUrlNewParseP( cfgOptionIdxStr(cfgOptRepoStorageHost, repoIdx), .type = httpProtocolTypeAny, .defaultType = httpProtocolTypeHttps); endpoint = httpUrlHost(url); port = httpUrlPort(url); protocolType = httpUrlProtocolType(url); if (cfgOptionIdxSource(cfgOptRepoAzureUriStyle, repoIdx) == cfgSourceDefault) uriStyle = storageAzureUriStylePath; } // Ensure the key is valid base64 when key type is shared const StorageAzureKeyType keyType = (StorageAzureKeyType)cfgOptionIdxSeq(cfgOptRepoAzureKeyType, repoIdx); const String *const key = cfgOptionIdxStrNull(cfgOptRepoAzureKey, repoIdx); if (keyType == storageAzureKeyTypeShared) { TRY_BEGIN() { bufNewDecode(encodingBase64, key); } CATCH(FormatError) { THROW_FMT( OptionInvalidValueError, "invalid value for '%s' option: %s\n" "HINT: value must be valid base64 when '%s = " CFGOPTVAL_REPO_AZURE_KEY_TYPE_SHARED_Z "'.", cfgOptionIdxName(cfgOptRepoAzureKey, repoIdx), errorMessage(), cfgOptionIdxName(cfgOptRepoAzureKeyType, repoIdx)); } TRY_END(); } // If port was specified, overwrite the parsed/default port if (cfgOptionIdxSource(cfgOptRepoStoragePort, repoIdx) != cfgSourceDefault) port = cfgOptionIdxUInt(cfgOptRepoStoragePort, repoIdx); MEM_CONTEXT_PRIOR_BEGIN() { result = storageAzureNew( cfgOptionIdxStr(cfgOptRepoPath, repoIdx), write, storageRepoTargetTime(), pathExpressionCallback, cfgOptionIdxStr(cfgOptRepoAzureContainer, repoIdx), cfgOptionIdxStr(cfgOptRepoAzureAccount, repoIdx), keyType, key, (size_t)cfgOptionIdxUInt64(cfgOptRepoStorageUploadChunkSize, repoIdx), cfgOptionIdxKvNull(cfgOptRepoStorageTag, repoIdx), endpoint, uriStyle, port, ioTimeoutMs(), protocolType, cfgOptionIdxBool(cfgOptRepoStorageVerifyTls, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaFile, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaPath, repoIdx), cfgOptionIdxUInt(cfgOptRepoStoragePrefetch, repoIdx), cfgOptionIdxUInt64(cfgOptRepoStorageReadOver, repoIdx)); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/azure/helper.h0000664000175100017510000000237415265101745020421 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage Helper ***********************************************************************************************************************************/ #ifndef STORAGE_AZURE_STORAGE_HELPER_H #define STORAGE_AZURE_STORAGE_HELPER_H #include "storage/azure/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN Storage *storageAzureHelper(unsigned int repoIdx, bool write, StoragePathExpressionCallback pathExpressionCallback); /*********************************************************************************************************************************** Storage helper for StorageHelper array passed to storageHelperInit() ***********************************************************************************************************************************/ #define STORAGE_AZURE_HELPER {.type = STORAGE_AZURE_TYPE, .helper = storageAzureHelper} #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/azure/read.c0000664000175100017510000001601615265101745020046 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage Read ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/client.h" #include "common/log.h" #include "common/type/object.h" #include "storage/azure/read.h" #include "storage/read.h" /*********************************************************************************************************************************** Azure query tokens ***********************************************************************************************************************************/ STRING_STATIC(AZURE_QUERY_VERSION_ID_STR, "versionid"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageReadAzure { const StorageReadInterface *interface; // Interface StorageAzure *storage; // Storage that created this object const String *name; // File name uint64_t offset; // Read offset const Variant *limit; // Read limit (NULL for no limit) const String *versionId; // Version id (NULL for most recent) HttpRequest *httpRequest; // HTTP request HttpResponse *httpResponse; // HTTP response }; /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadAzureOpenAsync(THIS_VOID) { THIS(StorageReadAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_AZURE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpRequest != NULL); // Wait for response MEM_CONTEXT_OBJ_BEGIN(this) { this->httpResponse = storageAzureResponseP(this->httpRequest, .allowMissing = true, .contentIo = true); httpRequestFree(this->httpRequest); this->httpRequest = NULL; } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN(BOOL, httpResponseCodeOk(this->httpResponse)); } static bool storageReadAzureOpen(THIS_VOID) { THIS(StorageReadAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_AZURE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpResponse == NULL); // Request the file MEM_CONTEXT_OBJ_BEGIN(this) { this->httpRequest = storageAzureRequestAsyncP( this->storage, HTTP_VERB_GET_STR, .path = this->name, .query = this->versionId != NULL ? httpQueryPut(httpQueryNewP(), AZURE_QUERY_VERSION_ID_STR, this->versionId) : NULL, .header = httpHeaderPutRange(httpHeaderNew(NULL), this->offset, this->limit)); } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN(BOOL, true); } /*********************************************************************************************************************************** Read from a file ***********************************************************************************************************************************/ static size_t storageReadAzure(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_AZURE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); (void)block; // Blocking is handled by the caller FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpResponse != NULL); ASSERT(buffer != NULL && !bufFull(buffer)); ASSERT(httpResponseIoRead(this->httpResponse) != NULL); FUNCTION_LOG_RETURN(SIZE, ioRead(httpResponseIoRead(this->httpResponse), buffer)); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadAzureEof(THIS_VOID) { THIS(StorageReadAzure); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_AZURE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->httpResponse != NULL); ASSERT(httpResponseIoRead(this->httpResponse) != NULL); FUNCTION_TEST_RETURN(BOOL, ioReadEof(httpResponseIoRead(this->httpResponse))); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageReadAzureClose(THIS_VOID) { THIS(StorageReadAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_AZURE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); httpRequestFree(this->httpRequest); this->httpRequest = NULL; httpResponseFree(this->httpResponse); this->httpResponse = NULL; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageReadInterface storageReadAzureInterface = { .close = storageReadAzureClose, .eof = storageReadAzureEof, .open = storageReadAzureOpen, .openAsync = storageReadAzureOpenAsync, .read = storageReadAzure, }; FN_EXTERN StorageReadAzure * storageReadAzureNew( StorageAzure *const storage, const String *const name, const uint64_t offset, const Variant *const limit, const String *const versionId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_AZURE, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_PARAM(STRING, versionId); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageReadAzure, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageReadAzure) { .interface = &storageReadAzureInterface, .storage = storage, .name = strDup(name), .offset = offset, .limit = varDup(limit), .versionId = strDup(versionId), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_AZURE, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/azure/read.h0000664000175100017510000000345615265101745020057 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage Read ***********************************************************************************************************************************/ #ifndef STORAGE_AZURE_READ_H #define STORAGE_AZURE_READ_H #include "storage/azure/storage.intern.h" #include "storage/read.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadAzure StorageReadAzure; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadAzure *storageReadAzureNew( StorageAzure *storage, const String *name, uint64_t offset, const Variant *limit, const String *versionId); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_AZURE_TYPE \ StorageReadAzure * #define FUNCTION_LOG_STORAGE_READ_AZURE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadAzure", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/azure/storage.c0000664000175100017510000014273715265101745020611 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage ***********************************************************************************************************************************/ #include #include #include "common/crypto/common.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/http/client.h" #include "common/io/http/common.h" #include "common/io/socket/client.h" #include "common/io/tls/client.h" #include "common/log.h" #include "common/regExp.h" #include "common/stat.h" #include "common/type/json.h" #include "common/type/object.h" #include "common/type/xml.h" #include "storage/azure/read.h" #include "storage/azure/write.h" /*********************************************************************************************************************************** Defaults ***********************************************************************************************************************************/ #define STORAGE_AZURE_DELETE_MAX 256 /*********************************************************************************************************************************** Azure http headers ***********************************************************************************************************************************/ STRING_STATIC(AZURE_HEADER_TAGS, "x-ms-tags"); STRING_STATIC(AZURE_HEADER_VERSION_STR, "x-ms-version"); STRING_STATIC(AZURE_HEADER_VERSION_VALUE_STR, "2024-08-04"); /*********************************************************************************************************************************** Azure query tokens ***********************************************************************************************************************************/ STRING_STATIC(AZURE_QUERY_MARKER_STR, "marker"); STRING_STATIC(AZURE_QUERY_BATCH_STR, "batch"); STRING_EXTERN(AZURE_QUERY_COMP_STR, AZURE_QUERY_COMP); STRING_STATIC(AZURE_QUERY_DELIMITER_STR, "delimiter"); STRING_STATIC(AZURE_QUERY_INCLUDE_STR, "include"); STRING_STATIC(AZURE_QUERY_PREFIX_STR, "prefix"); STRING_EXTERN(AZURE_QUERY_RESTYPE_STR, AZURE_QUERY_RESTYPE); STRING_STATIC(AZURE_QUERY_SIG_STR, "sig"); STRING_STATIC(AZURE_QUERY_VALUE_LIST_STR, "list"); STRING_EXTERN(AZURE_QUERY_VALUE_CONTAINER_STR, AZURE_QUERY_VALUE_CONTAINER); STRING_STATIC(AZURE_QUERY_VALUE_VERSIONS_STR, "versions"); STRING_STATIC(AZURE_QUERY_API_VERSION, "api-version"); STRING_STATIC(AZURE_QUERY_RESOURCE, "resource"); /*********************************************************************************************************************************** XML tags ***********************************************************************************************************************************/ STRING_STATIC(AZURE_XML_TAG_BLOB_PREFIX_STR, "BlobPrefix"); STRING_STATIC(AZURE_XML_TAG_BLOB_STR, "Blob"); STRING_STATIC(AZURE_XML_TAG_BLOBS_STR, "Blobs"); STRING_STATIC(AZURE_XML_TAG_CONTENT_LENGTH_STR, "Content-Length"); STRING_STATIC(AZURE_XML_TAG_LAST_MODIFIED_STR, "Last-Modified"); STRING_STATIC(AZURE_XML_TAG_NEXT_MARKER_STR, "NextMarker"); STRING_STATIC(AZURE_XML_TAG_NAME_STR, "Name"); STRING_STATIC(AZURE_XML_TAG_PROPERTIES_STR, "Properties"); STRING_STATIC(AZURE_XML_TAG_VERSION_ID_STR, "VersionId"); /*********************************************************************************************************************************** Constants required for Azure managed identities Documentation for the response format is found at: https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-to-use-vm-token#get-a-token-using-curl ***********************************************************************************************************************************/ STRING_STATIC(AZURE_CREDENTIAL_HOST_STR, "169.254.169.254"); #define AZURE_CREDENTIAL_PORT 80 #define AZURE_CREDENTIAL_PATH "/metadata/identity/oauth2/token" #define AZURE_CREDENTIAL_API_VERSION "2018-02-01" VARIANT_STRDEF_STATIC(AZURE_JSON_TAG_ACCESS_TOKEN_VAR, "access_token"); VARIANT_STRDEF_STATIC(AZURE_JSON_TAG_EXPIRES_IN_VAR, "expires_in"); /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_STATIC(AZURE_STAT_REMOVE_STR, "azure.rm"); STRING_STATIC(AZURE_STAT_REMOVE_BATCH_STR, "azure.rm.batch"); STRING_STATIC(AZURE_STAT_REMOVE_BATCH_PART_STR, "azure.rm.batch.part"); STRING_STATIC(AZURE_STAT_REMOVE_BATCH_RETRY_STR, "azure.rm.batch.retry"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageAzure { STORAGE_COMMON_MEMBER; HttpClient *httpClient; // Http client to service requests StringList *headerRedactList; // List of headers to redact from logging StringList *queryRedactList; // List of query keys to redact from logging StorageAzureKeyType keyType; // Key type (e.g. storageAzureKeyTypeShared) const String *container; // Container to store data in const String *account; // Account const Buffer *sharedKey; // Shared key const HttpQuery *sasKey; // SAS key const String *host; // Host name size_t blockSize; // Block size for multi-block upload unsigned int deleteMax; // Maximum objects that can be deleted in one request const String *tag; // Tags to be applied to objects const String *pathPrefix; // Account/container prefix uint64_t fileId; // Id to used to make file block identifiers unique // For Azure managed identities authentication HttpClient *credHttpClient; // HTTP client to service credential requests const String *credHost; // Credentials host String *accessToken; // Access token time_t accessTokenExpirationTime; // Time the access token expires }; /*********************************************************************************************************************************** Generate authorization header and add it to the supplied header list Based on the documentation at https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key ***********************************************************************************************************************************/ static void storageAzureAuth( StorageAzure *const this, const String *const verb, const String *const path, HttpQuery *const query, const String *const dateTime, HttpHeader *const httpHeader, const bool subRequest) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_AZURE, this); FUNCTION_TEST_PARAM(STRING, verb); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(HTTP_QUERY, query); FUNCTION_TEST_PARAM(STRING, dateTime); FUNCTION_TEST_PARAM(KEY_VALUE, httpHeader); FUNCTION_TEST_PARAM(BOOL, subRequest); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(verb != NULL); ASSERT(path != NULL); ASSERT(dateTime != NULL); ASSERT(httpHeader != NULL); ASSERT(httpHeaderGet(httpHeader, HTTP_HEADER_CONTENT_LENGTH_STR) != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Host header is required for all authentication types but must not be set on batch sub-requests if (!subRequest) httpHeaderPut(httpHeader, HTTP_HEADER_HOST_STR, this->host); // Date header is required for shared key authentication (for signing) if (this->keyType == storageAzureKeyTypeShared) httpHeaderPut(httpHeader, HTTP_HEADER_DATE_STR, dateTime); // Set version header but never on batch sub-requests, which inherit the version from the top-level batch request. SAS // specifies a version in the sv parameter, but that is not sufficient for batch, which rejects the request when the version // is not set explicitly. if (!subRequest) httpHeaderPut(httpHeader, AZURE_HEADER_VERSION_STR, AZURE_HEADER_VERSION_VALUE_STR); // Shared key authentication if (this->keyType == storageAzureKeyTypeShared) { ASSERT(this->sharedKey != NULL); // Generate canonical headers String *const headerCanonical = strNew(); const StringList *const headerKeyList = httpHeaderList(httpHeader); for (unsigned int headerKeyIdx = 0; headerKeyIdx < strLstSize(headerKeyList); headerKeyIdx++) { const String *const headerKey = strLstGet(headerKeyList, headerKeyIdx); if (!strBeginsWithZ(headerKey, "x-ms-")) continue; strCatFmt(headerCanonical, "%s:%s\n", strZ(headerKey), strZ(httpHeaderGet(httpHeader, headerKey))); } // Generate canonical query String *const queryCanonical = strNew(); if (query != NULL) { const StringList *const queryKeyList = httpQueryList(query); ASSERT(!strLstEmpty(queryKeyList)); for (unsigned int queryKeyIdx = 0; queryKeyIdx < strLstSize(queryKeyList); queryKeyIdx++) { const String *const queryKey = strLstGet(queryKeyList, queryKeyIdx); strCatFmt(queryCanonical, "\n%s:%s", strZ(queryKey), strZ(httpQueryGet(query, queryKey))); } } // Generate string to sign const String *const contentLength = httpHeaderGet(httpHeader, HTTP_HEADER_CONTENT_LENGTH_STR); const String *const contentMd5 = httpHeaderGet(httpHeader, HTTP_HEADER_CONTENT_MD5_STR); const String *const contentType = httpHeaderGet(httpHeader, HTTP_HEADER_CONTENT_TYPE_STR); const String *const range = httpHeaderGet(httpHeader, HTTP_HEADER_RANGE_STR); const String *const stringToSign = strNewFmt( "%s\n" // verb "\n" // content-encoding "\n" // content-language "%s\n" // content-length "%s\n" // content-md5 "%s\n" // content-type "%s\n" // date "\n" // If-Modified-Since "\n" // If-Match "\n" // If-None-Match "\n" // If-Unmodified-Since "%s\n" // range "%s" // Canonicalized headers "/%s%s" // Canonicalized account/path "%s", // Canonicalized query strZ(verb), strEq(contentLength, ZERO_STR) ? "" : strZ(contentLength), contentMd5 == NULL ? "" : strZ(contentMd5), contentType == NULL ? "" : strZ(contentType), strZ(dateTime), range == NULL ? "" : strZ(range), strZ(headerCanonical), strZ(this->account), strZ(path), strZ(queryCanonical)); // Generate authorization header httpHeaderPut( httpHeader, HTTP_HEADER_AUTHORIZATION_STR, strNewFmt( "SharedKey %s:%s", strZ(this->account), strZ(strNewEncode(encodingBase64, cryptoHmacOne(hashTypeSha256, this->sharedKey, BUFSTR(stringToSign)))))); } // Auto authentication else if (this->keyType == storageAzureKeyTypeAuto) { const time_t timeBegin = time(NULL); if (timeBegin >= this->accessTokenExpirationTime) { // Retrieve the access token via the Managed Identities endpoint HttpHeader *const authHeader = httpHeaderNew(NULL); httpHeaderAdd(authHeader, STRDEF("Metadata"), TRUE_STR); httpHeaderAdd(authHeader, HTTP_HEADER_HOST_STR, this->credHost); httpHeaderAdd(authHeader, HTTP_HEADER_CONTENT_LENGTH_STR, ZERO_STR); HttpQuery *const authQuery = httpQueryNewP(); httpQueryAdd(authQuery, AZURE_QUERY_API_VERSION, STRDEF(AZURE_CREDENTIAL_API_VERSION)); httpQueryAdd(authQuery, AZURE_QUERY_RESOURCE, strNewFmt("https://%s", strZ(this->host))); HttpRequest *const request = httpRequestNewP( this->credHttpClient, HTTP_VERB_GET_STR, STRDEF(AZURE_CREDENTIAL_PATH), .header = authHeader, .query = authQuery); HttpResponse *const response = httpRequestResponse(request, true); // Set the access_token on success and store an expiration time when we should re-fetch it if (httpResponseCodeOk(response)) { // Get credentials and expiration from the JSON response const KeyValue *const credential = varKv(jsonToVar(strNewBuf(httpResponseContent(response)))); const String *const accessToken = varStr(kvGet(credential, AZURE_JSON_TAG_ACCESS_TOKEN_VAR)); CHECK(FormatError, accessToken != NULL, "access token missing"); const Variant *const expiresInStr = kvGet(credential, AZURE_JSON_TAG_EXPIRES_IN_VAR); CHECK(FormatError, expiresInStr != NULL, "expiry missing"); MEM_CONTEXT_OBJ_BEGIN(this) { strCat(strTrunc(this->accessToken), accessToken); // Subtract http client timeout * 2 so the token does not expire in the middle of http retries const time_t clientTimeoutPeriod = ((time_t)(httpClientTimeout(this->httpClient) / MSEC_PER_SEC * 2)); const time_t expiresIn = (time_t)varInt64Force(expiresInStr); this->accessTokenExpirationTime = timeBegin + expiresIn - clientTimeoutPeriod; } MEM_CONTEXT_OBJ_END(); } else httpRequestError(request, response); } // Add authorization header httpHeaderPut(httpHeader, HTTP_HEADER_AUTHORIZATION_STR, strNewFmt("Bearer %s", strZ(this->accessToken))); } // SAS authentication else httpQueryMerge(query, this->sasKey); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Process Azure request ***********************************************************************************************************************************/ FN_EXTERN HttpRequest * storageAzureRequestAsync(StorageAzure *const this, const String *const verb, StorageAzureRequestAsyncParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(STRING, param.path); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_PARAM(LIST, param.contentList); FUNCTION_LOG_PARAM(BOOL, param.tag); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(verb != NULL); HttpRequest *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Prepend path prefix param.path = param.path == NULL ? this->pathPrefix : strNewFmt("%s%s", strZ(this->pathPrefix), strZ(param.path)); // Create header list HttpHeader *requestHeader = param.header == NULL ? httpHeaderNew(this->headerRedactList) : httpHeaderDup(param.header, this->headerRedactList); // Set content or construct multipart content const String *const date = httpDateFromTime(time(NULL)); const Buffer *content = param.content; if (param.contentList != NULL) { ASSERT(param.content == NULL); ASSERT(!lstEmpty(param.contentList)); HttpRequestMulti *const requestMulti = httpRequestMultiNew(); for (unsigned int contentIdx = 0; contentIdx < lstSize(param.contentList); contentIdx++) { const StorageAzureRequestPart *const requestPart = lstGet(param.contentList, contentIdx); HttpHeader *const partHeader = httpHeaderNew(this->headerRedactList); HttpQuery *const partQuery = this->sasKey != NULL ? httpQueryNewP(.redactList = this->queryRedactList) : NULL; const String *const path = strNewFmt("%s%s", strZ(this->pathPrefix), strZ(requestPart->path)); httpHeaderAdd(partHeader, HTTP_HEADER_CONTENT_LENGTH_STR, ZERO_STR); storageAzureAuth(this, requestPart->verb, path, partQuery, date, partHeader, true); httpRequestMultiAddP( requestMulti, strNewFmt("%u", contentIdx), requestPart->verb, path, .header = partHeader, .query = partQuery); } httpRequestMultiHeaderAdd(requestMulti, requestHeader); content = httpRequestMultiContent(requestMulti); } // Set content length httpHeaderAdd( requestHeader, HTTP_HEADER_CONTENT_LENGTH_STR, content == NULL || bufEmpty(content) ? ZERO_STR : strNewFmt("%zu", bufUsed(content))); // Calculate content-md5 header if there is content if (content != NULL) { httpHeaderAdd( requestHeader, HTTP_HEADER_CONTENT_MD5_STR, strNewEncode(encodingBase64, cryptoHashOne(hashTypeMd5, content))); } // Set tags when requested and available if (param.tag && this->tag != NULL) httpHeaderPut(requestHeader, AZURE_HEADER_TAGS, this->tag); // Encode path const String *const path = httpUriEncode(param.path, true); // Make a copy of the query so it can be modified HttpQuery *const query = this->sasKey != NULL && param.query == NULL ? httpQueryNewP(.redactList = this->queryRedactList) : httpQueryDupP(param.query, .redactList = this->queryRedactList); // Generate authorization header storageAzureAuth(this, verb, path, query, date, requestHeader, false); // Send request MEM_CONTEXT_PRIOR_BEGIN() { result = httpRequestNewP( this->httpClient, verb, path, .query = query, .header = requestHeader, .content = content); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_REQUEST, result); } FN_EXTERN HttpResponse * storageAzureResponse(HttpRequest *const request, const StorageAzureResponseParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_REQUEST, request); FUNCTION_LOG_PARAM(BOOL, param.allowMissing); FUNCTION_LOG_PARAM(BOOL, param.contentIo); FUNCTION_LOG_END(); ASSERT(request != NULL); HttpResponse *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get response result = httpRequestResponse(request, !param.contentIo); // Error if the request was not successful if (!httpResponseCodeOk(result) && (!param.allowMissing || httpResponseCode(result) != HTTP_RESPONSE_CODE_NOT_FOUND)) httpRequestError(request, result); // Move response to the prior context httpResponseMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } FN_EXTERN HttpResponse * storageAzureRequest(StorageAzure *const this, const String *const verb, const StorageAzureRequestParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(STRING, param.path); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_PARAM(LIST, param.contentList); FUNCTION_LOG_PARAM(BOOL, param.allowMissing); FUNCTION_LOG_PARAM(BOOL, param.contentIo); FUNCTION_LOG_PARAM(BOOL, param.tag); FUNCTION_LOG_END(); HttpRequest *const request = storageAzureRequestAsyncP( this, verb, .path = param.path, .header = param.header, .query = param.query, .content = param.content, .contentList = param.contentList, .tag = param.tag); HttpResponse *const result = storageAzureResponseP(request, .allowMissing = param.allowMissing, .contentIo = param.contentIo); httpRequestFree(request); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } /*********************************************************************************************************************************** General function for listing files to be used by other list routines Based on the documentation at https://learn.microsoft.com/en-us/rest/api/storageservices/list-blobs ***********************************************************************************************************************************/ static void storageAzureListInternal( StorageAzure *const this, const String *const path, const StorageInfoLevel level, const String *const expression, const bool recurse, const time_t targetTime, const StorageListCallback callback, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(STRING, expression); FUNCTION_LOG_PARAM(BOOL, recurse); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, callback); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Build the base prefix by stripping off the initial / const String *const basePrefix = strSize(path) == 1 ? EMPTY_STR : strNewFmt("%s/", strZ(strSub(path, 1))); // Get the expression prefix when possible to limit initial results const String *const expressionPrefix = regExpPrefix(expression); // If there is an expression prefix then use it to build the query prefix, otherwise query prefix is base prefix const String *queryPrefix; if (expressionPrefix == NULL) queryPrefix = basePrefix; else { if (strEmpty(basePrefix)) queryPrefix = expressionPrefix; else queryPrefix = strNewFmt("%s%s", strZ(basePrefix), strZ(expressionPrefix)); } // Create query HttpQuery *const query = httpQueryNewP(); // Add the delimiter to not recurse if (!recurse) httpQueryAdd(query, AZURE_QUERY_DELIMITER_STR, FSLASH_STR); // Add resource type httpQueryAdd(query, AZURE_QUERY_RESTYPE_STR, AZURE_QUERY_VALUE_CONTAINER_STR); // Add list comp httpQueryAdd(query, AZURE_QUERY_COMP_STR, AZURE_QUERY_VALUE_LIST_STR); // Don't specify empty prefix because it is the default if (!strEmpty(queryPrefix)) httpQueryAdd(query, AZURE_QUERY_PREFIX_STR, queryPrefix); // Add versions if (targetTime != 0) httpQueryAdd(query, AZURE_QUERY_INCLUDE_STR, AZURE_QUERY_VALUE_VERSIONS_STR); // Store last info so it can be updated across requests for versioning String *const nameLast = strNew(); String *const versionIdLast = strNew(); StorageInfo infoLast = {.level = level, .name = nameLast}; if (targetTime != 0) infoLast.versionId = versionIdLast; // Loop as long as a continuation marker returned HttpRequest *request = NULL; do { // Use an inner mem context here because we could potentially be retrieving millions of files so it is a good idea to // free memory at regular intervals MEM_CONTEXT_TEMP_BEGIN() { HttpResponse *response = NULL; // If there is an outstanding async request then wait for the response if (request != NULL) { response = storageAzureResponseP(request); httpRequestFree(request); request = NULL; } // Else get the response immediately from a sync request else response = storageAzureRequestP(this, HTTP_VERB_GET_STR, .query = query); const XmlNode *const xmlRoot = xmlDocumentRoot(xmlDocumentNewBuf(httpResponseContent(response))); // If a continuation marker exists then send an async request to get more data const String *const continuationMarker = xmlNodeContent( xmlNodeChild(xmlRoot, AZURE_XML_TAG_NEXT_MARKER_STR, false)); if (!strEq(continuationMarker, EMPTY_STR)) { httpQueryPut(query, AZURE_QUERY_MARKER_STR, continuationMarker); // Store request in the outer temp context MEM_CONTEXT_PRIOR_BEGIN() { request = storageAzureRequestAsyncP(this, HTTP_VERB_GET_STR, .query = query); } MEM_CONTEXT_PRIOR_END(); } // Get prefix list const XmlNode *const blobs = xmlNodeChild(xmlRoot, AZURE_XML_TAG_BLOBS_STR, true); const XmlNodeList *const blobPrefixList = xmlNodeChildList(blobs, AZURE_XML_TAG_BLOB_PREFIX_STR); for (unsigned int blobPrefixIdx = 0; blobPrefixIdx < xmlNodeLstSize(blobPrefixList); blobPrefixIdx++) { const XmlNode *const subPathNode = xmlNodeLstGet(blobPrefixList, blobPrefixIdx); // Get path name StorageInfo info = { .level = level, .name = xmlNodeContent(xmlNodeChild(subPathNode, AZURE_XML_TAG_NAME_STR, true)), .exists = true, }; // Strip off base prefix and final / info.name = strSubN(info.name, strSize(basePrefix), strSize(info.name) - strSize(basePrefix) - 1); // Add type info if requested if (level >= storageInfoLevelType) info.type = storageTypePath; // Callback with info callback(callbackData, &info); } // Get file list const XmlNodeList *const fileList = xmlNodeChildList(blobs, AZURE_XML_TAG_BLOB_STR); for (unsigned int fileIdx = 0; fileIdx < xmlNodeLstSize(fileList); fileIdx++) { const XmlNode *const fileNode = xmlNodeLstGet(fileList, fileIdx); const XmlNode *const property = xmlNodeChild(fileNode, AZURE_XML_TAG_PROPERTIES_STR, true); // Get file name and strip off the base prefix when present const String *name = xmlNodeContent(xmlNodeChild(fileNode, AZURE_XML_TAG_NAME_STR, true)); if (!strEmpty(basePrefix)) name = strSub(name, strSize(basePrefix)); // Return info for last file if new file if (infoLast.exists && !strEq(name, nameLast)) { callback(callbackData, &infoLast); infoLast.exists = false; } // If targeting by time exclude versions that are newer than targetTime. Note that the API does not provide // reliable delete markers so the filtering will also show files that have been deleted rather than replaced // with a new version. The problem with the delete markers is that Creation-Time/Last-Modified are set equal to // the times in the last version so we don't know when the file was deleted. It might be possible to use // VersionId for this purpose, since it appears to be a timestamp, but the field is described as "opaque" in the // documentation so it does not seem to be a good idea to use it. if (targetTime != 0) { infoLast.timeModified = httpDateToTime( xmlNodeContent(xmlNodeChild(property, AZURE_XML_TAG_LAST_MODIFIED_STR, true))); // Skip this version if it is newer than the time limit if (infoLast.timeModified > targetTime) continue; } // Update last name and set exists strCat(strTrunc(nameLast), name); infoLast.exists = true; // Add basic info if requested (no need to add type info since file is default type) if (level >= storageInfoLevelBasic) { infoLast.size = cvtZToUInt64( strZ(xmlNodeContent(xmlNodeChild(property, AZURE_XML_TAG_CONTENT_LENGTH_STR, true)))); if (targetTime == 0) { infoLast.timeModified = httpDateToTime( xmlNodeContent(xmlNodeChild(property, AZURE_XML_TAG_LAST_MODIFIED_STR, true))); } else { strCat( strTrunc(versionIdLast), xmlNodeContent(xmlNodeChild(fileNode, AZURE_XML_TAG_VERSION_ID_STR, true))); } } } } MEM_CONTEXT_TEMP_END(); } while (request != NULL); // Callback with last info if it exists if (infoLast.exists) callback(callbackData, &infoLast); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static StorageInfo storageAzureInfo(THIS_VOID, const String *const file, const StorageInfoLevel level, const StorageInterfaceInfoParam param) { THIS(StorageAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(ENUM, level); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); // Attempt to get file info HttpResponse *const httpResponse = storageAzureRequestP(this, HTTP_VERB_HEAD_STR, .path = file, .allowMissing = true); // Does the file exist? StorageInfo result = {.level = level, .exists = httpResponseCodeOk(httpResponse)}; // Add basic level info if requested and the file exists (no need to add type info since file is default type) if (result.level >= storageInfoLevelBasic && result.exists) { result.size = cvtZToUInt64(strZ(httpHeaderGet(httpResponseHeader(httpResponse), HTTP_HEADER_CONTENT_LENGTH_STR))); result.timeModified = httpDateToTime(httpHeaderGet(httpResponseHeader(httpResponse), HTTP_HEADER_LAST_MODIFIED_STR)); } httpResponseFree(httpResponse); FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ static void storageAzureListCallback(void *const callbackData, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(callbackData != NULL); ASSERT(info != NULL); storageLstAdd(callbackData, info); FUNCTION_TEST_RETURN_VOID(); } static StorageList * storageAzureList(THIS_VOID, const String *const path, const StorageInfoLevel level, const StorageInterfaceListParam param) { THIS(StorageAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(STRING, param.expression); FUNCTION_LOG_PARAM(TIME, param.targetTime); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); StorageList *const result = storageLstNew(level); storageAzureListInternal(this, path, level, param.expression, false, param.targetTime, storageAzureListCallback, result); FUNCTION_LOG_RETURN(STORAGE_LIST, result); } /**********************************************************************************************************************************/ static void * storageAzureNewRead(THIS_VOID, const String *const file, const StorageInterfaceNewReadParam param) { THIS(StorageAzure); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_PARAM(STRING, param.versionId); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); FUNCTION_LOG_RETURN(STORAGE_READ_AZURE, storageReadAzureNew(this, file, param.offset, param.limit, param.versionId)); } /**********************************************************************************************************************************/ static void * storageAzureNewWrite(THIS_VOID, const String *const file, const StorageInterfaceNewWriteParam param) { THIS(StorageAzure); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, file); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.createPath); ASSERT(param.truncate); ASSERT(param.user == NULL); ASSERT(param.group == NULL); ASSERT(param.timeModified == 0); FUNCTION_LOG_RETURN(STORAGE_WRITE_AZURE, storageWriteAzureNew(this, file, this->fileId++, this->blockSize)); } /**********************************************************************************************************************************/ typedef struct StorageAzurePathRemoveData { StorageAzure *this; // Storage object MemContext *memContext; // Mem context to create requests in HttpRequest *request; // Async remove request List *requestContentList; // Content list for async request List *contentList; // Content list currently being built const String *path; // Root path of remove } StorageAzurePathRemoveData; static void storageAzurePathRemoveInternal(StorageAzurePathRemoveData *const data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_END(); ASSERT(data != NULL); ASSERT(data->this != NULL); // Get response for async request if (data->request != NULL) { MEM_CONTEXT_TEMP_BEGIN() { HttpResponse *const response = storageAzureResponseP(data->request); HttpResponseMulti *const responseMulti = httpResponseMultiNew( httpResponseContent(response), httpHeaderGet(httpResponseHeader(response), HTTP_HEADER_CONTENT_TYPE_STR)); // Loop through all sub-requests. Response parts are mapped to the original request by ordinal position since the // service returns one response part per sub-request in request order. The echoed content-id is intentionally not used // because Azure omits it on some error responses, which would otherwise make a failed part impossible to retry. // A response with no parts at all is an error rather than a retry since the response was well formed enough to parse // but contained nothing that can be mapped to a sub-request HttpResponse *responsePart = httpResponseMultiNext(responseMulti); CHECK(FormatError, responsePart != NULL, "at least one response part is required"); for (unsigned int partIdx = 0; partIdx < lstSize(data->requestContentList); partIdx++) { // Retry when there is no response part for the sub-request or the part is not OK and not missing. A response part // is missing when the service returns fewer parts than were requested, which happens when the entire batch is // rejected, since none of the sub-requests have been run in that case. if (responsePart == NULL || (!httpResponseCodeOk(responsePart) && httpResponseCode(responsePart) != HTTP_RESPONSE_CODE_NOT_FOUND)) { const StorageAzureRequestPart *const content = lstGet(data->requestContentList, partIdx); // Retry remove statInc(AZURE_STAT_REMOVE_BATCH_RETRY_STR); httpResponseFree(storageAzureRequestP(data->this, content->verb, .path = content->path, .allowMissing = true)); } else statInc(AZURE_STAT_REMOVE_BATCH_PART_STR); // Get the next response part httpResponseFree(responsePart); responsePart = httpResponseMultiNext(responseMulti); } // Error when the service returned more response parts than there were sub-requests CHECK(FormatError, responsePart == NULL, "more response parts than sub-requests"); } MEM_CONTEXT_TEMP_END(); // Free request httpRequestFree(data->request); data->request = NULL; // Free content list lstFree(data->requestContentList); } // Send new async request if there is more to remove if (data->contentList != NULL) { statInc(AZURE_STAT_REMOVE_BATCH_STR); MEM_CONTEXT_BEGIN(data->memContext) { data->request = storageAzureRequestAsyncP( data->this, HTTP_VERB_POST_STR, .query = httpQueryAdd( httpQueryAdd(httpQueryNewP(), AZURE_QUERY_COMP_STR, AZURE_QUERY_BATCH_STR), AZURE_QUERY_RESTYPE_STR, AZURE_QUERY_VALUE_CONTAINER_STR), .contentList = data->contentList); } MEM_CONTEXT_END(); // Store the content list for use in error handling data->requestContentList = data->contentList; data->contentList = NULL; } FUNCTION_TEST_RETURN_VOID(); } static void storageAzurePathRemoveCallback(void *const callbackData, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(callbackData != NULL); ASSERT(info != NULL); // Only delete files since paths don't really exist if (info->type == storageTypeFile) { StorageAzurePathRemoveData *const data = callbackData; if (data->contentList == NULL) { MEM_CONTEXT_BEGIN(data->memContext) { data->contentList = lstNewP(sizeof(StorageAzureRequestPart)); } MEM_CONTEXT_END(); } MEM_CONTEXT_OBJ_BEGIN(data->contentList) { const StorageAzureRequestPart content = { .verb = HTTP_VERB_DELETE_STR, .path = strNewFmt("%s/%s", strZ(data->path), strZ(info->name)), }; lstAdd(data->contentList, &content); } MEM_CONTEXT_OBJ_END(); if (lstSize(data->contentList) == data->this->deleteMax) storageAzurePathRemoveInternal(data); } FUNCTION_TEST_RETURN_VOID(); } static bool storageAzurePathRemove(THIS_VOID, const String *const path, const bool recurse, const StorageInterfacePathRemoveParam param) { THIS(StorageAzure); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, path); (void)recurse; // There are no paths so removal is always recursive (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { StorageAzurePathRemoveData data = { .this = this, .memContext = memContextCurrent(), .path = strEq(path, FSLASH_STR) ? EMPTY_STR : path, }; MEM_CONTEXT_TEMP_BEGIN() { storageAzureListInternal(this, path, storageInfoLevelType, NULL, true, 0, storageAzurePathRemoveCallback, &data); // Call if there is more to be removed if (data.contentList != NULL) storageAzurePathRemoveInternal(&data); // Check response on last async request storageAzurePathRemoveInternal(&data); } MEM_CONTEXT_TEMP_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, true); } /**********************************************************************************************************************************/ static void storageAzureRemove(THIS_VOID, const String *const file, const StorageInterfaceRemoveParam param) { THIS(StorageAzure); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, file); (void)param; FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); statInc(AZURE_STAT_REMOVE_STR); httpResponseFree(storageAzureRequestP(this, HTTP_VERB_DELETE_STR, file, .allowMissing = true)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageInterface storageInterfaceAzure = { .feature = 1 << storageFeatureVersioning | 1 << storageFeatureReadRetry, .info = storageAzureInfo, .list = storageAzureList, .newRead = storageAzureNewRead, .newWrite = storageAzureNewWrite, .pathRemove = storageAzurePathRemove, .remove = storageAzureRemove, }; FN_EXTERN Storage * storageAzureNew( const String *const path, const bool write, const time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, const String *const container, const String *const account, const StorageAzureKeyType keyType, const String *const key, const size_t blockSize, const KeyValue *const tag, const String *const endpoint, const StorageAzureUriStyle uriStyle, const unsigned int port, const TimeMSec timeout, const HttpProtocolType protocolType, const bool verifyPeer, const String *const caFile, const String *const caPath, const unsigned int prefetch, const uint64_t readOver) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_PARAM(STRING, container); FUNCTION_TEST_PARAM(STRING, account); FUNCTION_LOG_PARAM(STRING_ID, keyType); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_LOG_PARAM(SIZE, blockSize); FUNCTION_LOG_PARAM(KEY_VALUE, tag); FUNCTION_LOG_PARAM(STRING, endpoint); FUNCTION_LOG_PARAM(ENUM, uriStyle); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_PARAM(ENUM, protocolType); FUNCTION_LOG_PARAM(BOOL, verifyPeer); FUNCTION_LOG_PARAM(STRING, caFile); FUNCTION_LOG_PARAM(STRING, caPath); FUNCTION_LOG_PARAM(UINT, prefetch); FUNCTION_LOG_PARAM(UINT64, readOver); FUNCTION_LOG_END(); ASSERT(path != NULL); ASSERT(container != NULL); ASSERT(account != NULL); ASSERT(endpoint != NULL); ASSERT(keyType == storageAzureKeyTypeAuto || key != NULL); ASSERT(blockSize != 0); OBJ_NEW_BEGIN(StorageAzure, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageAzure) { .interface = storageInterfaceAzure, .container = strDup(container), .account = strDup(account), .blockSize = blockSize, .host = uriStyle == storageAzureUriStyleHost ? strNewFmt("%s.%s", strZ(account), strZ(endpoint)) : strDup(endpoint), .pathPrefix = uriStyle == storageAzureUriStyleHost ? strNewFmt("/%s", strZ(container)) : strNewFmt("/%s/%s", strZ(account), strZ(container)), .deleteMax = STORAGE_AZURE_DELETE_MAX, .keyType = keyType, }; // Set prefetch and read over this->interface.prefetch = prefetch; this->interface.readOver = readOver; // Create tag query string if (tag != NULL) { HttpQuery *const query = httpQueryNewP(.kv = tag); this->tag = httpQueryRenderP(query); httpQueryFree(query); } // Initialization by key type switch (keyType) { // Create authentication client case storageAzureKeyTypeAuto: this->accessToken = strNew(); this->credHost = AZURE_CREDENTIAL_HOST_STR; this->credHttpClient = httpClientNew( sckClientNew(this->credHost, AZURE_CREDENTIAL_PORT, timeout, timeout), timeout); break; // Store shared key case storageAzureKeyTypeShared: this->sharedKey = bufNewDecode(encodingBase64, key); break; // Parse sas query case storageAzureKeyTypeSas: this->sasKey = httpQueryNewStr(key); break; } // Create the http client used to service requests. Use plain socket for HTTP, TLS for HTTPS. IoClient *ioClient; if (protocolType == httpProtocolTypeHttp) ioClient = sckClientNew(this->host, port, timeout, timeout); else { ioClient = tlsClientNewP( sckClientNew(this->host, port, timeout, timeout), this->host, timeout, timeout, verifyPeer, .caFile = caFile, .caPath = caPath); } this->httpClient = httpClientNew(ioClient, timeout); // Create list of redacted headers this->headerRedactList = strLstNew(); strLstAdd(this->headerRedactList, HTTP_HEADER_AUTHORIZATION_STR); strLstAdd(this->headerRedactList, HTTP_HEADER_DATE_STR); // Create list of redacted query keys this->queryRedactList = strLstNew(); strLstAdd(this->queryRedactList, AZURE_QUERY_SIG_STR); // Generate starting file id cryptoRandomBytes((uint8_t *)&this->fileId, sizeof(this->fileId)); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( STORAGE, storageNew(STORAGE_AZURE_TYPE, path, 0, 0, write, targetTime, pathExpressionFunction, this, this->interface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/azure/storage.h0000664000175100017510000000450215265101745020601 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage ***********************************************************************************************************************************/ #ifndef STORAGE_AZURE_STORAGE_H #define STORAGE_AZURE_STORAGE_H #include "common/io/http/url.h" #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_AZURE_TYPE STRID5("azure", 0x5957410) /*********************************************************************************************************************************** Key type ***********************************************************************************************************************************/ typedef enum { storageAzureKeyTypeShared, storageAzureKeyTypeSas, storageAzureKeyTypeAuto, } StorageAzureKeyType; /*********************************************************************************************************************************** URI style ***********************************************************************************************************************************/ typedef enum { storageAzureUriStyleHost, storageAzureUriStylePath, } StorageAzureUriStyle; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Storage *storageAzureNew( const String *path, bool write, time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, const String *container, const String *account, StorageAzureKeyType keyType, const String *key, size_t blockSize, const KeyValue *tag, const String *endpoint, StorageAzureUriStyle uriStyle, unsigned int port, TimeMSec timeout, HttpProtocolType protocolType, bool verifyPeer, const String *caFile, const String *caPath, unsigned int prefetch, uint64_t readOver); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/azure/storage.intern.h0000664000175100017510000001327015265101745022101 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage Internal ***********************************************************************************************************************************/ #ifndef STORAGE_AZURE_STORAGE_INTERN_H #define STORAGE_AZURE_STORAGE_INTERN_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageAzure StorageAzure; #include "common/io/http/request.h" #include "storage/azure/storage.h" /*********************************************************************************************************************************** Azure query tokens ***********************************************************************************************************************************/ #define AZURE_QUERY_COMP "comp" STRING_DECLARE(AZURE_QUERY_COMP_STR); #define AZURE_QUERY_RESTYPE "restype" STRING_DECLARE(AZURE_QUERY_RESTYPE_STR); #define AZURE_QUERY_VALUE_CONTAINER "container" STRING_DECLARE(AZURE_QUERY_VALUE_CONTAINER_STR); /*********************************************************************************************************************************** Multi-Part request data ***********************************************************************************************************************************/ typedef struct StorageAzureRequestPart { const String *path; // Request path const String *verb; // Verb (GET, PUT, etc) } StorageAzureRequestPart; /*********************************************************************************************************************************** Perform an Azure Request ***********************************************************************************************************************************/ // Perform async request typedef struct StorageAzureRequestAsyncParam { VAR_PARAM_HEADER; const String *path; // Request path const HttpHeader *header; // Request headers const HttpQuery *query; // Query parameters const Buffer *content; // Request content const List *contentList; // Request content part list bool tag; // Add tags when available? } StorageAzureRequestAsyncParam; #define storageAzureRequestAsyncP(this, verb, ...) \ storageAzureRequestAsync(this, verb, (StorageAzureRequestAsyncParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpRequest *storageAzureRequestAsync(StorageAzure *this, const String *verb, StorageAzureRequestAsyncParam param); // Get async response typedef struct StorageAzureResponseParam { VAR_PARAM_HEADER; bool allowMissing; // Allow missing files (caller can check response code) bool contentIo; // Is IoRead interface required to read content? } StorageAzureResponseParam; #define storageAzureResponseP(request, ...) \ storageAzureResponse(request, (StorageAzureResponseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpResponse *storageAzureResponse(HttpRequest *request, StorageAzureResponseParam param); typedef struct StorageAzureRequestParam { VAR_PARAM_HEADER; const String *path; // Request path const HttpHeader *header; // Request headers const HttpQuery *query; // Query parameters const Buffer *content; // Request content const List *contentList; // Request content part list bool allowMissing; // Allow missing files (caller can check response code) bool contentIo; // Is IoRead interface required to read content? bool tag; // Add tags when available? } StorageAzureRequestParam; #define storageAzureRequestP(this, verb, ...) \ storageAzureRequest(this, verb, (StorageAzureRequestParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpResponse *storageAzureRequest(StorageAzure *this, const String *verb, StorageAzureRequestParam param); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_AZURE_TYPE \ StorageAzure * #define FUNCTION_LOG_STORAGE_AZURE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageAzure", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/azure/write.c0000664000175100017510000002700415265101745020264 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage File Write ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/xml.h" #include "storage/azure/write.h" #include "storage/write.h" /*********************************************************************************************************************************** Block defaults based on limits at https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob ***********************************************************************************************************************************/ #define STORAGE_AZURE_SPLIT_DEFAULT 257 #define STORAGE_AZURE_SPLIT_MAX 9505 /*********************************************************************************************************************************** Azure HTTP headers ***********************************************************************************************************************************/ STRING_STATIC(AZURE_HEADER_BLOB_TYPE_STR, "x-ms-blob-type"); STRING_STATIC(AZURE_HEADER_VALUE_BLOCK_BLOB_STR, "BlockBlob"); /*********************************************************************************************************************************** Azure query tokens ***********************************************************************************************************************************/ STRING_STATIC(AZURE_QUERY_BLOCK_ID_STR, "blockid"); STRING_STATIC(AZURE_QUERY_VALUE_BLOCK_STR, "block"); STRING_STATIC(AZURE_QUERY_VALUE_BLOCK_LIST_STR, "blocklist"); /*********************************************************************************************************************************** XML tags ***********************************************************************************************************************************/ STRING_STATIC(AZURE_XML_TAG_BLOCK_LIST_STR, "BlockList"); STRING_STATIC(AZURE_XML_TAG_UNCOMMITTED_STR, "Uncommitted"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWriteAzure { const StorageWriteInterface *interface; // Interface StorageAzure *storage; // Storage that created this object const String *name; // File name HttpRequest *request; // Async block upload request uint64_t fileId; // Id to used to make file block identifiers unique size_t blockSize; // Size of blocks for multi-block upload Buffer *blockBuffer; // Block buffer (stores data until blockSize is reached) StringList *blockIdList; // List of uploaded block ids }; /*********************************************************************************************************************************** Flush bytes to upload block ***********************************************************************************************************************************/ static void storageWriteAzureBlock(StorageWriteAzure *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_AZURE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // If there is an outstanding async request then wait for the response. Since the part id has already been stored there is // nothing to do except make sure the request did not error. if (this->request != NULL) { httpResponseFree(storageAzureResponseP(this->request)); httpRequestFree(this->request); this->request = NULL; } FUNCTION_LOG_RETURN_VOID(); } static void storageWriteAzureBlockAsync(StorageWriteAzure *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_AZURE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->blockBuffer != NULL); ASSERT(bufSize(this->blockBuffer) > 0); MEM_CONTEXT_TEMP_BEGIN() { // Complete prior async request, if any storageWriteAzureBlock(this); // Create the block id list if (this->blockIdList == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->blockIdList = strLstNew(); } MEM_CONTEXT_OBJ_END(); } // Generate block id. Combine the block number with the provided file id to create a (hopefully) unique block id that won't // overlap with any other process. This is to prevent another process from overwriting our blocks. If two processes are // writing against the same file then there may be problems anyway but we need to at least ensure the result is consistent, // i.e. we get all of one file or all of the other depending on who writes last. const String *const blockId = strNewFmt("%016" PRIX64 "x%07u", this->fileId, strLstSize(this->blockIdList)); // Upload the block and add to block list HttpQuery *const query = httpQueryNewP(); httpQueryAdd(query, AZURE_QUERY_COMP_STR, AZURE_QUERY_VALUE_BLOCK_STR); httpQueryAdd(query, AZURE_QUERY_BLOCK_ID_STR, blockId); MEM_CONTEXT_OBJ_BEGIN(this) { this->request = storageAzureRequestAsyncP( this->storage, HTTP_VERB_PUT_STR, .path = this->name, .query = query, .content = this->blockBuffer); } MEM_CONTEXT_OBJ_END(); strLstAdd(this->blockIdList, blockId); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to internal buffer ***********************************************************************************************************************************/ static void storageWriteAzure(THIS_VOID, const Buffer *const buffer) { THIS(StorageWriteAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_AZURE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); // Continue until the write buffer has been exhausted size_t bytesTotal = 0; do { // Resize block buffer. This must be done in the loop because block size grows as blocks are written. storageWriteChunkBufferResize(buffer, bytesTotal, this->blockBuffer, this->blockSize); // Copy as many bytes as possible into the block buffer const size_t bytesNext = bufRemains(this->blockBuffer) > bufUsed(buffer) - bytesTotal ? bufUsed(buffer) - bytesTotal : bufRemains(this->blockBuffer); bufCatSub(this->blockBuffer, buffer, bytesTotal, bytesNext); bytesTotal += bytesNext; // If the block buffer is full then write it if (bufUsed(this->blockBuffer) == this->blockSize) { storageWriteAzureBlockAsync(this); bufUsedZero(this->blockBuffer); this->blockSize = storageWriteChunkSize( this->blockSize, STORAGE_AZURE_SPLIT_DEFAULT, STORAGE_AZURE_SPLIT_MAX, strLstSize(this->blockIdList)); } } while (bytesTotal != bufUsed(buffer)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWriteAzureClose(THIS_VOID) { THIS(StorageWriteAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_AZURE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close if the file has not already been closed if (this->blockBuffer != NULL) { MEM_CONTEXT_TEMP_BEGIN() { // If a multi-block upload was started we need to finish that way if (this->blockIdList != NULL) { // If there is anything left in the block buffer then write it if (!bufEmpty(this->blockBuffer)) storageWriteAzureBlockAsync(this); // Complete prior async request, if any storageWriteAzureBlock(this); // Generate the xml block list XmlDocument *const blockXml = xmlDocumentNew(AZURE_XML_TAG_BLOCK_LIST_STR); for (unsigned int blockIdx = 0; blockIdx < strLstSize(this->blockIdList); blockIdx++) { xmlNodeContentSet( xmlNodeAdd(xmlDocumentRoot(blockXml), AZURE_XML_TAG_UNCOMMITTED_STR), strLstGet(this->blockIdList, blockIdx)); } // Finalize the multi-block upload storageAzureRequestP( this->storage, HTTP_VERB_PUT_STR, .path = this->name, .query = httpQueryAdd(httpQueryNewP(), AZURE_QUERY_COMP_STR, AZURE_QUERY_VALUE_BLOCK_LIST_STR), .content = xmlDocumentBuf(blockXml), .tag = true); } // Else upload all the data in a single block else { storageAzureRequestP( this->storage, HTTP_VERB_PUT_STR, .path = this->name, httpHeaderAdd(httpHeaderNew(NULL), AZURE_HEADER_BLOB_TYPE_STR, AZURE_HEADER_VALUE_BLOCK_BLOB_STR), .content = this->blockBuffer, .tag = true); } bufFree(this->blockBuffer); this->blockBuffer = NULL; } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageWriteInterface storageWriteAzureInterface = { .close = storageWriteAzureClose, .write = storageWriteAzure, }; FN_EXTERN StorageWriteAzure * storageWriteAzureNew(StorageAzure *const storage, const String *const name, const uint64_t fileId, const size_t blockSize) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_AZURE, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, fileId); FUNCTION_LOG_PARAM(UINT64, blockSize); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageWriteAzure, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageWriteAzure) { .interface = &storageWriteAzureInterface, .storage = storage, .name = strDup(name), .fileId = fileId, .blockSize = blockSize, .blockBuffer = bufNew(0), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE_AZURE, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/azure/write.h0000664000175100017510000000343315265101745020271 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage File Write ***********************************************************************************************************************************/ #ifndef STORAGE_AZURE_WRITE_H #define STORAGE_AZURE_WRITE_H #include "storage/azure/storage.intern.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWriteAzure StorageWriteAzure; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWriteAzure *storageWriteAzureNew(StorageAzure *storage, const String *name, uint64_t fileId, size_t blockSize); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_WRITE_AZURE_TYPE \ StorageWriteAzure * #define FUNCTION_LOG_STORAGE_WRITE_AZURE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageWriteAzure", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/cifs/0000775000175100017510000000000015265101745016561 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/cifs/helper.c0000664000175100017510000000227115265101745020206 0ustar00runnerrunner/*********************************************************************************************************************************** CIFS Storage Helper ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "config/config.h" #include "storage/cifs/helper.h" /**********************************************************************************************************************************/ FN_EXTERN Storage * storageCifsHelper(const unsigned int repoIdx, const bool write, StoragePathExpressionCallback pathExpressionCallback) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM_P(VOID, pathExpressionCallback); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxStrId(cfgOptRepoType, repoIdx) == STORAGE_CIFS_TYPE); Storage *const result = storageCifsNew( cfgOptionIdxStr(cfgOptRepoPath, repoIdx), STORAGE_MODE_FILE_DEFAULT, STORAGE_MODE_PATH_DEFAULT, write, pathExpressionCallback); FUNCTION_LOG_RETURN(STORAGE, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/cifs/helper.h0000664000175100017510000000236515265101745020217 0ustar00runnerrunner/*********************************************************************************************************************************** CIFS Storage Helper ***********************************************************************************************************************************/ #ifndef STORAGE_CIFS_STORAGE_HELPER_H #define STORAGE_CIFS_STORAGE_HELPER_H #include "storage/cifs/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN Storage *storageCifsHelper(unsigned int repoIdx, bool write, StoragePathExpressionCallback pathExpressionCallback); /*********************************************************************************************************************************** Storage helper for StorageHelper array passed to storageHelperInit() ***********************************************************************************************************************************/ #define STORAGE_CIFS_HELPER {.type = STORAGE_CIFS_TYPE, .helper = storageCifsHelper} #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/cifs/storage.c0000664000175100017510000000232215265101745020370 0ustar00runnerrunner/*********************************************************************************************************************************** CIFS Storage ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/regExp.h" #include "storage/cifs/storage.h" #include "storage/posix/storage.intern.h" /**********************************************************************************************************************************/ FN_EXTERN Storage * storageCifsNew( const String *const path, const mode_t modeFile, const mode_t modePath, const bool write, StoragePathExpressionCallback pathExpressionFunction) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN( STORAGE, storagePosixNewInternal(STORAGE_CIFS_TYPE, path, modeFile, modePath, write, pathExpressionFunction, false, false)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/cifs/storage.h0000664000175100017510000000225015265101745020375 0ustar00runnerrunner/*********************************************************************************************************************************** CIFS Storage ***********************************************************************************************************************************/ #ifndef STORAGE_CIFS_STORAGE_H #define STORAGE_CIFS_STORAGE_H #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_CIFS_TYPE STRID5("cifs", 0x999230) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Storage *storageCifsNew( const String *path, mode_t modeFile, mode_t modePath, bool write, StoragePathExpressionCallback pathExpressionFunction); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/gcs/0000775000175100017510000000000015265101745016411 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/gcs/helper.c0000664000175100017510000000363415265101745020042 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage Helper ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "config/config.h" #include "storage/gcs/helper.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN Storage * storageGcsHelper(const unsigned int repoIdx, const bool write, StoragePathExpressionCallback pathExpressionCallback) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM_P(VOID, pathExpressionCallback); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxStrId(cfgOptRepoType, repoIdx) == STORAGE_GCS_TYPE); Storage *const result = storageGcsNew( cfgOptionIdxStr(cfgOptRepoPath, repoIdx), write, storageRepoTargetTime(), pathExpressionCallback, cfgOptionIdxStr(cfgOptRepoGcsBucket, repoIdx), (StorageGcsKeyType)cfgOptionIdxSeq(cfgOptRepoGcsKeyType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoGcsKey, repoIdx), (size_t)cfgOptionIdxUInt64(cfgOptRepoStorageUploadChunkSize, repoIdx), cfgOptionIdxKvNull(cfgOptRepoStorageTag, repoIdx), cfgOptionIdxStr(cfgOptRepoGcsEndpoint, repoIdx), ioTimeoutMs(), cfgOptionIdxBool(cfgOptRepoStorageVerifyTls, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaFile, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaPath, repoIdx), cfgOptionIdxStrNull(cfgOptRepoGcsUserProject, repoIdx), cfgOptionIdxUInt(cfgOptRepoStoragePrefetch, repoIdx), cfgOptionIdxUInt64(cfgOptRepoStorageReadOver, repoIdx)); FUNCTION_LOG_RETURN(STORAGE, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/gcs/helper.h0000664000175100017510000000235615265101745020047 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage Helper ***********************************************************************************************************************************/ #ifndef STORAGE_GCS_STORAGE_HELPER_H #define STORAGE_GCS_STORAGE_HELPER_H #include "storage/gcs/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN Storage *storageGcsHelper(unsigned int repoIdx, bool write, StoragePathExpressionCallback pathExpressionCallback); /*********************************************************************************************************************************** Storage helper for StorageHelper array passed to storageHelperInit() ***********************************************************************************************************************************/ #define STORAGE_GCS_HELPER {.type = STORAGE_GCS_TYPE, .helper = storageGcsHelper} #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/gcs/read.c0000664000175100017510000001612015265101745017470 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage Read ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/client.h" #include "common/io/read.h" #include "common/log.h" #include "common/type/object.h" #include "storage/gcs/read.h" #include "storage/read.h" /*********************************************************************************************************************************** GCS query tokens ***********************************************************************************************************************************/ STRING_STATIC(GCS_QUERY_ALT_STR, "alt"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageReadGcs { const StorageReadInterface *interface; // Interface StorageGcs *storage; // Storage that created this object const String *name; // File name uint64_t offset; // Read offset const Variant *limit; // Read limit (NULL for no limit) const String *versionId; // Version id (NULL for most recent) HttpRequest *httpRequest; // HTTP request HttpResponse *httpResponse; // HTTP response }; /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadGcsOpenAsync(THIS_VOID) { THIS(StorageReadGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_GCS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpRequest != NULL); // Wait for response MEM_CONTEXT_OBJ_BEGIN(this) { this->httpResponse = storageGcsResponseP(this->httpRequest, .allowMissing = true, .contentIo = true); httpRequestFree(this->httpRequest); this->httpRequest = NULL; } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN(BOOL, httpResponseCodeOk(this->httpResponse)); } static bool storageReadGcsOpen(THIS_VOID) { THIS(StorageReadGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_GCS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpResponse == NULL); // Request the file MEM_CONTEXT_OBJ_BEGIN(this) { HttpQuery *const query = httpQueryAdd(httpQueryNewP(), GCS_QUERY_ALT_STR, GCS_QUERY_MEDIA_STR); if (this->versionId != NULL) httpQueryAdd(query, varStr(GCS_JSON_GENERATION_VAR), this->versionId); this->httpRequest = storageGcsRequestAsyncP( this->storage, HTTP_VERB_GET_STR, .object = this->name, .header = httpHeaderPutRange(httpHeaderNew(NULL), this->offset, this->limit), .query = query); } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN(BOOL, true); } /*********************************************************************************************************************************** Read from a file ***********************************************************************************************************************************/ static size_t storageReadGcs(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_GCS, this); FUNCTION_LOG_PARAM(BUFFER, buffer); (void)block; // Blocking is handled by the caller FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpResponse != NULL); ASSERT(buffer != NULL && !bufFull(buffer)); ASSERT(httpResponseIoRead(this->httpResponse) != NULL); FUNCTION_LOG_RETURN(SIZE, ioRead(httpResponseIoRead(this->httpResponse), buffer)); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadGcsEof(THIS_VOID) { THIS(StorageReadGcs); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_GCS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->httpResponse != NULL); ASSERT(httpResponseIoRead(this->httpResponse) != NULL); FUNCTION_TEST_RETURN(BOOL, ioReadEof(httpResponseIoRead(this->httpResponse))); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageReadGcsClose(THIS_VOID) { THIS(StorageReadGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_GCS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); httpRequestFree(this->httpRequest); this->httpRequest = NULL; httpResponseFree(this->httpResponse); this->httpResponse = NULL; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageReadInterface storageReadGcsInterface = { .close = storageReadGcsClose, .eof = storageReadGcsEof, .open = storageReadGcsOpen, .openAsync = storageReadGcsOpenAsync, .read = storageReadGcs, }; FN_EXTERN StorageReadGcs * storageReadGcsNew( StorageGcs *const storage, const String *const name, const uint64_t offset, const Variant *const limit, const String *const versionId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_GCS, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_PARAM(STRING, versionId); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageReadGcs, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageReadGcs) { .interface = &storageReadGcsInterface, .storage = storage, .name = strDup(name), .offset = offset, .limit = varDup(limit), .versionId = strDup(versionId), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_GCS, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/gcs/read.h0000664000175100017510000000343015265101745017475 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage Read ***********************************************************************************************************************************/ #ifndef STORAGE_GCS_READ_H #define STORAGE_GCS_READ_H #include "storage/gcs/storage.intern.h" #include "storage/read.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadGcs StorageReadGcs; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadGcs *storageReadGcsNew( StorageGcs *storage, const String *name, uint64_t offset, const Variant *limit, const String *versionId); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_GCS_TYPE \ StorageReadGcs * #define FUNCTION_LOG_STORAGE_READ_GCS_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadGcs", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/gcs/storage.c0000664000175100017510000017720615265101745020236 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/crypto/common.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/http/client.h" #include "common/io/http/common.h" #include "common/io/http/url.h" #include "common/io/socket/client.h" #include "common/io/tls/client.h" #include "common/log.h" #include "common/regExp.h" #include "common/stat.h" #include "common/type/json.h" #include "common/type/object.h" #include "storage/gcs/read.h" #include "storage/gcs/write.h" #include "storage/posix/storage.h" /*********************************************************************************************************************************** Defaults ***********************************************************************************************************************************/ #define STORAGE_GCS_DELETE_MAX 100 /*********************************************************************************************************************************** HTTP headers ***********************************************************************************************************************************/ STRING_EXTERN(GCS_HEADER_UPLOAD_ID_STR, GCS_HEADER_UPLOAD_ID); STRING_STATIC(GCS_HEADER_METADATA_FLAVOR_STR, "metadata-flavor"); STRING_STATIC(GCS_HEADER_GOOGLE_STR, "Google"); /*********************************************************************************************************************************** Query tokens ***********************************************************************************************************************************/ STRING_STATIC(GCS_QUERY_DELIMITER_STR, "delimiter"); STRING_EXTERN(GCS_QUERY_FIELDS_STR, GCS_QUERY_FIELDS); STRING_EXTERN(GCS_QUERY_MEDIA_STR, GCS_QUERY_MEDIA); STRING_EXTERN(GCS_QUERY_NAME_STR, GCS_QUERY_NAME); STRING_STATIC(GCS_QUERY_PAGE_TOKEN_STR, "pageToken"); STRING_STATIC(GCS_QUERY_PREFIX_STR, "prefix"); STRING_EXTERN(GCS_QUERY_UPLOAD_ID_STR, GCS_QUERY_UPLOAD_ID); STRING_STATIC(GCS_QUERY_VERSIONS_STR, "versions"); STRING_STATIC(GCS_QUERY_USER_PROJECT_STR, "userProject"); /*********************************************************************************************************************************** JSON tokens ***********************************************************************************************************************************/ VARIANT_STRDEF_STATIC(GCS_JSON_ACCESS_TOKEN_VAR, "access_token"); VARIANT_STRDEF_STATIC(GCS_JSON_AUDIENCE_VAR, "audience"); VARIANT_STRDEF_STATIC(GCS_JSON_CLIENT_EMAIL_VAR, "client_email"); VARIANT_STRDEF_STATIC(GCS_JSON_CREDENTIAL_SOURCE_VAR, "credential_source"); VARIANT_STRDEF_STATIC(GCS_JSON_ERROR_VAR, "error"); VARIANT_STRDEF_STATIC(GCS_JSON_ERROR_DESCRIPTION_VAR, "error_description"); VARIANT_STRDEF_STATIC(GCS_JSON_EXPIRES_IN_VAR, "expires_in"); VARIANT_STRDEF_STATIC(GCS_JSON_FILE_VAR, "file"); VARIANT_STRDEF_STATIC(GCS_JSON_FORMAT_VAR, "format"); #define GCS_JSON_ITEMS "items" VARIANT_STRDEF_STATIC(GCS_JSON_ITEMS_VAR, GCS_JSON_ITEMS); VARIANT_STRDEF_EXTERN(GCS_JSON_GENERATION_VAR, GCS_JSON_GENERATION); VARIANT_STRDEF_EXTERN(GCS_JSON_MD5_HASH_VAR, GCS_JSON_MD5_HASH); VARIANT_STRDEF_EXTERN(GCS_JSON_NAME_VAR, GCS_JSON_NAME); #define GCS_JSON_NEXT_PAGE_TOKEN "nextPageToken" VARIANT_STRDEF_STATIC(GCS_JSON_NEXT_PAGE_TOKEN_VAR, GCS_JSON_NEXT_PAGE_TOKEN); #define GCS_JSON_PREFIXES "prefixes" VARIANT_STRDEF_STATIC(GCS_JSON_PREFIXES_VAR, GCS_JSON_PREFIXES); VARIANT_STRDEF_STATIC(GCS_JSON_PRIVATE_KEY_VAR, "private_key"); VARIANT_STRDEF_STATIC(GCS_JSON_SA_IMPERSONATION_URL_VAR, "service_account_impersonation_url"); VARIANT_STRDEF_EXTERN(GCS_JSON_SIZE_VAR, GCS_JSON_SIZE); VARIANT_STRDEF_STATIC(GCS_JSON_TOKEN_TYPE_VAR, "token_type"); VARIANT_STRDEF_STATIC(GCS_JSON_TOKEN_URI_VAR, "token_uri"); VARIANT_STRDEF_STATIC(GCS_JSON_TOKEN_URL_VAR, "token_url"); VARIANT_STRDEF_STATIC(GCS_JSON_TYPE_VAR, "type"); #define GCS_JSON_UPDATED "updated" VARIANT_STRDEF_STATIC(GCS_JSON_UPDATED_VAR, GCS_JSON_UPDATED); /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_STATIC(GCS_STAT_REMOVE_STR, "gcs.rm"); STRING_STATIC(GCS_STAT_REMOVE_BATCH_STR, "gcs.rm.batch"); STRING_STATIC(GCS_STAT_REMOVE_BATCH_PART_STR, "gcs.rm.batch.part"); STRING_STATIC(GCS_STAT_REMOVE_BATCH_RETRY_STR, "gcs.rm.batch.retry"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageGcs { STORAGE_COMMON_MEMBER; HttpClient *httpClient; // Http client to service requests StringList *headerRedactList; // List of headers to redact from logging StringList *queryRedactList; // List of query keys to redact from logging bool write; // Storage is writable const String *bucket; // Bucket to store data in const String *endpoint; // Endpoint size_t chunkSize; // Block size for resumable upload unsigned int deleteMax; // Maximum objects that can be deleted in one request const Buffer *tag; // Tags to be applied to objects const String *userProject; // Project ID StorageGcsKeyType keyType; // Auth key type const String *key; // Key (value depends on key type) const String *webIdTokenFile; // Web identity token file const String *webIdAudience; // Web identity provider audience String *token; // Token time_t tokenTimeExpire; // Token expiration time (if renewable auth) HttpUrl *authUrl; // URL for authentication server HttpClient *authClient; // Client to service auth requests }; /*********************************************************************************************************************************** Parse HTTP JSON response containing an authentication token and expiration Note that the function is intended to run directly in the caller's mem context and results will be placed in the caller's prior mem context. ***********************************************************************************************************************************/ typedef struct { String *tokenType; String *token; time_t timeExpire; } StorageGcsAuthTokenResult; typedef StorageGcsAuthTokenResult (*StorageGcsAuthFunction)(StorageGcs *this, time_t timeBegin); static StorageGcsAuthTokenResult storageGcsAuthToken(HttpRequest *const request, const time_t timeBegin) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_REQUEST, request); FUNCTION_TEST_PARAM(TIME, timeBegin); FUNCTION_TEST_END(); FUNCTION_AUDIT_STRUCT(); StorageGcsAuthTokenResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { // Get the response HttpResponse *const response = httpRequestResponse(request, true); const String *const contentType = httpHeaderGet(httpResponseHeader(response), HTTP_HEADER_CONTENT_TYPE_STR); // Error when the response is not OK and is not JSON, e.g. an HTML error page returned by a misconfigured authentication // server. Otherwise the JSON error fields checked below provide a more detailed report of the error. if (!httpResponseCodeOk(response) && (contentType == NULL || !strBeginsWithZ(contentType, "application/json"))) httpRequestError(request, response); const Variant *const responseVariant = jsonToVar(strNewBuf(httpResponseContent(response))); CHECK( FormatError, responseVariant != NULL && varType(responseVariant) == varTypeKeyValue, "token response must be an object"); const KeyValue *const kvResponse = varKv(responseVariant); // Check for an error const Variant *const error = kvGet(kvResponse, GCS_JSON_ERROR_VAR); if (error != NULL) { // Report flat OAuth-style errors, e.g. {"error":"invalid_grant","error_description":"..."} if (varType(error) == varTypeString) { const Variant *const description = kvGet(kvResponse, GCS_JSON_ERROR_DESCRIPTION_VAR); THROW_FMT( ProtocolError, "unable to get authentication token: [%s] %s", strZ(varStr(error)), description != NULL && varType(description) == varTypeString ? strZ(varStr(description)) : "null"); } // Some errors are nested objects, e.g. {"error":{"code":403,...}}, so report the entire response httpRequestError(request, response); } MEM_CONTEXT_PRIOR_BEGIN() { // Get token result.tokenType = strDup(varStr(kvGet(kvResponse, GCS_JSON_TOKEN_TYPE_VAR))); CHECK(FormatError, result.tokenType != NULL, "token type missing"); result.token = strDup(varStr(kvGet(kvResponse, GCS_JSON_ACCESS_TOKEN_VAR))); CHECK(FormatError, result.token != NULL, "access token missing"); // Get expiration const Variant *const expiresIn = kvGet(kvResponse, GCS_JSON_EXPIRES_IN_VAR); CHECK(FormatError, expiresIn != NULL, "expiry missing"); result.timeExpire = timeBegin + (time_t)varInt64Force(expiresIn); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_TYPE(StorageGcsAuthTokenResult, result); } /*********************************************************************************************************************************** Get authentication header for service keys Based on the documentation at https://developers.google.com/identity/protocols/oauth2/service-account#httprest ***********************************************************************************************************************************/ // Helper to construct a JSON Web Token static String * storageGcsAuthJwt(StorageGcs *const this, const time_t timeBegin) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_GCS, this); FUNCTION_TEST_PARAM(TIME, timeBegin); FUNCTION_TEST_END(); // Static header with dot delimiter String *const result = strCatZ(strNew(), "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9."); MEM_CONTEXT_TEMP_BEGIN() { // Load client email and private key const KeyValue *const kvKey = varKv( jsonToVar(strNewBuf(storageGetP(storageNewReadP(storagePosixNewP(FSLASH_STR), this->key))))); const String *const clientEmail = varStr(kvGet(kvKey, GCS_JSON_CLIENT_EMAIL_VAR)); const String *const privateKeyRaw = varStr(kvGet(kvKey, GCS_JSON_PRIVATE_KEY_VAR)); CHECK(FormatError, clientEmail != NULL && privateKeyRaw != NULL, "credentials missing"); // Add claim strCatEncode( result, encodingBase64Url, BUFSTR( strNewFmt( "{\"iss\":\"%s\",\"scope\":\"https://www.googleapis.com/auth/devstorage.read%s\",\"aud\":\"%s\"" ",\"exp\":%" PRIu64 ",\"iat\":%" PRIu64 "}", strZ(clientEmail), this->write ? "_write" : "_only", strZ(httpUrl(this->authUrl)), (uint64_t)timeBegin + 3600, (uint64_t)timeBegin))); // Sign with RSA key BIO *volatile bio = NULL; EVP_PKEY *volatile privateKey = NULL; EVP_MD_CTX *volatile sign = NULL; cryptoInit(); TRY_BEGIN() { // Load key bio = BIO_new(BIO_s_mem()); BIO_write(bio, strZ(privateKeyRaw), (int)strSize(privateKeyRaw)); privateKey = PEM_read_bio_PrivateKey(bio, NULL, NULL, NULL); cryptoError(privateKey == NULL, "unable to read PEM"); // Create signature sign = EVP_MD_CTX_new(); cryptoError(EVP_DigestSignInit(sign, NULL, EVP_sha256(), NULL, privateKey) <= 0, "unable to init"); cryptoError( EVP_DigestSignUpdate(sign, (const uint8_t *)strZ(result), (unsigned int)strSize(result)) <= 0, "unable to update"); size_t signatureLen = 0; cryptoError(EVP_DigestSignFinal(sign, NULL, &signatureLen) <= 0, "unable to get size"); Buffer *const signature = bufNew(signatureLen); bufUsedSet(signature, bufSize(signature)); cryptoError(EVP_DigestSignFinal(sign, bufPtr(signature), &signatureLen) <= 0, "unable to finalize"); // Add dot delimiter and signature strCatChr(result, '.'); strCatEncode(result, encodingBase64Url, signature); } FINALLY() { BIO_free(bio); EVP_MD_CTX_free(sign); EVP_PKEY_free(privateKey); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } static StorageGcsAuthTokenResult storageGcsAuthService(StorageGcs *const this, const time_t timeBegin) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_GCS, this); FUNCTION_TEST_PARAM(TIME, timeBegin); FUNCTION_TEST_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); ASSERT(timeBegin > 0); StorageGcsAuthTokenResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { const String *const content = strNewFmt( "grant_type=urn%%3Aietf%%3Aparams%%3Aoauth%%3Agrant-type%%3Ajwt-bearer&assertion=%s", strZ(storageGcsAuthJwt(this, timeBegin))); HttpHeader *const header = httpHeaderNew(NULL); httpHeaderAdd(header, HTTP_HEADER_HOST_STR, httpUrlHost(this->authUrl)); httpHeaderAdd(header, HTTP_HEADER_CONTENT_TYPE_STR, HTTP_HEADER_CONTENT_TYPE_APP_FORM_URL_STR); httpHeaderAdd(header, HTTP_HEADER_CONTENT_LENGTH_STR, strNewFmt("%zu", strSize(content))); HttpRequest *const request = httpRequestNewP( this->authClient, HTTP_VERB_POST_STR, httpUrlPath(this->authUrl), NULL, .header = header, .content = BUFSTR(content)); MEM_CONTEXT_PRIOR_BEGIN() { result = storageGcsAuthToken(request, timeBegin); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_TYPE(StorageGcsAuthTokenResult, result); } /*********************************************************************************************************************************** Exchange a web identity token for an access token using the STS token exchange API (https://docs.cloud.google.com/iam/docs/reference/sts/rest/v1/TopLevel/token). The resulting token authenticates the federated principal directly, https://docs.cloud.google.com/iam/docs/workload-identity-federation#direct-resource-access ***********************************************************************************************************************************/ static StorageGcsAuthTokenResult storageGcsAuthWebId(StorageGcs *const this, const time_t timeBegin) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_GCS, this); FUNCTION_TEST_PARAM(TIME, timeBegin); FUNCTION_TEST_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); ASSERT(timeBegin > 0); StorageGcsAuthTokenResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { // Reload on every exchange because the projected token may rotate const String *const subjectToken = strTrim(strNewBuf(storageGetP(storageNewReadP(storagePosixNewP(FSLASH_STR), this->webIdTokenFile)))); CHECK(FormatError, !strEmpty(subjectToken), "web identity token is empty"); // Build form content HttpQuery *const contentQuery = httpQueryNewP(); httpQueryAdd(contentQuery, STRDEF("audience"), this->webIdAudience); httpQueryAdd(contentQuery, STRDEF("grant_type"), STRDEF("urn:ietf:params:oauth:grant-type:token-exchange")); httpQueryAdd(contentQuery, STRDEF("requested_token_type"), STRDEF("urn:ietf:params:oauth:token-type:access_token")); httpQueryAdd( contentQuery, STRDEF("scope"), strNewFmt("https://www.googleapis.com/auth/devstorage.read%s", this->write ? "_write" : "_only")); httpQueryAdd(contentQuery, STRDEF("subject_token"), subjectToken); httpQueryAdd(contentQuery, STRDEF("subject_token_type"), STRDEF("urn:ietf:params:oauth:token-type:jwt")); String *const content = httpQueryRenderP(contentQuery); HttpHeader *const header = httpHeaderNew(NULL); httpHeaderAdd(header, HTTP_HEADER_HOST_STR, httpUrlHost(this->authUrl)); httpHeaderAdd(header, HTTP_HEADER_CONTENT_TYPE_STR, HTTP_HEADER_CONTENT_TYPE_APP_FORM_URL_STR); httpHeaderAdd(header, HTTP_HEADER_CONTENT_LENGTH_STR, strNewFmt("%zu", strSize(content))); HttpRequest *const request = httpRequestNewP( this->authClient, HTTP_VERB_POST_STR, httpUrlPath(this->authUrl), .header = header, .content = BUFSTR(content)); MEM_CONTEXT_PRIOR_BEGIN() { result = storageGcsAuthToken(request, timeBegin); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_TYPE(StorageGcsAuthTokenResult, result); } /*********************************************************************************************************************************** Get authentication token automatically for instances running in GCE. Based on the documentation at https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#applications ***********************************************************************************************************************************/ static StorageGcsAuthTokenResult storageGcsAuthAuto(StorageGcs *const this, const time_t timeBegin) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_GCS, this); FUNCTION_TEST_PARAM(TIME, timeBegin); FUNCTION_TEST_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); ASSERT(timeBegin > 0); StorageGcsAuthTokenResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { HttpHeader *const header = httpHeaderNew(NULL); httpHeaderAdd(header, HTTP_HEADER_HOST_STR, httpUrlHost(this->authUrl)); httpHeaderAdd(header, GCS_HEADER_METADATA_FLAVOR_STR, GCS_HEADER_GOOGLE_STR); httpHeaderAdd(header, HTTP_HEADER_CONTENT_LENGTH_STR, ZERO_STR); HttpRequest *const request = httpRequestNewP( this->authClient, HTTP_VERB_GET_STR, httpUrlPath(this->authUrl), NULL, .header = header); MEM_CONTEXT_PRIOR_BEGIN() { result = storageGcsAuthToken(request, timeBegin); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_TYPE(StorageGcsAuthTokenResult, result); } /*********************************************************************************************************************************** Generate authorization header and add it to the supplied header list ***********************************************************************************************************************************/ static void storageGcsAuth(StorageGcs *const this, HttpHeader *const httpHeader) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_GCS, this); FUNCTION_TEST_PARAM(HTTP_HEADER, httpHeader); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(httpHeader != NULL); ASSERT(httpHeaderGet(httpHeader, HTTP_HEADER_CONTENT_LENGTH_STR) != NULL); MEM_CONTEXT_TEMP_BEGIN() { StorageGcsAuthFunction authFunction = NULL; switch (this->keyType) { case storageGcsKeyTypeAuto: authFunction = storageGcsAuthAuto; break; case storageGcsKeyTypeService: authFunction = storageGcsAuthService; break; case storageGcsKeyTypeWebId: authFunction = storageGcsAuthWebId; break; case storageGcsKeyTypeToken: break; } // Get the token if it was not supplied by the user if (authFunction != NULL) { const time_t timeBegin = time(NULL); // If the current token has expired then request a new one if (timeBegin >= this->tokenTimeExpire) { const StorageGcsAuthTokenResult tokenResult = authFunction(this, timeBegin); MEM_CONTEXT_OBJ_BEGIN(this) { strFree(this->token); this->token = strNewFmt("%s %s", strZ(tokenResult.tokenType), strZ(tokenResult.token)); // Subtract http client timeout * 2 so the token does not expire in the middle of http retries this->tokenTimeExpire = tokenResult.timeExpire - ((time_t)(httpClientTimeout(this->httpClient) / MSEC_PER_SEC * 2)); } MEM_CONTEXT_OBJ_END(); } } // Add authorization header httpHeaderPut(httpHeader, HTTP_HEADER_AUTHORIZATION_STR, this->token); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Process Gcs request ***********************************************************************************************************************************/ // Helper to generate request path static String * storageGcsRequestPath(StorageGcs *const this, const String *const object, const bool bucket, const bool upload) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_GCS, this); FUNCTION_TEST_PARAM(STRING, object); FUNCTION_TEST_PARAM(BOOL, bucket); FUNCTION_TEST_PARAM(BOOL, upload); FUNCTION_TEST_END(); ASSERT(this != NULL); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { strCatFmt(result, "%s/storage/v1/b", upload ? "/upload" : ""); if (bucket) strCatFmt(result, "/%s/o", strZ(this->bucket)); if (object != NULL) strCatFmt(result, "/%s", strZ(httpUriEncode(strSub(object, 1), false))); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN HttpRequest * storageGcsRequestAsync(StorageGcs *const this, const String *const verb, StorageGcsRequestAsyncParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(BOOL, param.noBucket); FUNCTION_LOG_PARAM(BOOL, param.upload); FUNCTION_LOG_PARAM(BOOL, param.noAuth); FUNCTION_LOG_PARAM(BOOL, param.tag); FUNCTION_LOG_PARAM(STRING, param.path); FUNCTION_LOG_PARAM(STRING, param.object); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_PARAM(LIST, param.contentList); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(verb != NULL); ASSERT(!param.noBucket || param.object == NULL); HttpRequest *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Generate path const String *const path = param.path != NULL ? param.path : storageGcsRequestPath(this, param.object, !param.noBucket, param.upload); // Create header list HttpHeader *const requestHeader = param.header == NULL ? httpHeaderNew(this->headerRedactList) : httpHeaderDup(param.header, this->headerRedactList); // Add tags if (param.tag) { ASSERT(param.content == NULL); ASSERT(this->tag != NULL); httpHeaderPut(requestHeader, HTTP_HEADER_CONTENT_TYPE_STR, HTTP_HEADER_CONTENT_TYPE_JSON_STR); param.content = this->tag; } // Set host httpHeaderPut(requestHeader, HTTP_HEADER_HOST_STR, this->endpoint); // Set content or construct multipart content const Buffer *content = param.content; if (param.contentList != NULL) { ASSERT(param.content == NULL); HttpRequestMulti *const requestMulti = httpRequestMultiNew(); for (unsigned int contentIdx = 0; contentIdx < lstSize(param.contentList); contentIdx++) { const StorageGcsRequestPart *const requestPart = lstGet(param.contentList, contentIdx); HttpHeader *const partHeader = httpHeaderNew(this->headerRedactList); httpHeaderAdd(partHeader, HTTP_HEADER_CONTENT_LENGTH_STR, ZERO_STR); httpRequestMultiAddP( requestMulti, strNewFmt("%u", contentIdx), requestPart->verb, storageGcsRequestPath(this, requestPart->object, true, false), .header = partHeader); } httpRequestMultiHeaderAdd(requestMulti, requestHeader); content = httpRequestMultiContent(requestMulti); } // Set content length httpHeaderPut( requestHeader, HTTP_HEADER_CONTENT_LENGTH_STR, content == NULL || bufEmpty(content) ? ZERO_STR : strNewFmt("%zu", bufUsed(content))); // Make a copy of the query so it can be modified HttpQuery *const query = this->userProject != NULL && param.query == NULL ? httpQueryNewP(.redactList = this->queryRedactList) : httpQueryDupP(param.query, .redactList = this->queryRedactList); // Add user project if (this->userProject != NULL) httpQueryAdd(query, GCS_QUERY_USER_PROJECT_STR, this->userProject); // Generate authorization header if (!param.noAuth) storageGcsAuth(this, requestHeader); // Send request MEM_CONTEXT_PRIOR_BEGIN() { result = httpRequestNewP( this->httpClient, verb, path, .query = query, .header = requestHeader, .content = content); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_REQUEST, result); } FN_EXTERN HttpResponse * storageGcsResponse(HttpRequest *const request, const StorageGcsResponseParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_REQUEST, request); FUNCTION_LOG_PARAM(BOOL, param.allowMissing); FUNCTION_LOG_PARAM(BOOL, param.allowIncomplete); FUNCTION_LOG_PARAM(BOOL, param.contentIo); FUNCTION_LOG_END(); ASSERT(request != NULL); ASSERT(!param.allowMissing || !param.allowIncomplete); HttpResponse *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get response result = httpRequestResponse(request, !param.contentIo); // Error if the request was not successful if (!httpResponseCodeOk(result) && (!param.allowMissing || httpResponseCode(result) != HTTP_RESPONSE_CODE_NOT_FOUND) && (!param.allowIncomplete || httpResponseCode(result) != HTTP_RESPONSE_CODE_PERMANENT_REDIRECT)) httpRequestError(request, result); // Move response to the prior context httpResponseMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } FN_EXTERN HttpResponse * storageGcsRequest(StorageGcs *const this, const String *const verb, const StorageGcsRequestParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(BOOL, param.noBucket); FUNCTION_LOG_PARAM(BOOL, param.upload); FUNCTION_LOG_PARAM(BOOL, param.noAuth); FUNCTION_LOG_PARAM(BOOL, param.tag); FUNCTION_LOG_PARAM(STRING, param.path); FUNCTION_LOG_PARAM(STRING, param.object); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_PARAM(BOOL, param.allowMissing); FUNCTION_LOG_PARAM(BOOL, param.allowIncomplete); FUNCTION_LOG_PARAM(BOOL, param.contentIo); FUNCTION_LOG_END(); HttpRequest *const request = storageGcsRequestAsyncP( this, verb, .noBucket = param.noBucket, .upload = param.upload, .noAuth = param.noAuth, .tag = param.tag, .path = param.path, .object = param.object, .header = param.header, .query = param.query, .content = param.content, .contentList = param.contentList); HttpResponse *const result = storageGcsResponseP( request, .allowMissing = param.allowMissing, .allowIncomplete = param.allowIncomplete, .contentIo = param.contentIo); httpRequestFree(request); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } /*********************************************************************************************************************************** General function for listing files to be used by other list routines Based on the documentation at https://cloud.google.com/storage/docs/json_api/v1/objects/list ***********************************************************************************************************************************/ // Helper to convert YYYY-MM-DDTHH:MM:SS.MSECZ format to time_t. This format is very nearly ISO-8601 except for the inclusion of // milliseconds, which are discarded here. static time_t storageGcsCvtTime(const String *const time) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, time); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN( TIME, epochFromParts( cvtZSubNToInt(strZ(time), 0, 4), cvtZSubNToInt(strZ(time), 5, 2), cvtZSubNToInt(strZ(time), 8, 2), cvtZSubNToInt(strZ(time), 11, 2), cvtZSubNToInt(strZ(time), 14, 2), cvtZSubNToInt(strZ(time), 17, 2), 0)); } static void storageGcsInfoFile(StorageInfo *const info, const KeyValue *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_PARAM(KEY_VALUE, file); FUNCTION_TEST_END(); info->size = cvtZToUInt64(strZ(varStr(kvGet(file, GCS_JSON_SIZE_VAR)))); info->timeModified = storageGcsCvtTime(varStr(kvGet(file, GCS_JSON_UPDATED_VAR))); FUNCTION_TEST_RETURN_VOID(); } static void storageGcsListInternal( StorageGcs *const this, const String *const path, const StorageInfoLevel level, const String *const expression, const bool recurse, const time_t targetTime, StorageListCallback callback, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(STRING, expression); FUNCTION_LOG_PARAM(BOOL, recurse); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, callback); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Build the base prefix by stripping off the initial / const String *const basePrefix = strSize(path) == 1 ? EMPTY_STR : strNewFmt("%s/", strZ(strSub(path, 1))); // Get the expression prefix when possible to limit initial results const String *const expressionPrefix = regExpPrefix(expression); // If there is an expression prefix then use it to build the query prefix, otherwise query prefix is base prefix const String *queryPrefix; if (expressionPrefix == NULL) queryPrefix = basePrefix; else { if (strEmpty(basePrefix)) queryPrefix = expressionPrefix; else queryPrefix = strNewFmt("%s%s", strZ(basePrefix), strZ(expressionPrefix)); } // Create query HttpQuery *const query = httpQueryNewP(); // Add the delimiter to not recurse if (!recurse) httpQueryAdd(query, GCS_QUERY_DELIMITER_STR, FSLASH_STR); // Don't specify empty prefix because it is the default if (!strEmpty(queryPrefix)) httpQueryAdd(query, GCS_QUERY_PREFIX_STR, queryPrefix); // Specify fields to limit the amount of data returned String *const fields = strCatZ( strNew(), GCS_JSON_NEXT_PAGE_TOKEN "," GCS_JSON_PREFIXES "," GCS_JSON_ITEMS "(" GCS_JSON_NAME); if (level >= storageInfoLevelBasic || targetTime != 0) { if (level >= storageInfoLevelBasic) strCatZ(fields, "," GCS_JSON_SIZE); strCatZ(fields, "," GCS_JSON_UPDATED); if (targetTime != 0) strCatZ(fields, "," GCS_JSON_GENERATION); } strCatChr(fields, ')'); httpQueryAdd(query, GCS_QUERY_FIELDS_STR, fields); // Add versions if (targetTime != 0) httpQueryAdd(query, GCS_QUERY_VERSIONS_STR, TRUE_STR); // Store last info so it can be updated across requests for versioning String *const nameLast = strNew(); String *const versionIdLast = strNew(); StorageInfo infoLast = {.level = level, .name = nameLast}; if (targetTime != 0) infoLast.versionId = versionIdLast; // Loop as long as a continuation marker returned HttpRequest *request = NULL; do { // Use an inner mem context here because we could potentially be retrieving millions of files so it is a good idea to // free memory at regular intervals MEM_CONTEXT_TEMP_BEGIN() { HttpResponse *response = NULL; // If there is an outstanding async request then wait for the response if (request != NULL) { response = storageGcsResponseP(request); httpRequestFree(request); request = NULL; } // Else get the response immediately from a sync request else response = storageGcsRequestP(this, HTTP_VERB_GET_STR, .query = query); const KeyValue *const content = varKv(jsonToVar(strNewBuf(httpResponseContent(response)))); // If next page token exists then send an async request to get more data const String *const nextPageToken = varStr(kvGet(content, GCS_JSON_NEXT_PAGE_TOKEN_VAR)); if (nextPageToken != NULL) { httpQueryPut(query, GCS_QUERY_PAGE_TOKEN_STR, nextPageToken); // Store request in the outer temp context MEM_CONTEXT_PRIOR_BEGIN() { request = storageGcsRequestAsyncP(this, HTTP_VERB_GET_STR, .query = query); } MEM_CONTEXT_PRIOR_END(); } // Get prefix list const VariantList *const prefixList = varVarLst(kvGet(content, GCS_JSON_PREFIXES_VAR)); if (prefixList != NULL) { for (unsigned int prefixIdx = 0; prefixIdx < varLstSize(prefixList); prefixIdx++) { // Get path name StorageInfo info = { .level = level, .name = varStr(varLstGet(prefixList, prefixIdx)), .exists = true, }; // Strip off base prefix and final / info.name = strSubN(info.name, strSize(basePrefix), strSize(info.name) - strSize(basePrefix) - 1); // Add type info if requested if (level >= storageInfoLevelType) info.type = storageTypePath; // Callback with info callback(callbackData, &info); } } // Get file list const VariantList *const fileList = varVarLst(kvGet(content, GCS_JSON_ITEMS_VAR)); if (fileList != NULL) { for (unsigned int fileIdx = 0; fileIdx < varLstSize(fileList); fileIdx++) { const KeyValue *const file = varKv(varLstGet(fileList, fileIdx)); CHECK(FormatError, file != NULL, "file missing"); // Get file name and strip off the base prefix when present const String *name = varStr(kvGet(file, GCS_JSON_NAME_VAR)); CHECK(FormatError, name != NULL, "file name missing"); if (!strEmpty(basePrefix)) name = strSub(name, strSize(basePrefix)); // Return info for last file if new file if (infoLast.exists && !strEq(name, nameLast)) { callback(callbackData, &infoLast); infoLast.exists = false; } // If targeting by time exclude versions that are newer than targetTime. Note that the API does not provide // delete markers so the filtering will also show files that have been deleted rather than replaced with a // new version. if (targetTime != 0) { infoLast.timeModified = storageGcsCvtTime(varStr(kvGet(file, GCS_JSON_UPDATED_VAR))); // Skip this version if it is newer than the time limit if (infoLast.timeModified > targetTime) continue; } // Update last name and set exists strCat(strTrunc(nameLast), name); infoLast.exists = true; // Add basic level info if requested if (level >= storageInfoLevelBasic) { storageGcsInfoFile(&infoLast, file); if (targetTime != 0) strCat(strTrunc(versionIdLast), varStr(kvGet(file, GCS_JSON_GENERATION_VAR))); } } } } MEM_CONTEXT_TEMP_END(); } while (request != NULL); // Callback with last info if it exists if (infoLast.exists) callback(callbackData, &infoLast); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static StorageInfo storageGcsInfo(THIS_VOID, const String *const file, const StorageInfoLevel level, const StorageInterfaceInfoParam param) { THIS(StorageGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(ENUM, level); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); StorageInfo result = {.level = level}; MEM_CONTEXT_TEMP_BEGIN() { // Attempt to get file info HttpResponse *const httpResponse = storageGcsRequestP( this, HTTP_VERB_GET_STR, .object = file, .allowMissing = true, .query = httpQueryAdd( httpQueryNewP(), GCS_QUERY_FIELDS_STR, level >= storageInfoLevelBasic ? STRDEF(GCS_JSON_SIZE "," GCS_JSON_UPDATED) : EMPTY_STR)); // Does the file exist? result.exists = httpResponseCodeOk(httpResponse); // Add basic level info if requested and the file exists if (result.level >= storageInfoLevelBasic && result.exists) { result.type = storageTypeFile; storageGcsInfoFile(&result, varKv(jsonToVar(strNewBuf(httpResponseContent(httpResponse))))); } httpResponseFree(httpResponse); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ static void storageGcsListCallback(void *const callbackData, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(callbackData != NULL); ASSERT(info != NULL); storageLstAdd(callbackData, info); FUNCTION_TEST_RETURN_VOID(); } static StorageList * storageGcsList(THIS_VOID, const String *const path, const StorageInfoLevel level, const StorageInterfaceListParam param) { THIS(StorageGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(STRING, param.expression); FUNCTION_LOG_PARAM(TIME, param.targetTime); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); StorageList *const result = storageLstNew(level); storageGcsListInternal(this, path, level, param.expression, false, param.targetTime, storageGcsListCallback, result); FUNCTION_LOG_RETURN(STORAGE_LIST, result); } /**********************************************************************************************************************************/ static void * storageGcsNewRead(THIS_VOID, const String *const file, const StorageInterfaceNewReadParam param) { THIS(StorageGcs); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_PARAM(STRING, param.versionId); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); FUNCTION_LOG_RETURN(STORAGE_READ_GCS, storageReadGcsNew(this, file, param.offset, param.limit, param.versionId)); } /**********************************************************************************************************************************/ static void * storageGcsNewWrite(THIS_VOID, const String *const file, const StorageInterfaceNewWriteParam param) { THIS(StorageGcs); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, file); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.createPath); ASSERT(param.truncate); ASSERT(param.user == NULL); ASSERT(param.group == NULL); ASSERT(param.timeModified == 0); FUNCTION_LOG_RETURN(STORAGE_WRITE_GCS, storageWriteGcsNew(this, file, this->chunkSize, this->tag != NULL)); } /**********************************************************************************************************************************/ #define GCS_HEADER_CONTENTID_RESPONSE "response-" typedef struct StorageGcsPathRemoveData { StorageGcs *this; // Storage Object MemContext *memContext; // Mem context to create requests in HttpRequest *request; // Async remove request List *requestContentList; // Content list for async request List *contentList; // Content list currently being built const String *path; // Root path of remove } StorageGcsPathRemoveData; static void storageGcsPathRemoveInternal(StorageGcsPathRemoveData *const data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_END(); ASSERT(data != NULL); ASSERT(data->this != NULL); // Get response for async request if (data->request != NULL) { MEM_CONTEXT_TEMP_BEGIN() { HttpResponse *const response = storageGcsResponseP(data->request); HttpResponseMulti *const responseMulti = httpResponseMultiNew( httpResponseContent(response), httpHeaderGet(httpResponseHeader(response), HTTP_HEADER_CONTENT_TYPE_STR)); // Loop through all response parts HttpResponse *responsePart = httpResponseMultiNext(responseMulti); CHECK(FormatError, responsePart != NULL, "at least one response part is required"); do { // If not OK and not missing then retry if (!httpResponseCodeOk(responsePart) && httpResponseCode(responsePart) != HTTP_RESPONSE_CODE_NOT_FOUND) { // Extract and check content-id header const String *const contentId = httpHeaderGet(httpResponseHeader(responsePart), HTTP_HEADER_CONTENT_ID_STR); CHECK(FormatError, contentId != NULL, HTTP_HEADER_CONTENT_ID " header is not present"); CHECK_FMT( FormatError, strBeginsWithZ(contentId, GCS_HEADER_CONTENTID_RESPONSE), HTTP_HEADER_CONTENT_ID " header '%s' must begin with '" GCS_HEADER_CONTENTID_RESPONSE "'", strZ(contentId)); // Use content-id to get content const unsigned int contentIdx = cvtZToUInt(strZ(strSub(contentId, sizeof(GCS_HEADER_CONTENTID_RESPONSE) - 1))); const StorageGcsRequestPart *const content = lstGet(data->requestContentList, contentIdx); // Retry remove statInc(GCS_STAT_REMOVE_BATCH_RETRY_STR); storageGcsRequestP(data->this, content->verb, .object = content->object, .allowMissing = true); } else statInc(GCS_STAT_REMOVE_BATCH_PART_STR); httpResponseFree(responsePart); responsePart = httpResponseMultiNext(responseMulti); } while (responsePart != NULL); } MEM_CONTEXT_TEMP_END(); // Free request httpRequestFree(data->request); data->request = NULL; // Free content list lstFree(data->requestContentList); } // Send new async request if there is more to remove if (data->contentList != NULL) { statInc(GCS_STAT_REMOVE_BATCH_STR); MEM_CONTEXT_BEGIN(data->memContext) { data->request = storageGcsRequestAsyncP( data->this, HTTP_VERB_POST_STR, .path = STRDEF("/batch/storage/v1"), .contentList = data->contentList); } MEM_CONTEXT_END(); // Store the content list for use in error handling data->requestContentList = data->contentList; data->contentList = NULL; } FUNCTION_TEST_RETURN_VOID(); } static void storageGcsPathRemoveCallback(void *const callbackData, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(callbackData != NULL); ASSERT(info != NULL); // Only delete files since paths don't really exist if (info->type == storageTypeFile) { StorageGcsPathRemoveData *const data = callbackData; if (data->contentList == NULL) { MEM_CONTEXT_BEGIN(data->memContext) { data->contentList = lstNewP(sizeof(StorageGcsRequestPart)); } MEM_CONTEXT_END(); } MEM_CONTEXT_OBJ_BEGIN(data->contentList) { const StorageGcsRequestPart content = { .verb = HTTP_VERB_DELETE_STR, .object = strNewFmt("%s/%s", strZ(data->path), strZ(info->name)), }; lstAdd(data->contentList, &content); } MEM_CONTEXT_OBJ_END(); if (lstSize(data->contentList) == data->this->deleteMax) storageGcsPathRemoveInternal(data); } FUNCTION_TEST_RETURN_VOID(); } static bool storageGcsPathRemove(THIS_VOID, const String *const path, const bool recurse, const StorageInterfacePathRemoveParam param) { THIS(StorageGcs); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, path); (void)recurse; // There are no paths so removal is always recursive (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { StorageGcsPathRemoveData data = { .this = this, .memContext = memContextCurrent(), .path = strEq(path, FSLASH_STR) ? EMPTY_STR : path, }; MEM_CONTEXT_TEMP_BEGIN() { storageGcsListInternal(this, path, storageInfoLevelType, NULL, true, 0, storageGcsPathRemoveCallback, &data); // Call if there is more to be removed if (data.contentList != NULL) storageGcsPathRemoveInternal(&data); // Check response on last async request storageGcsPathRemoveInternal(&data); } MEM_CONTEXT_TEMP_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, true); } /**********************************************************************************************************************************/ static void storageGcsRemove(THIS_VOID, const String *const file, const StorageInterfaceRemoveParam param) { THIS(StorageGcs); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, file); (void)param; FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); statInc(GCS_STAT_REMOVE_STR); httpResponseFree(storageGcsRequestP(this, HTTP_VERB_DELETE_STR, .object = file, .allowMissing = true)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageInterface storageInterfaceGcs = { .feature = 1 << storageFeatureVersioning | 1 << storageFeatureReadRetry, .info = storageGcsInfo, .list = storageGcsList, .newRead = storageGcsNewRead, .newWrite = storageGcsNewWrite, .pathRemove = storageGcsPathRemove, .remove = storageGcsRemove, }; FN_EXTERN Storage * storageGcsNew( const String *const path, const bool write, const time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, const String *const bucket, const StorageGcsKeyType keyType, const String *const key, const size_t chunkSize, const KeyValue *const tag, const String *const endpoint, const TimeMSec timeout, const bool verifyPeer, const String *const caFile, const String *const caPath, const String *const userProject, const unsigned int prefetch, const uint64_t readOver) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_PARAM(STRING, bucket); FUNCTION_LOG_PARAM(ENUM, keyType); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_LOG_PARAM(SIZE, chunkSize); FUNCTION_LOG_PARAM(KEY_VALUE, tag); FUNCTION_LOG_PARAM(STRING, endpoint); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_PARAM(BOOL, verifyPeer); FUNCTION_LOG_PARAM(STRING, caFile); FUNCTION_LOG_PARAM(STRING, caPath); FUNCTION_LOG_PARAM(STRING, userProject); FUNCTION_LOG_PARAM(UINT, prefetch); FUNCTION_LOG_PARAM(UINT64, readOver); FUNCTION_LOG_END(); ASSERT(path != NULL); ASSERT(bucket != NULL); ASSERT(keyType == storageGcsKeyTypeAuto || key != NULL); ASSERT(chunkSize != 0); OBJ_NEW_BEGIN(StorageGcs, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageGcs) { .interface = storageInterfaceGcs, .write = write, .bucket = strDup(bucket), .keyType = keyType, .chunkSize = chunkSize, .deleteMax = STORAGE_GCS_DELETE_MAX, .userProject = strDup(userProject), }; // Set prefetch and read over this->interface.prefetch = prefetch; this->interface.readOver = readOver; // Create tag JSON buffer if (write && tag != NULL) { MEM_CONTEXT_TEMP_BEGIN() { JsonWrite *const tagJson = jsonWriteObjectBegin( jsonWriteKeyStrId(jsonWriteObjectBegin(jsonWriteNewP()), STRID5("metadata", 0xd0240d0ad0))); const StringList *const keyList = strLstSort(strLstNewVarLst(kvKeyList(tag)), sortOrderAsc); for (unsigned int keyIdx = 0; keyIdx < strLstSize(keyList); keyIdx++) { const String *const key = strLstGet(keyList, keyIdx); jsonWriteStr(jsonWriteKey(tagJson, key), varStr(kvGet(tag, VARSTR(key)))); } const String *const tagStr = jsonWriteResult(jsonWriteObjectEnd(jsonWriteObjectEnd(tagJson))); MEM_CONTEXT_PRIOR_BEGIN() { this->tag = bufDup(BUFSTR(tagStr)); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); } // Handle auth key types switch (keyType) { // Auto authentication for GCE instances case storageGcsKeyTypeAuto: { this->authUrl = httpUrlNewParseP( STRDEF("metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token"), .type = httpProtocolTypeHttp); this->authClient = httpClientNew( sckClientNew(httpUrlHost(this->authUrl), httpUrlPort(this->authUrl), timeout, timeout), timeout); break; } // Read data from file for service keys case storageGcsKeyTypeService: { MEM_CONTEXT_TEMP_BEGIN() { const KeyValue *const kvKey = varKv( jsonToVar(strNewBuf(storageGetP(storageNewReadP(storagePosixNewP(FSLASH_STR), key))))); const String *const uri = varStr(kvGet(kvKey, GCS_JSON_TOKEN_URI_VAR)); CHECK(FormatError, uri != NULL, "uri missing"); MEM_CONTEXT_PRIOR_BEGIN() { this->key = strDup(key); this->authUrl = httpUrlNewParseP(uri, .type = httpProtocolTypeHttps); this->authClient = httpClientNew( tlsClientNewP( sckClientNew(httpUrlHost(this->authUrl), httpUrlPort(this->authUrl), timeout, timeout), httpUrlHost(this->authUrl), timeout, timeout, verifyPeer, .caFile = caFile, .caPath = caPath), timeout); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); break; } // Store the authentication token case storageGcsKeyTypeToken: this->token = strDup(key); break; // Read data from an external account credential file, https://google.aip.dev/auth/4117. This is the file generated by // gcloud iam workload-identity-pools create-cred-config and contains the audience, token url, and token file. case storageGcsKeyTypeWebId: { ASSERT(key != NULL); MEM_CONTEXT_TEMP_BEGIN() { const Variant *const keyVariant = jsonToVar(strNewBuf(storageGetP(storageNewReadP(storagePosixNewP(FSLASH_STR), key)))); CHECK( FormatError, keyVariant != NULL && varType(keyVariant) == varTypeKeyValue, "not an external account credential file"); const KeyValue *const kvKey = varKv(keyVariant); const Variant *const typeVariant = kvGet(kvKey, GCS_JSON_TYPE_VAR); const String *const type = typeVariant != NULL && varType(typeVariant) == varTypeString ? varStr(typeVariant) : NULL; CHECK(FormatError, type != NULL && strEqZ(type, "external_account"), "not an external account credential file"); // Impersonation would require a second exchange so the federated identity must have direct access CHECK( FormatError, kvGet(kvKey, GCS_JSON_SA_IMPERSONATION_URL_VAR) == NULL, "service account impersonation is not supported"); const Variant *const audienceVariant = kvGet(kvKey, GCS_JSON_AUDIENCE_VAR); CHECK(FormatError, audienceVariant != NULL, "audience missing"); CHECK(FormatError, varType(audienceVariant) == varTypeString, "audience must be a string"); const String *const audience = varStr(audienceVariant); const Variant *const tokenUrlVariant = kvGet(kvKey, GCS_JSON_TOKEN_URL_VAR); CHECK(FormatError, tokenUrlVariant != NULL, "token url missing"); CHECK(FormatError, varType(tokenUrlVariant) == varTypeString, "token url must be a string"); const String *const tokenUrl = varStr(tokenUrlVariant); const Variant *const credentialSourceVariant = kvGet(kvKey, GCS_JSON_CREDENTIAL_SOURCE_VAR); CHECK(FormatError, credentialSourceVariant != NULL, "credential source missing"); CHECK( FormatError, varType(credentialSourceVariant) == varTypeKeyValue, "credential source must be an object"); const KeyValue *const credentialSource = varKv(credentialSourceVariant); const Variant *const tokenFileVariant = kvGet(credentialSource, GCS_JSON_FILE_VAR); CHECK(FormatError, tokenFileVariant != NULL, "only the file credential source is supported"); CHECK(FormatError, varType(tokenFileVariant) == varTypeString, "token file must be a string"); const String *const tokenFile = varStr(tokenFileVariant); // Only the default text format is supported const Variant *const format = kvGet(credentialSource, GCS_JSON_FORMAT_VAR); if (format != NULL) { CHECK( FormatError, varType(format) == varTypeKeyValue, "credential source format is not supported"); const Variant *const formatTypeVariant = kvGet(varKv(format), GCS_JSON_TYPE_VAR); const String *const formatType = formatTypeVariant != NULL && varType(formatTypeVariant) == varTypeString ? varStr(formatTypeVariant) : NULL; CHECK( FormatError, formatType != NULL && strEqZ(formatType, "text"), "credential source format is not supported"); } MEM_CONTEXT_PRIOR_BEGIN() { this->webIdTokenFile = strDup(tokenFile); this->webIdAudience = strDup(audience); this->authUrl = httpUrlNewParseP(tokenUrl, .type = httpProtocolTypeHttps); this->authClient = httpClientNew( tlsClientNewP( sckClientNew(httpUrlHost(this->authUrl), httpUrlPort(this->authUrl), timeout, timeout), httpUrlHost(this->authUrl), timeout, timeout, verifyPeer, .caFile = caFile, .caPath = caPath), timeout); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); break; } } // Parse the endpoint to extract the host, port, and protocol const HttpUrl *const url = httpUrlNewParseP(endpoint, .type = httpProtocolTypeAny, .defaultType = httpProtocolTypeHttps); const HttpProtocolType protocolType = httpUrlProtocolType(url); this->endpoint = httpUrlHost(url); // Create the http client used to service requests. Use plain socket for HTTP, TLS for HTTPS. IoClient *ioClient; if (protocolType == httpProtocolTypeHttp) ioClient = sckClientNew(this->endpoint, httpUrlPort(url), timeout, timeout); else { ioClient = tlsClientNewP( sckClientNew(this->endpoint, httpUrlPort(url), timeout, timeout), this->endpoint, timeout, timeout, verifyPeer, .caFile = caFile, .caPath = caPath); } this->httpClient = httpClientNew(ioClient, timeout); // Create list of redacted headers this->headerRedactList = strLstNew(); strLstAdd(this->headerRedactList, HTTP_HEADER_AUTHORIZATION_STR); strLstAdd(this->headerRedactList, GCS_HEADER_UPLOAD_ID_STR); // Create list of redacted query keys this->queryRedactList = strLstNew(); strLstAdd(this->queryRedactList, GCS_QUERY_UPLOAD_ID_STR); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( STORAGE, storageNew(STORAGE_GCS_TYPE, path, 0, 0, write, targetTime, pathExpressionFunction, this, this->interface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/gcs/storage.h0000664000175100017510000000353715265101745020236 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage ***********************************************************************************************************************************/ #ifndef STORAGE_GCS_STORAGE_H #define STORAGE_GCS_STORAGE_H #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_GCS_TYPE STRID5("gcs", 0x4c670) /*********************************************************************************************************************************** Key type ***********************************************************************************************************************************/ typedef enum { storageGcsKeyTypeService, storageGcsKeyTypeToken, storageGcsKeyTypeAuto, storageGcsKeyTypeWebId, } StorageGcsKeyType; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Storage *storageGcsNew( const String *path, bool write, time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, const String *bucket, StorageGcsKeyType keyType, const String *key, size_t blockSize, const KeyValue *tag, const String *endpoint, TimeMSec timeout, bool verifyPeer, const String *caFile, const String *caPath, const String *userProject, unsigned int prefetch, uint64_t readOver); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/gcs/storage.intern.h0000664000175100017510000001761715265101745021540 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage Internal ***********************************************************************************************************************************/ #ifndef STORAGE_GCS_STORAGE_INTERN_H #define STORAGE_GCS_STORAGE_INTERN_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageGcs StorageGcs; #include "common/io/http/request.h" #include "storage/gcs/storage.h" /*********************************************************************************************************************************** HTTP headers ***********************************************************************************************************************************/ #define GCS_HEADER_UPLOAD_ID "x-guploader-uploadid" STRING_DECLARE(GCS_HEADER_UPLOAD_ID_STR); /*********************************************************************************************************************************** Query tokens ***********************************************************************************************************************************/ #define GCS_QUERY_FIELDS "fields" STRING_DECLARE(GCS_QUERY_FIELDS_STR); #define GCS_QUERY_MEDIA "media" STRING_DECLARE(GCS_QUERY_MEDIA_STR); #define GCS_QUERY_NAME "name" STRING_DECLARE(GCS_QUERY_NAME_STR); #define GCS_QUERY_UPLOAD_ID "upload_id" STRING_DECLARE(GCS_QUERY_UPLOAD_ID_STR); /*********************************************************************************************************************************** JSON tokens ***********************************************************************************************************************************/ #define GCS_JSON_GENERATION "generation" VARIANT_DECLARE(GCS_JSON_GENERATION_VAR); #define GCS_JSON_MD5_HASH "md5Hash" VARIANT_DECLARE(GCS_JSON_MD5_HASH_VAR); #define GCS_JSON_NAME "name" VARIANT_DECLARE(GCS_JSON_NAME_VAR); #define GCS_JSON_SIZE "size" VARIANT_DECLARE(GCS_JSON_SIZE_VAR); /*********************************************************************************************************************************** Multi-Part request data ***********************************************************************************************************************************/ typedef struct StorageGcsRequestPart { const String *object; // Object to include in URI const String *verb; // Verb (GET, PUT, etc) } StorageGcsRequestPart; /*********************************************************************************************************************************** Perform a GCS Request ***********************************************************************************************************************************/ // Perform async request typedef struct StorageGcsRequestAsyncParam { VAR_PARAM_HEADER; bool noBucket; // Exclude bucket from the URI? bool upload; // Is an object upload? bool noAuth; // Exclude authentication header? bool tag; // Add tags when available? const String *path; // URI path (this overrides object) const String *object; // Object to include in URI const HttpHeader *header; // Request headers const HttpQuery *query; // Query parameters const Buffer *content; // Request content const List *contentList; // Request content part list } StorageGcsRequestAsyncParam; #define storageGcsRequestAsyncP(this, verb, ...) \ storageGcsRequestAsync(this, verb, (StorageGcsRequestAsyncParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpRequest *storageGcsRequestAsync(StorageGcs *this, const String *verb, StorageGcsRequestAsyncParam param); // Get async response typedef struct StorageGcsResponseParam { VAR_PARAM_HEADER; bool allowMissing; // Allow missing files (caller can check response code) bool allowIncomplete; // Allow incomplete resume (used for resumable upload) bool contentIo; // Is IoRead interface required to read content? } StorageGcsResponseParam; #define storageGcsResponseP(request, ...) \ storageGcsResponse(request, (StorageGcsResponseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpResponse *storageGcsResponse(HttpRequest *request, StorageGcsResponseParam param); typedef struct StorageGcsRequestParam { VAR_PARAM_HEADER; bool noBucket; // Exclude bucket from the URI? bool upload; // Is an object upload? bool noAuth; // Exclude authentication header? bool tag; // Add tags when available? const String *path; // URI path (this overrides object) const String *object; // Object to include in URI const HttpHeader *header; // Request headers const HttpQuery *query; // Query parameters const Buffer *content; // Request content const List *contentList; // Request content part list bool allowMissing; // Allow missing files (caller can check response code) bool allowIncomplete; // Allow incomplete resume (used for resumable upload) bool contentIo; // Is IoRead interface required to read content? } StorageGcsRequestParam; #define storageGcsRequestP(this, verb, ...) \ storageGcsRequest(this, verb, (StorageGcsRequestParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpResponse *storageGcsRequest(StorageGcs *this, const String *verb, StorageGcsRequestParam param); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_GCS_TYPE \ StorageGcs * #define FUNCTION_LOG_STORAGE_GCS_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageGcs", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/gcs/write.c0000664000175100017510000003307215265101745017714 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage File Write ***********************************************************************************************************************************/ #include #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/json.h" #include "common/type/keyValue.h" #include "common/type/object.h" #include "storage/gcs/write.h" #include "storage/write.h" /*********************************************************************************************************************************** Chunk defaults based on limits at https://cloud.google.com/storage/quotas#requests ***********************************************************************************************************************************/ #define STORAGE_GCS_SPLIT_DEFAULT 257 #define STORAGE_GCS_SPLIT_MAX 9505 /*********************************************************************************************************************************** GCS query tokens ***********************************************************************************************************************************/ STRING_STATIC(GCS_QUERY_UPLOAD_TYPE_STR, "uploadType"); STRING_STATIC(GCS_QUERY_RESUMABLE_STR, "resumable"); STRING_STATIC(GCS_QUERY_FIELDS_VALUE_STR, GCS_JSON_MD5_HASH "," GCS_JSON_SIZE); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWriteGcs { const StorageWriteInterface *interface; // Interface StorageGcs *storage; // Storage that created this object const String *name; // File name HttpRequest *request; // Async chunk upload request size_t chunkSize; // Size of chunks for resumable upload unsigned int chunkTotal; // Total chunks uploaded bool tag; // Are tags available? Buffer *chunkBuffer; // Block buffer (stores data until chunkSize is reached) const String *uploadId; // Id for resumable upload uint64_t uploadTotal; // Total bytes uploaded IoFilter *md5hash; // MD5 hash of file }; /*********************************************************************************************************************************** Verify upload ***********************************************************************************************************************************/ static void storageWriteGcsVerify(StorageWriteGcs *const this, HttpResponse *const response) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_GCS, this); FUNCTION_LOG_PARAM(HTTP_RESPONSE, response); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { KeyValue *const content = varKv(jsonToVar(strNewBuf(httpResponseContent(response)))); // Check the md5 hash const String *const md5base64 = varStr(kvGet(content, GCS_JSON_MD5_HASH_VAR)); CHECK(FormatError, md5base64 != NULL, "MD5 missing"); const Buffer *const md5actual = bufNewDecode(encodingBase64, md5base64); const Buffer *const md5expected = pckReadBinP(pckReadNew(ioFilterResult(this->md5hash))); if (!bufEq(md5actual, md5expected)) { THROW_FMT( FormatError, "expected md5 '%s' for '%s' but actual is '%s'", strZ(strNewEncode(encodingHex, md5expected)), strZ(this->name), strZ(strNewEncode(encodingHex, md5actual))); } // Check the size when available const String *const sizeStr = varStr(kvGet(content, GCS_JSON_SIZE_VAR)); if (sizeStr != NULL) { const uint64_t size = cvtZToUInt64(strZ(sizeStr)); if (size != this->uploadTotal) { THROW_FMT( FormatError, "expected size %" PRIu64 " for '%s' but actual is %" PRIu64, size, strZ(this->name), this->uploadTotal); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Flush bytes to upload chunk ***********************************************************************************************************************************/ static void storageWriteGcsBlock(StorageWriteGcs *const this, const bool done) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_GCS, this); FUNCTION_LOG_PARAM(BOOL, done); FUNCTION_LOG_END(); ASSERT(this != NULL); // If there is an outstanding async request then wait for the response to ensure the request did not error if (this->request != NULL) { HttpResponse *const response = storageGcsResponseP(this->request, .allowIncomplete = !done); // If done then verify the md5 checksum if (done) storageWriteGcsVerify(this, response); httpResponseFree(response); httpRequestFree(this->request); this->request = NULL; } FUNCTION_LOG_RETURN_VOID(); } static void storageWriteGcsBlockAsync(StorageWriteGcs *const this, const bool done) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_GCS, this); FUNCTION_LOG_PARAM(BOOL, done); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->chunkBuffer != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Complete prior async request, if any storageWriteGcsBlock(this, false); // Build query HttpQuery *const query = httpQueryNewP(); httpQueryAdd(query, GCS_QUERY_NAME_STR, strSub(this->name, 1)); httpQueryAdd(query, GCS_QUERY_UPLOAD_TYPE_STR, GCS_QUERY_RESUMABLE_STR); // Get the upload id if (this->uploadId == NULL) { HttpResponse *response = storageGcsRequestP( this->storage, HTTP_VERB_POST_STR, .upload = true, .tag = this->tag, .query = query); MEM_CONTEXT_OBJ_BEGIN(this) { this->uploadId = strDup(httpHeaderGet(httpResponseHeader(response), GCS_HEADER_UPLOAD_ID_STR)); CHECK(FormatError, this->uploadId != NULL, "upload id missing"); } MEM_CONTEXT_OBJ_END(); } // Add data to md5 hash if (!bufEmpty(this->chunkBuffer)) ioFilterProcessIn(this->md5hash, this->chunkBuffer); // Upload the chunk. If this is the last chunk then add the total bytes in the file to the range rather than the * added to // prior chunks. This indicates that the resumable upload is complete. If the last chunk is zero-size, then the byte range // is * to indicate that there is no more data to upload. HttpHeader *const header = httpHeaderAdd( httpHeaderNew(NULL), HTTP_HEADER_CONTENT_RANGE_STR, strNewFmt( HTTP_HEADER_CONTENT_RANGE_BYTES " %s/%s", bufUsed(this->chunkBuffer) == 0 ? "*" : zNewFmt("%" PRIu64 "-%" PRIu64, this->uploadTotal, this->uploadTotal + bufUsed(this->chunkBuffer) - 1), done ? zNewFmt("%" PRIu64, this->uploadTotal + bufUsed(this->chunkBuffer)) : "*")); httpQueryAdd(query, GCS_QUERY_UPLOAD_ID_STR, this->uploadId); // Add fields needed to verify to upload if (done) httpQueryAdd(query, GCS_QUERY_FIELDS_STR, GCS_QUERY_FIELDS_VALUE_STR); MEM_CONTEXT_OBJ_BEGIN(this) { this->request = storageGcsRequestAsyncP( this->storage, HTTP_VERB_PUT_STR, .upload = true, .noAuth = true, .header = header, .query = query, .content = this->chunkBuffer); } MEM_CONTEXT_OBJ_END(); this->uploadTotal += bufUsed(this->chunkBuffer); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to internal buffer ***********************************************************************************************************************************/ static void storageWriteGcs(THIS_VOID, const Buffer *const buffer) { THIS(StorageWriteGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_GCS, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); // Continue until the write buffer has been exhausted size_t bytesTotal = 0; do { // Resize chunk buffer. This must be done in the loop because chunk size grows as chunks are written. storageWriteChunkBufferResize(buffer, bytesTotal, this->chunkBuffer, this->chunkSize); // Copy as many bytes as possible into the chunk buffer const size_t bytesNext = bufRemains(this->chunkBuffer) > bufUsed(buffer) - bytesTotal ? bufUsed(buffer) - bytesTotal : bufRemains(this->chunkBuffer); bufCatSub(this->chunkBuffer, buffer, bytesTotal, bytesNext); bytesTotal += bytesNext; // If the chunk buffer is full then write it. It is possible that this is the last chunk and it would be better to wait, but // the chances of that are quite small so in general it is better to write now so there is less to write later. if (bufUsed(this->chunkBuffer) == this->chunkSize) { storageWriteGcsBlockAsync(this, false); bufUsedZero(this->chunkBuffer); this->chunkSize = storageWriteChunkSize(this->chunkSize, STORAGE_GCS_SPLIT_DEFAULT, STORAGE_GCS_SPLIT_MAX, ++this->chunkTotal); } } while (bytesTotal != bufUsed(buffer)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWriteGcsClose(THIS_VOID) { THIS(StorageWriteGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_GCS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close if the file has not already been closed if (this->chunkBuffer != NULL) { MEM_CONTEXT_TEMP_BEGIN() { // If a resumable upload was started then finish that way if (this->uploadId != NULL || this->tag) { // Write what is left in the chunk buffer storageWriteGcsBlockAsync(this, true); storageWriteGcsBlock(this, true); } // Else upload all the data in a single chunk else { // Add data to md5 hash if (bufUsed(this->chunkBuffer)) ioFilterProcessIn(this->md5hash, this->chunkBuffer); // Upload file HttpQuery *query = httpQueryNewP(); httpQueryAdd(query, GCS_QUERY_NAME_STR, strSub(this->name, 1)); httpQueryAdd(query, GCS_QUERY_UPLOAD_TYPE_STR, GCS_QUERY_MEDIA_STR); httpQueryAdd(query, GCS_QUERY_FIELDS_STR, GCS_QUERY_FIELDS_VALUE_STR); this->uploadTotal = bufUsed(this->chunkBuffer); storageWriteGcsVerify( this, storageGcsRequestP( this->storage, HTTP_VERB_POST_STR, .upload = true, .query = query, .content = this->chunkBuffer)); } bufFree(this->chunkBuffer); this->chunkBuffer = NULL; } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageWriteInterface storageWriteGcsInterface = { .close = storageWriteGcsClose, .write = storageWriteGcs, }; FN_EXTERN StorageWriteGcs * storageWriteGcsNew(StorageGcs *const storage, const String *const name, const size_t chunkSize, const bool tag) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_GCS, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, chunkSize); FUNCTION_LOG_PARAM(BOOL, tag); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageWriteGcs, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageWriteGcs) { .interface = &storageWriteGcsInterface, .storage = storage, .name = strDup(name), .chunkSize = chunkSize, .chunkBuffer = bufNew(0), .md5hash = cryptoHashNew(hashTypeMd5), .tag = tag, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE_GCS, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/gcs/write.h0000664000175100017510000000337615265101745017725 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage File Write ***********************************************************************************************************************************/ #ifndef STORAGE_GCS_WRITE_H #define STORAGE_GCS_WRITE_H #include "storage/gcs/storage.intern.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWriteGcs StorageWriteGcs; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWriteGcs *storageWriteGcsNew(StorageGcs *storage, const String *name, size_t chunkSize, bool tag); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_WRITE_GCS_TYPE \ StorageWriteGcs * #define FUNCTION_LOG_STORAGE_WRITE_GCS_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageWriteGcs", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/helper.c0000664000175100017510000005176615265101745017277 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Helper ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/io.h" #include "common/memContext.h" #include "common/regExp.h" #include "config/config.h" #include "protocol/helper.h" #include "storage/helper.h" #include "storage/posix/storage.h" #include "storage/remote/storage.h" /*********************************************************************************************************************************** Storage path constants ***********************************************************************************************************************************/ STRING_EXTERN(STORAGE_SPOOL_ARCHIVE_STR, STORAGE_SPOOL_ARCHIVE); STRING_EXTERN(STORAGE_SPOOL_ARCHIVE_IN_STR, STORAGE_SPOOL_ARCHIVE_IN); STRING_EXTERN(STORAGE_SPOOL_ARCHIVE_OUT_STR, STORAGE_SPOOL_ARCHIVE_OUT); STRING_EXTERN(STORAGE_REPO_ARCHIVE_STR, STORAGE_REPO_ARCHIVE); STRING_EXTERN(STORAGE_REPO_BACKUP_STR, STORAGE_REPO_BACKUP); STRING_EXTERN(STORAGE_PATH_ARCHIVE_STR, STORAGE_PATH_ARCHIVE); STRING_EXTERN(STORAGE_PATH_BACKUP_STR, STORAGE_PATH_BACKUP); /*********************************************************************************************************************************** Error message when writable storage is requested in dry-run mode ***********************************************************************************************************************************/ #define WRITABLE_WHILE_DRYRUN \ "unable to get writable storage in dry-run mode or before dry-run is initialized" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct StorageHelperLocal { MemContext *memContext; // Mem context for storage helper const StorageHelper *helperList; // List of helpers to create storage Storage *storageLocal; // Local read-only storage Storage *storageLocalWrite; // Local write storage Storage **storagePg; // PostgreSQL read-only storage Storage **storagePgWrite; // PostgreSQL write storage Storage **storageRepo; // Repository read-only storage Storage **storageRepoWrite; // Repository write storage Storage *storageSpool; // Spool read-only storage Storage *storageSpoolWrite; // Spool write storage String *stanza; // Stanza for storage bool stanzaInit; // Has the stanza been initialized? bool dryRunInit; // Has dryRun been initialized? If not disallow writes. bool dryRun; // Disallow writes in dry-run mode. RegExp *walRegExp; // Regular expression for identifying wal files } storageHelper; /*********************************************************************************************************************************** Create the storage helper memory context ***********************************************************************************************************************************/ static void storageHelperContextInit(void) { FUNCTION_TEST_VOID(); if (storageHelper.memContext == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(StorageHelper, .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX) { storageHelper.memContext = MEM_CONTEXT_NEW(); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void storageHelperInit(const StorageHelper *const helperList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, helperList); FUNCTION_TEST_END(); storageHelper.helperList = helperList; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void storageHelperDryRunInit(const bool dryRun) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, dryRun); FUNCTION_TEST_END(); storageHelper.dryRunInit = true; storageHelper.dryRun = dryRun; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Initialize the stanza and error if it changes ***********************************************************************************************************************************/ static void storageHelperStanzaInit(const bool stanzaRequired) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, stanzaRequired); FUNCTION_TEST_END(); // If the stanza is NULL and the storage has not already been initialized then initialize the stanza if (!storageHelper.stanzaInit) { if (stanzaRequired && cfgOptionStrNull(cfgOptStanza) == NULL) THROW(AssertError, "stanza cannot be NULL for this storage object"); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.stanza = strDup(cfgOptionStrNull(cfgOptStanza)); storageHelper.stanzaInit = true; } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN const Storage * storageLocal(void) { FUNCTION_TEST_VOID(); if (storageHelper.storageLocal == NULL) { storageHelperContextInit(); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageLocal = storagePosixNewP(FSLASH_STR); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storageLocal); } FN_EXTERN const Storage * storageLocalWrite(void) { FUNCTION_TEST_VOID(); if (storageHelper.storageLocalWrite == NULL) { storageHelperContextInit(); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageLocalWrite = storagePosixNewP(FSLASH_STR, .write = true); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storageLocalWrite); } /*********************************************************************************************************************************** Get pg storage for the specified host id ***********************************************************************************************************************************/ static Storage * storagePgGet(const unsigned int pgIdx, const bool write) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgIdx); FUNCTION_TEST_PARAM(BOOL, write); FUNCTION_TEST_END(); Storage *result; // Use remote storage if (!pgIsLocal(pgIdx)) { result = storageRemoteNew( STORAGE_MODE_FILE_DEFAULT, STORAGE_MODE_PATH_DEFAULT, write, 0, NULL, protocolRemoteGet(protocolStorageTypePg, pgIdx, true), cfgOptionUInt(cfgOptCompressLevelNetwork)); } // Use Posix storage else result = storagePosixNewP(cfgOptionIdxStr(cfgOptPgPath, pgIdx), .write = write); FUNCTION_TEST_RETURN(STORAGE, result); } /**********************************************************************************************************************************/ FN_EXTERN const Storage * storagePgIdx(const unsigned int pgIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgIdx); FUNCTION_TEST_END(); if (storageHelper.storagePg == NULL || storageHelper.storagePg[pgIdx] == NULL) { storageHelperContextInit(); MEM_CONTEXT_BEGIN(storageHelper.memContext) { if (storageHelper.storagePg == NULL) storageHelper.storagePg = memNewPtrArray(cfgOptionGroupIdxTotal(cfgOptGrpPg)); storageHelper.storagePg[pgIdx] = storagePgGet(pgIdx, false); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storagePg[pgIdx]); } FN_EXTERN const Storage * storagePg(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STORAGE, storagePgIdx(cfgOptionGroupIdxDefault(cfgOptGrpPg))); } FN_EXTERN const Storage * storagePgIdxWrite(const unsigned int pgIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgIdx); FUNCTION_TEST_END(); // Writes not allowed in dry-run mode if (!storageHelper.dryRunInit || storageHelper.dryRun) THROW(AssertError, WRITABLE_WHILE_DRYRUN); if (storageHelper.storagePgWrite == NULL || storageHelper.storagePgWrite[pgIdx] == NULL) { storageHelperContextInit(); MEM_CONTEXT_BEGIN(storageHelper.memContext) { if (storageHelper.storagePgWrite == NULL) storageHelper.storagePgWrite = memNewPtrArray(cfgOptionGroupIdxTotal(cfgOptGrpPg)); storageHelper.storagePgWrite[pgIdx] = storagePgGet(pgIdx, true); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storagePgWrite[pgIdx]); } FN_EXTERN const Storage * storagePgWrite(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STORAGE, storagePgIdxWrite(cfgOptionGroupIdxDefault(cfgOptGrpPg))); } /*********************************************************************************************************************************** Create the WAL regular expression ***********************************************************************************************************************************/ static void storageHelperRepoInit(void) { FUNCTION_TEST_VOID(); if (storageHelper.walRegExp == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { storageHelper.walRegExp = regExpNew(STRDEF("^[0-F]{24}")); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Construct a repo path from an expression and path ***********************************************************************************************************************************/ static String * storageRepoPathExpression(const String *const expression, const String *const path) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, expression); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_END(); ASSERT(expression != NULL); String *const result = strNew(); if (strEq(expression, STORAGE_REPO_ARCHIVE_STR)) { // Construct the base path if (storageHelper.stanza != NULL) strCatFmt(result, STORAGE_PATH_ARCHIVE "/%s", strZ(storageHelper.stanza)); else strCat(result, STORAGE_PATH_ARCHIVE_STR); // If a subpath should be appended, determine if it is WAL path, else just append the subpath if (path != NULL) { StringList *const pathSplit = strLstNewSplitZ(path, "/"); const String *const file = strLstSize(pathSplit) == 2 ? strLstGet(pathSplit, 1) : NULL; if (file != NULL && regExpMatch(storageHelper.walRegExp, file)) strCatFmt(result, "/%s/%.16s/%s", strZ(strLstGet(pathSplit, 0)), strZ(file), strZ(file)); else strCatFmt(result, "/%s", strZ(path)); strLstFree(pathSplit); } } else if (strEq(expression, STORAGE_REPO_BACKUP_STR)) { // Construct the base path if (storageHelper.stanza != NULL) strCatFmt(result, STORAGE_PATH_BACKUP "/%s", strZ(storageHelper.stanza)); else strCatZ(result, STORAGE_PATH_BACKUP); // Append subpath if provided if (path != NULL) strCatFmt(result, "/%s", strZ(path)); } else THROW_FMT(AssertError, "invalid expression '%s'", strZ(expression)); ASSERT(result != NULL); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Get the repo storage ***********************************************************************************************************************************/ static Storage * storageRepoGet(const unsigned int repoIdx, const bool write) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_PARAM(BOOL, write); FUNCTION_TEST_END(); Storage *result = NULL; // Use remote storage if (!repoIsLocal(repoIdx)) { result = storageRemoteNew( STORAGE_MODE_FILE_DEFAULT, STORAGE_MODE_PATH_DEFAULT, write, storageRepoTargetTime(), storageRepoPathExpression, protocolRemoteGet(protocolStorageTypeRepo, repoIdx, true), cfgOptionUInt(cfgOptCompressLevelNetwork)); } // Use local storage else { // Search for the helper const StringId type = cfgOptionIdxStrId(cfgOptRepoType, repoIdx); if (storageHelper.helperList != NULL) { for (const StorageHelper *helper = storageHelper.helperList; helper->type != 0; helper++) { if (helper->type == type) { result = helper->helper(repoIdx, write, storageRepoPathExpression); break; } } } // If no helper was found then try Posix if (result == NULL) { CHECK(AssertError, type == STORAGE_POSIX_TYPE, "invalid storage type"); result = storagePosixNewP( cfgOptionIdxStr(cfgOptRepoPath, repoIdx), .write = write, .pathExpressionFunction = storageRepoPathExpression, .noSymLink = cfgOptionValid(cfgOptRepoSymlink) ? !cfgOptionIdxBool(cfgOptRepoSymlink, repoIdx) : true); } } FUNCTION_TEST_RETURN(STORAGE, result); } /**********************************************************************************************************************************/ FN_EXTERN const Storage * storageRepoIdx(const unsigned int repoIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_END(); if (storageHelper.storageRepo == NULL) { storageHelperContextInit(); storageHelperStanzaInit(false); storageHelperRepoInit(); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageRepo = memNewPtrArray(cfgOptionGroupIdxTotal(cfgOptGrpRepo)); } MEM_CONTEXT_END(); } if (storageHelper.storageRepo[repoIdx] == NULL) { MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageRepo[repoIdx] = storageRepoGet(repoIdx, false); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storageRepo[repoIdx]); } FN_EXTERN const Storage * storageRepo(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STORAGE, storageRepoIdx(cfgOptionGroupIdxDefault(cfgOptGrpRepo))); } FN_EXTERN const Storage * storageRepoIdxWrite(const unsigned int repoIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_END(); // Writes not allowed in dry-run mode if (!storageHelper.dryRunInit || storageHelper.dryRun) THROW(AssertError, WRITABLE_WHILE_DRYRUN); if (storageHelper.storageRepoWrite == NULL) { storageHelperContextInit(); storageHelperStanzaInit(false); storageHelperRepoInit(); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageRepoWrite = memNewPtrArray(cfgOptionGroupIdxTotal(cfgOptGrpRepo)); } MEM_CONTEXT_END(); } if (storageHelper.storageRepoWrite[repoIdx] == NULL) { MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageRepoWrite[repoIdx] = storageRepoGet(repoIdx, true); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storageRepoWrite[repoIdx]); } FN_EXTERN const Storage * storageRepoWrite(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STORAGE, storageRepoIdxWrite(cfgOptionGroupIdxDefault(cfgOptGrpRepo))); } /**********************************************************************************************************************************/ FN_EXTERN time_t storageRepoTargetTime(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(TIME, cfgOptionTest(cfgOptRepoTargetTime) ? cvtZToTime(strZ(cfgOptionStr(cfgOptRepoTargetTime))) : 0); } /*********************************************************************************************************************************** Spool storage path expression ***********************************************************************************************************************************/ static String * storageSpoolPathExpression(const String *const expression, const String *const path) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, expression); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_END(); ASSERT(expression != NULL); ASSERT(storageHelper.stanza != NULL); String *result = NULL; if (strEqZ(expression, STORAGE_SPOOL_ARCHIVE)) { if (path == NULL) result = strNewFmt(STORAGE_PATH_ARCHIVE "/%s", strZ(storageHelper.stanza)); else result = strNewFmt(STORAGE_PATH_ARCHIVE "/%s/%s", strZ(storageHelper.stanza), strZ(path)); } else if (strEqZ(expression, STORAGE_SPOOL_ARCHIVE_IN)) { if (path == NULL) result = strNewFmt(STORAGE_PATH_ARCHIVE "/%s/in", strZ(storageHelper.stanza)); else result = strNewFmt(STORAGE_PATH_ARCHIVE "/%s/in/%s", strZ(storageHelper.stanza), strZ(path)); } else if (strEqZ(expression, STORAGE_SPOOL_ARCHIVE_OUT)) { if (path == NULL) result = strNewFmt(STORAGE_PATH_ARCHIVE "/%s/out", strZ(storageHelper.stanza)); else result = strNewFmt(STORAGE_PATH_ARCHIVE "/%s/out/%s", strZ(storageHelper.stanza), strZ(path)); } else THROW_FMT(AssertError, "invalid expression '%s'", strZ(expression)); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN const Storage * storageSpool(void) { FUNCTION_TEST_VOID(); if (storageHelper.storageSpool == NULL) { storageHelperContextInit(); storageHelperStanzaInit(true); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageSpool = storagePosixNewP( cfgOptionStr(cfgOptSpoolPath), .pathExpressionFunction = storageSpoolPathExpression); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storageSpool); } FN_EXTERN const Storage * storageSpoolWrite(void) { FUNCTION_TEST_VOID(); // Writes not allowed in dry-run mode if (!storageHelper.dryRunInit || storageHelper.dryRun) THROW(AssertError, WRITABLE_WHILE_DRYRUN); if (storageHelper.storageSpoolWrite == NULL) { storageHelperContextInit(); storageHelperStanzaInit(true); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageSpoolWrite = storagePosixNewP( cfgOptionStr(cfgOptSpoolPath), .write = true, .pathExpressionFunction = storageSpoolPathExpression); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storageSpoolWrite); } /**********************************************************************************************************************************/ FN_EXTERN void storageHelperFree(void) { FUNCTION_TEST_VOID(); if (storageHelper.memContext != NULL) memContextFree(storageHelper.memContext); storageHelper = (struct StorageHelperLocal){.memContext = NULL, .helperList = storageHelper.helperList}; FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/helper.h0000664000175100017510000000711715265101745017273 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Helper ***********************************************************************************************************************************/ #ifndef STORAGE_HELPER_H #define STORAGE_HELPER_H #include "storage/storage.h" /*********************************************************************************************************************************** Storage path constants ***********************************************************************************************************************************/ #define STORAGE_SPOOL_ARCHIVE "" STRING_DECLARE(STORAGE_SPOOL_ARCHIVE_STR); #define STORAGE_SPOOL_ARCHIVE_IN "" STRING_DECLARE(STORAGE_SPOOL_ARCHIVE_IN_STR); #define STORAGE_SPOOL_ARCHIVE_OUT "" STRING_DECLARE(STORAGE_SPOOL_ARCHIVE_OUT_STR); #define STORAGE_REPO_ARCHIVE "" STRING_DECLARE(STORAGE_REPO_ARCHIVE_STR); #define STORAGE_REPO_BACKUP "" STRING_DECLARE(STORAGE_REPO_BACKUP_STR); #define STORAGE_PATH_ARCHIVE "archive" STRING_DECLARE(STORAGE_PATH_ARCHIVE_STR); #define STORAGE_PATH_BACKUP "backup" STRING_DECLARE(STORAGE_PATH_BACKUP_STR); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize dry-run for the current command. No writes are allowed until dry-run has been initialized and no writes are allowed // after initialization if dry-run is true. Note that storageLocalWrite() is exempt from this rule. The primary purpose is to // prevent damage to the repository from an error in dry-run coding in the individual commands. FN_EXTERN void storageHelperDryRunInit(bool dryRun); // Initialize helpers to create storage other than built-in Posix FN_EXTERN void storageHelperInit(const StorageHelper *helperList); // Local storage object. Writable local storage should be used very sparingly. If writes are not needed then always use // storageLocal() or a specific storage object instead. FN_EXTERN const Storage *storageLocal(void); FN_EXTERN const Storage *storageLocalWrite(void); // PostgreSQL storage by cfgOptGrpPg index FN_EXTERN const Storage *storagePgIdx(unsigned int pgIdx); FN_EXTERN const Storage *storagePgIdxWrite(unsigned int pgIdx); // PostgreSQL storage default (calculated from the pg option, when set, or the first cfgOptGrpPg index) FN_EXTERN const Storage *storagePg(void); FN_EXTERN const Storage *storagePgWrite(void); // Repository storage by cfgOptGrpRepo index FN_EXTERN const Storage *storageRepoIdx(unsigned int repoIdx); FN_EXTERN const Storage *storageRepoIdxWrite(unsigned int repoIdx); // Repository storage default (calculated from the repo option, when set, or the first cfgOptGrpPg index) FN_EXTERN const Storage *storageRepo(void); FN_EXTERN const Storage *storageRepoWrite(void); // Return target time for repository storage, if any FN_EXTERN time_t storageRepoTargetTime(void); // Spool storage FN_EXTERN const Storage *storageSpool(void); FN_EXTERN const Storage *storageSpoolWrite(void); // Free cached storage objects FN_EXTERN void storageHelperFree(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/info.h0000664000175100017510000001066615265101745016752 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Info ***********************************************************************************************************************************/ #ifndef STORAGE_INFO_H #define STORAGE_INFO_H #include /*********************************************************************************************************************************** Specify the level of information required when calling functions that return StorageInfo Each level includes the level below it, i.e. level storageInfoLevelBasic includes storageInfoLevelType which includes storageInfoLevelExists. ***********************************************************************************************************************************/ typedef enum { // The info type is determined by driver capabilities. This mimics the prior behavior where drivers would always return as much // information as they could. storageInfoLevelDefault = 0, // Only test for existence. All drivers must support this level. storageInfoLevelExists, // Include file type, e.g. storageTypeFile. All drivers must support this level. storageInfoLevelType, // Basic information. All drivers support this level. storageInfoLevelBasic, // Detailed information that is generally only available from filesystems such as Posix storageInfoLevelDetail, } StorageInfoLevel; /*********************************************************************************************************************************** Storage types. The values are used in the remote protocol so must not be changed. ***********************************************************************************************************************************/ typedef enum { storageTypeFile = 0, storageTypePath = 1, storageTypeLink = 2, storageTypeSpecial = 3, } StorageType; /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageInfo { // Set when info type >= storageInfoLevelExists const String *name; // Name of path/file/link StorageInfoLevel level; // Level of information provided bool exists; // Does the path/file/link exist? // Set when info type >= storageInfoLevelType (undefined at lower levels) StorageType type; // Type file/path/link) // Set when info type >= storageInfoLevelBasic (undefined at lower levels) uint64_t size; // Size (path/link is 0) time_t timeModified; // Time file was last modified const String *versionId; // Version id when versioning enabled // Set when info type >= storageInfoLevelDetail (undefined at lower levels) mode_t mode; // Mode of path/file/link uid_t userId; // User that owns the file gid_t groupId; // Group that owns the file const String *user; // Name of user that owns the file const String *group; // Name of group that owns the file const String *linkDestination; // Destination if this is a link } StorageInfo; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_INFO_TYPE \ StorageInfo #define FUNCTION_LOG_STORAGE_INFO_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "StorageInfo", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/iterator.c0000664000175100017510000002640315265101745017637 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Iterator ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/regExp.h" #include "common/type/list.h" #include "storage/iterator.h" #include "storage/list.h" #include "storage/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageIterator { void *driver; // Storage driver const String *path; // Path to iterate StorageInfoLevel level; // Info level bool recurse; // Recurse into paths SortOrder sortOrder; // Sort order time_t targetTime; // List max version <= time const String *expression; // Match expression RegExp *regExp; // Parsed match expression List *stack; // Stack of info lists bool returnedNext; // Next info was returned StorageInfo infoNext; // Info to be returned by next String *nameNext; // Name for next info }; // Path info list typedef struct StorageIteratorInfo { String *pathSub; // Subpath StorageList *list; // Storage info list unsigned int listIdx; // Current index in info list bool pathContentSkip; // Skip reading path content } StorageIteratorInfo; /*********************************************************************************************************************************** Check a path and add it to the stack if it exists and has content ***********************************************************************************************************************************/ typedef struct StorageItrPathAddResult { bool exists; // Does the path exist? bool content; // Does the path have content? } StorageItrPathAddResult; static StorageItrPathAddResult storageItrPathAdd(StorageIterator *const this, const String *const pathSub) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_ITERATOR, this); FUNCTION_LOG_PARAM(STRING, pathSub); FUNCTION_LOG_END(); ASSERT(this != NULL); StorageItrPathAddResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { // Get path content StorageList *const list = storageInterfaceListP( this->driver, pathSub == NULL ? this->path : strNewFmt("%s/%s", strZ(this->path), strZ(pathSub)), this->level, .expression = this->expression, .targetTime = this->targetTime); // If path exists if (list != NULL) { result.exists = true; // If the path has content if (!storageLstEmpty(list)) { result.content = true; // Sort if requested if (this->sortOrder != sortOrderNone) storageLstSort(list, this->sortOrder); // Add path to top of stack MEM_CONTEXT_OBJ_BEGIN(this->stack) { OBJ_NEW_BASE_BEGIN(StorageIteratorInfo, .childQty = MEM_CONTEXT_QTY_MAX) { StorageIteratorInfo *const listInfo = OBJ_NEW_ALLOC(); *listInfo = (StorageIteratorInfo) { .pathSub = strDup(pathSub), .list = storageLstMove(list, memContextCurrent()), }; lstAdd(this->stack, &listInfo); } OBJ_NEW_END(); } MEM_CONTEXT_OBJ_END(); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN StorageIterator * storageItrNew( void *const driver, const String *const path, const StorageInfoLevel level, const bool errorOnMissing, const bool nullOnMissing, const bool recurse, const SortOrder sortOrder, const time_t targetTime, const String *const expression) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(BOOL, errorOnMissing); FUNCTION_LOG_PARAM(BOOL, nullOnMissing); FUNCTION_LOG_PARAM(BOOL, recurse); FUNCTION_LOG_PARAM(ENUM, sortOrder); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(STRING, expression); FUNCTION_LOG_END(); ASSERT(driver != NULL); ASSERT(path != NULL); ASSERT(!recurse || level >= storageInfoLevelType); StorageIterator *this = NULL; MEM_CONTEXT_TEMP_BEGIN() { OBJ_NEW_BASE_BEGIN(StorageIterator, .childQty = MEM_CONTEXT_QTY_MAX) { this = OBJ_NEW_ALLOC(); *this = (StorageIterator) { .driver = driver, .path = strDup(path), .level = level, .recurse = recurse, .sortOrder = sortOrder, .targetTime = targetTime, .expression = strDup(expression), .stack = lstNewP(sizeof(StorageIteratorInfo *)), .nameNext = strNew(), .returnedNext = true, }; // Compile regular expression if (this->expression != NULL) this->regExp = regExpNew(this->expression); // If root path does not exist if (!storageItrPathAdd(this, NULL).exists) { // Throw an error when requested if (errorOnMissing) THROW_FMT(PathMissingError, STORAGE_ERROR_LIST_INFO_MISSING, strZ(this->path)); // Return NULL when requested if (nullOnMissing) this = NULL; } } OBJ_NEW_END(); storageItrMove(this, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_ITERATOR, this); } /**********************************************************************************************************************************/ FN_EXTERN bool storageItrMore(StorageIterator *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_ITERATOR, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // If next has not been called then return true if (!this->returnedNext) FUNCTION_TEST_RETURN(BOOL, true); // Search stack for info while (lstSize(this->stack) != 0) { // Always pull from the top of the stack StorageIteratorInfo *const listInfo = *(StorageIteratorInfo **)lstGetLast(this->stack); // Search list for info for (; listInfo->listIdx < storageLstSize(listInfo->list); listInfo->listIdx++) { // Next info this->infoNext = storageLstGet(listInfo->list, listInfo->listIdx); // Update info name when in subpath if (listInfo->pathSub != NULL) { strCatFmt(strTrunc(this->nameNext), "%s/%s", strZ(listInfo->pathSub), strZ(this->infoNext.name)); this->infoNext.name = this->nameNext; } // Does the path have content? const bool pathContent = this->infoNext.type == storageTypePath && this->recurse && !listInfo->pathContentSkip && storageItrPathAdd(this, this->infoNext.name).content; // Clear path content skip flag if it was set on a previous iteration listInfo->pathContentSkip = false; // Skip info if it does match the provided expression if (this->regExp != NULL && !regExpMatch(this->regExp, this->infoNext.name)) { // Path content may match the expression even if the path does not. Break so the content on the top of the stack // will be checked and increment the index so that path will be skipped. if (pathContent) { listInfo->listIdx++; break; } // Skip info and continue continue; } // When sort order is descending the path will need to be output after the content. Break so that content gets checked // but set a flag so on the next iteration the path will be output but the content skipped. if (pathContent && this->sortOrder == sortOrderDesc) { listInfo->pathContentSkip = true; break; } // Return next info this->returnedNext = false; listInfo->listIdx++; FUNCTION_TEST_RETURN(BOOL, true); } // If no more info then free the list. This check is required because we may break out of the above loop early. // // Only free when the list is at the top of the stack. It is possible that this is the last entry in the current list but // it is a path that must be checked for entries. In that case the path entries will end up at the top of the stack so we'll // need to wait to free the list containing the path until the path entries have been processed. if (listInfo->listIdx >= storageLstSize(listInfo->list) && listInfo == *(StorageIteratorInfo **)lstGetLast(this->stack)) { objFree(listInfo); lstRemoveLast(this->stack); } } FUNCTION_TEST_RETURN(BOOL, false); } /**********************************************************************************************************************************/ FN_EXTERN StorageInfo storageItrNext(StorageIterator *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_ITERATOR, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(!this->returnedNext); this->returnedNext = true; FUNCTION_TEST_RETURN(STORAGE_INFO, this->infoNext); } /**********************************************************************************************************************************/ FN_EXTERN void storageItrToLog(const StorageIterator *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{stack: "); lstToLog(this->stack, debugLog); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/iterator.h0000664000175100017510000000620115265101745017636 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Iterator ***********************************************************************************************************************************/ #ifndef STORAGE_ITERATOR_H #define STORAGE_ITERATOR_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageIterator StorageIterator; #include "common/type/string.h" #include "storage/info.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageIterator *storageItrNew( void *driver, const String *path, StorageInfoLevel level, bool errorOnMissing, bool nullOnMissing, bool recurse, SortOrder sortOrder, time_t targetTime, const String *expression); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Is there more info to be retrieved from the iterator? FN_EXTERN bool storageItrMore(StorageIterator *this); // Move to a new parent mem context FN_INLINE_ALWAYS StorageIterator * storageItrMove(StorageIterator *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Get next info. An error will be thrown if there is no more data so use storageItrMore() to check. Note that StorageInfo pointer // members (e.g. name) will be undefined after the next call to storageItrMore(). FN_EXTERN StorageInfo storageItrNext(StorageIterator *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void storageItrFree(StorageIterator *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void storageItrToLog(const StorageIterator *this, StringStatic *debugLog); #define FUNCTION_LOG_STORAGE_ITERATOR_TYPE \ StorageIterator * #define FUNCTION_LOG_STORAGE_ITERATOR_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, storageItrToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/list.c0000664000175100017510000002364515265101745016766 0ustar00runnerrunner/*********************************************************************************************************************************** Storage List ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/type/blob.h" #include "common/type/list.h" #include "storage/list.h" #include "storage/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageList { StorageListPub pub; // Publicly accessible variables StringList *ownerList; // List of users/groups Blob *blob; // Blob of info data String *name; // Current info name String *linkDestination; // Current link destination String *versionId; // Current version id }; /*********************************************************************************************************************************** A more space efficient version of StorageInfo. Each level is contained in a struct to ensure alignment when only using part of the struct to store info. This also keeps the size calculations accurate if members are added to a level. ***********************************************************************************************************************************/ typedef struct StorageListInfo { // Set when info type >= storageInfoLevelExists struct { const char *name; // Name of path/file/link } exists; // Set when info type >= storageInfoLevelType (undefined at lower levels). Mode is only provided at higher detail levels but // included here to save space on 64-bit architectures struct { // Set when info type >= storageInfoLevelType (undefined at lower levels) StorageType type; // Type file/path/link) // Set when info type >= storageInfoLevelDetail (undefined at lower levels) mode_t mode; // Mode of path/file/link } type; // Set when info type >= storageInfoLevelBasic (undefined at lower levels) struct { uint64_t size; // Size (path/link is 0) time_t timeModified; // Time file was last modified const char *versionId; // Version id when versioning enabled } basic; // Set when info type >= storageInfoLevelDetail (undefined at lower levels) struct { const String *user; // Name of user that owns the file const String *group; // Name of group that owns the file uid_t userId; // User that owns the file uid_t groupId; // Group that owns the file const char *linkDestination; // Destination if this is a link } detail; } StorageListInfo; // Determine size of struct to store in the list. We only need to store the members that are used by the current level. static const uint8_t storageLstInfoSize[storageInfoLevelDetail + 1] = { 0, // storageInfoLevelDefault (should not be used) offsetof(StorageListInfo, type), // storageInfoLevelExists offsetof(StorageListInfo, basic), // storageInfoLevelType offsetof(StorageListInfo, detail), // storageInfoLevelBasic sizeof(StorageListInfo), // storageInfoLevelDetail }; /**********************************************************************************************************************************/ FN_EXTERN StorageList * storageLstNew(const StorageInfoLevel level) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, level); FUNCTION_TEST_END(); ASSERT(level != storageInfoLevelDefault); OBJ_NEW_BEGIN(StorageList, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageList) { .pub = { .list = lstNewP(storageLstInfoSize[level], .comparator = lstComparatorZ), .level = level, }, .ownerList = strLstNew(), .blob = blbNew(), .name = strNew(), .linkDestination = strNew(), .versionId = strNew(), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(STORAGE_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN void storageLstInsert(StorageList *const this, const unsigned int idx, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_LIST, this); FUNCTION_TEST_PARAM(UINT, idx); FUNCTION_TEST_PARAM_P(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(info != NULL); MEM_CONTEXT_OBJ_BEGIN(this->pub.list) { StorageListInfo listInfo = {.exists = {.name = blbAdd(this->blob, strZ(info->name), strSize(info->name) + 1)}}; switch (storageLstLevel(this)) { case storageInfoLevelDetail: { listInfo.type.mode = info->mode; listInfo.detail.user = strLstAddIfMissing(this->ownerList, info->user); listInfo.detail.group = strLstAddIfMissing(this->ownerList, info->group); listInfo.detail.userId = info->userId; listInfo.detail.groupId = info->groupId; if (info->linkDestination != NULL) { listInfo.detail.linkDestination = blbAdd( this->blob, strZ(info->linkDestination), strSize(info->linkDestination) + 1); } __attribute__((fallthrough)); } case storageInfoLevelBasic: { listInfo.basic.size = info->size; listInfo.basic.timeModified = info->timeModified; if (info->versionId != NULL) listInfo.basic.versionId = blbAdd(this->blob, strZ(info->versionId), strSize(info->versionId) + 1); __attribute__((fallthrough)); } case storageInfoLevelType: listInfo.type.type = info->type; __attribute__((fallthrough)); default: break; } lstInsert(this->pub.list, idx, &listInfo); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN StorageInfo storageLstGet(const StorageList *const this, const unsigned int idx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_LIST, this); FUNCTION_TEST_PARAM(UINT, idx); FUNCTION_TEST_END(); ASSERT(this != NULL); const StorageListInfo *const listInfo = lstGet(this->pub.list, idx); StorageInfo result = { .name = strCatZ(strTrunc(this->name), listInfo->exists.name), .exists = true, .level = storageLstLevel(this) }; switch (result.level) { case storageInfoLevelDetail: { result.mode = listInfo->type.mode; result.user = listInfo->detail.user; result.group = listInfo->detail.group; result.userId = listInfo->detail.userId; result.groupId = listInfo->detail.groupId; if (listInfo->detail.linkDestination != NULL) result.linkDestination = strCatZ(strTrunc(this->linkDestination), listInfo->detail.linkDestination); __attribute__((fallthrough)); } case storageInfoLevelBasic: { result.size = listInfo->basic.size; result.timeModified = listInfo->basic.timeModified; if (listInfo->basic.versionId != NULL) result.versionId = strCatZ(strTrunc(this->versionId), listInfo->basic.versionId); __attribute__((fallthrough)); } case storageInfoLevelType: result.type = listInfo->type.type; __attribute__((fallthrough)); default: break; } FUNCTION_TEST_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ FN_EXTERN StorageInfo storageLstFind(const StorageList *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_LIST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); const char *const namePtr = strZ(name); const unsigned int listIdx = lstFindIdx(this->pub.list, &namePtr); if (listIdx == LIST_NOT_FOUND) FUNCTION_TEST_RETURN(STORAGE_INFO, (StorageInfo){.exists = false}); FUNCTION_TEST_RETURN(STORAGE_INFO, storageLstGet(this, listIdx)); } /**********************************************************************************************************************************/ FN_EXTERN void storageLstToLog(const StorageList *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{size: %u}", lstSize(this->pub.list)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/list.h0000664000175100017510000001047615265101745016771 0ustar00runnerrunner/*********************************************************************************************************************************** Storage List ***********************************************************************************************************************************/ #ifndef STORAGE_LIST_H #define STORAGE_LIST_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageList StorageList; #include "common/type/string.h" #include "storage/info.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageList *storageLstNew(StorageInfoLevel level); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct StorageListPub { List *list; // Storage list StorageInfoLevel level; // Storage info level } StorageListPub; // Empty? FN_INLINE_ALWAYS bool storageLstEmpty(const StorageList *const this) { return lstEmpty(THIS_PUB(StorageList)->list); } // Storage info level FN_INLINE_ALWAYS StorageInfoLevel storageLstLevel(const StorageList *const this) { return THIS_PUB(StorageList)->level; } // List size FN_INLINE_ALWAYS unsigned int storageLstSize(const StorageList *const this) { return lstSize(THIS_PUB(StorageList)->list); } // List size FN_INLINE_ALWAYS StorageList * storageLstSort(StorageList *const this, const SortOrder sortOrder) { return (StorageList *)lstSort(THIS_PUB(StorageList)->list, sortOrder); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Insert info FN_EXTERN void storageLstInsert(StorageList *this, unsigned int idx, const StorageInfo *info); // Add info FN_INLINE_ALWAYS void storageLstAdd(StorageList *const this, const StorageInfo *const info) { storageLstInsert(this, storageLstSize(this), info); } // Get info. Note that StorageInfo pointer members (e.g. name) will be undefined after the next call to storageLstGet(). FN_EXTERN StorageInfo storageLstGet(const StorageList *this, unsigned int idx); // Find info. Note that StorageInfo pointer members (e.g. name) will be undefined after the next call to storageLstGet(). FN_EXTERN StorageInfo storageLstFind(const StorageList *this, const String *name); // Move to a new parent mem context FN_INLINE_ALWAYS StorageList * storageLstMove(StorageList *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void storageLstFree(StorageList *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void storageLstToLog(const StorageList *this, StringStatic *debugLog); #define FUNCTION_LOG_STORAGE_LIST_TYPE \ StorageList * #define FUNCTION_LOG_STORAGE_LIST_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, storageLstToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/posix/0000775000175100017510000000000015265101745016777 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/posix/read.c0000664000175100017510000002125515265101745020063 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage Read ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/io/read.h" #include "common/log.h" #include "common/type/object.h" #include "storage/posix/read.h" #include "storage/read.h" /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ struct StorageReadPosix { const StorageReadInterface *interface; // Interface StoragePosix *storage; // Storage that created this object const String *name; // File name uint64_t offset; // Read offset const Variant *limit; // Read limit (NULL for no limit) int fd; // File descriptor uint64_t current; // Current bytes read from file bool eof; }; /*********************************************************************************************************************************** Close file descriptor ***********************************************************************************************************************************/ static void storageReadPosixFreeResource(THIS_VOID) { THIS(StorageReadPosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_POSIX, this); FUNCTION_LOG_END(); ASSERT(this != NULL); if (this->fd != -1) THROW_ON_SYS_ERROR_FMT(close(this->fd) == -1, FileCloseError, STORAGE_ERROR_READ_CLOSE, strZ(this->name)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadPosixOpen(THIS_VOID) { THIS(StorageReadPosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_POSIX, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->fd == -1); // Open the file this->fd = open(strZ(this->name), O_RDONLY, 0); // Handle errors except missing, which is reported to the caller via the result if (this->fd == -1) { if (errno != ENOENT) // {vm_covered} THROW_SYS_ERROR_FMT(FileOpenError, STORAGE_ERROR_READ_OPEN, strZ(this->name)); // {vm_covered} } // Else success else { // Set free callback to ensure the file descriptor is freed memContextCallbackSet(objMemContext(this), storageReadPosixFreeResource, this); // Seek to offset if (this->offset != 0) { THROW_ON_SYS_ERROR_FMT( lseek(this->fd, (off_t)this->offset, SEEK_SET) == -1, FileOpenError, STORAGE_ERROR_READ_SEEK, this->offset, strZ(this->name)); } } FUNCTION_LOG_RETURN(BOOL, this->fd != -1); } /*********************************************************************************************************************************** Read from a file ***********************************************************************************************************************************/ static size_t storageReadPosix(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadPosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_POSIX, this); FUNCTION_LOG_PARAM(BUFFER, buffer); (void)block; // Blocking is handled by the caller FUNCTION_LOG_END(); ASSERT(this != NULL && this->fd != -1); ASSERT(buffer != NULL && !bufFull(buffer)); // Read if EOF has not been reached ssize_t actualBytes = 0; if (!this->eof) { // Rather than enable/disable limit checking just use a big number when there is no limit. We can feel pretty confident that // no files will be > UINT64_MAX in size. const uint64_t limit = this->limit == NULL ? UINT64_MAX : varUInt64(this->limit); // Determine expected bytes to read. If remaining size in the buffer would exceed the limit then reduce the expected read. size_t expectedBytes = bufRemains(buffer); if (this->current + expectedBytes > limit) expectedBytes = (size_t)(limit - this->current); // Read from file actualBytes = read(this->fd, bufRemainsPtr(buffer), expectedBytes); // Error occurred during read if (actualBytes == -1) THROW_SYS_ERROR_FMT(FileReadError, "unable to read '%s'", strZ(this->name)); // Update amount of buffer used bufUsedInc(buffer, (size_t)actualBytes); this->current += (uint64_t)actualBytes; // If less data than expected was read or the limit has been reached then EOF. The file may not actually be EOF but we are // not concerned with files that are growing. Just read up to the point where the file is being extended. if ((size_t)actualBytes != expectedBytes || this->current == limit) this->eof = true; } FUNCTION_LOG_RETURN(SIZE, (size_t)actualBytes); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageReadPosixClose(THIS_VOID) { THIS(StorageReadPosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_POSIX, this); FUNCTION_LOG_END(); ASSERT(this != NULL); memContextCallbackClear(objMemContext(this)); storageReadPosixFreeResource(this); this->fd = -1; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadPosixEof(THIS_VOID) { THIS(StorageReadPosix); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_POSIX, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->eof); } /*********************************************************************************************************************************** Get file descriptor ***********************************************************************************************************************************/ static int storageReadPosixFd(const THIS_VOID) { THIS(const StorageReadPosix); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_POSIX, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->fd); } /**********************************************************************************************************************************/ static const StorageReadInterface storageReadPosixInterface = { .close = storageReadPosixClose, .eof = storageReadPosixEof, .fd = storageReadPosixFd, .open = storageReadPosixOpen, .read = storageReadPosix, }; FN_EXTERN StorageReadPosix * storageReadPosixNew(StoragePosix *const storage, const String *const name, const uint64_t offset, const Variant *const limit) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_END(); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageReadPosix, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (StorageReadPosix) { .interface = &storageReadPosixInterface, .storage = storage, .name = strDup(name), .offset = offset, .limit = varDup(limit), .fd = -1, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_POSIX, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/posix/read.h0000664000175100017510000000342015265101745020062 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage Read ***********************************************************************************************************************************/ #ifndef STORAGE_POSIX_READ_H #define STORAGE_POSIX_READ_H #include "storage/posix/storage.intern.h" #include "storage/read.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadPosix StorageReadPosix; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadPosix *storageReadPosixNew(StoragePosix *storage, const String *name, uint64_t offset, const Variant *limit); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_POSIX_TYPE \ StorageReadPosix * #define FUNCTION_LOG_STORAGE_READ_POSIX_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadPosix", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/posix/storage.c0000664000175100017510000006054715265101745020623 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include #include #include "common/debug.h" #include "common/log.h" #include "common/regExp.h" #include "common/user.h" #include "storage/posix/read.h" #include "storage/posix/storage.intern.h" #include "storage/posix/write.h" /*********************************************************************************************************************************** Define PATH_MAX if it is not defined ***********************************************************************************************************************************/ #ifndef PATH_MAX #define PATH_MAX (4 * 1024) #endif /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StoragePosix { STORAGE_COMMON_MEMBER; }; /**********************************************************************************************************************************/ static StorageInfo storagePosixInfo(THIS_VOID, const String *const file, const StorageInfoLevel level, const StorageInterfaceInfoParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); (void)this; FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(BOOL, param.followLink); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); ASSERT(file != NULL); StorageInfo result = {.level = level}; // Stat the file to check if it exists struct stat statFile; if ((param.followLink ? stat(strZ(file), &statFile) : lstat(strZ(file), &statFile)) == -1) { if (errno != ENOENT) // {vm_covered} THROW_SYS_ERROR_FMT(FileOpenError, STORAGE_ERROR_INFO, strZ(file)); // {vm_covered} } // On success the file exists else { result.exists = true; // Add type info (no need set file type since it is the default) if (result.level >= storageInfoLevelType && !S_ISREG(statFile.st_mode)) { if (S_ISDIR(statFile.st_mode)) result.type = storageTypePath; else if (S_ISLNK(statFile.st_mode)) result.type = storageTypeLink; else result.type = storageTypeSpecial; } // Add basic level info if (result.level >= storageInfoLevelBasic) { result.timeModified = statFile.st_mtime; if (result.type == storageTypeFile) result.size = (uint64_t)statFile.st_size; } // Add detail level info if (result.level >= storageInfoLevelDetail) { result.groupId = statFile.st_gid; result.group = groupNameFromId(result.groupId); result.userId = statFile.st_uid; result.user = userNameFromId(result.userId); result.mode = statFile.st_mode & (S_IRWXU | S_IRWXG | S_IRWXO); if (result.type == storageTypeLink) { char linkDestination[PATH_MAX]; ssize_t linkDestinationSize = 0; THROW_ON_SYS_ERROR_FMT( (linkDestinationSize = readlink(strZ(file), linkDestination, sizeof(linkDestination) - 1)) == -1, FileReadError, "unable to get destination for link '%s'", strZ(file)); result.linkDestination = strNewZN(linkDestination, (size_t)linkDestinationSize); } } } FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ static void storagePosixLinkCreate( THIS_VOID, const String *const target, const String *const linkPath, const StorageInterfaceLinkCreateParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); (void)this; FUNCTION_LOG_PARAM(STRING, target); FUNCTION_LOG_PARAM(STRING, linkPath); FUNCTION_LOG_PARAM(ENUM, param.linkType); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(target != NULL); ASSERT(linkPath != NULL); // Create symlink if (param.linkType == storageLinkSym) { THROW_ON_SYS_ERROR_FMT( symlink(strZ(target), strZ(linkPath)) == -1, FileOpenError, "unable to create symlink '%s' to '%s'", strZ(linkPath), strZ(target)); } // Else create hardlink else { ASSERT(param.linkType == storageLinkHard); THROW_ON_SYS_ERROR_FMT( link(strZ(target), strZ(linkPath)) == -1, FileOpenError, "unable to create hardlink '%s' to '%s'", strZ(linkPath), strZ(target)); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ // Helper function to get info for a file if it exists. This logic can't live directly in storagePosixList() because there is a race // condition where a file might exist while listing the directory but it is gone before stat() can be called. In order to get // complete test coverage this function must be split out. static void storagePosixListEntry( StoragePosix *const this, StorageList *const list, const String *const path, const char *const name, const StorageInfoLevel level) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_POSIX, this); FUNCTION_TEST_PARAM(STORAGE_LIST, list); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(STRINGZ, name); FUNCTION_TEST_PARAM(ENUM, level); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(this != NULL); ASSERT(list != NULL); ASSERT(path != NULL); ASSERT(name != NULL); StorageInfo info = storageInterfaceInfoP(this, strNewFmt("%s/%s", strZ(path), name), level); if (info.exists) { info.name = STR(name); storageLstAdd(list, &info); } FUNCTION_TEST_RETURN_VOID(); } static StorageList * storagePosixList(THIS_VOID, const String *const path, const StorageInfoLevel level, const StorageInterfaceListParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); ASSERT(param.targetTime == 0); StorageList *result = NULL; // Open the directory for read DIR *const dir = opendir(strZ(path)); // If the directory could not be opened process errors and report missing directories if (dir == NULL) { if (errno != ENOENT) // {vm_covered} THROW_SYS_ERROR_FMT(PathOpenError, STORAGE_ERROR_LIST_INFO, strZ(path)); // {vm_covered} } // Directory was found else { result = storageLstNew(level); TRY_BEGIN() { MEM_CONTEXT_TEMP_RESET_BEGIN() { // Read the directory entries const struct dirent *dirEntry = readdir(dir); while (dirEntry != NULL) { // Always skip . and .. if (!strEqZ(DOT_STR, dirEntry->d_name) && !strEqZ(DOTDOT_STR, dirEntry->d_name)) { // If only making a list of files that exist then no need to go get detailed info which requires calling // stat() and is therefore relatively slow if (level == storageInfoLevelExists) { const StorageInfo storageInfo = { .name = STR(dirEntry->d_name), .level = storageInfoLevelExists, .exists = true, }; storageLstAdd(result, &storageInfo); } // Else more info is required which requires a call to stat() else storagePosixListEntry(this, result, path, dirEntry->d_name, level); } // Get next entry dirEntry = readdir(dir); // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } FINALLY() { closedir(dir); } TRY_END(); } FUNCTION_LOG_RETURN(STORAGE_LIST, result); } /**********************************************************************************************************************************/ static bool storagePosixMove(THIS_VOID, StorageRead *const source, StorageWrite *const destination, const StorageInterfaceMoveParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STORAGE_READ, source); FUNCTION_LOG_PARAM(STORAGE_WRITE, destination); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(source != NULL); ASSERT(destination != NULL); bool result = true; MEM_CONTEXT_TEMP_BEGIN() { const String *const sourceFile = storageReadName(source); const String *const destinationFile = storageWriteName(destination); const String *const destinationPath = strPath(destinationFile); // Attempt to move the file if (rename(strZ(sourceFile), strZ(destinationFile)) == -1) { // Determine which file/path is missing if (errno == ENOENT) // {vm_covered} { // Check if the source is missing. Rename does not follow links so there is no need to set followLink. if (!storageInterfaceInfoP(this, sourceFile, storageInfoLevelExists).exists) THROW_SYS_ERROR_FMT(FileMissingError, "unable to move missing source '%s'", strZ(sourceFile)); if (!storageWriteCreatePath(destination)) { THROW_SYS_ERROR_FMT( PathMissingError, "unable to move '%s' to missing path '%s'", strZ(sourceFile), strZ(destinationPath)); } storageInterfacePathCreateP(this, destinationPath, false, false, storageWriteModePath(destination)); result = storageInterfaceMoveP(this, source, destination); } // Else the destination is on a different device so a copy will be needed else if (errno == EXDEV) // {vm_covered} { result = false; // {vm_covered} } else { THROW_SYS_ERROR_FMT( // {vm_covered} FileMoveError, "unable to move '%s' to '%s'", strZ(sourceFile), strZ(destinationFile)); // {vm_covered} } } // Sync paths on success else { // Sync source path if the destination path was synced and the paths are not equal if (storageWriteSyncPath(destination)) { const String *const sourcePath = strPath(sourceFile); if (!strEq(destinationPath, sourcePath)) storageInterfacePathSyncP(this, sourcePath); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ static void * storagePosixNewRead(THIS_VOID, const String *const file, const StorageInterfaceNewReadParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.versionId == NULL); FUNCTION_LOG_RETURN(STORAGE_READ_POSIX, storageReadPosixNew(this, file, param.offset, param.limit)); } /**********************************************************************************************************************************/ static void * storagePosixNewWrite(THIS_VOID, const String *const file, const StorageInterfaceNewWriteParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(MODE, param.modeFile); FUNCTION_LOG_PARAM(MODE, param.modePath); FUNCTION_LOG_PARAM(STRING, param.user); FUNCTION_LOG_PARAM(STRING, param.group); FUNCTION_LOG_PARAM(TIME, param.timeModified); FUNCTION_LOG_PARAM(BOOL, param.createPath); FUNCTION_LOG_PARAM(BOOL, param.syncFile); FUNCTION_LOG_PARAM(BOOL, param.syncPath); FUNCTION_LOG_PARAM(BOOL, param.atomic); FUNCTION_LOG_PARAM(BOOL, param.truncate); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); FUNCTION_LOG_RETURN( STORAGE_WRITE_POSIX, storageWritePosixNew( this, file, param.modeFile, param.modePath, param.user, param.group, param.timeModified, param.createPath, param.syncFile, this->interface.pathSync != NULL ? param.syncPath : false, param.atomic, param.truncate)); } /**********************************************************************************************************************************/ static void storagePosixPathCreate( THIS_VOID, const String *const path, const bool errorOnExists, const bool noParentCreate, const mode_t mode, const StorageInterfacePathCreateParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, errorOnExists); FUNCTION_LOG_PARAM(BOOL, noParentCreate); FUNCTION_LOG_PARAM(MODE, mode); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); // Attempt to create the directory if (mkdir(strZ(path), mode) == -1) { // If the parent path does not exist then create it if allowed if (errno == ENOENT && !noParentCreate) { String *const pathParent = strPath(path); storageInterfacePathCreateP(this, pathParent, errorOnExists, noParentCreate, mode); storageInterfacePathCreateP(this, path, errorOnExists, noParentCreate, mode); strFree(pathParent); } // Ignore path exists if allowed else if (errno != EEXIST || errorOnExists) THROW_SYS_ERROR_FMT(PathCreateError, "unable to create path '%s'", strZ(path)); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static bool storagePosixPathRemove(THIS_VOID, const String *const path, const bool recurse, const StorageInterfacePathRemoveParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, recurse); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); bool result = true; MEM_CONTEXT_TEMP_BEGIN() { // Recurse if requested if (recurse) { StorageList *const list = storageInterfaceListP(this, path, storageInfoLevelExists); if (list != NULL) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int listIdx = 0; listIdx < storageLstSize(list); listIdx++) { const String *const file = strNewFmt("%s/%s", strZ(path), strZ(storageLstGet(list, listIdx).name)); // Rather than stat the file to discover what type it is, just try to unlink it and see what happens if (unlink(strZ(file)) == -1) // {vm_covered} { // These errors indicate that the entry is actually a path so we'll try to delete it that way if (errno == EPERM || errno == EISDIR) // {uncovered_branch - no EPERM on tested systems} { storageInterfacePathRemoveP(this, file, true); } // Else error else THROW_SYS_ERROR_FMT(PathRemoveError, STORAGE_ERROR_FILE_REMOVE, strZ(file)); // {vm_covered} } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } } // Delete the path if (rmdir(strZ(path)) == -1) { if (errno != ENOENT) // {vm_covered} THROW_SYS_ERROR_FMT(PathRemoveError, STORAGE_ERROR_PATH_REMOVE, strZ(path)); // {vm_covered} // Path does not exist result = false; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ static void storagePosixPathSync(THIS_VOID, const String *const path, const StorageInterfacePathSyncParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); (void)this; FUNCTION_LOG_PARAM(STRING, path); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); // Open directory and handle errors const int fd = open(strZ(path), O_RDONLY, 0); // Handle errors if (fd == -1) { if (errno == ENOENT) // {vm_covered} THROW_FMT(PathMissingError, STORAGE_ERROR_PATH_SYNC_MISSING, strZ(path)); else THROW_SYS_ERROR_FMT(PathOpenError, STORAGE_ERROR_PATH_SYNC_OPEN, strZ(path)); // {vm_covered} } else { // Attempt to sync the directory if (fsync(fd) == -1) { const int errNo = errno; // Close the file descriptor to free resources but don't check for failure close(fd); THROW_SYS_ERROR_CODE_FMT(errNo, PathSyncError, STORAGE_ERROR_PATH_SYNC, strZ(path)); } THROW_ON_SYS_ERROR_FMT(close(fd) == -1, PathCloseError, STORAGE_ERROR_PATH_SYNC_CLOSE, strZ(path)); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static void storagePosixRemove(THIS_VOID, const String *const file, const StorageInterfaceRemoveParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); (void)this; FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(BOOL, param.errorOnMissing); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); // Attempt to unlink the file if (unlink(strZ(file)) == -1) { if (param.errorOnMissing || errno != ENOENT) // {vm_covered} THROW_SYS_ERROR_FMT(FileRemoveError, STORAGE_ERROR_FILE_REMOVE, strZ(file)); // {vm_covered} } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageInterface storageInterfacePosix = { .feature = 1 << storageFeaturePath, .info = storagePosixInfo, .linkCreate = storagePosixLinkCreate, .list = storagePosixList, .move = storagePosixMove, .newRead = storagePosixNewRead, .newWrite = storagePosixNewWrite, .pathCreate = storagePosixPathCreate, .pathRemove = storagePosixPathRemove, .pathSync = storagePosixPathSync, .remove = storagePosixRemove, }; FN_EXTERN Storage * storagePosixNewInternal( const StringId type, const String *const path, const mode_t modeFile, const mode_t modePath, const bool write, StoragePathExpressionCallback pathExpressionFunction, const bool pathSync, const bool symLink) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_PARAM(BOOL, pathSync); FUNCTION_LOG_PARAM(BOOL, symLink); FUNCTION_LOG_END(); ASSERT(type != 0); ASSERT(path != NULL); ASSERT(modeFile != 0); ASSERT(modePath != 0); // Initialize user module userInit(); // Create the object OBJ_NEW_BEGIN(StoragePosix, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StoragePosix) { .interface = storageInterfacePosix, }; // Disable path sync when not supported if (!pathSync) this->interface.pathSync = NULL; // If this is a posix driver then add link features if (type == STORAGE_POSIX_TYPE) { this->interface.feature |= 1 << storageFeatureHardLink | (unsigned int)symLink << storageFeatureSymLink | 1 << storageFeaturePathSync | 1 << storageFeatureInfoDetail | 1 << storageFeatureFileRemoveMissing; } } OBJ_NEW_END(); FUNCTION_LOG_RETURN( STORAGE, storageNew(type, path, modeFile, modePath, write, 0, pathExpressionFunction, this, this->interface)); } FN_EXTERN Storage * storagePosixNew(const String *const path, const StoragePosixNewParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(MODE, param.modeFile); FUNCTION_LOG_PARAM(MODE, param.modePath); FUNCTION_LOG_PARAM(BOOL, param.write); FUNCTION_LOG_PARAM(BOOL, param.noSymLink); FUNCTION_LOG_PARAM(FUNCTIONP, param.pathExpressionFunction); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN( STORAGE, storagePosixNewInternal( STORAGE_POSIX_TYPE, path, param.modeFile == 0 ? STORAGE_MODE_FILE_DEFAULT : param.modeFile, param.modePath == 0 ? STORAGE_MODE_PATH_DEFAULT : param.modePath, param.write, param.pathExpressionFunction, true, !param.noSymLink)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/posix/storage.h0000664000175100017510000000315715265101745020622 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage ***********************************************************************************************************************************/ #ifndef STORAGE_POSIX_STORAGE_H #define STORAGE_POSIX_STORAGE_H #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_POSIX_TYPE STRID5("posix", 0x184cdf00) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct StoragePosixNewParam { VAR_PARAM_HEADER; bool write; bool noSymLink; // Do not create symlinks on this storage mode_t modeFile; mode_t modePath; StoragePathExpressionCallback *pathExpressionFunction; } StoragePosixNewParam; #define storagePosixNewP(path, ...) \ storagePosixNew(path, (StoragePosixNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN Storage *storagePosixNew(const String *path, StoragePosixNewParam param); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/posix/storage.intern.h0000664000175100017510000000351715265101745022120 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage Internal ***********************************************************************************************************************************/ #ifndef STORAGE_POSIX_STORAGE_INTERN_H #define STORAGE_POSIX_STORAGE_INTERN_H #include "storage/posix/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StoragePosix StoragePosix; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Storage *storagePosixNewInternal( StringId type, const String *path, mode_t modeFile, mode_t modePath, bool write, StoragePathExpressionCallback pathExpressionFunction, bool pathSync, bool symLink); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_POSIX_TYPE \ StoragePosix * #define FUNCTION_LOG_STORAGE_POSIX_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StoragePosix *", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/posix/write.c0000664000175100017510000003173415265101745020305 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage File write ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/debug.h" #include "common/io/write.h" #include "common/log.h" #include "common/type/object.h" #include "common/user.h" #include "storage/posix/write.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWritePosix { const StorageWriteInterface *interface; // Interface StoragePosix *storage; // Storage that created this object const String *name; // File name mode_t modeFile; // File mode mode_t modePath; // Path mode bool createPath; // Create path if missing bool atomic; // Atomic write bool truncate; // Truncate existing file bool syncPath; // Sync path after write bool syncFile; // Sync file after write const String *user; // User owner const String *group; // Group owner time_t timeModified; // Modified time const String *nameTmp; const String *path; int fd; // File descriptor }; /*********************************************************************************************************************************** File open constants Since open is called more than once use constants to make sure these parameters are always the same ***********************************************************************************************************************************/ #define FILE_OPEN_PURPOSE "write" /*********************************************************************************************************************************** Close file descriptor ***********************************************************************************************************************************/ static void storageWritePosixFreeResource(THIS_VOID) { THIS(StorageWritePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_POSIX, this); FUNCTION_LOG_END(); ASSERT(this != NULL); THROW_ON_SYS_ERROR_FMT(close(this->fd) == -1, FileCloseError, STORAGE_ERROR_WRITE_CLOSE, strZ(this->nameTmp)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static const char * storageWritePosixOpenOwner(const String *const owner, const char *const defaultOwner) { return owner == NULL ? defaultOwner : strZ(owner); } static const char * storageWritePosixOpenOwnerId(const String *const owner, const unsigned int ownerId) { return owner == NULL ? "" : zNewFmt("[%u]", ownerId); } static void storageWritePosixOpen(THIS_VOID) { THIS(StorageWritePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_POSIX, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->fd == -1); // Open the file const int flags = O_CREAT | O_WRONLY | (this->truncate ? O_TRUNC : 0); this->fd = open(strZ(this->nameTmp), flags, this->modeFile); // Attempt to create the path if it is missing if (this->fd == -1 && errno == ENOENT && this->createPath) // {vm_covered} { // Create the path storageInterfacePathCreateP(this->storage, this->path, false, false, this->modePath); // Open file again this->fd = open(strZ(this->nameTmp), flags, this->modeFile); } // Handle errors if (this->fd == -1) { if (errno == ENOENT) // {vm_covered} THROW_FMT(FileMissingError, STORAGE_ERROR_WRITE_MISSING, strZ(this->name)); else THROW_SYS_ERROR_FMT(FileOpenError, STORAGE_ERROR_WRITE_OPEN, strZ(this->name)); // {vm_covered} } // Set free callback to ensure the file descriptor is freed memContextCallbackSet(objMemContext(this), storageWritePosixFreeResource, this); // Update user/group owner if (this->user != NULL || this->group != NULL) { MEM_CONTEXT_TEMP_BEGIN() { const StorageInfo info = storageInterfaceInfoP( this->storage, this->nameTmp, storageInfoLevelDetail, .followLink = true); ASSERT(info.exists); uid_t updateUserId = userIdFromName(this->user); gid_t updateGroupId = groupIdFromName(this->group); if (updateUserId == (uid_t)-1) updateUserId = info.userId; if (updateGroupId == (gid_t)-1) updateGroupId = info.groupId; // Continue if one of the owners would be changed if (updateUserId != info.userId || updateGroupId != info.groupId) { THROW_ON_SYS_ERROR_FMT( chown(strZ(this->nameTmp), updateUserId, updateGroupId) == -1, FileOwnerError, "unable to set ownership for '%s' to %s%s:%s%s from %s[%u]:%s[%u]", strZ(this->nameTmp), storageWritePosixOpenOwner(this->user, "[none]"), storageWritePosixOpenOwnerId(this->user, updateUserId), storageWritePosixOpenOwner(this->group, "[none]"), storageWritePosixOpenOwnerId(this->group, updateGroupId), storageWritePosixOpenOwner(info.user, "[unknown]"), info.userId, storageWritePosixOpenOwner(info.group, "[unknown]"), info.groupId); } } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to the file ***********************************************************************************************************************************/ static void storageWritePosix(THIS_VOID, const Buffer *const buffer) { THIS(StorageWritePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_POSIX, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(this->fd != -1); // Write the data if (write(this->fd, bufPtrConst(buffer), bufUsed(buffer)) != (ssize_t)bufUsed(buffer)) THROW_SYS_ERROR_FMT(FileWriteError, "unable to write '%s'", strZ(this->nameTmp)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Seek to specified position relative to beginning of file ***********************************************************************************************************************************/ static void storageWritePosixSeek(THIS_VOID, const uint64_t position) { THIS(StorageWritePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_POSIX, this); FUNCTION_LOG_PARAM(UINT64, position); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->fd != -1); THROW_ON_SYS_ERROR_FMT( lseek(this->fd, (off_t)position, SEEK_SET) == -1, FileWriteError, STORAGE_ERROR_WRITE_SEEK, position, strZ(this->nameTmp)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWritePosixClose(THIS_VOID) { THIS(StorageWritePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_POSIX, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close if the file has not already been closed if (this->fd != -1) { // Sync the file if (this->syncFile) THROW_ON_SYS_ERROR_FMT(fsync(this->fd) == -1, FileSyncError, STORAGE_ERROR_WRITE_SYNC, strZ(this->nameTmp)); // Close the file memContextCallbackClear(objMemContext(this)); THROW_ON_SYS_ERROR_FMT(close(this->fd) == -1, FileCloseError, STORAGE_ERROR_WRITE_CLOSE, strZ(this->nameTmp)); this->fd = -1; // Update modified time if (this->timeModified != 0) { THROW_ON_SYS_ERROR_FMT( utime( strZ(this->nameTmp), &((struct utimbuf){.actime = this->timeModified, .modtime = this->timeModified})) == -1, FileInfoError, "unable to set time for '%s'", strZ(this->nameTmp)); } // Rename from temp file if (this->atomic) { if (rename(strZ(this->nameTmp), strZ(this->name)) == -1) THROW_SYS_ERROR_FMT(FileMoveError, "unable to move '%s' to '%s'", strZ(this->nameTmp), strZ(this->name)); } // Sync the path if (this->syncPath) storageInterfacePathSyncP(this->storage, this->path); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get file descriptor ***********************************************************************************************************************************/ static int storageWritePosixFd(const THIS_VOID) { THIS(const StorageWritePosix); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_WRITE_POSIX, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->fd); } /**********************************************************************************************************************************/ static const StorageWriteInterface storageWritePosixInterface = { .close = storageWritePosixClose, .fd = storageWritePosixFd, .open = storageWritePosixOpen, .seek = storageWritePosixSeek, .write = storageWritePosix, }; FN_EXTERN StorageWritePosix * storageWritePosixNew( StoragePosix *const storage, const String *const name, const mode_t modeFile, const mode_t modePath, const String *const user, const String *const group, const time_t timeModified, const bool createPath, const bool syncFile, const bool syncPath, const bool atomic, const bool truncate) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(STRING, user); FUNCTION_LOG_PARAM(STRING, group); FUNCTION_LOG_PARAM(TIME, timeModified); FUNCTION_LOG_PARAM(BOOL, createPath); FUNCTION_LOG_PARAM(BOOL, syncFile); FUNCTION_LOG_PARAM(BOOL, syncPath); FUNCTION_LOG_PARAM(BOOL, atomic); FUNCTION_LOG_PARAM(BOOL, truncate); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); ASSERT(modeFile != 0); ASSERT(modePath != 0); OBJ_NEW_BEGIN(StorageWritePosix, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (StorageWritePosix) { .interface = &storageWritePosixInterface, .storage = storage, .name = strDup(name), .modeFile = modeFile, .modePath = modePath, .createPath = createPath, .atomic = atomic, .truncate = truncate, .syncFile = syncFile, .syncPath = syncPath, .user = strDup(user), .group = strDup(group), .timeModified = timeModified, .nameTmp = atomic ? strNewFmt("%s." STORAGE_FILE_TEMP_EXT, strZ(name)) : strDup(name), .path = strPath(name), .fd = -1, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE_POSIX, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/posix/write.h0000664000175100017510000000365415265101745020312 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage File write ***********************************************************************************************************************************/ #ifndef STORAGE_POSIX_WRITE_H #define STORAGE_POSIX_WRITE_H #include "storage/posix/storage.intern.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWritePosix StorageWritePosix; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWritePosix *storageWritePosixNew( StoragePosix *storage, const String *name, mode_t modeFile, mode_t modePath, const String *user, const String *group, time_t timeModified, bool createPath, bool syncFile, bool syncPath, bool atomic, bool truncate); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_WRITE_POSIX_TYPE \ StorageWritePosix * #define FUNCTION_LOG_STORAGE_WRITE_POSIX_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageWritePosix", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/read.c0000664000175100017510000003371615265101745016726 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Read Interface ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/wait.h" #include "storage/read.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageRead { StorageReadPub pub; // Publicly accessible variables const Storage *storage; // Storage void *driver; // Driver uint64_t bytesRead; // Bytes that have been successfully read bool retry; // Are read retries allowed? bool compressible; // Is the read compressible? bool async; // Async (deferred) open result enabled bool openAsync; // Async open pending completion before first read }; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_INTERFACE_TYPE \ StorageReadInterface #define FUNCTION_LOG_STORAGE_READ_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "StorageReadInterface", buffer, bufferSize) /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadOpen(THIS_VOID) { THIS(StorageRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = false; // Open if not versioned or if versionId is not null if (!this->pub.version || this->pub.versionId != NULL) { result = storageReadDriverInterface(this->driver)->open(this->driver); // If the driver open was async then it is not yet known if the file exists if (result && storageReadDriverInterface(this->driver)->openAsync != NULL) { // Defer completing the open until the first read so that multiple opens can be in flight at once, but only when async // is enabled (multi-read prefetch). Otherwise complete the open now so existence is known as soon as the read is // opened, which is what a single read expects. Async is never combined with ignore missing, see storageReadNew(). if (this->async) this->openAsync = true; else result = storageReadDriverInterface(this->driver)->openAsync(this->driver); } } // Error when the file is missing and missing files are not ignored if (!result && !this->pub.ignoreMissing) THROW_FMT(FileMissingError, STORAGE_ERROR_READ_MISSING, strZ(storageReadName(this))); FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Read from a file and retry when there is a read failure ***********************************************************************************************************************************/ static size_t storageRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); const size_t bufUsedBegin = bufUsed(buffer); size_t result = 0; MEM_CONTEXT_TEMP_BEGIN() { // Use a specific number of tries here instead of a timeout because the underlying operations already have timeouts and // failures will generally happen after some delay, so it is not clear what timeout would be appropriate here. unsigned int try = this->retry ? 3 : 1; // While tries remaining while (try > 0) { TRY_BEGIN() { // Complete async open before first read if (this->openAsync) { if (!storageReadDriverInterface(this->driver)->openAsync(this->driver)) THROW_FMT(FileMissingError, STORAGE_ERROR_READ_MISSING, strZ(storageReadName(this))); this->openAsync = false; } storageReadDriverInterface(this->driver)->read(this->driver, buffer, block); // Account for bytes that have been read result += bufUsed(buffer) - bufUsedBegin; this->bytesRead += bufUsed(buffer) - bufUsedBegin; // Set try to 1 to exit the loop try = 1; } CATCH_ANY() { // If there is another try remaining then close the file and reopen it to the new position, taking into account any // bytes that have already been read. Do not retry when the file is missing. if (try > 1 && !errorInstanceOf(&FileMissingError)) { // Close the file storageReadDriverInterface(this->driver)->close(this->driver); // Ignore partial reads and restart from the last successful read bufUsedSet(buffer, bufUsedBegin); // Driver with new offset/limit MEM_CONTEXT_OBJ_BEGIN(this) { objFree(this->driver); this->driver = storageInterfaceNewReadP( storageDriver(this->storage), this->pub.name, .compressible = this->compressible, .offset = this->pub.offset + this->bytesRead, .limit = this->pub.limit != NULL ? varNewUInt64(varUInt64(this->pub.limit) - this->bytesRead) : NULL, .versionId = this->pub.versionId); // The filter group is never set on the recreated driver, so retry must already be disabled for any driver // that requires one (see storageReadNew()) ASSERT(storageReadDriverInterface(this->driver)->filterGroup == NULL); // Open should always return true either because a) the file still exists or b) the driver is async and // existence will be checked later. This is a CHECK() because currently there are not drivers that have // retry without async. CHECK_FMT( FileMissingError, storageReadDriverInterface(this->driver)->open(this->driver), STORAGE_ERROR_READ_MISSING, strZ(storageReadName(this))); // Complete async open on the next try this->openAsync = storageReadDriverInterface(this->driver)->openAsync != NULL; } MEM_CONTEXT_OBJ_END(); } else RETHROW(); } TRY_END(); try--; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(SIZE, result); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageReadClose(THIS_VOID) { THIS(StorageRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); storageReadDriverInterface(this->driver)->close(this->driver); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadEof(THIS_VOID) { THIS(StorageRead); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Eof is not known until the async open has completed, so return false FUNCTION_TEST_RETURN(BOOL, this->openAsync ? false : storageReadDriverInterface(this->driver)->eof(this->driver)); } /*********************************************************************************************************************************** Get file descriptor ***********************************************************************************************************************************/ static int storageReadFd(const THIS_VOID) { THIS(const StorageRead); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, storageReadDriverInterface(this->driver)->fd(this->driver)); } /**********************************************************************************************************************************/ static const IoReadInterface storageIoReadInterface = { .open = storageReadOpen, .read = storageRead, .close = storageReadClose, .eof = storageReadEof, .fd = storageReadFd, }; FN_EXTERN StorageRead * storageReadNew( const Storage *const storage, const String *const name, const bool ignoreMissing, const bool async, const bool compressible, const uint64_t offset, const Variant *const limit, const bool version, const String *const versionId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(BOOL, ignoreMissing); FUNCTION_LOG_PARAM(BOOL, async); FUNCTION_LOG_PARAM(BOOL, compressible); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_PARAM(BOOL, version); FUNCTION_LOG_PARAM(STRING, versionId); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(storage != NULL); ASSERT(name != NULL); // Async (deferred) open cannot be used with ignore missing files ASSERT(!async || !ignoreMissing); OBJ_NEW_BEGIN(StorageRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageRead) { .storage = storage, .driver = storageInterfaceNewReadP( storageDriver(storage), name, .compressible = compressible, .offset = offset, .limit = limit, .versionId = versionId), .retry = storageFeature(storage, storageFeatureReadRetry), .compressible = compressible, .async = async, .pub = { .type = storageType(storage), .name = strDup(name), .ignoreMissing = ignoreMissing, .offset = offset, .limit = varDup(limit), .version = version, .versionId = strDup(versionId), }, }; ASSERT(storageReadDriverInterface(this->driver)->eof != NULL); ASSERT(storageReadDriverInterface(this->driver)->close != NULL); ASSERT(storageReadDriverInterface(this->driver)->open != NULL); ASSERT(storageReadDriverInterface(this->driver)->read != NULL); // Remove fd method if it does not exist in the driver IoReadInterface storageIoReadInterfaceCopy = storageIoReadInterface; if (storageReadDriverInterface(this->driver)->fd == NULL) storageIoReadInterfaceCopy.fd = NULL; this->pub.io = ioReadNew(this, storageIoReadInterfaceCopy); // Set filter group when interface function exists if (storageReadDriverInterface(this->driver)->filterGroup != NULL) { storageReadDriverInterface(this->driver)->filterGroup(this->driver, ioReadFilterGroup(storageReadIo(this))); // Disable retry when the driver requires the filter group. The filters are passed to the driver when the read is // opened, e.g. to be run on a remote host, so they cannot be run again for a retry. Also, bytes read are counted after // the filters have been applied so the offset to restart the read cannot be determined. Retry in this case is expected // to be handled by the driver, e.g. the remote host retries reads on object stores. this->retry = false; } } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ, this); } /**********************************************************************************************************************************/ FN_EXTERN void storageReadToLog(const StorageRead *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "); strStcResultSizeInc(debugLog, strIdToLog(storageReadType(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt( debugLog, ", name: %s, ignoreMissing: %s}", strZ(storageReadName(this)), cvtBoolToConstZ(storageReadIgnoreMissing(this))); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/read.h0000664000175100017510000001031615265101745016722 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Read Interface ***********************************************************************************************************************************/ #ifndef STORAGE_READ_H #define STORAGE_READ_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageRead StorageRead; #include "common/io/read.h" #include "common/type/object.h" #include "common/type/stringId.h" #include "storage/read.intern.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_INLINE_ALWAYS StorageRead * storageReadMove(StorageRead *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct StorageReadPub { StringId type; // Storage type const String *name; // File name uint64_t offset; // Where to start reading in the file Variant *limit; // Limit how many bytes are read (NULL for no limit) bool ignoreMissing; // Ignore missing file? bool version; // Read version const String *versionId; // File version to read IoRead *io; // Read interface } StorageReadPub; // Should a missing file be ignored? FN_INLINE_ALWAYS bool storageReadIgnoreMissing(const StorageRead *const this) { return THIS_PUB(StorageRead)->ignoreMissing; } // Read interface FN_INLINE_ALWAYS IoRead * storageReadIo(const StorageRead *const this) { return THIS_PUB(StorageRead)->io; } // Is there a read limit? NULL for no limit. FN_INLINE_ALWAYS const Variant * storageReadLimit(const StorageRead *const this) { return THIS_PUB(StorageRead)->limit; } // File name FN_INLINE_ALWAYS const String * storageReadName(const StorageRead *const this) { return THIS_PUB(StorageRead)->name; } // Is there a read limit? NULL for no limit. FN_INLINE_ALWAYS uint64_t storageReadOffset(const StorageRead *const this) { return THIS_PUB(StorageRead)->offset; } // Get file type FN_INLINE_ALWAYS StringId storageReadType(const StorageRead *const this) { return THIS_PUB(StorageRead)->type; } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void storageReadFree(StorageRead *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void storageReadToLog(const StorageRead *this, StringStatic *debugLog); #define FUNCTION_LOG_STORAGE_READ_TYPE \ StorageRead * #define FUNCTION_LOG_STORAGE_READ_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, storageReadToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/read.intern.h0000664000175100017510000000453415265101745020225 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Read Interface Internal ***********************************************************************************************************************************/ #ifndef STORAGE_READ_INTERN_H #define STORAGE_READ_INTERN_H #include "common/io/read.h" #include "storage/storage.h" /*********************************************************************************************************************************** Interface ***********************************************************************************************************************************/ typedef struct StorageReadInterface { void (*filterGroup)(void *driver, IoFilterGroup *filterGroup); // Set filter group (optional) bool (*open)(void *driver); // Open read (async when openAsync is implemented) bool (*openAsync)(void *driver); // Complete async open and report if file exists (optional) size_t (*read)(void *driver, Buffer *buffer, bool block); // Read bytes bool (*eof)(void *driver); // Is read eof? int (*fd)(const void *driver); // Read file descriptor (optional) void (*close)(void *driver); // Close read } StorageReadInterface; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageRead *storageReadNew( const Storage *storage, const String *name, bool ignoreMissing, bool async, bool compressible, uint64_t offset, const Variant *limit, bool version, const String *versionId); /*********************************************************************************************************************************** Getters ***********************************************************************************************************************************/ FN_INLINE_ALWAYS const StorageReadInterface * storageReadDriverInterface(const void *const driver) { return *(const StorageReadInterface *const *)driver; } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/readMulti.c0000664000175100017510000006047515265101745017743 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Read Multi ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/io.h" #include "common/io/limitRead.h" #include "common/log.h" #include "common/memContext.h" #include "storage/read.intern.h" #include "storage/readMulti.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageReadMulti { StorageReadMultiPub pub; // Publicly accessible variables const Storage *storage; // Storage void *driver; // Driver String *versionFile; // Last file a version id was resolved for (target by time) String *versionId; // Version id resolved for versionFile }; /*********************************************************************************************************************************** Default driver that queues reads directly on the storage ***********************************************************************************************************************************/ typedef struct StorageReadMultiDefault { const StorageReadMultiInterface *interface; // Driver interface (must be first member) const Storage *storage; // Storage List *requestList; // List of read requests List *queue; // Queue of reads uint64_t readOver; // Bytes to read over rather than open file with new offset unsigned int queueMax; // Max size of read queue bool eof; // End-of-file indicator #ifdef DEBUG bool opened; // Has open been called? Reads may not be added after open. #endif } StorageReadMultiDefault; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_MULTI_DEFAULT_TYPE \ StorageReadMultiDefault * #define FUNCTION_LOG_STORAGE_READ_MULTI_DEFAULT_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadMultiDefault", buffer, bufferSize) /*********************************************************************************************************************************** Structure to store read requests ***********************************************************************************************************************************/ typedef struct StorageReadMultiRequest { const String *file; bool compressible; // Is the file compressible? uint64_t offset; // Where to start reading in the file uint64_t limit; // Limit bytes to read from the file const String *versionId; // Version to read (NULL for current version) List *rangeList; // Ranges for read over uint64_t rangeRead; // Bytes read in the current range } StorageReadMultiRequest; /*********************************************************************************************************************************** Structure to store ranges for read over ***********************************************************************************************************************************/ typedef struct StorageReadMultiRange { uint64_t offset; // Where to start reading in the file uint64_t limit; // Limit bytes to read from the file } StorageReadMultiRange; /*********************************************************************************************************************************** Constant to indicate that there is no limit ***********************************************************************************************************************************/ #define STORAGE_READ_MULTI_NO_LIMIT UINT64_MAX /*********************************************************************************************************************************** Manage the read queue ***********************************************************************************************************************************/ static void storageReadMultiDefaultQueue(StorageReadMultiDefault *const this, const bool prelim) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI_DEFAULT, this); FUNCTION_LOG_PARAM(BOOL, prelim); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!prelim || !lstEmpty(this->requestList)); // If the read queue is less than max then queue more reads if (lstSize(this->queue) < this->queueMax) { // Look for request to queue but skip the last request during prelim queueing since it might not be complete. Add prelim to // requestIdx rather than subtracting from list size to avoid an unsigned underflow when the list is empty. for (unsigned int requestIdx = lstSize(this->queue); requestIdx + prelim < lstSize(this->requestList); requestIdx++) { const StorageReadMultiRequest *const request = lstGet(this->requestList, requestIdx); // Open the read request. The version id, when needed, was already resolved when the request was added. MEM_CONTEXT_OBJ_BEGIN(this->queue) { StorageRead *const read = storageReadNew( this->storage, request->file, false, true, request->compressible, request->offset, request->limit != STORAGE_READ_MULTI_NO_LIMIT ? VARUINT64(request->limit) : NULL, request->versionId != NULL, request->versionId); ioReadOpen(storageReadIo(read)); lstAdd(this->queue, &read); } MEM_CONTEXT_OBJ_END(); // Break when queue is full if (lstSize(this->queue) == this->queueMax) break; } } // If the queue is empty and not in prelim queueing then eof if (lstEmpty(this->queue) && !prelim) this->eof = true; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Open read ***********************************************************************************************************************************/ static bool storageReadMultiDefaultOpen(THIS_VOID) { THIS(StorageReadMultiDefault); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI_DEFAULT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); storageReadMultiDefaultQueue(this, false); #ifdef DEBUG this->opened = true; #endif FUNCTION_LOG_RETURN(BOOL, true); } /*********************************************************************************************************************************** Read from multiple files as if they were one file, skipping unused bytes ***********************************************************************************************************************************/ static size_t storageReadMultiDefault(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadMultiDefault); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI_DEFAULT, this); FUNCTION_LOG_PARAM(BUFFER, buffer); (void)block; // Blocking is handled by the caller FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->eof); ASSERT(!lstEmpty(this->requestList)); ASSERT(!lstEmpty(this->queue)); ASSERT(buffer != NULL && bufEmpty(buffer)); size_t result = 0; MEM_CONTEXT_TEMP_BEGIN() { StorageReadMultiRequest *const request = lstGet(this->requestList, 0); const StorageReadMultiRange *const range = request->rangeList == NULL ? NULL : lstGet(request->rangeList, 0); StorageRead *const read = *(StorageRead **)lstGet(this->queue, 0); IoRead *const readIo = storageReadIo(read); // Set buffer limit so we do not read into unused bytes bufLimitClear(buffer); if (range != NULL && lstSize(request->rangeList) > 1) { uint64_t rangeRemains = range->limit - request->rangeRead; if (bufRemains(buffer) > rangeRemains) bufLimitSet(buffer, (size_t)rangeRemains); } // Read range result = ioRead(readIo, buffer); // Check range completion if (range != NULL) { request->rangeRead += result; // If range is complete and there are more ranges to process if (range->limit - request->rangeRead == 0 && lstSize(request->rangeList) > 1) { // Read over unused bytes StorageReadMultiRange *const rangeNext = lstGet(request->rangeList, 1); IoRead *const readOver = ioLimitReadNew(readIo, rangeNext->offset - (range->offset + range->limit)); ioReadDrain(readOver); // Remove range and reset read lstRemoveIdx(request->rangeList, 0); request->rangeRead = 0; } } // On eof close read and update queue if (ioReadEof(readIo)) { ioReadClose(readIo); storageReadFree(read); lstRemoveIdx(this->requestList, 0); lstRemoveIdx(this->queue, 0); storageReadMultiDefaultQueue(this, false); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(SIZE, result); } /*********************************************************************************************************************************** Close queued files ***********************************************************************************************************************************/ static void storageReadMultiDefaultClose(THIS_VOID) { THIS(StorageReadMultiDefault); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI_DEFAULT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); for (unsigned int queueIdx = 0; queueIdx < lstSize(this->queue); queueIdx++) ioReadClose(storageReadIo(*(StorageRead **)lstGet(this->queue, queueIdx))); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Has read reached EOF? ***********************************************************************************************************************************/ static bool storageReadMultiDefaultEof(THIS_VOID) { THIS(StorageReadMultiDefault); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_MULTI_DEFAULT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->eof); } /*********************************************************************************************************************************** Add a file to the read ***********************************************************************************************************************************/ static void storageReadMultiDefaultAdd(THIS_VOID, const String *const file, const StorageReadMultiAddParam param) { THIS(StorageReadMultiDefault); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI_DEFAULT, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(BOOL, param.compressible); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_PARAM(STRING, param.versionId); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(!this->opened); // Check if new request can be combined with prior request StorageReadMultiRequest *const requestPrior = lstEmpty(this->requestList) ? NULL : lstGetLast(this->requestList); if (requestPrior != NULL && strEq(requestPrior->file, file)) { CHECK(AssertError, requestPrior->limit != STORAGE_READ_MULTI_NO_LIMIT, "cannot add request after request with no limit"); CHECK(AssertError, param.limit != NULL, "request with no limit must be first"); CHECK_FMT( AssertError, param.offset >= requestPrior->offset + requestPrior->limit, "new request offset %" PRIu64 " must be after prior request (offset %" PRIu64 ", limit %" PRIu64 ")", param.offset, requestPrior->offset, requestPrior->limit); // If the new offset is within the allowed gap from prior range end then extend the prior range const uint64_t limit = varUInt64(param.limit); const uint64_t requestPriorEnd = requestPrior->offset + requestPrior->limit; if (param.offset - requestPriorEnd <= this->readOver) { // If the new range offset is not exactly after the prior range end then add to the subrange list if (param.offset != requestPriorEnd) { // If the subrange list does not exist then add it if (requestPrior->rangeList == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { requestPrior->rangeList = lstNewP(sizeof(StorageReadMultiRange)); } MEM_CONTEXT_OBJ_END(); // Add the prior range to the range list lstAdd( requestPrior->rangeList, &(StorageReadMultiRange){.offset = requestPrior->offset, .limit = requestPrior->limit}); } lstAdd(requestPrior->rangeList, &(StorageReadMultiRange){.offset = param.offset, .limit = limit}); } // Else combine with prior range if it exists else if (requestPrior->rangeList != NULL) { StorageReadMultiRange *const range = lstGetLast(requestPrior->rangeList); range->limit = param.offset - range->offset + limit; } // Combine limit with prior limit requestPrior->limit = param.offset - requestPrior->offset + limit; FUNCTION_LOG_RETURN_VOID(); } } // Add request if not combined with prior MEM_CONTEXT_OBJ_BEGIN(this->requestList) { const StorageReadMultiRequest request = { .file = requestPrior != NULL && strEq(requestPrior->file, file) ? requestPrior->file : strDup(file), .compressible = param.compressible, .offset = param.offset, .limit = param.limit == NULL ? STORAGE_READ_MULTI_NO_LIMIT : varUInt64(param.limit), .versionId = strDup(param.versionId), }; lstAdd(this->requestList, &request); } MEM_CONTEXT_OBJ_END(); // Update queue storageReadMultiDefaultQueue(this, true); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** New multi-file read ***********************************************************************************************************************************/ static const StorageReadMultiInterface storageReadMultiDefaultInterface = { .add = storageReadMultiDefaultAdd, .close = storageReadMultiDefaultClose, .eof = storageReadMultiDefaultEof, .open = storageReadMultiDefaultOpen, .read = storageReadMultiDefault, }; static void * storageReadMultiDefaultNew(const Storage *const storage, const unsigned int prefetch, const uint64_t readOver) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(UINT, prefetch); FUNCTION_LOG_PARAM(UINT64, readOver); FUNCTION_LOG_END(); ASSERT(storage != NULL); OBJ_NEW_BEGIN(StorageReadMultiDefault, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageReadMultiDefault) { .interface = &storageReadMultiDefaultInterface, .storage = storage, .requestList = lstNewP(sizeof(StorageReadMultiRequest), .comparator = lstComparatorStr), .queue = lstNewP(sizeof(StorageRead *)), .queueMax = prefetch + 1, .readOver = readOver, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_MULTI_DEFAULT, this); } /*********************************************************************************************************************************** Open read interface ***********************************************************************************************************************************/ static bool storageReadMultiOpen(THIS_VOID) { THIS(StorageReadMulti); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(BOOL, storageReadMultiDriverInterface(this->driver)->open(this->driver)); } /*********************************************************************************************************************************** Read interface ***********************************************************************************************************************************/ static size_t storageReadMulti(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadMulti); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); FUNCTION_LOG_RETURN(SIZE, storageReadMultiDriverInterface(this->driver)->read(this->driver, buffer, block)); } /*********************************************************************************************************************************** Close interface ***********************************************************************************************************************************/ static void storageReadMultiClose(THIS_VOID) { THIS(StorageReadMulti); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI, this); FUNCTION_LOG_END(); ASSERT(this != NULL); storageReadMultiDriverInterface(this->driver)->close(this->driver); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Eof interface ***********************************************************************************************************************************/ static bool storageReadMultiEof(THIS_VOID) { THIS(StorageReadMulti); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_MULTI, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, storageReadMultiDriverInterface(this->driver)->eof(this->driver)); } /**********************************************************************************************************************************/ FN_EXTERN void storageReadMultiAdd(StorageReadMulti *const this, const String *const fileExp, StorageReadMultiAddParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI, this); FUNCTION_LOG_PARAM(STRING, fileExp); FUNCTION_LOG_PARAM(BOOL, param.compressible); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_PARAM(STRING, param.versionId); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(fileExp != NULL && !strEmpty(fileExp)); ASSERT(param.limit == NULL || varType(param.limit) == varTypeUInt64); MEM_CONTEXT_TEMP_BEGIN() { const String *const path = storagePathP(this->storage, fileExp); // If targeting by time and the version has not already been resolved then look up the version id if (param.versionId == NULL && storageTargetTime(this->storage) != 0) { // References to a file are added together so reuse the version id resolved for the prior file when it is the same, // rather than looking it up again. Otherwise resolve the version id as of the target time and cache it for reuse. if (strEmpty(this->versionFile) || !strEq(this->versionFile, fileExp)) { const String *const versionId = storageInfoP(this->storage, fileExp, .ignoreMissing = true).versionId; if (versionId == NULL) THROW_FMT(FileMissingError, STORAGE_ERROR_READ_MISSING, strZ(path)); MEM_CONTEXT_OBJ_BEGIN(this) { this->versionFile = strCat(strTrunc(this->versionFile), fileExp); this->versionId = strCat(strTrunc(this->versionId), versionId); } MEM_CONTEXT_OBJ_END(); } param.versionId = this->versionId; } storageReadMultiDriverInterface(this->driver)->add(this->driver, path, param); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const IoReadInterface storageIoReadMultiInterface = { .close = storageReadMultiClose, .eof = storageReadMultiEof, .open = storageReadMultiOpen, .read = storageReadMulti, }; FN_EXTERN StorageReadMulti * storageReadMultiNew(const Storage *const storage, const unsigned int prefetch, const uint64_t readOver) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(UINT, prefetch); FUNCTION_LOG_PARAM(UINT64, readOver); FUNCTION_LOG_END(); ASSERT(storage != NULL); OBJ_NEW_BEGIN(StorageReadMulti, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageReadMulti) { .storage = storage, .driver = storageInterface(storage).newReadMulti != NULL ? storageInterface(storage).newReadMulti(storageDriver(storage)) : storageReadMultiDefaultNew(storage, prefetch, readOver), .versionFile = strNew(), .versionId = strNew(), }; ASSERT(storageReadMultiDriverInterface(this->driver)->add != NULL); ASSERT(storageReadMultiDriverInterface(this->driver)->close != NULL); ASSERT(storageReadMultiDriverInterface(this->driver)->eof != NULL); ASSERT(storageReadMultiDriverInterface(this->driver)->open != NULL); ASSERT(storageReadMultiDriverInterface(this->driver)->read != NULL); this->pub.io = ioReadNew(this, storageIoReadMultiInterface); // Set filter group when interface function exists if (storageReadMultiDriverInterface(this->driver)->filterGroup != NULL) { storageReadMultiDriverInterface(this->driver)->filterGroup( this->driver, ioReadFilterGroup(storageReadMultiIo(this))); } } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_MULTI, this); } /**********************************************************************************************************************************/ FN_EXTERN void storageReadMultiToLog(const StorageReadMulti *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{storage: "); storageToLog(this->storage, debugLog); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/readMulti.h0000664000175100017510000000733715265101745017746 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Read Multi Files and Ranges ***********************************************************************************************************************************/ #ifndef STORAGE_READ_MULTI_H #define STORAGE_READ_MULTI_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadMulti StorageReadMulti; #include "common/type/object.h" #include "common/type/param.h" #include "common/type/variant.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ typedef struct StorageReadMultiAddParam { VAR_PARAM_HEADER; // Is the file compressible? bool compressible; // Where to start reading in the file const uint64_t offset; // Limit bytes to read from the file (must be varTypeUInt64). NULL for no limit. const Variant *limit; // Target a specific file version (NULL for current version). This is resolved automatically when the storage is time-targeted // but may also be set explicitly, e.g. when the version has already been resolved on the client side of a remote. const String *versionId; } StorageReadMultiAddParam; #include "storage/readMulti.intern.h" #define storageReadMultiAddP(this, pathExp, ...) \ storageReadMultiAdd(this, pathExp, (StorageReadMultiAddParam){VAR_PARAM_INIT, __VA_ARGS__}) // Add a file to the read. All files must be added before the io is opened since some drivers, e.g. remote, send the entire request // list to be processed when the io is opened. FN_EXTERN void storageReadMultiAdd(StorageReadMulti *this, const String *fileExp, StorageReadMultiAddParam param); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Read interface FN_INLINE_ALWAYS IoRead * storageReadMultiIo(const StorageReadMulti *const this) { return THIS_PUB(StorageReadMulti)->io; } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void storageReadMultiFree(StorageReadMulti *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void storageReadMultiToLog(const StorageReadMulti *this, StringStatic *debugLog); #define FUNCTION_LOG_STORAGE_READ_MULTI_TYPE \ StorageReadMulti * #define FUNCTION_LOG_STORAGE_READ_MULTI_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, storageReadMultiToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/readMulti.intern.h0000664000175100017510000000441715265101745021240 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Read Multi Internal ***********************************************************************************************************************************/ #ifndef STORAGE_READ_MULTI_INTERN_H #define STORAGE_READ_MULTI_INTERN_H #include "common/io/read.h" #include "storage/storage.h" /*********************************************************************************************************************************** Interface ***********************************************************************************************************************************/ typedef struct StorageReadMultiInterface { void (*add)(void *driver, const String *file, StorageReadMultiAddParam param); // Add a file to the read void (*filterGroup)(void *driver, IoFilterGroup *filterGroup); // Set filter group (optional) bool (*open)(void *driver); // Open read size_t (*read)(void *driver, Buffer *buffer, bool block); // Read bytes bool (*eof)(void *driver); // Is read eof? void (*close)(void *driver); // Close read } StorageReadMultiInterface; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadMulti *storageReadMultiNew(const Storage *storage, unsigned int prefetch, uint64_t readOver); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct StorageReadMultiPub { IoRead *io; // Read interface } StorageReadMultiPub; FN_INLINE_ALWAYS const StorageReadMultiInterface * storageReadMultiDriverInterface(const void *const driver) { return *(const StorageReadMultiInterface *const *)driver; } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/remote/0000775000175100017510000000000015265101745017130 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/remote/protocol.c0000664000175100017510000006354515265101745021152 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage Protocol Handler ***********************************************************************************************************************************/ #include #include "common/compress/helper.h" #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/regExp.h" #include "common/type/pack.h" #include "config/config.h" #include "protocol/helper.h" #include "storage/helper.h" #include "storage/remote/protocol.h" #include "storage/storage.intern.h" #include "storage/write.intern.h" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct { MemContext *memContext; // Mem context const Storage *storage; // Storage void *driver; // Storage driver used for requests const List *filterHandler; // Filter handler list } storageRemoteProtocolLocal; /*********************************************************************************************************************************** Set filter handlers ***********************************************************************************************************************************/ FN_EXTERN void storageRemoteFilterHandlerSet(const List *const filterHandler) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VOID, filterHandler); FUNCTION_TEST_END(); ASSERT(filterHandler != NULL); ASSERT(!lstEmpty(filterHandler)); storageRemoteProtocolLocal.filterHandler = filterHandler; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Set filter group based on passed filters ***********************************************************************************************************************************/ static void storageRemoteFilterGroup(IoFilterGroup *const filterGroup, const Pack *const filterPack) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_TEST_PARAM(PACK, filterPack); FUNCTION_TEST_END(); ASSERT(filterGroup != NULL); ASSERT(filterPack != NULL); MEM_CONTEXT_TEMP_BEGIN() { PackRead *const filterList = pckReadNew(filterPack); while (!pckReadNullP(filterList)) { const StringId filterKey = pckReadStrIdP(filterList); const Pack *const filterParam = pckReadPackP(filterList); // If a compression filter IoFilter *const filter = compressFilterPack(filterKey, filterParam); if (filter != NULL) { ioFilterGroupAdd(filterGroup, filter); } // Else a filter handler else { ASSERT(storageRemoteProtocolLocal.filterHandler != NULL); // Search for a filter handler unsigned int filterIdx = 0; for (; filterIdx < lstSize(storageRemoteProtocolLocal.filterHandler); filterIdx++) { const StorageRemoteFilterHandler *const filterHandler = lstGet( storageRemoteProtocolLocal.filterHandler, filterIdx); // If a match create the filter if (filterHandler->type == filterKey) { // Create a filter with parameters if (filterHandler->handlerParam != NULL) { ASSERT(filterParam != NULL); ioFilterGroupAdd(filterGroup, filterHandler->handlerParam(filterParam)); } // Else create a filter without parameters else { ASSERT(filterHandler->handlerNoParam != NULL); ASSERT(filterParam == NULL); ioFilterGroupAdd(filterGroup, filterHandler->handlerNoParam()); } // Break on filter match break; } } // Error when the filter was not found if (filterIdx == lstSize(storageRemoteProtocolLocal.filterHandler)) THROW_FMT(AssertError, "unable to add filter '%s'", zNewStrId(filterKey)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemoteFeatureProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); (void)param; FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(param == NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Get storage based on remote type const Storage *const storage = cfgOptionStrId(cfgOptRemoteType) == protocolStorageTypeRepo ? storageRepoWrite() : storagePgWrite(); // Store local variables in the server context if (storageRemoteProtocolLocal.memContext == NULL) { MEM_CONTEXT_PRIOR_BEGIN() { MEM_CONTEXT_NEW_BEGIN(StorageRemoteProtocol, .childQty = MEM_CONTEXT_QTY_MAX) { storageRemoteProtocolLocal.memContext = memContextCurrent(); storageRemoteProtocolLocal.storage = storage; storageRemoteProtocolLocal.driver = storageDriver(storage); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_PRIOR_END(); } // Return storage features PackWrite *const data = protocolServerResultData(result); pckWriteStrP(data, storagePathP(storage, NULL)); pckWriteU64P(data, storageInterface(storage).feature); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ typedef struct StorageRemoteInfoProtocolWriteData { time_t timeModifiedLast; // timeModified from last call mode_t modeLast; // mode from last call uid_t userIdLast; // userId from last call gid_t groupIdLast; // groupId from last call String *user; // user from last call String *group; // group from last call } StorageRemoteInfoProtocolWriteData; // Helper to write storage info into the protocol. This function is not called unless the info exists so no need to write exists or // check for level == storageInfoLevelExists. // // Fields that do not change from one call to the next are omitted to save bandwidth. static void storageRemoteInfoProtocolPut( StorageRemoteInfoProtocolWriteData *const data, PackWrite *const write, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(PACK_WRITE, write); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(data != NULL); ASSERT(write != NULL); ASSERT(info != NULL); // Write type and time pckWriteU32P(write, info->type); pckWriteTimeP(write, info->timeModified - data->timeModifiedLast); // Write size for files if (info->type == storageTypeFile) pckWriteU64P(write, info->size); // Write version pckWriteStrP(write, info->versionId); // Write fields needed for detail level if (info->level >= storageInfoLevelDetail) { // Write mode pckWriteModeP(write, info->mode, .defaultValue = data->modeLast); // Write user id/name pckWriteU32P(write, info->userId, .defaultValue = data->userIdLast); if (info->user == NULL) // {vm_covered} pckWriteBoolP(write, true); // {vm_covered} else { pckWriteBoolP(write, false); pckWriteStrP(write, info->user, .defaultValue = data->user); } // Write group id/name pckWriteU32P(write, info->groupId, .defaultValue = data->groupIdLast); if (info->group == NULL) // {vm_covered} pckWriteBoolP(write, true); // {vm_covered} else { pckWriteBoolP(write, false); pckWriteStrP(write, info->group, .defaultValue = data->group); } // Write link destination if (info->type == storageTypeLink) pckWriteStrP(write, info->linkDestination); } // Store defaults to use for the next call data->timeModifiedLast = info->timeModified; data->modeLast = info->mode; data->userIdLast = info->userId; data->groupIdLast = info->groupId; if (info->user != NULL && !strEq(info->user, data->user)) // {vm_covered} { strFree(data->user); data->user = strDup(info->user); } if (info->group != NULL && !strEq(info->group, data->group)) // {vm_covered} { strFree(data->group); data->group = strDup(info->group); } FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN ProtocolServerResult * storageRemoteInfoProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Get file info const String *const file = pckReadStrP(param); const StorageInfoLevel level = (StorageInfoLevel)pckReadU32P(param); const bool followLink = pckReadBoolP(param); StorageInfo info = storageInterfaceInfoP(storageRemoteProtocolLocal.driver, file, level, .followLink = followLink); // Write file info to protocol PackWrite *const data = protocolServerResultData(result); pckWriteBoolP(data, info.exists, .defaultWrite = true); if (info.exists) storageRemoteInfoProtocolPut(&(StorageRemoteInfoProtocolWriteData){0}, data, &info); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemoteLinkCreateProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); MEM_CONTEXT_TEMP_BEGIN() { const String *const target = pckReadStrP(param); const String *const linkPath = pckReadStrP(param); const StorageLinkType linkType = (StorageLinkType)pckReadU32P(param); storageInterfaceLinkCreateP(storageRemoteProtocolLocal.driver, target, linkPath, .linkType = linkType); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, NULL); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemoteListProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(.extra = ioBufferSize()); MEM_CONTEXT_TEMP_BEGIN() { const String *const path = pckReadStrP(param); const StorageInfoLevel level = (StorageInfoLevel)pckReadU32P(param); const time_t targetTime = pckReadTimeP(param); StorageRemoteInfoProtocolWriteData writeData = {0}; StorageList *const list = storageInterfaceListP(storageRemoteProtocolLocal.driver, path, level, .targetTime = targetTime); PackWrite *const data = protocolServerResultData(result); // Indicate whether or not the path was found pckWriteBoolP(data, list != NULL, .defaultWrite = true); // Put list if (list != NULL) { for (unsigned int listIdx = 0; listIdx < storageLstSize(list); listIdx++) { const StorageInfo info = storageLstGet(list, listIdx); pckWriteObjBeginP(data); pckWriteStrP(data, info.name); storageRemoteInfoProtocolPut(&writeData, data, &info); pckWriteObjEndP(data); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ static bool storageRemoteReadInternal(IoRead *const read, PackWrite *const packWrite) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(PACK_WRITE, packWrite); FUNCTION_LOG_END(); ASSERT(read != NULL); ASSERT(packWrite != NULL); FUNCTION_AUDIT_HELPER(); // Read chunk and send to client Buffer *const buffer = bufNew(ioBufferSize()); ioRead(read, buffer); pckWriteBoolP(packWrite, bufEmpty(buffer)); if (!bufEmpty(buffer)) pckWriteBinP(packWrite, buffer); // On eof bool result = true; if (ioReadEof(read)) { // Close read (needed to get filter results) ioReadClose(read); // Write filter results pckWritePackP(packWrite, ioFilterGroupResultAll(ioReadFilterGroup(read))); // Let the server know to close the session result = false; } FUNCTION_LOG_RETURN(BOOL, result); } FN_EXTERN ProtocolServerResult * storageRemoteReadOpenProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(.extra = ioBufferSize()); MEM_CONTEXT_TEMP_BEGIN() { const String *file = pckReadStrP(param); const uint64_t offset = pckReadU64P(param); const Variant *const limit = pckReadNullP(param) ? NULL : VARUINT64(pckReadU64P(param)); const String *const versionId = pckReadStrP(param); const Pack *const filter = pckReadPackP(param); // Create the read object ignoring missing since the client decides whether a missing file is an error StorageRead *const fileRead = storageReadNew( storageRemoteProtocolLocal.storage, file, true, false, false, offset, limit, versionId != NULL, versionId); // Set filter group based on passed filters storageRemoteFilterGroup(ioReadFilterGroup(storageReadIo(fileRead)), filter); // Determine if file exists PackWrite *const data = protocolServerResultData(result); const bool exists = ioReadOpen(storageReadIo(fileRead)); pckWriteBoolP(data, exists, .defaultWrite = true); // If the file exists if (exists) { // If there is more to read then set session data if (storageRemoteReadInternal(storageReadIo(fileRead), data)) protocolServerResultSessionDataSet(result, fileRead); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } FN_EXTERN ProtocolServerResult * storageRemoteReadProtocol(PackRead *const param, void *const fileRead) { FUNCTION_LOG_BEGIN(logLevelDebug); (void)param; FUNCTION_LOG_PARAM(STORAGE_READ, fileRead); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(param == NULL); ASSERT(fileRead != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(.extra = ioBufferSize()); if (!storageRemoteReadInternal(storageReadIo(fileRead), protocolServerResultData(result))) protocolServerResultCloseSet(result); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemoteReadMultiOpenProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(.extra = ioBufferSize()); MEM_CONTEXT_TEMP_BEGIN() { // Create the multi read and set filter group based on passed filters StorageReadMulti *const fileRead = storageNewReadMultiP(storageRemoteProtocolLocal.storage); storageRemoteFilterGroup(ioReadFilterGroup(storageReadMultiIo(fileRead)), pckReadPackP(param)); // Add requested files while (!pckReadNullP(param)) { const String *const file = pckReadStrP(param); const bool compressible = pckReadBoolP(param); const uint64_t offset = pckReadU64P(param); const Variant *const limit = pckReadNullP(param) ? NULL : VARUINT64(pckReadU64P(param)); const String *const versionId = pckReadStrP(param); storageReadMultiAddP( fileRead, file, .compressible = compressible, .offset = offset, .limit = limit, .versionId = versionId); } // Open read ioReadOpen(storageReadMultiIo(fileRead)); // If there is more to read then set session data if (storageRemoteReadInternal(storageReadMultiIo(fileRead), protocolServerResultData(result))) protocolServerResultSessionDataSet(result, fileRead); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } FN_EXTERN ProtocolServerResult * storageRemoteReadMultiProtocol(PackRead *const param, void *const fileRead) { FUNCTION_LOG_BEGIN(logLevelDebug); (void)param; FUNCTION_LOG_PARAM(STORAGE_READ_MULTI, fileRead); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(param == NULL); ASSERT(fileRead != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(.extra = ioBufferSize()); if (!storageRemoteReadInternal(storageReadMultiIo(fileRead), protocolServerResultData(result))) protocolServerResultCloseSet(result); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemoteWriteOpenProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Create the write object const String *const file = pckReadStrP(param); const mode_t modeFile = pckReadModeP(param); const mode_t modePath = pckReadModeP(param); const String *const user = pckReadStrP(param); const String *const group = pckReadStrP(param); const time_t timeModified = pckReadTimeP(param); const bool createPath = pckReadBoolP(param); const bool syncFile = pckReadBoolP(param); const bool syncPath = pckReadBoolP(param); const bool atomic = pckReadBoolP(param); const Pack *const filter = pckReadPackP(param); StorageWrite *const fileWrite = storageWriteNew( storageRemoteProtocolLocal.storage, file, modeFile, modePath, user, group, timeModified, createPath, syncFile, syncPath, atomic, true, false); // Set filter group based on passed filters storageRemoteFilterGroup(ioWriteFilterGroup(storageWriteIo(fileWrite)), filter); // Open file ioWriteOpen(storageWriteIo(fileWrite)); // Set session data protocolServerResultSessionDataSet(result, fileWrite); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } FN_EXTERN ProtocolServerResult * storageRemoteWriteProtocol(PackRead *const param, void *const fileWrite) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_PARAM(STORAGE_WRITE, fileWrite); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(fileWrite != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); MEM_CONTEXT_TEMP_BEGIN() { Buffer *const buffer = pckReadBinP(param); ioWrite(storageWriteIo(fileWrite), buffer); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, NULL); } FN_EXTERN ProtocolServerResult * storageRemoteWriteCloseProtocol(PackRead *const param, void *const fileWrite) { FUNCTION_LOG_BEGIN(logLevelDebug); (void)param; FUNCTION_LOG_PARAM(STORAGE_WRITE, fileWrite); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(param == NULL); ASSERT(fileWrite != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { ioWriteClose(storageWriteIo(fileWrite)); // Send filter results pckWritePackP(protocolServerResultData(result), ioFilterGroupResultAll(ioWriteFilterGroup(storageWriteIo(fileWrite)))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemotePathCreateProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); MEM_CONTEXT_TEMP_BEGIN() { const String *const path = pckReadStrP(param); const bool errorOnExists = pckReadBoolP(param); const bool noParentCreate = pckReadBoolP(param); const mode_t mode = pckReadModeP(param); storageInterfacePathCreateP(storageRemoteProtocolLocal.driver, path, errorOnExists, noParentCreate, mode); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, NULL); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemotePathRemoveProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { const String *const path = pckReadStrP(param); const bool recurse = pckReadBoolP(param); pckWriteBoolP( protocolServerResultData(result), storageInterfacePathRemoveP(storageRemoteProtocolLocal.driver, path, recurse), .defaultWrite = true); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemotePathSyncProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); MEM_CONTEXT_TEMP_BEGIN() { const String *const path = pckReadStrP(param); storageInterfacePathSyncP(storageRemoteProtocolLocal.driver, path); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, NULL); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemoteRemoveProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); MEM_CONTEXT_TEMP_BEGIN() { const String *const file = pckReadStrP(param); const bool errorOnMissing = pckReadBoolP(param); storageInterfaceRemoveP(storageRemoteProtocolLocal.driver, file, .errorOnMissing = errorOnMissing); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, NULL); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/remote/protocol.h0000664000175100017510000001371015265101745021144 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage Protocol Handler ***********************************************************************************************************************************/ #ifndef STORAGE_REMOTE_PROTOCOL_H #define STORAGE_REMOTE_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process storage protocol requests FN_EXTERN ProtocolServerResult *storageRemoteFeatureProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteInfoProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteLinkCreateProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteListProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemotePathCreateProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemotePathRemoveProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemotePathSyncProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteReadOpenProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteReadProtocol(PackRead *param, void *fileRead); FN_EXTERN ProtocolServerResult *storageRemoteReadMultiOpenProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteReadMultiProtocol(PackRead *param, void *fileRead); FN_EXTERN ProtocolServerResult *storageRemoteRemoveProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteWriteOpenProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteWriteProtocol(PackRead *param, void *fileWrite); FN_EXTERN ProtocolServerResult *storageRemoteWriteCloseProtocol(PackRead *param, void *fileWrite); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_STORAGE_FEATURE STRID5("s-f", 0x1b730) #define PROTOCOL_COMMAND_STORAGE_INFO STRID5("s-i", 0x27730) #define PROTOCOL_COMMAND_STORAGE_LINK_CREATE STRID5("s-lc", 0x1b3730) #define PROTOCOL_COMMAND_STORAGE_LIST STRID5("s-l", 0x33730) #define PROTOCOL_COMMAND_STORAGE_PATH_CREATE STRID5("s-pc", 0x1c3730) #define PROTOCOL_COMMAND_STORAGE_PATH_REMOVE STRID5("s-pr", 0x943730) #define PROTOCOL_COMMAND_STORAGE_PATH_SYNC STRID5("s-ps", 0x9c3730) #define PROTOCOL_COMMAND_STORAGE_READ STRID5("s-rd", 0x24b730) #define PROTOCOL_COMMAND_STORAGE_READ_MULTI STRID5("s-rdm", 0xd24b730) #define PROTOCOL_COMMAND_STORAGE_REMOVE STRID5("s-r", 0x4b730) #define PROTOCOL_COMMAND_STORAGE_WRITE STRID5("s-wr", 0x95f730) #define PROTOCOL_SERVER_HANDLER_STORAGE_REMOTE_LIST \ {.command = PROTOCOL_COMMAND_STORAGE_FEATURE, .process = storageRemoteFeatureProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_INFO, .process = storageRemoteInfoProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_LINK_CREATE, .process = storageRemoteLinkCreateProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_LIST, .process = storageRemoteListProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_PATH_CREATE, .process = storageRemotePathCreateProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_PATH_REMOVE, .process = storageRemotePathRemoveProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_PATH_SYNC, .process = storageRemotePathSyncProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_READ, .open = storageRemoteReadOpenProtocol, .processSession = storageRemoteReadProtocol},\ {.command = PROTOCOL_COMMAND_STORAGE_READ_MULTI, .open = storageRemoteReadMultiOpenProtocol, \ .processSession = storageRemoteReadMultiProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_REMOVE, .process = storageRemoteRemoveProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_WRITE, .open = storageRemoteWriteOpenProtocol, \ .processSession = storageRemoteWriteProtocol, .close = storageRemoteWriteCloseProtocol}, /*********************************************************************************************************************************** Filters that may be passed to a remote ***********************************************************************************************************************************/ typedef IoFilter *(*StorageRemoteFilterHandlerParam)(const Pack *paramList); typedef IoFilter *(*StorageRemoteFilterHandlerNoParam)(void); typedef struct StorageRemoteFilterHandler { StringId type; // Filter type StorageRemoteFilterHandlerParam handlerParam; // Handler for filter with parameters StorageRemoteFilterHandlerNoParam handlerNoParam; // Handler with no parameters } StorageRemoteFilterHandler; FN_EXTERN void storageRemoteFilterHandlerSet(const List *filterHandler); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/remote/read.c0000664000175100017510000002751215265101745020216 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage Read ***********************************************************************************************************************************/ #include #include #include #include "common/compress/helper.h" #include "common/debug.h" #include "common/io/io.h" #include "common/io/read.h" #include "common/log.h" #include "common/memContext.h" #include "common/type/convert.h" #include "common/type/object.h" #include "storage/read.h" #include "storage/remote/protocol.h" #include "storage/remote/read.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageReadRemote { const StorageReadInterface *interface; // Interface StorageRemote *storage; // Storage that created this object IoFilterGroup *filterGroup; // Filter group const String *name; // File name uint64_t offset; // Read offset const Variant *limit; // Read limit (NULL for no limit) const String *versionId; // Version id (NULL for most recent) StorageReadRemoteStream stream; // Protocol stream state bool compressible; // Is this file compressible? unsigned int compressLevel; // Level to use for compression }; /*********************************************************************************************************************************** Read from the protocol stream ***********************************************************************************************************************************/ FN_EXTERN void storageReadRemoteStreamInternal(StorageReadRemoteStream *const this, IoFilterGroup *const filterGroup, PackRead *const packRead) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, this); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_LOG_PARAM(PACK_READ, packRead); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(filterGroup != NULL); ASSERT(packRead != NULL); FUNCTION_AUDIT_HELPER(); // If not done then read the next chunk if (!pckReadBoolP(packRead)) { MEM_CONTEXT_BEGIN(this->memContext) { this->chunk = pckReadBinP(packRead); this->remaining = bufUsed(this->chunk); } MEM_CONTEXT_END(); } // If eof then get results if (protocolClientSessionClosed(this->session)) { ioFilterGroupResultAllSet(filterGroup, pckReadPackP(packRead)); this->eofFound = true; if (this->remaining == 0) this->eof = true; } #ifdef DEBUG this->protocolReadBytes += this->remaining; #endif FUNCTION_LOG_RETURN_VOID(); } FN_EXTERN size_t storageReadRemoteStreamRead(StorageReadRemoteStream *const this, IoFilterGroup *const filterGroup, Buffer *const buffer) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, this); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL && !bufFull(buffer)); size_t result = 0; // Read if eof has not been reached if (!this->eof) { do { // If no bytes remaining then read a new chunk if (this->remaining == 0) { MEM_CONTEXT_TEMP_BEGIN() { if (!protocolClientSessionQueued(this->session)) protocolClientSessionRequestAsyncP(this->session); storageReadRemoteStreamInternal(this, filterGroup, protocolClientSessionResponse(this->session)); if (!this->eofFound) protocolClientSessionRequestAsyncP(this->session); } MEM_CONTEXT_TEMP_END(); } // Read if not eof if (!this->eof) { // Copy as much as possible into the output buffer const size_t remains = this->remaining < bufRemains(buffer) ? this->remaining : bufRemains(buffer); bufCatSub(buffer, this->chunk, bufUsed(this->chunk) - this->remaining, remains); result += remains; this->remaining -= remains; // If there is no more to copy from the chunk then free it if (this->remaining == 0) { bufFree(this->chunk); this->chunk = NULL; if (this->eofFound) this->eof = true; } } } while (!this->eof && !bufFull(buffer)); } FUNCTION_LOG_RETURN(SIZE, result); } /*********************************************************************************************************************************** Read from a file ***********************************************************************************************************************************/ static size_t storageReadRemote(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_REMOTE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); (void)block; // Blocking is handled by the caller FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL && !bufFull(buffer)); FUNCTION_LOG_RETURN(SIZE, storageReadRemoteStreamRead(&this->stream, this->filterGroup, buffer)); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadRemoteEof(THIS_VOID) { THIS(StorageReadRemote); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_REMOTE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->stream.eof); } /*********************************************************************************************************************************** Set filter group ***********************************************************************************************************************************/ static void storageReadRemoteFilterGroup(THIS_VOID, IoFilterGroup *const filterGroup) { THIS(StorageReadRemote); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_REMOTE, this); FUNCTION_TEST_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_TEST_END(); this->filterGroup = filterGroup; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadRemoteOpen(THIS_VOID) { THIS(StorageReadRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_REMOTE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { // If the file is compressible add compression filter on the remote if (this->compressible) ioFilterGroupAdd(this->filterGroup, compressFilterP(compressTypeLz4, (int)this->compressLevel, .raw = true)); PackWrite *const param = protocolPackNew(); pckWriteStrP(param, this->name); pckWriteU64P(param, this->offset); if (this->limit == NULL) pckWriteNullP(param); else pckWriteU64P(param, varUInt64(this->limit)); pckWriteStrP(param, this->versionId); pckWritePackP(param, ioFilterGroupParamAll(this->filterGroup)); // If the file exists PackRead *const packRead = protocolClientSessionOpenP(this->stream.session, .param = param); result = pckReadBoolP(packRead); if (result) { // Clear filters since they will be run on the remote side ioFilterGroupClear(this->filterGroup); // If the file is compressible add decompression filter locally if (this->compressible) ioFilterGroupAdd(this->filterGroup, decompressFilterP(compressTypeLz4, .raw = true)); // Read the first chunk or eof storageReadRemoteStreamInternal(&this->stream, this->filterGroup, packRead); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Close the file and read any remaining data. It is possible that all data has been read but if the amount of data is exactly divisible by the buffer size then the eof may not have been received. ***********************************************************************************************************************************/ static void storageReadRemoteClose(THIS_VOID) { THIS(StorageReadRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_REMOTE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); protocolClientSessionCancel(this->stream.session); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageReadInterface storageReadRemoteInterface = { .close = storageReadRemoteClose, .eof = storageReadRemoteEof, .filterGroup = storageReadRemoteFilterGroup, .open = storageReadRemoteOpen, .read = storageReadRemote, }; FN_EXTERN StorageReadRemote * storageReadRemoteNew( StorageRemote *const storage, ProtocolClient *const client, const String *const name, const bool compressible, const unsigned int compressLevel, const uint64_t offset, const Variant *const limit, const String *const versionId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_REMOTE, storage); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(BOOL, compressible); FUNCTION_LOG_PARAM(UINT, compressLevel); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_PARAM(STRING, versionId); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(client != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageReadRemote, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (StorageReadRemote) { .interface = &storageReadRemoteInterface, .storage = storage, .stream = { .memContext = memContextCurrent(), .session = protocolClientSessionNewP(client, PROTOCOL_COMMAND_STORAGE_READ, .async = true), }, .name = strDup(name), .compressible = compressible, .compressLevel = compressLevel, .offset = offset, .limit = varDup(limit), .versionId = strDup(versionId), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_REMOTE, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/remote/read.h0000664000175100017510000000663315265101745020224 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage Read ***********************************************************************************************************************************/ #ifndef STORAGE_REMOTE_READ_H #define STORAGE_REMOTE_READ_H #include "protocol/client.h" #include "storage/read.h" #include "storage/remote/storage.intern.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadRemote StorageReadRemote; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadRemote *storageReadRemoteNew( StorageRemote *storage, ProtocolClient *client, const String *name, bool compressible, unsigned int compressLevel, uint64_t offset, const Variant *limit, const String *versionId); /*********************************************************************************************************************************** Stream state and functions shared with the remote multi read ***********************************************************************************************************************************/ typedef struct StorageReadRemoteStream { MemContext *memContext; // Object context used to allocate chunks ProtocolClientSession *session; // Protocol session for requests size_t remaining; // Bytes remaining to be read in chunk Buffer *chunk; // Chunk currently being read bool eof; // Has the read reached eof? bool eofFound; // Eof found but a chunk is remaining to be read #ifdef DEBUG uint64_t protocolReadBytes; // How many bytes were read from the protocol layer? #endif } StorageReadRemoteStream; // Process a response pack containing chunk data and/or eof with filter results FN_EXTERN void storageReadRemoteStreamInternal(StorageReadRemoteStream *this, IoFilterGroup *filterGroup, PackRead *packRead); // Read from the session stream FN_EXTERN size_t storageReadRemoteStreamRead(StorageReadRemoteStream *this, IoFilterGroup *filterGroup, Buffer *buffer); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_REMOTE_TYPE \ StorageReadRemote * #define FUNCTION_LOG_STORAGE_READ_REMOTE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadRemote", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/remote/readMulti.c0000664000175100017510000002475715265101745021241 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage Read Multi Forward the entire request list to the remote in a single session open so the remote can run the multi read against its locally attached storage, where prefetch and read over are most effective, and stream the result back as one continuous stream. ***********************************************************************************************************************************/ #include #include "common/compress/helper.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/type/object.h" #include "storage/remote/protocol.h" #include "storage/remote/read.h" #include "storage/remote/readMulti.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageReadMultiRemote { const StorageReadMultiInterface *interface; // Interface (must be first member) StorageRemote *storage; // Storage that created this object IoFilterGroup *filterGroup; // Filter group List *requestList; // List of read requests unsigned int compressLevel; // Level to use for compression bool compressible; // Is any file compressible? StorageReadRemoteStream stream; // Protocol stream state }; /*********************************************************************************************************************************** Structure to store read requests until open ***********************************************************************************************************************************/ typedef struct StorageReadMultiRemoteRequest { const String *file; // File to read bool compressible; // Is the file compressible? uint64_t offset; // Where to start reading in the file const Variant *limit; // Limit bytes to read from the file (NULL for no limit) const String *versionId; // Version to read (NULL for current version) } StorageReadMultiRemoteRequest; /*********************************************************************************************************************************** Add a file to the read ***********************************************************************************************************************************/ static void storageReadMultiRemoteAdd(THIS_VOID, const String *const file, const StorageReadMultiAddParam param) { THIS(StorageReadMultiRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI_REMOTE, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(BOOL, param.compressible); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_PARAM(STRING, param.versionId); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); MEM_CONTEXT_OBJ_BEGIN(this->requestList) { const StorageReadMultiRemoteRequest storageReadMultiRemoteRequest = { .file = strDup(file), .compressible = param.compressible, .offset = param.offset, .limit = varDup(param.limit), .versionId = strDup(param.versionId), }; lstAdd(this->requestList, &storageReadMultiRemoteRequest); } MEM_CONTEXT_OBJ_END(); // If any file is compressible then the stream will be compressed for transport if (this->compressLevel > 0 && param.compressible) this->compressible = true; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Set filter group ***********************************************************************************************************************************/ static void storageReadMultiRemoteFilterGroup(THIS_VOID, IoFilterGroup *const filterGroup) { THIS(StorageReadMultiRemote); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_MULTI_REMOTE, this); FUNCTION_TEST_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_TEST_END(); this->filterGroup = filterGroup; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Open the read by sending the request list to the remote ***********************************************************************************************************************************/ static bool storageReadMultiRemoteOpen(THIS_VOID) { THIS(StorageReadMultiRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI_REMOTE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { // If the stream is compressible add compression filter on the remote if (this->compressible) ioFilterGroupAdd(this->filterGroup, compressFilterP(compressTypeLz4, (int)this->compressLevel, .raw = true)); PackWrite *const param = protocolPackNew(); pckWritePackP(param, ioFilterGroupParamAll(this->filterGroup)); // Add requests for (unsigned int requestIdx = 0; requestIdx < lstSize(this->requestList); requestIdx++) { const StorageReadMultiRemoteRequest *const request = lstGet(this->requestList, requestIdx); pckWriteStrP(param, request->file); pckWriteBoolP(param, request->compressible); pckWriteU64P(param, request->offset); if (request->limit == NULL) pckWriteNullP(param); else pckWriteU64P(param, varUInt64(request->limit)); pckWriteStrP(param, request->versionId); } PackRead *const packRead = protocolClientSessionOpenP(this->stream.session, .param = param); // Clear filters since they will be run on the remote side ioFilterGroupClear(this->filterGroup); // If the stream is compressible add decompression filter locally if (this->compressible) ioFilterGroupAdd(this->filterGroup, decompressFilterP(compressTypeLz4, .raw = true)); // Read the first chunk or eof storageReadRemoteStreamInternal(&this->stream, this->filterGroup, packRead); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, true); } /*********************************************************************************************************************************** Read from the remote stream ***********************************************************************************************************************************/ static size_t storageReadMultiRemote(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadMultiRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI_REMOTE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); (void)block; // Blocking is handled by the caller FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL && !bufFull(buffer)); FUNCTION_LOG_RETURN(SIZE, storageReadRemoteStreamRead(&this->stream, this->filterGroup, buffer)); } /*********************************************************************************************************************************** Has read reached EOF? ***********************************************************************************************************************************/ static bool storageReadMultiRemoteEof(THIS_VOID) { THIS(StorageReadMultiRemote); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_MULTI_REMOTE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->stream.eof); } /*********************************************************************************************************************************** Close the read ***********************************************************************************************************************************/ static void storageReadMultiRemoteClose(THIS_VOID) { THIS(StorageReadMultiRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_MULTI_REMOTE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); protocolClientSessionCancel(this->stream.session); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageReadMultiInterface storageReadMultiRemoteInterface = { .add = storageReadMultiRemoteAdd, .close = storageReadMultiRemoteClose, .eof = storageReadMultiRemoteEof, .filterGroup = storageReadMultiRemoteFilterGroup, .open = storageReadMultiRemoteOpen, .read = storageReadMultiRemote, }; FN_EXTERN StorageReadMultiRemote * storageReadMultiRemoteNew(StorageRemote *const storage, ProtocolClient *const client, const unsigned int compressLevel) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_REMOTE, storage); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client); FUNCTION_LOG_PARAM(UINT, compressLevel); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(client != NULL); OBJ_NEW_BEGIN(StorageReadMultiRemote, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (StorageReadMultiRemote) { .interface = &storageReadMultiRemoteInterface, .storage = storage, .requestList = lstNewP(sizeof(StorageReadMultiRemoteRequest)), .compressLevel = compressLevel, .stream = { .memContext = memContextCurrent(), .session = protocolClientSessionNewP(client, PROTOCOL_COMMAND_STORAGE_READ_MULTI, .async = true), }, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_MULTI_REMOTE, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/remote/readMulti.h0000664000175100017510000000355315265101745021235 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage Read Multi ***********************************************************************************************************************************/ #ifndef STORAGE_REMOTE_READ_MULTI_H #define STORAGE_REMOTE_READ_MULTI_H #include "protocol/client.h" #include "storage/readMulti.h" #include "storage/remote/storage.intern.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadMultiRemote StorageReadMultiRemote; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadMultiRemote *storageReadMultiRemoteNew( StorageRemote *storage, ProtocolClient *client, unsigned int compressLevel); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_MULTI_REMOTE_TYPE \ StorageReadMultiRemote * #define FUNCTION_LOG_STORAGE_READ_MULTI_REMOTE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadMultiRemote", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/remote/storage.c0000664000175100017510000004427115265101745020750 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/type/object.h" #include "common/type/pack.h" #include "storage/remote/protocol.h" #include "storage/remote/read.h" #include "storage/remote/readMulti.h" #include "storage/remote/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageRemote { STORAGE_COMMON_MEMBER; ProtocolClient *client; // Protocol client unsigned int compressLevel; // Protocol compression level }; /**********************************************************************************************************************************/ typedef struct StorageRemoteInfoData { MemContext *memContext; // Mem context used to store values from last call time_t timeModifiedLast; // timeModified from last call mode_t modeLast; // mode from last call uid_t userIdLast; // userId from last call gid_t groupIdLast; // groupId from last call String *user; // user from last call String *group; // group from last call } StorageRemoteInfoData; // Helper to get storage info from the protocol output static void storageRemoteInfoGet(StorageRemoteInfoData *const data, PackRead *const read, StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(PACK_READ, read); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(data != NULL); ASSERT(read != NULL); ASSERT(info != NULL); // Read type and time modified info->type = pckReadU32P(read); info->timeModified = pckReadTimeP(read) + data->timeModifiedLast; // Read size for files if (info->type == storageTypeFile) info->size = pckReadU64P(read); // Read version info->versionId = pckReadStrP(read); // Read fields needed for detail level if (info->level >= storageInfoLevelDetail) { // Read mode info->mode = pckReadModeP(read, .defaultValue = data->modeLast); // Read user id/name info->userId = pckReadU32P(read, .defaultValue = data->userIdLast); if (pckReadBoolP(read)) // {vm_covered} info->user = NULL; // {vm_covered} else info->user = pckReadStrP(read, .defaultValue = data->user); // Read group id/name info->groupId = pckReadU32P(read, .defaultValue = data->groupIdLast); if (pckReadBoolP(read)) // {vm_covered} info->group = NULL; // {vm_covered} else info->group = pckReadStrP(read, .defaultValue = data->group); // Read link destination if (info->type == storageTypeLink) info->linkDestination = pckReadStrP(read); } // Store defaults to use for the next call data->timeModifiedLast = info->timeModified; data->modeLast = info->mode; data->userIdLast = info->userId; data->groupIdLast = info->groupId; if (info->user != NULL && !strEq(info->user, data->user)) // {vm_covered} { strFree(data->user); MEM_CONTEXT_BEGIN(data->memContext) { data->user = strDup(info->user); } MEM_CONTEXT_END(); } if (info->group != NULL && !strEq(info->group, data->group)) // {vm_covered} { strFree(data->group); MEM_CONTEXT_BEGIN(data->memContext) { data->group = strDup(info->group); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_VOID(); } static StorageInfo storageRemoteInfo(THIS_VOID, const String *file, StorageInfoLevel level, StorageInterfaceInfoParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(BOOL, param.followLink); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); StorageInfo result = {.level = level}; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, file); pckWriteU32P(commandParam, level); pckWriteBoolP(commandParam, param.followLink); // Read info from protocol PackRead *const read = protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_INFO, .param = commandParam); result.exists = pckReadBoolP(read); if (result.exists) { StorageRemoteInfoData parseData = {.memContext = memContextCurrent()}; // Read info from protocol into prior context MEM_CONTEXT_PRIOR_BEGIN() { storageRemoteInfoGet(&parseData, read, &result); } MEM_CONTEXT_PRIOR_END(); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ static void storageRemoteLinkCreate( THIS_VOID, const String *const target, const String *const linkPath, const StorageInterfaceLinkCreateParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, target); FUNCTION_LOG_PARAM(STRING, linkPath); FUNCTION_LOG_PARAM(ENUM, param.linkType); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(target != NULL); ASSERT(linkPath != NULL); MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, target); pckWriteStrP(commandParam, linkPath); pckWriteU32P(commandParam, param.linkType); protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_LINK_CREATE, .param = commandParam); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static StorageList * storageRemoteList(THIS_VOID, const String *const path, const StorageInfoLevel level, const StorageInterfaceListParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(TIME, param.targetTime); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); StorageList *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, path); pckWriteU32P(commandParam, level); pckWriteTimeP(commandParam, param.targetTime); // Read list StorageRemoteInfoData parseData = {.memContext = memContextCurrent()}; PackRead *const read = protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_LIST, .param = commandParam); if (pckReadBoolP(read)) { result = storageLstNew(level); MEM_CONTEXT_TEMP_RESET_BEGIN() { while (pckReadNext(read)) { pckReadObjBeginP(read); StorageInfo info = {.exists = true, .level = level, .name = pckReadStrP(read)}; storageRemoteInfoGet(&parseData, read, &info); storageLstAdd(result, &info); pckReadObjEndP(read); // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); storageLstMove(result, memContextPrior()); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_LIST, result); } /**********************************************************************************************************************************/ static void * storageRemoteNewRead(THIS_VOID, const String *const file, const StorageInterfaceNewReadParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(BOOL, param.compressible); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_PARAM(STRING, param.versionId); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); FUNCTION_LOG_RETURN( STORAGE_READ_REMOTE, storageReadRemoteNew( this, this->client, file, this->compressLevel > 0 ? param.compressible : false, this->compressLevel, param.offset, param.limit, param.versionId)); } /**********************************************************************************************************************************/ static void * storageRemoteNewReadMulti(THIS_VOID) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(STORAGE_READ_MULTI_REMOTE, storageReadMultiRemoteNew(this, this->client, this->compressLevel)); } /**********************************************************************************************************************************/ static void * storageRemoteNewWrite(THIS_VOID, const String *const file, const StorageInterfaceNewWriteParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(MODE, param.modeFile); FUNCTION_LOG_PARAM(MODE, param.modePath); FUNCTION_LOG_PARAM(STRING, param.user); FUNCTION_LOG_PARAM(STRING, param.group); FUNCTION_LOG_PARAM(TIME, param.timeModified); FUNCTION_LOG_PARAM(BOOL, param.createPath); FUNCTION_LOG_PARAM(BOOL, param.syncFile); FUNCTION_LOG_PARAM(BOOL, param.syncPath); FUNCTION_LOG_PARAM(BOOL, param.atomic); FUNCTION_LOG_PARAM(BOOL, param.compressible); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.truncate); FUNCTION_LOG_RETURN( STORAGE_WRITE_REMOTE, storageWriteRemoteNew( this, this->client, file, param.modeFile, param.modePath, param.user, param.group, param.timeModified, param.createPath, param.syncFile, param.syncPath, param.atomic, this->compressLevel > 0 ? param.compressible : false, this->compressLevel)); } /**********************************************************************************************************************************/ static void storageRemotePathCreate( THIS_VOID, const String *const path, const bool errorOnExists, const bool noParentCreate, const mode_t mode, const StorageInterfacePathCreateParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, errorOnExists); FUNCTION_LOG_PARAM(BOOL, noParentCreate); FUNCTION_LOG_PARAM(MODE, mode); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, path); pckWriteBoolP(commandParam, errorOnExists); pckWriteBoolP(commandParam, noParentCreate); pckWriteModeP(commandParam, mode); protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_PATH_CREATE, .param = commandParam); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static bool storageRemotePathRemove(THIS_VOID, const String *const path, const bool recurse, const StorageInterfacePathRemoveParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, recurse); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, path); pckWriteBoolP(commandParam, recurse); result = pckReadBoolP(protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_PATH_REMOVE, .param = commandParam)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ static void storageRemotePathSync(THIS_VOID, const String *const path, const StorageInterfacePathSyncParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, path); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, path); protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_PATH_SYNC, .param = commandParam); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static void storageRemoteRemove(THIS_VOID, const String *const file, const StorageInterfaceRemoveParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(BOOL, param.errorOnMissing); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, file); pckWriteBoolP(commandParam, param.errorOnMissing); protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_REMOVE, .param = commandParam); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageInterface storageInterfaceRemote = { .info = storageRemoteInfo, .linkCreate = storageRemoteLinkCreate, .list = storageRemoteList, .newRead = storageRemoteNewRead, .newReadMulti = storageRemoteNewReadMulti, .newWrite = storageRemoteNewWrite, .pathCreate = storageRemotePathCreate, .pathRemove = storageRemotePathRemove, .pathSync = storageRemotePathSync, .remove = storageRemoteRemove, }; FN_EXTERN Storage * storageRemoteNew( const mode_t modeFile, const mode_t modePath, const bool write, const time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, ProtocolClient *const client, const unsigned int compressLevel) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client); FUNCTION_LOG_PARAM(UINT, compressLevel); FUNCTION_LOG_END(); ASSERT(modeFile != 0); ASSERT(modePath != 0); ASSERT(client != NULL); const String *path; OBJ_NEW_BEGIN(StorageRemote, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageRemote) { .client = client, .compressLevel = compressLevel, .interface = storageInterfaceRemote, }; // Get storage features from the remote MEM_CONTEXT_TEMP_BEGIN() { // Execute command and get result PackRead *const result = protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_FEATURE); // Get path in parent context MEM_CONTEXT_PRIOR_BEGIN() { path = pckReadStrP(result); } MEM_CONTEXT_PRIOR_END(); this->interface.feature = pckReadU64P(result); } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( STORAGE, storageNew( STORAGE_REMOTE_TYPE, path, modeFile, modePath, write, targetTime, pathExpressionFunction, this, this->interface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/remote/storage.h0000664000175100017510000000241015265101745020742 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage ***********************************************************************************************************************************/ #ifndef STORAGE_REMOTE_STORAGE_H #define STORAGE_REMOTE_STORAGE_H #include "protocol/client.h" #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_REMOTE_TYPE STRID5("remote", 0xb47b4b20) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Storage *storageRemoteNew( mode_t modeFile, mode_t modePath, bool write, time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, ProtocolClient *client, unsigned int compressLevel); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/remote/storage.intern.h0000664000175100017510000000254415265101745022250 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage Internal ***********************************************************************************************************************************/ #ifndef STORAGE_REMOTE_STORAGE_INTERN_H #define STORAGE_REMOTE_STORAGE_INTERN_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageRemote StorageRemote; #include "storage/remote/storage.h" /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_REMOTE_TYPE \ StorageRemote * #define FUNCTION_LOG_STORAGE_REMOTE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageRemote", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/remote/write.c0000664000175100017510000002270215265101745020431 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage File write ***********************************************************************************************************************************/ #include #include "common/compress/helper.h" #include "common/debug.h" #include "common/io/io.h" #include "common/io/write.h" #include "common/log.h" #include "common/type/object.h" #include "storage/remote/protocol.h" #include "storage/remote/write.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWriteRemote { const StorageWriteInterface *interface; // Interface StorageRemote *storage; // Storage that created this object ProtocolClient *client; // Protocol client to make requests with ProtocolClientSession *session; // Protocol session for requests IoFilterGroup *filterGroup; // Filter group const String *name; // File name mode_t modeFile; // File mode mode_t modePath; // Path mode const String *user; // User name const String *group; // Group name time_t timeModified; // Time modified bool createPath; // Create path bool syncFile; // Sync file bool syncPath; // Sync path bool atomic; // Atomic write bool compressible; // Is this file compressible? unsigned int compressLevel; // Level to use for compression #ifdef DEBUG uint64_t protocolWriteBytes; // How many bytes were written to the protocol layer? #endif }; /*********************************************************************************************************************************** Set filter group ***********************************************************************************************************************************/ static void storageWriteRemoteFilterGroup(THIS_VOID, IoFilterGroup *const filterGroup) { THIS(StorageWriteRemote); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_WRITE_REMOTE, this); FUNCTION_TEST_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_TEST_END(); this->filterGroup = filterGroup; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static void storageWriteRemoteOpen(THIS_VOID) { THIS(StorageWriteRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_REMOTE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { // If the file is compressible add decompression filter on the remote if (this->compressible) { ioFilterGroupInsert( this->filterGroup, 0, decompressFilterP(compressTypeLz4, .raw = true)); } PackWrite *const param = protocolPackNew(); pckWriteStrP(param, this->name); pckWriteModeP(param, this->modeFile); pckWriteModeP(param, this->modePath); pckWriteStrP(param, this->user); pckWriteStrP(param, this->group); pckWriteTimeP(param, this->timeModified); pckWriteBoolP(param, this->createPath); pckWriteBoolP(param, this->syncFile); pckWriteBoolP(param, this->syncPath); pckWriteBoolP(param, this->atomic); pckWritePackP(param, ioFilterGroupParamAll(this->filterGroup)); protocolClientSessionOpenP(this->session, .param = param); // Clear filters since they will be run on the remote side ioFilterGroupClear(this->filterGroup); // If the file is compressible add compression filter locally if (this->compressible) { ioFilterGroupAdd( this->filterGroup, compressFilterP(compressTypeLz4, (int)this->compressLevel, .raw = true)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to the file ***********************************************************************************************************************************/ static void storageWriteRemote(THIS_VOID, const Buffer *const buffer) { THIS(StorageWriteRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_REMOTE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); MEM_CONTEXT_TEMP_BEGIN() { if (protocolClientSessionQueued(this->session)) protocolClientSessionResponse(this->session); PackWrite *const param = pckWriteNewP(.size = PROTOCOL_PACK_DEFAULT_SIZE + bufUsed(buffer)); pckWriteBinP(param, buffer); protocolClientSessionRequestAsyncP(this->session, .param = param); } MEM_CONTEXT_TEMP_END(); #ifdef DEBUG this->protocolWriteBytes += bufUsed(buffer); #endif FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWriteRemoteClose(THIS_VOID) { THIS(StorageWriteRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_REMOTE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close if the file has not already been closed if (this->client != NULL) { MEM_CONTEXT_TEMP_BEGIN() { if (protocolClientSessionQueued(this->session)) protocolClientSessionResponse(this->session); ioFilterGroupResultAllSet( this->filterGroup, pckReadPackP(protocolClientSessionClose(this->session))); } MEM_CONTEXT_TEMP_END(); this->client = NULL; } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageWriteInterface storageWriteRemoteInterface = { .close = storageWriteRemoteClose, .filterGroup = storageWriteRemoteFilterGroup, .open = storageWriteRemoteOpen, .write = storageWriteRemote, }; FN_EXTERN StorageWriteRemote * storageWriteRemoteNew( StorageRemote *const storage, ProtocolClient *const client, const String *const name, const mode_t modeFile, const mode_t modePath, const String *const user, const String *const group, const time_t timeModified, const bool createPath, const bool syncFile, const bool syncPath, const bool atomic, const bool compressible, const unsigned int compressLevel) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_REMOTE, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(STRING, user); FUNCTION_LOG_PARAM(STRING, group); FUNCTION_LOG_PARAM(TIME, timeModified); FUNCTION_LOG_PARAM(BOOL, createPath); FUNCTION_LOG_PARAM(BOOL, syncFile); FUNCTION_LOG_PARAM(BOOL, syncPath); FUNCTION_LOG_PARAM(BOOL, atomic); FUNCTION_LOG_PARAM(BOOL, compressible); FUNCTION_LOG_PARAM(UINT, compressLevel); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(client != NULL); ASSERT(name != NULL); ASSERT(modeFile != 0); ASSERT(modePath != 0); OBJ_NEW_BEGIN(StorageWriteRemote, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (StorageWriteRemote) { .interface = &storageWriteRemoteInterface, .storage = storage, .client = client, .session = protocolClientSessionNewP(client, PROTOCOL_COMMAND_STORAGE_WRITE, .async = true), .name = strDup(name), .modeFile = modeFile, .modePath = modePath, .user = strDup(user), .group = strDup(group), .timeModified = timeModified, .createPath = createPath, .syncFile = syncFile, .syncPath = syncPath, .atomic = atomic, .compressible = compressible, .compressLevel = compressLevel, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE_REMOTE, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/remote/write.h0000664000175100017510000000402015265101745020427 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage File write ***********************************************************************************************************************************/ #ifndef STORAGE_REMOTE_WRITE_H #define STORAGE_REMOTE_WRITE_H #include "protocol/client.h" #include "storage/remote/storage.intern.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWriteRemote StorageWriteRemote; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWriteRemote *storageWriteRemoteNew( StorageRemote *storage, ProtocolClient *client, const String *name, mode_t modeFile, mode_t modePath, const String *user, const String *group, time_t timeModified, bool createPath, bool syncFile, bool syncPath, bool atomic, bool compressible, unsigned int compressLevel); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_WRITE_REMOTE_TYPE \ StorageWriteRemote * #define FUNCTION_LOG_STORAGE_WRITE_REMOTE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageWriteRemote", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/s3/0000775000175100017510000000000015265101745016162 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/s3/helper.c0000664000175100017510000001515615265101745017615 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage Helper ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/http/url.h" #include "common/io/io.h" #include "common/log.h" #include "config/config.h" #include "storage/helper.h" #include "storage/posix/storage.h" #include "storage/s3/helper.h" /**********************************************************************************************************************************/ FN_EXTERN Storage * storageS3Helper(const unsigned int repoIdx, const bool write, StoragePathExpressionCallback pathExpressionCallback) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM_P(VOID, pathExpressionCallback); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxStrId(cfgOptRepoType, repoIdx) == STORAGE_S3_TYPE); Storage *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Parse the endpoint url const HttpUrl *const url = httpUrlNewParseP( cfgOptionIdxStr(cfgOptRepoS3Endpoint, repoIdx), .type = httpProtocolTypeAny, .defaultType = httpProtocolTypeHttps); const String *const endPoint = httpUrlHost(url); unsigned int port = httpUrlPort(url); HttpProtocolType protocolType = httpUrlProtocolType(url); // If host was specified then use it const String *host = NULL; if (cfgOptionIdxSource(cfgOptRepoStorageHost, repoIdx) != cfgSourceDefault) { const HttpUrl *const url = httpUrlNewParseP( cfgOptionIdxStr(cfgOptRepoStorageHost, repoIdx), .type = httpProtocolTypeAny, .defaultType = httpProtocolTypeHttps); host = httpUrlHost(url); port = httpUrlPort(url); protocolType = httpUrlProtocolType(url); } // If port was specified, overwrite the parsed/default port if (cfgOptionIdxSource(cfgOptRepoStoragePort, repoIdx) != cfgSourceDefault) port = cfgOptionIdxUInt(cfgOptRepoStoragePort, repoIdx); // Get role and token const StorageS3KeyType keyType = (StorageS3KeyType)cfgOptionIdxSeq(cfgOptRepoS3KeyType, repoIdx); const String *role = cfgOptionIdxStrNull(cfgOptRepoS3Role, repoIdx); const String *tokenFile = NULL; const String *credUrl = NULL; // If web identity authentication then load the role and token filename from environment variables documented here: // https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html if (keyType == storageS3KeyTypeWebId) { #define S3_ENV_AWS_ROLE_ARN "AWS_ROLE_ARN" #define S3_ENV_AWS_WEB_IDENTITY_TOKEN_FILE "AWS_WEB_IDENTITY_TOKEN_FILE" const char *const roleZ = getenv(S3_ENV_AWS_ROLE_ARN); const char *const webIdTokenFileZ = getenv(S3_ENV_AWS_WEB_IDENTITY_TOKEN_FILE); if (roleZ == NULL || webIdTokenFileZ == NULL) { THROW_FMT( OptionInvalidError, "option '%s' is '" CFGOPTVAL_REPO_S3_KEY_TYPE_WEB_ID_Z "' but '" S3_ENV_AWS_ROLE_ARN "' and" " '" S3_ENV_AWS_WEB_IDENTITY_TOKEN_FILE "' are not set", cfgOptionIdxName(cfgOptRepoS3KeyType, repoIdx)); } role = strNewZ(roleZ); tokenFile = strNewZ(webIdTokenFileZ); } // If pod identity authentication then load the credentials url and token filename from environment variables documented // here: https://docs.aws.amazon.com/eks/latest/userguide/pod-id-how-it-works.html else if (keyType == storageS3KeyTypePodId) { #define S3_ENV_AWS_CONTAINER_CREDENTIALS_FULL_URI "AWS_CONTAINER_CREDENTIALS_FULL_URI" #define S3_ENV_AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE" const char *const credUrlZ = getenv(S3_ENV_AWS_CONTAINER_CREDENTIALS_FULL_URI); const char *const podIdTokenFileZ = getenv(S3_ENV_AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE); if (credUrlZ == NULL || podIdTokenFileZ == NULL) { THROW_FMT( OptionInvalidError, "option '%s' is '" CFGOPTVAL_REPO_S3_KEY_TYPE_POD_ID_Z "' but '" S3_ENV_AWS_CONTAINER_CREDENTIALS_FULL_URI "'" " and '" S3_ENV_AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE "' are not set", cfgOptionIdxName(cfgOptRepoS3KeyType, repoIdx)); } credUrl = strNewZ(credUrlZ); tokenFile = strNewZ(podIdTokenFileZ); } // If process credential authentication then get the command from config const StringList *const credCmd = keyType == storageS3KeyTypeProcess ? strLstNewVarLst(cfgOptionIdxLst(cfgOptRepoS3ProcessCmd, repoIdx)) : NULL; MEM_CONTEXT_PRIOR_BEGIN() { result = storageS3New( cfgOptionIdxStr(cfgOptRepoPath, repoIdx), write, storageRepoTargetTime(), pathExpressionCallback, cfgOptionIdxStr(cfgOptRepoS3Bucket, repoIdx), endPoint, cfgOptionIdxStr(cfgOptRepoS3Region, repoIdx), cfgOptionIdxStr(cfgOptRepoS3Service, repoIdx), keyType, (StorageS3UriStyle)cfgOptionIdxSeq(cfgOptRepoS3UriStyle, repoIdx), cfgOptionIdxStrNull(cfgOptRepoS3Key, repoIdx), cfgOptionIdxStrNull(cfgOptRepoS3KeySecret, repoIdx), cfgOptionIdxStrNull(cfgOptRepoS3Token, repoIdx), cfgOptionIdxStrNull(cfgOptRepoS3KmsKeyId, repoIdx), cfgOptionIdxStrNull(cfgOptRepoS3SseCustomerKey, repoIdx), role, tokenFile, credUrl, credCmd, cfgOptionIdxStrNull(cfgOptRepoS3StsHost, repoIdx), (size_t)cfgOptionIdxUInt64(cfgOptRepoStorageUploadChunkSize, repoIdx), cfgOptionIdxKvNull(cfgOptRepoStorageTag, repoIdx), host, port, ioTimeoutMs(), protocolType, cfgOptionIdxBool(cfgOptRepoStorageVerifyTls, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaFile, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaPath, repoIdx), cfgOptionIdxBool(cfgOptRepoS3RequesterPays, repoIdx), cfgOptionIdxUInt(cfgOptRepoStoragePrefetch, repoIdx), cfgOptionIdxUInt64(cfgOptRepoStorageReadOver, repoIdx)); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/s3/helper.h0000664000175100017510000000234715265101745017620 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage Helper ***********************************************************************************************************************************/ #ifndef STORAGE_S3_STORAGE_HELPER_H #define STORAGE_S3_STORAGE_HELPER_H #include "storage/s3/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN Storage *storageS3Helper(unsigned int repoIdx, bool write, StoragePathExpressionCallback pathExpressionCallback); /*********************************************************************************************************************************** Storage helper for StorageHelper array passed to storageHelperInit() ***********************************************************************************************************************************/ #define STORAGE_S3_HELPER {.type = STORAGE_S3_TYPE, .helper = storageS3Helper} #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/s3/read.c0000664000175100017510000001557015265101745017251 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage Read Based on the documentation at https://cloud.google.com/storage/docs/json_api/v1/objects/get ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/client.h" #include "common/log.h" #include "common/type/object.h" #include "storage/read.h" #include "storage/s3/read.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ #define STORAGE_READ_S3_TYPE StorageReadS3 #define STORAGE_READ_S3_PREFIX storageReadS3 struct StorageReadS3 { const StorageReadInterface *interface; // Interface StorageS3 *storage; // Storage that created this object const String *name; // File name uint64_t offset; // Read offset const Variant *limit; // Read limit (NULL for no limit) const String *versionId; // Version id (NULL for most recent) HttpRequest *httpRequest; // HTTP request HttpResponse *httpResponse; // HTTP response }; /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadS3OpenAsync(THIS_VOID) { THIS(StorageReadS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_S3, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpRequest != NULL); // Wait for response MEM_CONTEXT_OBJ_BEGIN(this) { this->httpResponse = storageS3ResponseP(this->httpRequest, .allowMissing = true, .contentIo = true); httpRequestFree(this->httpRequest); this->httpRequest = NULL; } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN(BOOL, httpResponseCodeOk(this->httpResponse)); } static bool storageReadS3Open(THIS_VOID) { THIS(StorageReadS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_S3, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpResponse == NULL); // Request the file MEM_CONTEXT_OBJ_BEGIN(this) { this->httpRequest = storageS3RequestAsyncP( this->storage, HTTP_VERB_GET_STR, this->name, .header = httpHeaderPutRange(httpHeaderNew(NULL), this->offset, this->limit), .query = this->versionId != NULL ? httpQueryPut(httpQueryNewP(), STRDEF("versionId"), this->versionId) : NULL, .sseC = true); } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN(BOOL, true); } /*********************************************************************************************************************************** Read from a file ***********************************************************************************************************************************/ static size_t storageReadS3(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_S3, this); FUNCTION_LOG_PARAM(BUFFER, buffer); (void)block; // Blocking is handled by the caller FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpResponse != NULL); ASSERT(buffer != NULL && !bufFull(buffer)); ASSERT(httpResponseIoRead(this->httpResponse) != NULL); FUNCTION_LOG_RETURN(SIZE, ioRead(httpResponseIoRead(this->httpResponse), buffer)); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadS3Eof(THIS_VOID) { THIS(StorageReadS3); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_S3, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->httpResponse != NULL); ASSERT(httpResponseIoRead(this->httpResponse) != NULL); FUNCTION_TEST_RETURN(BOOL, ioReadEof(httpResponseIoRead(this->httpResponse))); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageReadS3Close(THIS_VOID) { THIS(StorageReadS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_S3, this); FUNCTION_LOG_END(); ASSERT(this != NULL); httpRequestFree(this->httpRequest); this->httpRequest = NULL; httpResponseFree(this->httpResponse); this->httpResponse = NULL; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageReadInterface storageReadS3Interface = { .close = storageReadS3Close, .eof = storageReadS3Eof, .open = storageReadS3Open, .openAsync = storageReadS3OpenAsync, .read = storageReadS3, }; FN_EXTERN StorageReadS3 * storageReadS3New( StorageS3 *const storage, const String *const name, const uint64_t offset, const Variant *const limit, const String *const versionId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_S3, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_PARAM(STRING, versionId); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); ASSERT(limit == NULL || varUInt64(limit) > 0); OBJ_NEW_BEGIN(StorageReadS3, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageReadS3) { .interface = &storageReadS3Interface, .storage = storage, .name = strDup(name), .offset = offset, .limit = varDup(limit), .versionId = strDup(versionId), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_S3, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/s3/read.h0000664000175100017510000000341515265101745017251 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage Read ***********************************************************************************************************************************/ #ifndef STORAGE_S3_READ_H #define STORAGE_S3_READ_H #include "storage/read.h" #include "storage/s3/storage.intern.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadS3 StorageReadS3; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadS3 *storageReadS3New( StorageS3 *storage, const String *name, uint64_t offset, const Variant *limit, const String *versionId); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_S3_TYPE \ StorageReadS3 * #define FUNCTION_LOG_STORAGE_READ_S3_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadS3", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/s3/storage.c0000664000175100017510000021117415265101745020000 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage ***********************************************************************************************************************************/ #include #include #include "common/crypto/hash.h" #include "common/debug.h" #include "common/exec.h" #include "common/io/http/client.h" #include "common/io/http/common.h" #include "common/io/socket/client.h" #include "common/io/tls/client.h" #include "common/log.h" #include "common/regExp.h" #include "common/type/json.h" #include "common/type/object.h" #include "common/type/xml.h" #include "storage/posix/storage.h" #include "storage/s3/read.h" #include "storage/s3/write.h" /*********************************************************************************************************************************** Defaults ***********************************************************************************************************************************/ #define STORAGE_S3_DELETE_MAX 1000 /*********************************************************************************************************************************** S3 HTTP headers ***********************************************************************************************************************************/ STRING_STATIC(S3_HEADER_CONTENT_SHA256_STR, "x-amz-content-sha256"); STRING_STATIC(S3_HEADER_DATE_STR, "x-amz-date"); STRING_STATIC(S3_HEADER_TOKEN_STR, "x-amz-security-token"); STRING_STATIC(S3_HEADER_SRVSDENC_STR, "x-amz-server-side-encryption"); STRING_STATIC(S3_HEADER_SRVSDENC_KMS_STR, "aws:kms"); STRING_STATIC(S3_HEADER_SRVSDENC_KMSKEYID_STR, "x-amz-server-side-encryption-aws-kms-key-id"); STRING_STATIC(S3_HEADER_SSECUSTKEY_ALGO_STR, "x-amz-server-side-encryption-customer-algorithm"); STRING_STATIC(S3_HEADER_SSECUSTKEY_AES256_STR, "AES256"); STRING_STATIC(S3_HEADER_SSECUSTKEY_KEY_STR, "x-amz-server-side-encryption-customer-key"); STRING_STATIC(S3_HEADER_SSECUSTKEY_KEY_MD5_STR, "x-amz-server-side-encryption-customer-key-md5"); STRING_STATIC(S3_HEADER_TAGGING, "x-amz-tagging"); STRING_STATIC(S3_HEADER_REQUEST_PAYER, "x-amz-request-payer"); STRING_STATIC(S3_HEADER_REQUEST_PAYER_STR, "requester"); /*********************************************************************************************************************************** S3 query tokens ***********************************************************************************************************************************/ STRING_STATIC(S3_QUERY_CONTINUATION_TOKEN_STR, "continuation-token"); STRING_STATIC(S3_QUERY_DELETE_STR, "delete"); STRING_STATIC(S3_QUERY_DELIMITER_STR, "delimiter"); STRING_STATIC(S3_QUERY_KEY_MARKER_STR, "key-marker"); STRING_STATIC(S3_QUERY_LIST_TYPE_STR, "list-type"); STRING_STATIC(S3_QUERY_PREFIX_STR, "prefix"); STRING_STATIC(S3_QUERY_VERSION_ID_MARKER_STR, "version-id-marker"); STRING_STATIC(S3_QUERY_VERSIONS_STR, "versions"); STRING_STATIC(S3_QUERY_VALUE_LIST_TYPE_2_STR, "2"); /*********************************************************************************************************************************** XML tags ***********************************************************************************************************************************/ STRING_STATIC(S3_XML_TAG_COMMON_PREFIXES_STR, "CommonPrefixes"); STRING_STATIC(S3_XML_TAG_CONTENTS_STR, "Contents"); STRING_STATIC(S3_XML_TAG_DELETE_STR, "Delete"); STRING_STATIC(S3_XML_TAG_DELETE_MARKER_STR, "DeleteMarker"); STRING_STATIC(S3_XML_TAG_ERROR_STR, "Error"); STRING_STATIC(S3_XML_TAG_IS_TRUNCATED_STR, "IsTruncated"); STRING_STATIC(S3_XML_TAG_KEY_STR, "Key"); STRING_STATIC(S3_XML_TAG_LAST_MODIFIED_STR, "LastModified"); #define S3_XML_TAG_NEXT_CONTINUATION_TOKEN "NextContinuationToken" STRING_STATIC(S3_XML_TAG_NEXT_CONTINUATION_TOKEN_STR, S3_XML_TAG_NEXT_CONTINUATION_TOKEN); #define S3_XML_TAG_NEXT_KEY_MARKER "NextKeyMarker" STRING_STATIC(S3_XML_TAG_NEXT_KEY_MARKER_STR, S3_XML_TAG_NEXT_KEY_MARKER); STRING_STATIC(S3_XML_TAG_NEXT_VERSION_ID_MARKER_STR, "NextVersionIdMarker"); STRING_STATIC(S3_XML_TAG_OBJECT_STR, "Object"); STRING_STATIC(S3_XML_TAG_PREFIX_STR, "Prefix"); STRING_STATIC(S3_XML_TAG_QUIET_STR, "Quiet"); STRING_STATIC(S3_XML_TAG_SIZE_STR, "Size"); STRING_STATIC(S3_XML_TAG_VERSION_STR, "Version"); STRING_STATIC(S3_XML_TAG_VERSION_ID_STR, "VersionId"); /*********************************************************************************************************************************** AWS authentication v4 constants ***********************************************************************************************************************************/ #define AWS4 "AWS4" #define AWS4_REQUEST "aws4_request" BUFFER_STRDEF_STATIC(AWS4_REQUEST_BUF, AWS4_REQUEST); #define AWS4_HMAC_SHA256 "AWS4-HMAC-SHA256" /*********************************************************************************************************************************** Starting date for signing string so it will be regenerated on the first request ***********************************************************************************************************************************/ STRING_STATIC(YYYYMMDD_STR, "YYYYMMDD"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageS3 { STORAGE_COMMON_MEMBER; HttpClient *httpClient; // HTTP client to service requests StringList *headerRedactList; // List of headers to redact from logging const String *bucket; // Bucket to store data in const String *region; // e.g. us-east-1 const String *signingService; // SigV4 signing service StorageS3KeyType keyType; // Key type (e.g. storageS3KeyTypeShared) String *accessKey; // Access key String *secretAccessKey; // Secret access key String *securityToken; // Security token, if any const String *kmsKeyId; // Server-side encryption key const String *sseCustomerKey; // Base64 of SSE-C encryption key const String *sseCustomerKeyMd5; // Base64 of MD5 of SSE-C key size_t partSize; // Part size for multi-part upload const String *tag; // Tags to be applied to objects unsigned int deleteMax; // Maximum objects that can be deleted in one request StorageS3UriStyle uriStyle; // Path or host style URIs const String *bucketEndpoint; // Set to {bucket}.{endpoint} bool requesterPays; // Requester pays? // For retrieving temporary security credentials HttpClient *credHttpClient; // HTTP client to service credential requests const String *credHost; // Credentials host const String *credPath; // Credential url path const String *credRole; // Role to use for credential requests const String *tokenFile; // File with token to use for web-id/pod-id credential requests const StringList *credCmd; // Command and parameters for process credential requests time_t credExpirationTime; // Time the temporary credentials expire // Current signing key and date it is valid for const String *signingKeyDate; // Date of cached signing key (so we know when to regenerate) const Buffer *signingKey; // Cached signing key }; /*********************************************************************************************************************************** Expected ISO-8601 data/time size ***********************************************************************************************************************************/ #define ISO_8601_DATE_TIME_SIZE 16 /*********************************************************************************************************************************** Generate authorization header and add it to the supplied header list Based on the excellent documentation at http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html. ***********************************************************************************************************************************/ static void storageS3Auth( StorageS3 *const this, const String *const verb, const String *const path, const HttpQuery *const query, const String *const dateTime, HttpHeader *const httpHeader, const String *const payloadHash) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_S3, this); FUNCTION_TEST_PARAM(STRING, verb); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(HTTP_QUERY, query); FUNCTION_TEST_PARAM(STRING, dateTime); FUNCTION_TEST_PARAM(KEY_VALUE, httpHeader); FUNCTION_TEST_PARAM(STRING, payloadHash); FUNCTION_TEST_END(); ASSERT(verb != NULL); ASSERT(path != NULL); ASSERT(dateTime != NULL); ASSERT(httpHeader != NULL); ASSERT(payloadHash != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Get date from datetime const String *const date = strSubN(dateTime, 0, 8); // Set required headers httpHeaderPut(httpHeader, S3_HEADER_CONTENT_SHA256_STR, payloadHash); httpHeaderPut(httpHeader, S3_HEADER_DATE_STR, dateTime); httpHeaderPut(httpHeader, HTTP_HEADER_HOST_STR, this->bucketEndpoint); if (this->securityToken != NULL) httpHeaderPut(httpHeader, S3_HEADER_TOKEN_STR, this->securityToken); // Generate canonical request and signed headers const StringList *const headerList = strLstSort(strLstDup(httpHeaderList(httpHeader)), sortOrderAsc); String *signedHeaders = NULL; String *const canonicalRequest = strCatFmt( strNew(), "%s\n%s\n%s\n", strZ(verb), strZ(path), query == NULL ? "" : strZ(httpQueryRenderP(query))); for (unsigned int headerIdx = 0; headerIdx < strLstSize(headerList); headerIdx++) { const String *const headerKey = strLstGet(headerList, headerIdx); const String *const headerKeyLower = strLower(strDup(headerKey)); // Skip the authorization (exists on retry) and content-length headers since they do not need to be signed if (strEq(headerKeyLower, HTTP_HEADER_AUTHORIZATION_STR) || strEq(headerKeyLower, HTTP_HEADER_CONTENT_LENGTH_STR)) continue; strCatFmt(canonicalRequest, "%s:%s\n", strZ(headerKeyLower), strZ(httpHeaderGet(httpHeader, headerKey))); if (signedHeaders == NULL) signedHeaders = strCat(strNew(), headerKeyLower); else strCatFmt(signedHeaders, ";%s", strZ(headerKeyLower)); } strCatFmt(canonicalRequest, "\n%s\n%s", strZ(signedHeaders), strZ(payloadHash)); // Generate string to sign const String *const stringToSign = strNewFmt( AWS4_HMAC_SHA256 "\n%s\n%s/%s/%s/" AWS4_REQUEST "\n%s", strZ(dateTime), strZ(date), strZ(this->region), strZ(this->signingService), strZ(strNewEncode(encodingHex, cryptoHashOne(hashTypeSha256, BUFSTR(canonicalRequest))))); // Generate signing key. This key only needs to be regenerated every seven days but we'll do it once a day to keep the // logic simple. It's a relatively expensive operation so we'd rather not do it for every request. // If the cached signing key has expired (or has none been generated) then regenerate it if (!strEq(date, this->signingKeyDate)) { const Buffer *const dateKey = cryptoHmacOne( hashTypeSha256, BUFSTR(strNewFmt(AWS4 "%s", strZ(this->secretAccessKey))), BUFSTR(date)); const Buffer *const regionKey = cryptoHmacOne(hashTypeSha256, dateKey, BUFSTR(this->region)); const Buffer *const serviceKey = cryptoHmacOne(hashTypeSha256, regionKey, BUFSTR(this->signingService)); // Switch to the object context so signing key and date are not lost MEM_CONTEXT_OBJ_BEGIN(this) { this->signingKey = cryptoHmacOne(hashTypeSha256, serviceKey, AWS4_REQUEST_BUF); this->signingKeyDate = strDup(date); } MEM_CONTEXT_OBJ_END(); } // Generate authorization header const String *const authorization = strNewFmt( AWS4_HMAC_SHA256 " Credential=%s/%s/%s/%s/" AWS4_REQUEST ",SignedHeaders=%s,Signature=%s", strZ(this->accessKey), strZ(date), strZ(this->region), strZ(this->signingService), strZ(signedHeaders), strZ(strNewEncode(encodingHex, cryptoHmacOne(hashTypeSha256, this->signingKey, BUFSTR(stringToSign))))); httpHeaderPut(httpHeader, HTTP_HEADER_AUTHORIZATION_STR, authorization); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Convert YYYY-MM-DDTHH:MM:SS.MSECZ format to time_t. This format is very nearly ISO-8601 except for the inclusion of milliseconds, which are discarded here. ***********************************************************************************************************************************/ static time_t storageS3CvtTime(const String *const time) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, time); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN( TIME, epochFromParts( cvtZSubNToInt(strZ(time), 0, 4), cvtZSubNToInt(strZ(time), 5, 2), cvtZSubNToInt(strZ(time), 8, 2), cvtZSubNToInt(strZ(time), 11, 2), cvtZSubNToInt(strZ(time), 14, 2), cvtZSubNToInt(strZ(time), 17, 2), 0)); } /*********************************************************************************************************************************** Automatically get credentials for an associated IAM role Documentation for the response format is found at: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html Documentation for IMDSv2 tokens: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html ***********************************************************************************************************************************/ STRING_STATIC(S3_CREDENTIAL_HOST_STR, "169.254.169.254"); #define S3_CREDENTIAL_PORT 80 #define S3_CREDENTIAL_PATH "/latest/meta-data/iam/security-credentials" #define S3_CREDENTIAL_RENEW_SEC (5 * 60) VARIANT_STRDEF_STATIC(S3_JSON_TAG_ACCESS_KEY_ID_VAR, "AccessKeyId"); VARIANT_STRDEF_STATIC(S3_JSON_TAG_CODE_VAR, "Code"); VARIANT_STRDEF_STATIC(S3_JSON_TAG_EXPIRATION_VAR, "Expiration"); VARIANT_STRDEF_STATIC(S3_JSON_TAG_SECRET_ACCESS_KEY_VAR, "SecretAccessKey"); VARIANT_STRDEF_STATIC(S3_JSON_TAG_TOKEN_VAR, "Token"); VARIANT_STRDEF_STATIC(S3_JSON_VALUE_SUCCESS_VAR, "Success"); static void storageS3AuthAuto(StorageS3 *const this, HttpHeader *const header) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(HTTP_HEADER, header); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // Retrieve the IMDSv2 token. Do not store the token because it will likely be expired by the time the credentials expire // since credentials have a much longer default expiration. The TTL needs to be long enough to survive retries of the two // metadata queries. HttpHeader *const tokenHeader = httpHeaderDup(header, NULL); httpHeaderAdd( tokenHeader, STRDEF("x-aws-ec2-metadata-token-ttl-seconds"), strNewFmt("%" PRIu64, httpClientTimeout(this->credHttpClient) / 1000 * 3)); HttpRequest *request = httpRequestNewP( this->credHttpClient, HTTP_VERB_PUT_STR, STRDEF("/latest/api/token"), .header = tokenHeader); HttpResponse *response = httpRequestResponse(request, true); // Set IMDSv2 token on success. If the token request fails for any reason assume that IMDSv2 is not supported. If the // instance only supports IMDSv2 then subsequent metadata requests will fail because of the missing token. if (httpResponseCodeOk(response)) httpHeaderAdd(header, STRDEF("x-aws-ec2-metadata-token"), strNewBuf(httpResponseContent(response))); // If the role was not set explicitly or retrieved previously then retrieve it if (this->credRole == NULL) { // Request the role HttpRequest *const request = httpRequestNewP( this->credHttpClient, HTTP_VERB_GET_STR, STRDEF(S3_CREDENTIAL_PATH), .header = header); HttpResponse *const response = httpRequestResponse(request, true); // Not found likely means no role is associated with this instance if (httpResponseCode(response) == HTTP_RESPONSE_CODE_NOT_FOUND) { THROW( ProtocolError, "role to retrieve temporary credentials not found\n" "HINT: is a valid IAM role associated with this instance?"); } // Else an error that we can't handle else if (!httpResponseCodeOk(response)) httpRequestError(request, response); // Get role from the text response MEM_CONTEXT_OBJ_BEGIN(this) { this->credRole = strNewBuf(httpResponseContent(response)); } MEM_CONTEXT_OBJ_END(); } // Retrieve the temp credentials request = httpRequestNewP( this->credHttpClient, HTTP_VERB_GET_STR, strNewFmt(S3_CREDENTIAL_PATH "/%s", strZ(this->credRole)), .header = header); response = httpRequestResponse(request, true); // Not found likely means the role is not valid if (httpResponseCode(response) == HTTP_RESPONSE_CODE_NOT_FOUND) { THROW_FMT( ProtocolError, "role '%s' not found\n" "HINT: is '%s' a valid IAM role associated with this instance?", strZ(this->credRole), strZ(this->credRole)); } // Else an error that we can't handle else if (!httpResponseCodeOk(response)) httpRequestError(request, response); // Get credentials from the JSON response const KeyValue *const credential = varKv(jsonToVar(strNewBuf(httpResponseContent(response)))); MEM_CONTEXT_OBJ_BEGIN(this) { // Check the code field for errors const Variant *const code = kvGetDefault(credential, S3_JSON_TAG_CODE_VAR, VARSTRDEF("code field is missing")); CHECK(FormatError, code != NULL, "error code missing"); if (!varEq(code, S3_JSON_VALUE_SUCCESS_VAR)) THROW_FMT(FormatError, "unable to retrieve temporary credentials: %s", strZ(varStr(code))); // Make sure the required values are present CHECK(FormatError, kvGet(credential, S3_JSON_TAG_ACCESS_KEY_ID_VAR) != NULL, "access key missing"); CHECK(FormatError, kvGet(credential, S3_JSON_TAG_SECRET_ACCESS_KEY_VAR) != NULL, "secret access key missing"); CHECK(FormatError, kvGet(credential, S3_JSON_TAG_TOKEN_VAR) != NULL, "token missing"); // Copy credentials this->accessKey = strDup(varStr(kvGet(credential, S3_JSON_TAG_ACCESS_KEY_ID_VAR))); this->secretAccessKey = strDup(varStr(kvGet(credential, S3_JSON_TAG_SECRET_ACCESS_KEY_VAR))); this->securityToken = strDup(varStr(kvGet(credential, S3_JSON_TAG_TOKEN_VAR))); } MEM_CONTEXT_OBJ_END(); // Update expiration time CHECK(FormatError, kvGet(credential, S3_JSON_TAG_EXPIRATION_VAR) != NULL, "expiration missing"); this->credExpirationTime = storageS3CvtTime(varStr(kvGet(credential, S3_JSON_TAG_EXPIRATION_VAR))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Automatically get credentials for an associated web identity service account Documentation is found at: https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html ***********************************************************************************************************************************/ static void storageS3AuthWebId(StorageS3 *const this, const HttpHeader *const header) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(HTTP_HEADER, header); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // Load the token from the given file for each request since the token may be updated during execution const String *const webIdToken = strNewBuf(storageGetP(storageNewReadP(storagePosixNewP(FSLASH_STR), this->tokenFile))); // Get credentials. Redact the web identity token so it is not exposed if the query is logged, e.g. in an error. StringList *const queryRedactList = strLstNew(); strLstAddZ(queryRedactList, "WebIdentityToken"); HttpQuery *const query = httpQueryNewP(.redactList = queryRedactList); httpQueryAdd(query, STRDEF("Action"), STRDEF("AssumeRoleWithWebIdentity")); httpQueryAdd(query, STRDEF("RoleArn"), this->credRole); httpQueryAdd(query, STRDEF("RoleSessionName"), STRDEF(PROJECT_NAME)); httpQueryAdd(query, STRDEF("Version"), STRDEF("2011-06-15")); httpQueryAdd(query, STRDEF("WebIdentityToken"), webIdToken); HttpRequest *const request = httpRequestNewP( this->credHttpClient, HTTP_VERB_GET_STR, FSLASH_STR, .header = header, .query = query); HttpResponse *const response = httpRequestResponse(request, true); // Error if the request was not successful, e.g. the web identity token is invalid. Without this the error would be a // confusing report that AssumeRoleWithWebIdentityResult is missing rather than the actual error returned by AWS. if (!httpResponseCodeOk(response)) httpRequestError(request, response); // Copy credentials const XmlNode *const xmlCred = xmlNodeChild( xmlNodeChild( xmlDocumentRoot(xmlDocumentNewBuf(httpResponseContent(response))), STRDEF("AssumeRoleWithWebIdentityResult"), true), STRDEF("Credentials"), true); const XmlNode *const accessKeyNode = xmlNodeChild(xmlCred, STRDEF("AccessKeyId"), true); const XmlNode *const secretAccessKeyNode = xmlNodeChild(xmlCred, STRDEF("SecretAccessKey"), true); const XmlNode *const securityTokenNode = xmlNodeChild(xmlCred, STRDEF("SessionToken"), true); MEM_CONTEXT_OBJ_BEGIN(this) { this->accessKey = xmlNodeContent(accessKeyNode); this->secretAccessKey = xmlNodeContent(secretAccessKeyNode); this->securityToken = xmlNodeContent(securityTokenNode); } MEM_CONTEXT_OBJ_END(); // Update expiration time this->credExpirationTime = storageS3CvtTime(xmlNodeContent(xmlNodeChild(xmlCred, STRDEF("Expiration"), true))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Automatically get credentials for an associated pod identity The AWS documentation does not appear to provide a clear example of how to perform this authorization without using their SDK but this article proved helpful: https://securitylabs.datadoghq.com/articles/eks-pod-identity-deep-dive ***********************************************************************************************************************************/ static void storageS3AuthPodId(StorageS3 *const this, const HttpHeader *const header) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(HTTP_HEADER, header); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // Load the token from the given file for each request since the token may be updated during execution const String *const podIdToken = strNewBuf(storageGetP(storageNewReadP(storagePosixNewP(FSLASH_STR), this->tokenFile))); // Get credentials. Redact the authorization header, which holds the token, so it is not exposed if the header is logged, // e.g. in an error. HttpRequest *const request = httpRequestNewP( this->credHttpClient, HTTP_VERB_GET_STR, this->credPath, .header = httpHeaderAdd(httpHeaderDup(header, this->headerRedactList), HTTP_HEADER_AUTHORIZATION_STR, podIdToken)); HttpResponse *const response = httpRequestResponse(request, true); // Error if the request was not successful, e.g. the pod identity token is invalid. Without this the error would be a // confusing report about missing JSON keys rather than the actual error returned by AWS. if (!httpResponseCodeOk(response)) httpRequestError(request, response); const KeyValue *const kvResponse = varKv(jsonToVar(strNewBuf(httpResponseContent(response)))); // Copy credentials MEM_CONTEXT_OBJ_BEGIN(this) { this->accessKey = strDup(varStr(kvGet(kvResponse, VARSTRDEF("AccessKeyId")))); CHECK(FormatError, this->accessKey != NULL, "access key missing"); this->secretAccessKey = strDup(varStr(kvGet(kvResponse, VARSTRDEF("SecretAccessKey")))); CHECK(FormatError, this->secretAccessKey != NULL, "secret access key missing"); this->securityToken = strDup(varStr(kvGet(kvResponse, VARSTRDEF("Token")))); CHECK(FormatError, this->securityToken != NULL, "token missing"); } MEM_CONTEXT_OBJ_END(); // Update expiration time const String *const credExpirationTime = varStr(kvGet(kvResponse, VARSTRDEF("Expiration"))); CHECK(FormatError, credExpirationTime != NULL, "expiration missing"); this->credExpirationTime = storageS3CvtTime(credExpirationTime); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get credentials from a process credential provider The process is executed and the JSON output is parsed for AccessKeyId, SecretAccessKey, SessionToken, and Expiration. Based on the "process credential provider" documentation at https://docs.aws.amazon.com/sdkref/latest/guide/feature-process-credentials.html ***********************************************************************************************************************************/ VARIANT_STRDEF_STATIC(S3_JSON_TAG_SESSION_TOKEN_VAR, "SessionToken"); static void storageS3AuthProcess(StorageS3 *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // Execute the credential process and parse JSON output const String *const output = execOne(this->credCmd); const KeyValue *const kvResponse = varKv(jsonToVar(output)); // Copy credentials MEM_CONTEXT_OBJ_BEGIN(this) { this->accessKey = strDup(varStr(kvGet(kvResponse, S3_JSON_TAG_ACCESS_KEY_ID_VAR))); CHECK(FormatError, this->accessKey != NULL, "access key missing"); this->secretAccessKey = strDup(varStr(kvGet(kvResponse, S3_JSON_TAG_SECRET_ACCESS_KEY_VAR))); CHECK(FormatError, this->secretAccessKey != NULL, "secret access key missing"); this->securityToken = strDup(varStr(kvGet(kvResponse, S3_JSON_TAG_SESSION_TOKEN_VAR))); CHECK(FormatError, this->securityToken != NULL, "session token missing"); } MEM_CONTEXT_OBJ_END(); // Update expiration time const String *const credExpirationTime = varStr(kvGet(kvResponse, S3_JSON_TAG_EXPIRATION_VAR)); CHECK(FormatError, credExpirationTime != NULL, "expiration missing"); this->credExpirationTime = storageS3CvtTime(credExpirationTime); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Process S3 request ***********************************************************************************************************************************/ FN_EXTERN HttpRequest * storageS3RequestAsync(StorageS3 *const this, const String *const verb, const String *path, const StorageS3RequestAsyncParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_PARAM(BOOL, param.sseKms); FUNCTION_LOG_PARAM(BOOL, param.sseC); FUNCTION_LOG_PARAM(BOOL, param.tag); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(verb != NULL); ASSERT(path != NULL); HttpRequest *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { HttpHeader *const requestHeader = param.header == NULL ? httpHeaderNew(this->headerRedactList) : httpHeaderDup(param.header, this->headerRedactList); // Set content length httpHeaderAdd( requestHeader, HTTP_HEADER_CONTENT_LENGTH_STR, param.content == NULL || bufEmpty(param.content) ? ZERO_STR : strNewFmt("%zu", bufUsed(param.content))); // Calculate content-md5 header if there is content if (param.content != NULL) { httpHeaderAdd( requestHeader, HTTP_HEADER_CONTENT_MD5_STR, strNewEncode(encodingBase64, cryptoHashOne(hashTypeMd5, param.content))); } // Set requester pays when requested if (this->requesterPays) httpHeaderPut(requestHeader, S3_HEADER_REQUEST_PAYER, S3_HEADER_REQUEST_PAYER_STR); // Set KMS headers when requested if (param.sseKms && this->kmsKeyId != NULL) { httpHeaderPut(requestHeader, S3_HEADER_SRVSDENC_STR, S3_HEADER_SRVSDENC_KMS_STR); httpHeaderPut(requestHeader, S3_HEADER_SRVSDENC_KMSKEYID_STR, this->kmsKeyId); } // Set SSE-C headers when requested if (param.sseC && this->sseCustomerKey != NULL) { httpHeaderPut(requestHeader, S3_HEADER_SSECUSTKEY_ALGO_STR, S3_HEADER_SSECUSTKEY_AES256_STR); httpHeaderPut(requestHeader, S3_HEADER_SSECUSTKEY_KEY_STR, this->sseCustomerKey); httpHeaderPut(requestHeader, S3_HEADER_SSECUSTKEY_KEY_MD5_STR, this->sseCustomerKeyMd5); } // Set tags when requested and available if (param.tag && this->tag != NULL) httpHeaderPut(requestHeader, S3_HEADER_TAGGING, this->tag); // When using path-style URIs the bucket name needs to be prepended if (this->uriStyle == storageS3UriStylePath) path = strNewFmt("/%s%s", strZ(this->bucket), strZ(path)); // If temp credentials will be expiring soon then renew them if (this->credExpirationTime != 0 && (this->credExpirationTime - time(NULL)) < S3_CREDENTIAL_RENEW_SEC) { // Free old credentials strFree(this->accessKey); strFree(this->secretAccessKey); strFree(this->securityToken); // Process credential authentication (no HTTP client needed) if (this->keyType == storageS3KeyTypeProcess) { storageS3AuthProcess(this); } // Else HTTP-based authentication methods else { // Set content-length and host headers HttpHeader *credHeader = httpHeaderNew(NULL); httpHeaderAdd(credHeader, HTTP_HEADER_CONTENT_LENGTH_STR, ZERO_STR); httpHeaderAdd(credHeader, HTTP_HEADER_HOST_STR, this->credHost); switch (this->keyType) { // Auto authentication case storageS3KeyTypeAuto: storageS3AuthAuto(this, credHeader); break; // Pod identity authentication case storageS3KeyTypePodId: storageS3AuthPodId(this, credHeader); break; // Web identity authentication default: { ASSERT(this->keyType == storageS3KeyTypeWebId); storageS3AuthWebId(this, credHeader); break; } } } // Reset the signing key date so the signing key gets regenerated this->signingKeyDate = YYYYMMDD_STR; } // Encode path path = httpUriEncode(path, true); // Generate authorization header storageS3Auth( this, verb, path, param.query, strNewTimeP("%Y%m%dT%H%M%SZ", time(NULL), .utc = true), requestHeader, strNewEncode( encodingHex, param.content == NULL || bufEmpty(param.content) ? HASH_TYPE_SHA256_ZERO_BUF : cryptoHashOne(hashTypeSha256, param.content))); // Send request MEM_CONTEXT_PRIOR_BEGIN() { result = httpRequestNewP( this->httpClient, verb, path, .query = param.query, .header = requestHeader, .content = param.content); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_REQUEST, result); } FN_EXTERN HttpResponse * storageS3Response(HttpRequest *const request, const StorageS3ResponseParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_REQUEST, request); FUNCTION_LOG_PARAM(BOOL, param.allowMissing); FUNCTION_LOG_PARAM(BOOL, param.contentIo); FUNCTION_LOG_END(); ASSERT(request != NULL); HttpResponse *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get response result = httpRequestResponse(request, !param.contentIo); // Error if the request was not successful if (!httpResponseCodeOk(result) && (!param.allowMissing || httpResponseCode(result) != HTTP_RESPONSE_CODE_NOT_FOUND)) httpRequestError(request, result); // Move response to the prior context httpResponseMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } FN_EXTERN HttpResponse * storageS3Request(StorageS3 *const this, const String *const verb, const String *const path, const StorageS3RequestParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_PARAM(BOOL, param.allowMissing); FUNCTION_LOG_PARAM(BOOL, param.contentIo); FUNCTION_LOG_PARAM(BOOL, param.sseKms); FUNCTION_LOG_PARAM(BOOL, param.sseC); FUNCTION_LOG_PARAM(BOOL, param.tag); FUNCTION_LOG_END(); HttpRequest *const request = storageS3RequestAsyncP( this, verb, path, .header = param.header, .query = param.query, .content = param.content, .sseKms = param.sseKms, .sseC = param.sseC, .tag = param.tag); HttpResponse *const result = storageS3ResponseP( request, .allowMissing = param.allowMissing, .contentIo = param.contentIo); httpRequestFree(request); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } /*********************************************************************************************************************************** General function for listing files to be used by other list routines ***********************************************************************************************************************************/ static void storageS3ListInternal( StorageS3 *const this, const String *const path, const StorageInfoLevel level, const String *const expression, const bool recurse, const time_t targetTime, StorageListCallback callback, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(STRING, expression); FUNCTION_LOG_PARAM(BOOL, recurse); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, callback); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Build the base prefix by stripping off the initial / const String *const basePrefix = strSize(path) == 1 ? EMPTY_STR : strNewFmt("%s/", strZ(strSub(path, 1))); // Get the expression prefix when possible to limit initial results const String *const expressionPrefix = regExpPrefix(expression); // If there is an expression prefix then use it to build the query prefix, otherwise query prefix is base prefix const String *queryPrefix; if (expressionPrefix == NULL) queryPrefix = basePrefix; else { if (strEmpty(basePrefix)) queryPrefix = expressionPrefix; else queryPrefix = strNewFmt("%s%s", strZ(basePrefix), strZ(expressionPrefix)); } // Create query HttpQuery *const query = httpQueryNewP(); // Add the delimiter to not recurse if (!recurse) httpQueryAdd(query, S3_QUERY_DELIMITER_STR, FSLASH_STR); // Use list type 2 or versions as specified if (targetTime != 0) httpQueryAdd(query, S3_QUERY_VERSIONS_STR, EMPTY_STR); else httpQueryAdd(query, S3_QUERY_LIST_TYPE_STR, S3_QUERY_VALUE_LIST_TYPE_2_STR); // Don't specify empty prefix because it is the default if (!strEmpty(queryPrefix)) httpQueryAdd(query, S3_QUERY_PREFIX_STR, queryPrefix); // Store last info so it can be updated across requests for versioning String *const nameLast = strNew(); String *const versionIdLast = strNew(); StorageInfo infoLast = {.level = level, .name = nameLast}; if (targetTime != 0) infoLast.versionId = versionIdLast; // Loop as long as a continuation token returned HttpRequest *request = NULL; do { // Use an inner mem context here because we could potentially be retrieving millions of files so it is a good idea to // free memory at regular intervals MEM_CONTEXT_TEMP_BEGIN() { HttpResponse *response = NULL; // If there is an outstanding async request then wait for the response if (request != NULL) { response = storageS3ResponseP(request); httpRequestFree(request); request = NULL; } // Else get the response immediately from a sync request else response = storageS3RequestP(this, HTTP_VERB_GET_STR, FSLASH_STR, .query = query); const XmlNode *const xmlRoot = xmlDocumentRoot(xmlDocumentNewBuf(httpResponseContent(response))); // If list is truncated then send an async request to get more data if (strEq(xmlNodeContent(xmlNodeChild(xmlRoot, S3_XML_TAG_IS_TRUNCATED_STR, true)), TRUE_STR)) { // Build the query for the next page from the base query since the markers are set per page HttpQuery *const queryNext = httpQueryDupP(query); // Version listing is paginated with key and version id markers if (targetTime != 0) { const String *const nextKeyMarker = xmlNodeContent( xmlNodeChild(xmlRoot, S3_XML_TAG_NEXT_KEY_MARKER_STR, true)); CHECK(FormatError, !strEmpty(nextKeyMarker), S3_XML_TAG_NEXT_KEY_MARKER " may not be empty"); httpQueryAdd(queryNext, S3_QUERY_KEY_MARKER_STR, nextKeyMarker); // The version id marker may be missing or empty when the next result is a common prefix. In that case the // next page starts at the key after the key marker, which is what is needed. const String *const nextVersionIdMarker = xmlNodeContent( xmlNodeChild(xmlRoot, S3_XML_TAG_NEXT_VERSION_ID_MARKER_STR, false)); if (nextVersionIdMarker != NULL && !strEmpty(nextVersionIdMarker)) httpQueryAdd(queryNext, S3_QUERY_VERSION_ID_MARKER_STR, nextVersionIdMarker); } // Else object listing is paginated with a continuation token else { const String *const nextContinuationToken = xmlNodeContent( xmlNodeChild(xmlRoot, S3_XML_TAG_NEXT_CONTINUATION_TOKEN_STR, true)); CHECK( FormatError, !strEmpty(nextContinuationToken), S3_XML_TAG_NEXT_CONTINUATION_TOKEN " may not be empty"); httpQueryAdd(queryNext, S3_QUERY_CONTINUATION_TOKEN_STR, nextContinuationToken); } // Store request in the outer temp context MEM_CONTEXT_PRIOR_BEGIN() { request = storageS3RequestAsyncP(this, HTTP_VERB_GET_STR, FSLASH_STR, .query = queryNext); } MEM_CONTEXT_PRIOR_END(); } // Get prefix list const XmlNodeList *const subPathList = xmlNodeChildList(xmlRoot, S3_XML_TAG_COMMON_PREFIXES_STR); for (unsigned int subPathIdx = 0; subPathIdx < xmlNodeLstSize(subPathList); subPathIdx++) { const XmlNode *const subPathNode = xmlNodeLstGet(subPathList, subPathIdx); // Get path name StorageInfo info = { .level = level, .name = xmlNodeContent(xmlNodeChild(subPathNode, S3_XML_TAG_PREFIX_STR, true)), .exists = true, }; // Strip off base prefix and final / info.name = strSubN(info.name, strSize(basePrefix), strSize(info.name) - strSize(basePrefix) - 1); // Add type info if requested if (level >= storageInfoLevelType) info.type = storageTypePath; // Callback with info callback(callbackData, &info); } // Get file list const XmlNodeList *fileList; if (targetTime != 0) { StringList *const nameList = strLstNew(); strLstAdd(nameList, S3_XML_TAG_VERSION_STR); strLstAdd(nameList, S3_XML_TAG_DELETE_MARKER_STR); fileList = xmlNodeChildListMulti(xmlRoot, nameList); } else fileList = xmlNodeChildList(xmlRoot, S3_XML_TAG_CONTENTS_STR); for (unsigned int fileIdx = 0; fileIdx < xmlNodeLstSize(fileList); fileIdx++) { const XmlNode *const fileNode = xmlNodeLstGet(fileList, fileIdx); // Get file name and strip off the base prefix when present const String *name = xmlNodeContent(xmlNodeChild(fileNode, S3_XML_TAG_KEY_STR, true)); if (!strEmpty(basePrefix)) name = strSub(name, strSize(basePrefix)); // Return info for last file if new file if (infoLast.exists && !strEq(name, nameLast)) { callback(callbackData, &infoLast); infoLast.exists = false; } // If targeting by time if (targetTime != 0) { // Skip later versions infoLast.timeModified = storageS3CvtTime( xmlNodeContent(xmlNodeChild(fileNode, S3_XML_TAG_LAST_MODIFIED_STR, true))); if (infoLast.timeModified > targetTime) continue; // If a version has already been returned (or delete marker found) then skip this version if (strEq(infoLast.name, name)) continue; // If most recent version is a delete marker then the file will not be returned const bool deleteMarker = strEq(xmlNodeName(fileNode), S3_XML_TAG_DELETE_MARKER_STR); if (deleteMarker) { strCat(strTrunc(nameLast), name); infoLast.exists = false; continue; } } // Update last name and set exists strCat(strTrunc(nameLast), name); infoLast.exists = true; // Add basic info if requested (no need to add type info since file is default type) if (level >= storageInfoLevelBasic) { if (targetTime != 0) { strCat( strTrunc(versionIdLast), xmlNodeContent(xmlNodeChild(fileNode, S3_XML_TAG_VERSION_ID_STR, true))); } else { infoLast.timeModified = storageS3CvtTime( xmlNodeContent(xmlNodeChild(fileNode, S3_XML_TAG_LAST_MODIFIED_STR, true))); } infoLast.size = cvtZToUInt64(strZ(xmlNodeContent(xmlNodeChild(fileNode, S3_XML_TAG_SIZE_STR, true)))); } } } MEM_CONTEXT_TEMP_END(); } while (request != NULL); // Callback with last info if it exists if (infoLast.exists) callback(callbackData, &infoLast); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static StorageInfo storageS3Info(THIS_VOID, const String *const file, const StorageInfoLevel level, const StorageInterfaceInfoParam param) { THIS(StorageS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(ENUM, level); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); // Attempt to get file info HttpResponse *const httpResponse = storageS3RequestP(this, HTTP_VERB_HEAD_STR, file, .allowMissing = true, .sseC = true); // Does the file exist? StorageInfo result = {.level = level, .exists = httpResponseCodeOk(httpResponse)}; // Add basic level info if requested and the file exists (no need to add type info since file is default type) if (result.level >= storageInfoLevelBasic && result.exists) { const HttpHeader *const httpHeader = httpResponseHeader(httpResponse); const String *const contentLength = httpHeaderGet(httpHeader, HTTP_HEADER_CONTENT_LENGTH_STR); CHECK(FormatError, contentLength != NULL, "content length missing"); result.size = cvtZToUInt64(strZ(contentLength)); const String *const lastModified = httpHeaderGet(httpHeader, HTTP_HEADER_LAST_MODIFIED_STR); CHECK(FormatError, lastModified != NULL, "last modified missing"); result.timeModified = httpDateToTime(lastModified); } httpResponseFree(httpResponse); FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ static void storageS3ListCallback(void *const callbackData, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(callbackData != NULL); ASSERT(info != NULL); storageLstAdd(callbackData, info); FUNCTION_TEST_RETURN_VOID(); } static StorageList * storageS3List(THIS_VOID, const String *const path, const StorageInfoLevel level, const StorageInterfaceListParam param) { THIS(StorageS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(STRING, param.expression); FUNCTION_LOG_PARAM(TIME, param.targetTime); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); StorageList *const result = storageLstNew(level); storageS3ListInternal( this, path, level, param.expression, false, param.targetTime, storageS3ListCallback, result); FUNCTION_LOG_RETURN(STORAGE_LIST, result); } /**********************************************************************************************************************************/ static void * storageS3NewRead(THIS_VOID, const String *const file, const StorageInterfaceNewReadParam param) { THIS(StorageS3); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_PARAM(STRING, param.versionId); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); FUNCTION_LOG_RETURN(STORAGE_READ_S3, storageReadS3New(this, file, param.offset, param.limit, param.versionId)); } /**********************************************************************************************************************************/ static void * storageS3NewWrite(THIS_VOID, const String *const file, const StorageInterfaceNewWriteParam param) { THIS(StorageS3); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, file); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.createPath); ASSERT(param.truncate); ASSERT(param.user == NULL); ASSERT(param.group == NULL); ASSERT(param.timeModified == 0); FUNCTION_LOG_RETURN(STORAGE_WRITE_S3, storageWriteS3New(this, file, this->partSize)); } /**********************************************************************************************************************************/ typedef struct StorageS3PathRemoveData { StorageS3 *this; // Storage object MemContext *memContext; // Mem context to create xml document in unsigned int size; // Size of delete request HttpRequest *request; // Async delete request XmlDocument *xml; // Delete xml const String *path; // Root path of remove } StorageS3PathRemoveData; static HttpRequest * storageS3PathRemoveInternal(StorageS3 *const this, HttpRequest *const request, XmlDocument *const xml) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_S3, this); FUNCTION_TEST_PARAM(HTTP_REQUEST, request); FUNCTION_TEST_PARAM(XML_DOCUMENT, xml); FUNCTION_TEST_END(); ASSERT(this != NULL); // Get response for async request if (request != NULL) { MEM_CONTEXT_TEMP_BEGIN() { const Buffer *const response = httpResponseContent(storageS3ResponseP(request)); // Nothing is returned when there are no errors if (!bufEmpty(response)) { const XmlNodeList *const errorList = xmlNodeChildList( xmlDocumentRoot(xmlDocumentNewBuf(response)), S3_XML_TAG_ERROR_STR); // Attempt to remove errored files one at a time rather than retrying the batch for (unsigned int errorIdx = 0; errorIdx < xmlNodeLstSize(errorList); errorIdx++) { storageS3RequestP( this, HTTP_VERB_DELETE_STR, strNewFmt( "/%s", strZ(xmlNodeContent(xmlNodeChild(xmlNodeLstGet(errorList, errorIdx), S3_XML_TAG_KEY_STR, true))))); } } } MEM_CONTEXT_TEMP_END(); httpRequestFree(request); } // Send new async request if there is more to remove HttpRequest *result = NULL; if (xml != NULL) { HttpQuery *const query = httpQueryAdd(httpQueryNewP(), S3_QUERY_DELETE_STR, EMPTY_STR); Buffer *const content = xmlDocumentBuf(xml); result = storageS3RequestAsyncP(this, HTTP_VERB_POST_STR, FSLASH_STR, .query = query, .content = content); httpQueryFree(query); bufFree(content); } FUNCTION_TEST_RETURN(HTTP_REQUEST, result); } static void storageS3PathRemoveCallback(void *const callbackData, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(callbackData != NULL); ASSERT(info != NULL); // Only delete files since paths don't really exist if (info->type == storageTypeFile) { StorageS3PathRemoveData *const data = (StorageS3PathRemoveData *)callbackData; // If there is something to delete then create the request if (data->xml == NULL) { MEM_CONTEXT_BEGIN(data->memContext) { data->xml = xmlDocumentNew(S3_XML_TAG_DELETE_STR); xmlNodeContentSet(xmlNodeAdd(xmlDocumentRoot(data->xml), S3_XML_TAG_QUIET_STR), TRUE_STR); } MEM_CONTEXT_END(); } // Add to delete list MEM_CONTEXT_TEMP_BEGIN() { xmlNodeContentSet( xmlNodeAdd(xmlNodeAdd(xmlDocumentRoot(data->xml), S3_XML_TAG_OBJECT_STR), S3_XML_TAG_KEY_STR), strNewFmt("%s%s", strZ(data->path), strZ(info->name))); } MEM_CONTEXT_TEMP_END(); data->size++; // Delete list when it is full if (data->size == data->this->deleteMax) { MEM_CONTEXT_BEGIN(data->memContext) { data->request = storageS3PathRemoveInternal(data->this, data->request, data->xml); } MEM_CONTEXT_END(); xmlDocumentFree(data->xml); data->xml = NULL; data->size = 0; } } FUNCTION_TEST_RETURN_VOID(); } static bool storageS3PathRemove(THIS_VOID, const String *const path, const bool recurse, const StorageInterfacePathRemoveParam param) { THIS(StorageS3); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, path); (void)recurse; // There are no paths so removal is always recursive (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { StorageS3PathRemoveData data = { .this = this, .memContext = memContextCurrent(), .path = strEq(path, FSLASH_STR) ? EMPTY_STR : strNewFmt("%s/", strZ(strSub(path, 1))), }; storageS3ListInternal(this, path, storageInfoLevelType, NULL, true, 0, storageS3PathRemoveCallback, &data); // Call if there is more to be removed if (data.xml != NULL) data.request = storageS3PathRemoveInternal(this, data.request, data.xml); // Check response on last async request storageS3PathRemoveInternal(this, data.request, NULL); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, true); } /**********************************************************************************************************************************/ static void storageS3Remove(THIS_VOID, const String *const file, const StorageInterfaceRemoveParam param) { THIS(StorageS3); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, file); (void)param; FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); httpResponseFree(storageS3RequestP(this, HTTP_VERB_DELETE_STR, file)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageInterface storageInterfaceS3 = { .feature = 1 << storageFeatureVersioning | 1 << storageFeatureReadRetry, .info = storageS3Info, .list = storageS3List, .newRead = storageS3NewRead, .newWrite = storageS3NewWrite, .pathRemove = storageS3PathRemove, .remove = storageS3Remove, }; FN_EXTERN Storage * storageS3New( const String *const path, const bool write, const time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, const String *const bucket, const String *const endPoint, const String *const region, const String *const service, const StorageS3KeyType keyType, const StorageS3UriStyle uriStyle, const String *const accessKey, const String *const secretAccessKey, const String *const securityToken, const String *const kmsKeyId, const String *sseCustomerKey, const String *const credRole, const String *const tokenFile, const String *const credUrl, const StringList *const credCmd, const String *const stsHost, const size_t partSize, const KeyValue *const tag, const String *host, const unsigned int port, const TimeMSec timeout, const HttpProtocolType protocolType, const bool verifyPeer, const String *const caFile, const String *const caPath, const bool requesterPays, const unsigned int prefetch, const uint64_t readOver) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_PARAM(STRING, bucket); FUNCTION_LOG_PARAM(STRING, endPoint); FUNCTION_LOG_PARAM(STRING, region); FUNCTION_LOG_PARAM(STRING, service); FUNCTION_LOG_PARAM(ENUM, keyType); FUNCTION_LOG_PARAM(ENUM, uriStyle); FUNCTION_LOG_PARAM(STRING, stsHost); FUNCTION_TEST_PARAM(STRING, accessKey); FUNCTION_TEST_PARAM(STRING, secretAccessKey); FUNCTION_TEST_PARAM(STRING, securityToken); FUNCTION_TEST_PARAM(STRING, kmsKeyId); FUNCTION_TEST_PARAM(STRING, sseCustomerKey); FUNCTION_TEST_PARAM(STRING, credRole); FUNCTION_TEST_PARAM(STRING, tokenFile); FUNCTION_TEST_PARAM(STRING, credUrl); FUNCTION_TEST_PARAM(STRING_LIST, credCmd); FUNCTION_LOG_PARAM(SIZE, partSize); FUNCTION_LOG_PARAM(KEY_VALUE, tag); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_PARAM(ENUM, protocolType); FUNCTION_LOG_PARAM(BOOL, verifyPeer); FUNCTION_LOG_PARAM(STRING, caFile); FUNCTION_LOG_PARAM(STRING, caPath); FUNCTION_LOG_PARAM(BOOL, requesterPays); FUNCTION_LOG_PARAM(UINT, prefetch); FUNCTION_LOG_PARAM(UINT64, readOver); FUNCTION_LOG_END(); ASSERT(path != NULL); ASSERT(bucket != NULL); ASSERT(endPoint != NULL); ASSERT(region != NULL); ASSERT(service != NULL); ASSERT(partSize != 0); OBJ_NEW_BEGIN(StorageS3, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageS3) { .interface = storageInterfaceS3, .bucket = strDup(bucket), .region = strDup(region), .signingService = strDup(service), .keyType = keyType, .kmsKeyId = strDup(kmsKeyId), .requesterPays = requesterPays, .sseCustomerKey = strDup(sseCustomerKey), .partSize = partSize, .deleteMax = STORAGE_S3_DELETE_MAX, .uriStyle = uriStyle, .bucketEndpoint = uriStyle == storageS3UriStyleHost ? strNewFmt("%s.%s", strZ(bucket), strZ(endPoint)) : strDup(endPoint), // Force the signing key to be generated on the first run .signingKeyDate = YYYYMMDD_STR, }; // Set prefetch and read over this->interface.prefetch = prefetch; this->interface.readOver = readOver; // Create tag query string if (write && tag != NULL) { HttpQuery *const query = httpQueryNewP(.kv = tag); this->tag = httpQueryRenderP(query); httpQueryFree(query); } // Create the http client used to service requests. Use plain socket for HTTP, TLS for HTTPS. if (host == NULL) host = this->bucketEndpoint; IoClient *ioClient; if (protocolType == httpProtocolTypeHttp) ioClient = sckClientNew(host, port, timeout, timeout); else { ioClient = tlsClientNewP( sckClientNew(host, port, timeout, timeout), host, timeout, timeout, verifyPeer, .caFile = caFile, .caPath = caPath); } this->httpClient = httpClientNew(ioClient, timeout); // Initialize authentication switch (this->keyType) { // Create the HTTP client used to retrieve temporary security credentials case storageS3KeyTypeAuto: { ASSERT(accessKey == NULL && secretAccessKey == NULL && securityToken == NULL); this->credRole = strDup(credRole); this->credHost = S3_CREDENTIAL_HOST_STR; this->credExpirationTime = time(NULL); this->credHttpClient = httpClientNew( sckClientNew(this->credHost, S3_CREDENTIAL_PORT, timeout, timeout), timeout); break; } // Create the HTTP client used to retrieve web identity security credentials case storageS3KeyTypeWebId: { ASSERT(accessKey == NULL && secretAccessKey == NULL && securityToken == NULL); ASSERT(credRole != NULL); ASSERT(tokenFile != NULL); const HttpUrl *const stsHostUrl = httpUrlNewParseP(stsHost, .type = httpProtocolTypeHttps); this->credRole = strDup(credRole); this->tokenFile = strDup(tokenFile); this->credHost = httpUrlHost(stsHostUrl); this->credExpirationTime = time(NULL); this->credHttpClient = httpClientNew( tlsClientNewP( sckClientNew(this->credHost, httpUrlPort(stsHostUrl), timeout, timeout), this->credHost, timeout, timeout, true, .caFile = caFile, .caPath = caPath), timeout); break; } // Create the HTTP client used to retrieve pod identity security credentials case storageS3KeyTypePodId: { ASSERT(accessKey == NULL && secretAccessKey == NULL && securityToken == NULL); ASSERT(credUrl != NULL); ASSERT(tokenFile != NULL); const HttpUrl *const url = httpUrlNewParseP(credUrl); this->tokenFile = strDup(tokenFile); this->credHost = httpUrlHost(url); this->credPath = strDup(httpUrlPath(url)); this->credExpirationTime = time(NULL); this->credHttpClient = httpClientNew(sckClientNew(this->credHost, httpUrlPort(url), timeout, timeout), timeout); break; } // Initialize process credential provider case storageS3KeyTypeProcess: { ASSERT(accessKey == NULL && secretAccessKey == NULL && securityToken == NULL); ASSERT(credCmd != NULL); this->credCmd = strLstDup(credCmd); this->credExpirationTime = time(NULL); break; } // Set shared key credentials default: { ASSERT(this->keyType == storageS3KeyTypeShared); ASSERT(accessKey != NULL && secretAccessKey != NULL); this->accessKey = strDup(accessKey); this->secretAccessKey = strDup(secretAccessKey); this->securityToken = strDup(securityToken); break; } } // Generate SSE customer key MD5 hash if (this->sseCustomerKey != NULL) { this->sseCustomerKeyMd5 = strNewEncode( encodingBase64, cryptoHashOne(hashTypeMd5, bufNewDecode(encodingBase64, this->sseCustomerKey))); } // Create list of redacted headers this->headerRedactList = strLstNew(); strLstAdd(this->headerRedactList, HTTP_HEADER_AUTHORIZATION_STR); strLstAdd(this->headerRedactList, S3_HEADER_DATE_STR); strLstAdd(this->headerRedactList, S3_HEADER_SSECUSTKEY_KEY_STR); strLstAdd(this->headerRedactList, S3_HEADER_SSECUSTKEY_KEY_MD5_STR); strLstAdd(this->headerRedactList, S3_HEADER_TOKEN_STR); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( STORAGE, storageNew(STORAGE_S3_TYPE, path, 0, 0, write, targetTime, pathExpressionFunction, this, this->interface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/s3/storage.h0000664000175100017510000000520315265101745017777 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage ***********************************************************************************************************************************/ #ifndef STORAGE_S3_STORAGE_H #define STORAGE_S3_STORAGE_H #include "common/io/http/url.h" #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_S3_TYPE STRID6("s3", 0x7d31) /*********************************************************************************************************************************** Key type ***********************************************************************************************************************************/ typedef enum { storageS3KeyTypeShared, storageS3KeyTypeAuto, storageS3KeyTypeWebId, storageS3KeyTypePodId, storageS3KeyTypeProcess, } StorageS3KeyType; /*********************************************************************************************************************************** URI style ***********************************************************************************************************************************/ typedef enum { storageS3UriStyleHost, storageS3UriStylePath, } StorageS3UriStyle; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Storage *storageS3New( const String *path, bool write, time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, const String *bucket, const String *endPoint, const String *region, const String *service, StorageS3KeyType keyType, StorageS3UriStyle uriStyle, const String *accessKey, const String *secretAccessKey, const String *securityToken, const String *kmsKeyId, const String *sseCustomerKey, const String *credRole, const String *tokenFile, const String *credUrl, const StringList *credCmd, const String *stsHost, size_t partSize, const KeyValue *tag, const String *host, unsigned int port, TimeMSec timeout, HttpProtocolType protocolType, bool verifyPeer, const String *caFile, const String *caPath, bool requesterPays, unsigned int prefetch, uint64_t readOver); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/s3/storage.intern.h0000664000175100017510000001113615265101745021277 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage Internal ***********************************************************************************************************************************/ #ifndef STORAGE_S3_STORAGE_INTERN_H #define STORAGE_S3_STORAGE_INTERN_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageS3 StorageS3; #include "common/io/http/request.h" #include "storage/s3/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Perform async request typedef struct StorageS3RequestAsyncParam { VAR_PARAM_HEADER; const HttpHeader *header; // Headers const HttpQuery *query; // Query parameters const Buffer *content; // Request content bool sseKms; // Enable server-side encryption? bool sseC; // Enable server-side encryption with customer-provided keys? bool tag; // Add tags when available? } StorageS3RequestAsyncParam; #define storageS3RequestAsyncP(this, verb, path, ...) \ storageS3RequestAsync(this, verb, path, (StorageS3RequestAsyncParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpRequest *storageS3RequestAsync( StorageS3 *this, const String *verb, const String *path, StorageS3RequestAsyncParam param); // Get async response typedef struct StorageS3ResponseParam { VAR_PARAM_HEADER; bool allowMissing; // Allow missing files (caller can check response code) bool contentIo; // Is IoRead interface required to read content? } StorageS3ResponseParam; #define storageS3ResponseP(request, ...) \ storageS3Response(request, (StorageS3ResponseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpResponse *storageS3Response(HttpRequest *request, StorageS3ResponseParam param); // Perform sync request typedef struct StorageS3RequestParam { VAR_PARAM_HEADER; const HttpHeader *header; // Headers const HttpQuery *query; // Query parameters const Buffer *content; // Request content bool allowMissing; // Allow missing files (caller can check response code) bool contentIo; // Is IoRead interface required to read content? bool sseKms; // Enable server-side encryption? bool sseC; // Enable server-side encryption with customer-provided keys? bool tag; // Add tags when available? } StorageS3RequestParam; #define storageS3RequestP(this, verb, path, ...) \ storageS3Request(this, verb, path, (StorageS3RequestParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpResponse *storageS3Request(StorageS3 *this, const String *verb, const String *path, StorageS3RequestParam param); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_S3_TYPE \ StorageS3 * #define FUNCTION_LOG_STORAGE_S3_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageS3", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/s3/write.c0000664000175100017510000002652715265101745017474 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage File Write ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/xml.h" #include "storage/s3/write.h" #include "storage/write.h" /*********************************************************************************************************************************** Part defaults based on limits at https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html ***********************************************************************************************************************************/ #define STORAGE_S3_SPLIT_DEFAULT 205 #define STORAGE_S3_SPLIT_MAX 9557 /*********************************************************************************************************************************** S3 query tokens ***********************************************************************************************************************************/ STRING_STATIC(S3_QUERY_PART_NUMBER_STR, "partNumber"); STRING_STATIC(S3_QUERY_UPLOADS_STR, "uploads"); STRING_STATIC(S3_QUERY_UPLOAD_ID_STR, "uploadId"); /*********************************************************************************************************************************** XML tags ***********************************************************************************************************************************/ STRING_STATIC(S3_XML_TAG_ETAG_STR, "ETag"); STRING_STATIC(S3_XML_TAG_UPLOAD_ID_STR, "UploadId"); STRING_STATIC(S3_XML_TAG_COMPLETE_MULTIPART_UPLOAD_STR, "CompleteMultipartUpload"); STRING_STATIC(S3_XML_TAG_PART_STR, "Part"); STRING_STATIC(S3_XML_TAG_PART_NUMBER_STR, "PartNumber"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWriteS3 { const StorageWriteInterface *interface; // Interface StorageS3 *storage; // Storage that created this object const String *name; // File name HttpRequest *request; // Async request size_t partSize; Buffer *partBuffer; const String *uploadId; StringList *uploadPartList; }; /*********************************************************************************************************************************** Flush bytes to upload part ***********************************************************************************************************************************/ static void storageWriteS3Part(StorageWriteS3 *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_S3, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // If there is an outstanding async request then wait for the response and store the part id if (this->request != NULL) { HttpResponse *const response = storageS3ResponseP(this->request); strLstAdd(this->uploadPartList, httpHeaderGet(httpResponseHeader(response), HTTP_HEADER_ETAG_STR)); ASSERT(strLstGet(this->uploadPartList, strLstSize(this->uploadPartList) - 1) != NULL); httpResponseFree(response); httpRequestFree(this->request); this->request = NULL; } FUNCTION_LOG_RETURN_VOID(); } static void storageWriteS3PartAsync(StorageWriteS3 *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_S3, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->partBuffer != NULL); ASSERT(bufSize(this->partBuffer) > 0); MEM_CONTEXT_TEMP_BEGIN() { // Complete prior async request, if any storageWriteS3Part(this); // Get the upload id if we have not already if (this->uploadId == NULL) { // Initiate mult-part upload const XmlNode *const xmlRoot = xmlDocumentRoot( xmlDocumentNewBuf( httpResponseContent( storageS3RequestP( this->storage, HTTP_VERB_POST_STR, this->name, .query = httpQueryAdd(httpQueryNewP(), S3_QUERY_UPLOADS_STR, EMPTY_STR), .sseKms = true, .sseC = true, .tag = true)))); // Store the upload id MEM_CONTEXT_OBJ_BEGIN(this) { this->uploadId = xmlNodeContent(xmlNodeChild(xmlRoot, S3_XML_TAG_UPLOAD_ID_STR, true)); this->uploadPartList = strLstNew(); } MEM_CONTEXT_OBJ_END(); } // Upload the part async HttpQuery *const query = httpQueryNewP(); httpQueryAdd(query, S3_QUERY_UPLOAD_ID_STR, this->uploadId); httpQueryAdd(query, S3_QUERY_PART_NUMBER_STR, strNewFmt("%u", strLstSize(this->uploadPartList) + 1)); MEM_CONTEXT_OBJ_BEGIN(this) { this->request = storageS3RequestAsyncP( this->storage, HTTP_VERB_PUT_STR, this->name, .query = query, .content = this->partBuffer, .sseC = true); } MEM_CONTEXT_OBJ_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to internal buffer ***********************************************************************************************************************************/ static void storageWriteS3(THIS_VOID, const Buffer *const buffer) { THIS(StorageWriteS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_S3, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); // Continue until the write buffer has been exhausted size_t bytesTotal = 0; do { // Resize part buffer. This must be done in the loop because part size grows as parts are written. storageWriteChunkBufferResize(buffer, bytesTotal, this->partBuffer, this->partSize); // Copy as many bytes as possible into the part buffer const size_t bytesNext = bufRemains(this->partBuffer) > bufUsed(buffer) - bytesTotal ? bufUsed(buffer) - bytesTotal : bufRemains(this->partBuffer); bufCatSub(this->partBuffer, buffer, bytesTotal, bytesNext); bytesTotal += bytesNext; // If the part buffer is full then write it if (bufUsed(this->partBuffer) == this->partSize) { storageWriteS3PartAsync(this); bufUsedZero(this->partBuffer); this->partSize = storageWriteChunkSize( this->partSize, STORAGE_S3_SPLIT_DEFAULT, STORAGE_S3_SPLIT_MAX, strLstSize(this->uploadPartList)); } } while (bytesTotal != bufUsed(buffer)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWriteS3Close(THIS_VOID) { THIS(StorageWriteS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_S3, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close if the file has not already been closed if (this->partBuffer != NULL) { MEM_CONTEXT_TEMP_BEGIN() { // If a multi-part upload was started we need to finish that way if (this->uploadId != NULL) { // If there is anything left in the part buffer then write it if (!bufEmpty(this->partBuffer)) storageWriteS3PartAsync(this); // Complete prior async request, if any storageWriteS3Part(this); // Generate the xml part list XmlDocument *const partList = xmlDocumentNew(S3_XML_TAG_COMPLETE_MULTIPART_UPLOAD_STR); for (unsigned int partIdx = 0; partIdx < strLstSize(this->uploadPartList); partIdx++) { XmlNode *const partNode = xmlNodeAdd(xmlDocumentRoot(partList), S3_XML_TAG_PART_STR); xmlNodeContentSet(xmlNodeAdd(partNode, S3_XML_TAG_PART_NUMBER_STR), strNewFmt("%u", partIdx + 1)); xmlNodeContentSet(xmlNodeAdd(partNode, S3_XML_TAG_ETAG_STR), strLstGet(this->uploadPartList, partIdx)); } // Finalize the multi-part upload HttpRequest *const request = storageS3RequestAsyncP( this->storage, HTTP_VERB_POST_STR, this->name, .query = httpQueryAdd(httpQueryNewP(), S3_QUERY_UPLOAD_ID_STR, this->uploadId), .content = xmlDocumentBuf(partList)); HttpResponse *const response = storageS3ResponseP(request); // Error when no etag in the result. This indicates that the request did not succeed despite the success code. if (xmlNodeChild( xmlDocumentRoot(xmlDocumentNewBuf(httpResponseContent(response))), S3_XML_TAG_ETAG_STR, false) == NULL) { httpRequestError(request, response); } } // Else upload all the data in a single put else { storageS3RequestP( this->storage, HTTP_VERB_PUT_STR, this->name, .content = this->partBuffer, .sseKms = true, .sseC = true, .tag = true); } bufFree(this->partBuffer); this->partBuffer = NULL; } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageWriteInterface storageWriteS3Interface = { .close = storageWriteS3Close, .write = storageWriteS3, }; FN_EXTERN StorageWriteS3 * storageWriteS3New(StorageS3 *const storage, const String *const name, const size_t partSize) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_S3, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(SIZE, partSize); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageWriteS3, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageWriteS3) { .interface = &storageWriteS3Interface, .storage = storage, .name = strDup(name), .partSize = partSize, .partBuffer = bufNew(0), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE_S3, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/s3/write.h0000664000175100017510000000335015265101745017466 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage File Write ***********************************************************************************************************************************/ #ifndef STORAGE_S3_WRITE_H #define STORAGE_S3_WRITE_H #include "storage/s3/storage.intern.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWriteS3 StorageWriteS3; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWriteS3 *storageWriteS3New(StorageS3 *storage, const String *name, size_t partSize); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_WRITE_S3_TYPE \ StorageWriteS3 * #define FUNCTION_LOG_STORAGE_WRITE_S3_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageWriteS3", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/sftp/0000775000175100017510000000000015265101745016611 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/sftp/helper.c0000664000175100017510000000450015265101745020233 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage Helper ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBSSH2 #include "common/debug.h" #include "common/log.h" #include "config/config.h" #include "storage/sftp/helper.h" /**********************************************************************************************************************************/ FN_EXTERN Storage * storageSftpHelper(const unsigned int repoIdx, const bool write, StoragePathExpressionCallback pathExpressionCallback) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM_P(VOID, pathExpressionCallback); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxStrId(cfgOptRepoType, repoIdx) == STORAGE_SFTP_TYPE); Storage *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { const StringList *const knownHosts = strLstNewVarLst(cfgOptionIdxLst(cfgOptRepoSftpKnownHost, repoIdx)); MEM_CONTEXT_PRIOR_BEGIN() { result = storageSftpNewP( cfgOptionIdxStr(cfgOptRepoPath, repoIdx), cfgOptionIdxStr(cfgOptRepoSftpHost, repoIdx), cfgOptionIdxUInt(cfgOptRepoSftpHostPort, repoIdx), cfgOptionIdxStr(cfgOptRepoSftpHostUser, repoIdx), cfgOptionUInt64(cfgOptIoTimeout), cfgOptionIdxStr(cfgOptRepoSftpPrivateKeyFile, repoIdx), cfgOptionIdxStrId(cfgOptRepoSftpHostKeyHashType, repoIdx), .write = write, .pathExpressionFunction = pathExpressionCallback, .modeFile = STORAGE_MODE_FILE_DEFAULT, .modePath = STORAGE_MODE_PATH_DEFAULT, .keyPub = cfgOptionIdxStrNull(cfgOptRepoSftpPublicKeyFile, repoIdx), .keyPassphrase = cfgOptionIdxStrNull(cfgOptRepoSftpPrivateKeyPassphrase, repoIdx), .hostKeyCheckType = cfgOptionIdxStrId(cfgOptRepoSftpHostKeyCheckType, repoIdx), .hostFingerprint = cfgOptionIdxStrNull(cfgOptRepoSftpHostFingerprint, repoIdx), .knownHosts = knownHosts); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE, result); } #endif // HAVE_LIBSSH2 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/sftp/helper.h0000664000175100017510000000244215265101745020243 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage Helper ***********************************************************************************************************************************/ #ifndef STORAGE_SFTP_STORAGE_HELPER_H #define STORAGE_SFTP_STORAGE_HELPER_H #ifdef HAVE_LIBSSH2 #include "storage/sftp/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN Storage *storageSftpHelper(unsigned int repoIdx, bool write, StoragePathExpressionCallback pathExpressionCallback); /*********************************************************************************************************************************** Storage helper for StorageHelper array passed to storageHelperInit() ***********************************************************************************************************************************/ #define STORAGE_SFTP_HELPER {.type = STORAGE_SFTP_TYPE, .helper = storageSftpHelper} #endif // HAVE_LIBSSH2 #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/sftp/read.c0000664000175100017510000002457315265101745017703 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage Read ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBSSH2 #include "common/debug.h" #include "common/io/session.h" #include "common/log.h" #include "storage/read.h" #include "storage/sftp/read.h" /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ struct StorageReadSftp { const StorageReadInterface *interface; // Interface StorageSftp *storage; // Storage that created this object const String *name; // File name uint64_t offset; // Read offset const Variant *limit; // Read limit (NULL for no limit) LIBSSH2_SFTP_HANDLE *sftpHandle; // LibSsh2 session sftp handle LIBSSH2_SFTP_ATTRIBUTES *attr; // LibSsh2 file attributes uint64_t current; // Current bytes read from file bool eof; // Did we reach end of file }; /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadSftpOpen(THIS_VOID) { THIS(StorageReadSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Retry the open once if the connection was lost (e.g. the server dropped an idle connection) and reopened successfully unsigned int retry = 0; int connErrno; do { do { this->sftpHandle = libssh2_sftp_open_ex( storageSftpSessionSftp(this->storage), strZ(this->name), (unsigned int)strSize(this->name), LIBSSH2_FXF_READ, 0, LIBSSH2_SFTP_OPENFILE); } while (this->sftpHandle == NULL && storageSftpWaitFd(this->storage, (connErrno = libssh2_session_last_errno(storageSftpSession(this->storage))))); } while (this->sftpHandle == NULL && retry++ == 0 && storageSftpReconnect(this->storage, connErrno)); if (this->sftpHandle == NULL) { const int rc = libssh2_session_last_errno(storageSftpSession(this->storage)); const uint64_t sftpErr = libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)); // Throw on any error except missing, which is reported to the caller via the result if (rc != LIBSSH2_ERROR_SFTP_PROTOCOL || sftpErr != LIBSSH2_FX_NO_SUCH_FILE) { storageSftpEvalLibSsh2Error( rc, sftpErr, &FileOpenError, strNewFmt(STORAGE_ERROR_READ_OPEN, strZ(this->name)), NULL); } } // Else success else { // Seek to offset, libssh2_sftp_seek64 returns void if (this->offset != 0) libssh2_sftp_seek64(this->sftpHandle, this->offset); } FUNCTION_LOG_RETURN(BOOL, this->sftpHandle != NULL); } /*********************************************************************************************************************************** Read from a file ***********************************************************************************************************************************/ static size_t storageReadSftp(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_SFTP, this); FUNCTION_LOG_PARAM(BUFFER, buffer); (void)block; // Blocking is handled by the caller FUNCTION_LOG_END(); ASSERT(this != NULL && this->sftpHandle != NULL); ASSERT(buffer != NULL && !bufFull(buffer)); ssize_t actualBytes = 0; // Read if EOF has not been reached if (!this->eof) { // Rather than enable/disable limit checking just use a big number when there is no limit. We can feel pretty confident that // no files will be > UINT64_MAX in size. const uint64_t limit = this->limit == NULL ? UINT64_MAX : varUInt64(this->limit); // Determine expected bytes to read. If remaining size in the buffer would exceed the limit then reduce the expected read. size_t expectedBytes = bufRemains(buffer); if (this->current + expectedBytes > limit) expectedBytes = (size_t)(limit - this->current); bufLimitSet(buffer, expectedBytes); ssize_t rc = 0; // Read until EOF or buffer is full do { do { rc = libssh2_sftp_read(this->sftpHandle, (char *)bufRemainsPtr(buffer), bufRemains(buffer)); } while (storageSftpWaitFd(this->storage, rc)); // Break on EOF or error if (rc <= 0) break; // Account/shift for bytes read bufUsedInc(buffer, (size_t)rc); } while (!bufFull(buffer)); // Total bytes read into the buffer actualBytes = (ssize_t)bufUsed(buffer); // Error occurred during read if (rc < 0) { if (rc == LIBSSH2_ERROR_SFTP_PROTOCOL) { uint64_t sftpErr = 0; // libssh2 sftp lseek seems to return LIBSSH2_FX_BAD_MESSAGE on a seek too far if ((sftpErr = libssh2_sftp_last_error(storageSftpSessionSftp(this->storage))) == LIBSSH2_FX_BAD_MESSAGE && this->offset > 0) { THROW_FMT(FileOpenError, STORAGE_ERROR_READ_SEEK, this->offset, strZ(this->name)); } else THROW_FMT(FileReadError, "unable to read '%s': sftp errno [%" PRIu64 "]", strZ(this->name), sftpErr); } else if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileReadError, "timeout reading '%s'", strZ(this->name)); else { storageSftpEvalLibSsh2Error( (int)rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileReadError, strNewFmt("unable to read '%s'", strZ(this->name)), NULL); } } // Update amount of buffer used this->current += (uint64_t)actualBytes; // If less data than expected was read or the limit has been reached then EOF. The file may not actually be EOF but we are // not concerned with files that are growing. Just read up to the point where the file is being extended. if ((size_t)actualBytes != expectedBytes || this->current == limit) this->eof = true; } FUNCTION_LOG_RETURN(SIZE, (size_t)actualBytes); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageReadSftpClose(THIS_VOID) { THIS(StorageReadSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); if (this->sftpHandle != NULL) { int rc; // Close the file do { rc = libssh2_sftp_close(this->sftpHandle); } while (storageSftpWaitFd(this->storage, rc)); if (rc != 0) { if (rc != LIBSSH2_ERROR_EAGAIN) THROW_FMT( FileCloseError, STORAGE_ERROR_READ_CLOSE ": libssh2 errno [%d]%s", strZ(this->name), rc, rc == LIBSSH2_ERROR_SFTP_PROTOCOL ? strZ(strNewFmt(": sftp errno [%lu]", libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)))) : ""); else { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileCloseError, strNewFmt("timeout closing file '%s'", strZ(this->name)), NULL); } } } this->sftpHandle = NULL; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadSftpEof(THIS_VOID) { THIS(StorageReadSftp); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_SFTP, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->eof); } /**********************************************************************************************************************************/ static const StorageReadInterface storageReadSftpInterface = { .close = storageReadSftpClose, .eof = storageReadSftpEof, .open = storageReadSftpOpen, .read = storageReadSftp, }; FN_EXTERN StorageReadSftp * storageReadSftpNew( StorageSftp *const storage, const String *const name, const uint64_t offset, const Variant *const limit) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_END(); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageReadSftp, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageReadSftp) { .interface = &storageReadSftpInterface, .storage = storage, .name = strDup(name), .offset = offset, .limit = varDup(limit), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_SFTP, this); } #endif // HAVE_LIBSSH2 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/sftp/read.h0000664000175100017510000000340515265101745017677 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage Read ***********************************************************************************************************************************/ #ifndef STORAGE_SFTP_READ_H #define STORAGE_SFTP_READ_H #include "storage/read.h" #include "storage/sftp/storage.intern.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadSftp StorageReadSftp; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadSftp *storageReadSftpNew(StorageSftp *storage, const String *name, uint64_t offset, const Variant *limit); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_SFTP_TYPE \ StorageReadSftp * #define FUNCTION_LOG_STORAGE_READ_SFTP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadSftp", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/sftp/storage.c0000664000175100017510000017622315265101745020434 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBSSH2 #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/client.h" #include "common/io/fd.h" #include "common/io/session.h" #include "common/io/socket/client.h" #include "common/log.h" #include "common/regExp.h" #include "common/user.h" #include "storage/posix/storage.h" #include "storage/sftp/read.h" #include "storage/sftp/storage.intern.h" #include "storage/sftp/write.h" /*********************************************************************************************************************************** Define PATH_MAX if it is not defined ***********************************************************************************************************************************/ #ifndef PATH_MAX #define PATH_MAX (4 * 1024) #endif /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageSftp { STORAGE_COMMON_MEMBER; IoClient *ioClient; // IoClient (socket) used to (re)open sessions IoSession *ioSession; // IoSession (socket) connection to SFTP server LIBSSH2_SESSION *session; // LibSsh2 session LIBSSH2_SFTP *sftpSession; // LibSsh2 session sftp session TimeMSec timeout; // Session timeout // Connection parameters retained so the session can be reopened after the server drops an idle connection const String *host; // SFTP host unsigned int port; // SFTP port const String *user; // SFTP user const String *keyPriv; // Private key file path const String *keyPub; // Public key file path const String *keyPassphrase; // Private key passphrase StringId hostKeyHashType; // Host key hash type StringId hostKeyCheckType; // Host key check type const String *hostFingerprint; // Configured host fingerprint const StringList *knownHosts; // Known hosts file list }; /*********************************************************************************************************************************** Return known host key type based on host key type ***********************************************************************************************************************************/ static int storageSftpKnownHostKeyType(const int hostKeyType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, hostKeyType); FUNCTION_TEST_END(); int result; switch (hostKeyType) { case LIBSSH2_HOSTKEY_TYPE_RSA: result = LIBSSH2_KNOWNHOST_KEY_SSHRSA; break; case LIBSSH2_HOSTKEY_TYPE_DSS: result = LIBSSH2_KNOWNHOST_KEY_SSHDSS; break; #ifdef LIBSSH2_HOSTKEY_TYPE_ECDSA_256 case LIBSSH2_HOSTKEY_TYPE_ECDSA_256: result = LIBSSH2_KNOWNHOST_KEY_ECDSA_256; break; #endif #ifdef LIBSSH2_HOSTKEY_TYPE_ECDSA_384 case LIBSSH2_HOSTKEY_TYPE_ECDSA_384: result = LIBSSH2_KNOWNHOST_KEY_ECDSA_384; break; #endif #ifdef LIBSSH2_HOSTKEY_TYPE_ECDSA_521 case LIBSSH2_HOSTKEY_TYPE_ECDSA_521: result = LIBSSH2_KNOWNHOST_KEY_ECDSA_521; break; #endif #ifdef LIBSSH2_HOSTKEY_TYPE_ED25519 case LIBSSH2_HOSTKEY_TYPE_ED25519: result = LIBSSH2_KNOWNHOST_KEY_ED25519; break; #endif default: result = 0; break; } FUNCTION_TEST_RETURN(INT, result); } /*********************************************************************************************************************************** Return error message based on error code ***********************************************************************************************************************************/ static const char * libssh2SftpErrorMsg(const uint64_t error) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, error); FUNCTION_TEST_END(); const char *result; // SFTP error status codes (returned by libssh2_sftp_last_error() switch (error) { case LIBSSH2_FX_EOF: result = "eof"; break; case LIBSSH2_FX_NO_SUCH_FILE: result = "no such file"; break; case LIBSSH2_FX_PERMISSION_DENIED: result = "permission denied"; break; case LIBSSH2_FX_FAILURE: result = "failure"; break; case LIBSSH2_FX_BAD_MESSAGE: result = "bad message"; break; case LIBSSH2_FX_NO_CONNECTION: result = "no connection"; break; case LIBSSH2_FX_CONNECTION_LOST: result = "connection lost"; break; case LIBSSH2_FX_OP_UNSUPPORTED: result = "operation unsupported"; break; case LIBSSH2_FX_INVALID_HANDLE: result = "invalid handle"; break; case LIBSSH2_FX_NO_SUCH_PATH: result = "no such path"; break; case LIBSSH2_FX_FILE_ALREADY_EXISTS: result = "file already exists"; break; case LIBSSH2_FX_WRITE_PROTECT: result = "write protect"; break; case LIBSSH2_FX_NO_MEDIA: result = "no media"; break; case LIBSSH2_FX_NO_SPACE_ON_FILESYSTEM: result = "no space on filesystem"; break; case LIBSSH2_FX_QUOTA_EXCEEDED: result = "quota exceeded"; break; case LIBSSH2_FX_UNKNOWN_PRINCIPAL: result = "unknown principal"; break; case LIBSSH2_FX_LOCK_CONFLICT: result = "lock conflict"; break; case LIBSSH2_FX_DIR_NOT_EMPTY: result = "directory not empty"; break; case LIBSSH2_FX_NOT_A_DIRECTORY: result = "not a directory"; break; case LIBSSH2_FX_INVALID_FILENAME: result = "invalid filename"; break; case LIBSSH2_FX_LINK_LOOP: result = "link loop"; break; default: result = "unknown error"; break; } FUNCTION_TEST_RETURN_CONST(STRINGZ, result); } /*********************************************************************************************************************************** Return a match failed message based on known host check failure type ***********************************************************************************************************************************/ static const char * storageSftpKnownHostCheckpFailureMsg(const int rc) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, rc); FUNCTION_TEST_END(); const char *result; switch (rc) { case LIBSSH2_KNOWNHOST_CHECK_FAILURE: result = "LIBSSH2_KNOWNHOST_CHECK_FAILURE"; break; case LIBSSH2_KNOWNHOST_CHECK_NOTFOUND: result = "not found in known hosts files: LIBSSH2_KNOWNHOST_CHECK_NOTFOUND"; break; case LIBSSH2_KNOWNHOST_CHECK_MISMATCH: result = "mismatch in known hosts files: LIBSSH2_KNOWNHOST_CHECK_MISMATCH"; break; default: result = "unknown failure"; break; } FUNCTION_TEST_RETURN_CONST(STRINGZ, result); } /**********************************************************************************************************************************/ static String * storageSftpLibSsh2SessionLastError(LIBSSH2_SESSION *const libSsh2Session) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, libSsh2Session); FUNCTION_TEST_END(); String *result; char *libSsh2ErrMsg; int libSsh2ErrMsgLen; const int rc = libssh2_session_last_error(libSsh2Session, &libSsh2ErrMsg, &libSsh2ErrMsgLen, 0); if (libSsh2ErrMsgLen != 0) result = strNewZN(libSsh2ErrMsg, (size_t)libSsh2ErrMsgLen); else result = strNewFmt("libssh2 no session error message provided [%d]", rc); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Rewrite the user's known_hosts file with a new entry ***********************************************************************************************************************************/ static void storageSftpUpdateKnownHostsFile( StorageSftp *const this, const int hostKeyType, const String *const host, const char *const hostKey, const size_t hostKeyLen) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(INT, hostKeyType); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(STRINGZ, hostKey); FUNCTION_LOG_PARAM(SIZE, hostKeyLen); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // Init a known host collection for the user's known_hosts file const char *const userKnownHostsFile = strZ(strNewFmt("%s%s", strZ(userHome()), "/.ssh/known_hosts")); LIBSSH2_KNOWNHOSTS *const userKnownHostsList = libssh2_knownhost_init(this->session); LOG_WARN_FMT("host '%s' not found in known hosts files, attempting to add host to '%s'", strZ(host), userKnownHostsFile); if (userKnownHostsList == NULL) { // Get the libssh2 error message and emit warning const int rc = libssh2_session_last_errno(this->session); LOG_WARN_FMT( "libssh2_knownhost_init failed for '%s' for update: libssh2 errno [%d] %s", userKnownHostsFile, rc, strZ(storageSftpLibSsh2SessionLastError(this->session))); } else { // Read the user's known_hosts file entries into the collection. libssh2_knownhost_readfile() returns the number of // successfully loaded hosts or a negative value on error, an empty known hosts file will return 0. int rc; if ((rc = libssh2_knownhost_readfile(userKnownHostsList, userKnownHostsFile, LIBSSH2_KNOWNHOST_FILE_OPENSSH)) < 0) { // Missing known_hosts file will return LIBSSH2_ERROR_FILE. Possibly issues other than missing may return this. if (rc == LIBSSH2_ERROR_FILE) { // If user's known_hosts file is non-existent, create an empty one for libssh2 to operate on const Storage *const sshStorage = storagePosixNewP( strNewFmt("%s%s", strZ(userHome()), "/.ssh"), .modeFile = 0600, .modePath = 0700, .write = true); if (!storageExistsP(sshStorage, strNewFmt("%s", "known_hosts"))) storagePutP(storageNewWriteP(sshStorage, strNewFmt("%s", "known_hosts")), NULL); // Try to load the user's known_hosts file entries into the collection again rc = libssh2_knownhost_readfile(userKnownHostsList, userKnownHostsFile, LIBSSH2_KNOWNHOST_FILE_OPENSSH); } } // If the user's known_hosts file was read successfully, add the host to the collection and rewrite the file if (rc >= 0) { // Check for a supported known host key type const int knownHostKeyType = storageSftpKnownHostKeyType(hostKeyType); if (knownHostKeyType != 0) { // Add host to the internal list if (libssh2_knownhost_addc( userKnownHostsList, strZ(host), NULL, hostKey, hostKeyLen, strZ(strNewZ("Generated from " PROJECT_NAME)), strlen(strZ(strNewZ("Generated from " PROJECT_NAME))), LIBSSH2_KNOWNHOST_TYPE_PLAIN | LIBSSH2_KNOWNHOST_KEYENC_RAW | knownHostKeyType, NULL) == 0) { // Rewrite the updated known_hosts file rc = libssh2_knownhost_writefile(userKnownHostsList, userKnownHostsFile, LIBSSH2_KNOWNHOST_FILE_OPENSSH); if (rc != 0) LOG_WARN_FMT(PROJECT_NAME " unable to write '%s' for update", userKnownHostsFile); else LOG_WARN_FMT(PROJECT_NAME " added new host '%s' to '%s'", strZ(host), userKnownHostsFile); } else LOG_WARN_FMT(PROJECT_NAME " failed to add '%s' to known_hosts internal list", strZ(host)); } else LOG_WARN_FMT("unsupported key type [%d], unable to update knownhosts for '%s'", hostKeyType, strZ(host)); } else { // On readfile failure warn that we're unable to update the user's known_hosts file LOG_WARN_FMT( "libssh2 unable to read '%s' for update: libssh2 errno [%d] %s\n" "HINT: does '%s' exist with proper permissions?", userKnownHostsFile, rc, strZ(storageSftpLibSsh2SessionLastError(this->session)), userKnownHostsFile); } } // Free the user's known hosts list if (userKnownHostsList) libssh2_knownhost_free(userKnownHostsList); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Free libssh2 resources ***********************************************************************************************************************************/ static void storageSftpLibSsh2SessionFreeResource(THIS_VOID) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); int rc; if (this->sftpSession != NULL) { do { rc = libssh2_sftp_shutdown(this->sftpSession); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc != LIBSSH2_ERROR_EAGAIN) THROW_FMT( ServiceError, "failed to shutdown sftpSession: libssh2 errno [%d]%s", rc, rc == LIBSSH2_ERROR_SFTP_PROTOCOL ? strZ(strNewFmt(": sftp errno [%lu] %s", libssh2_sftp_last_error(this->sftpSession), libssh2SftpErrorMsg(libssh2_sftp_last_error(this->sftpSession)))) : ""); else THROW_FMT( ServiceError, "timeout shutting down sftpSession: libssh2 errno [%d]", rc); } } if (this->session != NULL) { do { rc = libssh2_session_disconnect_ex(this->session, SSH_DISCONNECT_BY_APPLICATION, PROJECT_NAME " instance shutdown", ""); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc != LIBSSH2_ERROR_EAGAIN) THROW_FMT(ServiceError, "failed to disconnect libssh2 session: libssh2 errno [%d]", rc); else THROW_FMT(ServiceError, "timeout disconnecting libssh2 session: libssh2 errno [%d]", rc); } do { rc = libssh2_session_free(this->session); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc != LIBSSH2_ERROR_EAGAIN) THROW_FMT(ServiceError, "failed to free libssh2 session: libssh2 errno [%d]", rc); else THROW_FMT(ServiceError, "timeout freeing libssh2 session: libssh2 errno [%d]", rc); } } libssh2_exit(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN noreturn void storageSftpEvalLibSsh2Error( const int ssh2Errno, const uint64_t sftpErrno, const ErrorType *const errorType, const String *const message, const String *const hint) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, ssh2Errno); FUNCTION_TEST_PARAM(UINT64, sftpErrno); FUNCTION_TEST_PARAM(ERROR_TYPE, errorType); FUNCTION_TEST_PARAM(STRING, message); FUNCTION_TEST_PARAM(STRING, hint); FUNCTION_TEST_END(); ASSERT(errorType != NULL); THROWP_FMT( errorType, "%s%s%s%s", message != NULL ? zNewFmt("%s%s", strZ(message), ssh2Errno == 0 ? "" : ": ") : "", ssh2Errno == 0 ? "" : zNewFmt("libssh2 error [%d]", ssh2Errno), ssh2Errno == LIBSSH2_ERROR_SFTP_PROTOCOL ? zNewFmt(": sftp error [%" PRIu64 "] %s", sftpErrno, libssh2SftpErrorMsg(sftpErrno)) : "", hint != NULL ? zNewFmt("\n%s", strZ(hint)) : ""); FUNCTION_TEST_NO_RETURN(); } /*********************************************************************************************************************************** Call in a loop whenever a libssh2 call might return LIBSSH2_ERROR_EAGAIN. We handle checking the rc from the libssh2 call here and will immediately exit out if it isn't LIBSSH2_ERROR_EAGAIN. Note that LIBSSH2_ERROR_EAGAIN can still be set after this call -- if that happens then there was a timeout while waiting for the fd to be ready. ***********************************************************************************************************************************/ FN_EXTERN bool storageSftpWaitFd(StorageSftp *const this, const int64_t rc) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_PARAM(INT64, rc); FUNCTION_TEST_END(); if (rc != LIBSSH2_ERROR_EAGAIN) FUNCTION_TEST_RETURN(BOOL, false); const int direction = libssh2_session_block_directions(this->session); const bool waitingRead = direction & LIBSSH2_SESSION_BLOCK_INBOUND; const bool waitingWrite = direction & LIBSSH2_SESSION_BLOCK_OUTBOUND; if (!waitingRead && !waitingWrite) FUNCTION_TEST_RETURN(BOOL, true); FUNCTION_TEST_RETURN(BOOL, fdReady(ioSessionFd(this->ioSession), waitingRead, waitingWrite, this->timeout)); } /**********************************************************************************************************************************/ static bool storageSftpLibSsh2FxNoSuchFile(THIS_VOID, const int rc) { THIS(StorageSftp); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_PARAM(INT, rc); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN( BOOL, rc == LIBSSH2_ERROR_SFTP_PROTOCOL && libssh2_sftp_last_error(this->sftpSession) == LIBSSH2_FX_NO_SUCH_FILE); } /**********************************************************************************************************************************/ static StorageInfo storageSftpInfo(THIS_VOID, const String *const file, const StorageInfoLevel level, const StorageInterfaceInfoParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(BOOL, param.followLink); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); ASSERT(file != NULL); StorageInfo result = {.level = level}; // Stat the file to check if it exists. Note that this function is also called per entry while a directory listing handle is // open (see storageSftpList()), so it must not reconnect on a lost connection -- doing so would free and invalidate the open // directory handle. A lost connection here is reported as an error and recovered by reconnecting at a higher-level operation. LIBSSH2_SFTP_ATTRIBUTES attr; int rc; do { rc = libssh2_sftp_stat_ex( this->sftpSession, strZ(file), (unsigned int)strSize(file), param.followLink ? LIBSSH2_SFTP_STAT : LIBSSH2_SFTP_LSTAT, &attr); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileOpenError, "timeout opening '%s'", strZ(file)); // Throw libssh2 on errors other than no such file if (!storageSftpLibSsh2FxNoSuchFile(this, rc)) { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &FileOpenError, strNewFmt(STORAGE_ERROR_INFO, strZ(file)), NULL); } } // Else the file exists else { result.exists = true; // Add type info (no need set file type since it is the default) if (result.level >= storageInfoLevelType && !LIBSSH2_SFTP_S_ISREG(attr.permissions)) { if (LIBSSH2_SFTP_S_ISDIR(attr.permissions)) result.type = storageTypePath; else if (LIBSSH2_SFTP_S_ISLNK(attr.permissions)) result.type = storageTypeLink; else result.type = storageTypeSpecial; } // Add basic level info if (result.level >= storageInfoLevelBasic) { if ((attr.flags & LIBSSH2_SFTP_ATTR_ACMODTIME) != 0) result.timeModified = (time_t)attr.mtime; if (result.type == storageTypeFile) if ((attr.flags & LIBSSH2_SFTP_ATTR_SIZE) != 0) result.size = (uint64_t)attr.filesize; } // Add detail level info if (result.level >= storageInfoLevelDetail) { if ((attr.flags & LIBSSH2_SFTP_ATTR_UIDGID) != 0) { result.groupId = (unsigned int)attr.gid; result.userId = (unsigned int)attr.uid; } if ((attr.flags & LIBSSH2_SFTP_ATTR_PERMISSIONS) != 0) result.mode = attr.permissions & (LIBSSH2_SFTP_S_IRWXU | LIBSSH2_SFTP_S_IRWXG | LIBSSH2_SFTP_S_IRWXO); if (result.type == storageTypeLink) { char linkDestination[PATH_MAX] = {0}; ssize_t linkDestinationSize = 0; do { linkDestinationSize = libssh2_sftp_symlink_ex( this->sftpSession, strZ(file), (unsigned int)strSize(file), linkDestination, PATH_MAX - 1, LIBSSH2_SFTP_READLINK); } while (storageSftpWaitFd(this, linkDestinationSize)); if (linkDestinationSize == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileReadError, "timeout getting destination for link '%s'", strZ(file)); if (linkDestinationSize < 0) { storageSftpEvalLibSsh2Error( (int)linkDestinationSize, libssh2_sftp_last_error(this->sftpSession), &FileReadError, strNewFmt("unable to get destination for link '%s'", strZ(file)), NULL); } result.linkDestination = strNewZN(linkDestination, (size_t)linkDestinationSize); } } } FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /*********************************************************************************************************************************** Build known hosts file list. If knownHosts is empty build the default file list, otherwise build the list provided. knownHosts requires full path and/or leading tilde path entries. ***********************************************************************************************************************************/ static StringList * storageSftpKnownHostsFilesList(const StringList *const knownHosts) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_LIST, knownHosts); FUNCTION_LOG_END(); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { if (strLstEmpty(knownHosts)) { // Create default file list strLstAddFmt(result, "%s%s", strZ(userHome()), "/.ssh/known_hosts"); strLstAddFmt(result, "%s%s", strZ(userHome()), "/.ssh/known_hosts2"); strLstAddZ(result, "/etc/ssh/ssh_known_hosts"); strLstAddZ(result, "/etc/ssh/ssh_known_hosts2"); } else { // Process the known host list entries and add them to the result list for (unsigned int listIdx = 0; listIdx < strLstSize(knownHosts); listIdx++) { // Get the trimmed file path and add it to the result list const String *const filePath = strTrim(strLstGet(knownHosts, listIdx)); if (strBeginsWithZ(filePath, "~/")) { // Replace leading tilde with space, trim space, prepend user home path and add to the result list strLstAddFmt( result, "%s%s", strZ(userHome()), strZ(strTrim(strSub(filePath, (size_t)strChr(filePath, '~') + 1)))); } else strLstAdd(result, filePath); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ static String * storageSftpExpandTildePath(const String *const tildePath) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, tildePath); FUNCTION_TEST_END(); String *const result = strNew(); // Append to user home directory path substring after the tilde MEM_CONTEXT_TEMP_BEGIN() { strCatFmt(result, "%s%s", strZ(userHome()), strZ(strSub(tildePath, (size_t)strChr(tildePath, '~') + 1))); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ // Helper function to get info for a file if it exists. This logic can't live directly in storageSftpList() because there is a race // condition where a file might exist while listing the directory but it is gone before stat() can be called. In order to get // complete test coverage this function must be split out. static void storageSftpListEntry( StorageSftp *const this, StorageList *const list, const String *const path, const char *const name, const StorageInfoLevel level) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_PARAM(STORAGE_LIST, list); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(STRINGZ, name); FUNCTION_TEST_PARAM(ENUM, level); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(this != NULL); ASSERT(list != NULL); ASSERT(path != NULL); ASSERT(name != NULL); StorageInfo info = storageInterfaceInfoP(this, strNewFmt("%s/%s", strZ(path), name), level); if (info.exists) { info.name = STR(name); storageLstAdd(list, &info); } FUNCTION_TEST_RETURN_VOID(); } static StorageList * storageSftpList(THIS_VOID, const String *const path, const StorageInfoLevel level, const StorageInterfaceListParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); ASSERT(param.targetTime == 0); StorageList *result = NULL; // Open the directory for read, retrying the open once if the connection was lost and reopened successfully LIBSSH2_SFTP_HANDLE *sftpHandle; unsigned int retry = 0; int connErrno; do { do { sftpHandle = libssh2_sftp_open_ex( this->sftpSession, strZ(path), (unsigned int)strSize(path), 0, 0, LIBSSH2_SFTP_OPENDIR); } while (sftpHandle == NULL && storageSftpWaitFd(this, (connErrno = libssh2_session_last_errno(this->session)))); } while (sftpHandle == NULL && retry++ == 0 && storageSftpReconnect(this, connErrno)); // If the directory could not be opened process errors and report missing directories if (sftpHandle == NULL) { const int rc = libssh2_session_last_errno(this->session); if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileReadError, "timeout opening directory '%s'", strZ(path)); // If sftpHandle == NULL is due to LIBSSH2_FX_NO_SUCH_FILE, do not throw error here, return NULL result if (!storageSftpLibSsh2FxNoSuchFile(this, rc)) { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &PathOpenError, strNewFmt(STORAGE_ERROR_LIST_INFO, strZ(path)), NULL); } } else { // Directory was found result = storageLstNew(level); TRY_BEGIN() { MEM_CONTEXT_TEMP_RESET_BEGIN() { LIBSSH2_SFTP_ATTRIBUTES attr; char filename[PATH_MAX] = {0}; int len; // Read the directory entries do { len = libssh2_sftp_readdir_ex(sftpHandle, filename, PATH_MAX - 1, NULL, 0, &attr); if (len > 0) { filename[len] = '\0'; // Always skip . and .. if (!strEqZ(DOT_STR, filename) && !strEqZ(DOTDOT_STR, filename)) { if (level == storageInfoLevelExists) { const StorageInfo storageInfo = { .name = STR(filename), .level = storageInfoLevelExists, .exists = true, }; storageLstAdd(result, &storageInfo); } else storageSftpListEntry(this, result, path, filename, level); } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } while (len > 0 || storageSftpWaitFd(this, len)); } MEM_CONTEXT_TEMP_END(); } FINALLY() { int rc; do { rc = libssh2_sftp_closedir(sftpHandle); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc != LIBSSH2_ERROR_EAGAIN) { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &PathCloseError, strNewFmt("unable to close path '%s' after listing", strZ(path)), NULL); } else THROW_FMT(PathCloseError, "timeout closing path '%s' after listing", strZ(path)); } sftpHandle = NULL; } TRY_END(); } FUNCTION_LOG_RETURN(STORAGE_LIST, result); } /**********************************************************************************************************************************/ static void storageSftpRemove(THIS_VOID, const String *const file, const StorageInterfaceRemoveParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, file); (void)param; FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(!param.errorOnMissing); // Attempt to unlink the file int rc; do { rc = libssh2_sftp_unlink_ex(this->sftpSession, strZ(file), (unsigned int)strSize(file)); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileRemoveError, "timeout removing '%s'", strZ(file)); if (!storageSftpLibSsh2FxNoSuchFile(this, rc)) { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &FileRemoveError, strNewFmt(STORAGE_ERROR_FILE_REMOVE, strZ(file)), NULL); } } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static void * storageSftpNewRead(THIS_VOID, const String *const file, const StorageInterfaceNewReadParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.versionId == NULL); FUNCTION_LOG_RETURN(STORAGE_READ_SFTP, storageReadSftpNew(this, file, param.offset, param.limit)); } /**********************************************************************************************************************************/ static void * storageSftpNewWrite(THIS_VOID, const String *const file, const StorageInterfaceNewWriteParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(MODE, param.modeFile); FUNCTION_LOG_PARAM(MODE, param.modePath); FUNCTION_LOG_PARAM(STRING, param.user); FUNCTION_LOG_PARAM(STRING, param.group); FUNCTION_LOG_PARAM(TIME, param.timeModified); FUNCTION_LOG_PARAM(BOOL, param.createPath); FUNCTION_LOG_PARAM(BOOL, param.syncFile); FUNCTION_LOG_PARAM(BOOL, param.syncPath); FUNCTION_LOG_PARAM(BOOL, param.atomic); FUNCTION_LOG_PARAM(BOOL, param.truncate); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.createPath); ASSERT(param.truncate); ASSERT(param.user == NULL); ASSERT(param.group == NULL); ASSERT(param.timeModified == 0); FUNCTION_LOG_RETURN( STORAGE_WRITE_SFTP, storageWriteSftpNew( this, file, param.modeFile, param.modePath, param.createPath, param.syncFile, param.atomic)); } /**********************************************************************************************************************************/ static void storageSftpPathCreate( THIS_VOID, const String *const path, const bool errorOnExists, const bool noParentCreate, const mode_t mode, const StorageInterfacePathCreateParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, errorOnExists); FUNCTION_LOG_PARAM(BOOL, noParentCreate); FUNCTION_LOG_PARAM(MODE, mode); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); int rc; // Attempt to create the directory do { rc = libssh2_sftp_mkdir_ex(this->sftpSession, strZ(path), (unsigned int)strSize(path), (int)mode); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(PathCreateError, "timeout creating path '%s'", strZ(path)); if (rc == LIBSSH2_ERROR_SFTP_PROTOCOL) { const uint64_t sftpErrno = libssh2_sftp_last_error(this->sftpSession); // libssh2 may return LIBSSH2_FX_FAILURE if the directory already exists if (sftpErrno == LIBSSH2_FX_FAILURE || sftpErrno == LIBSSH2_FX_FILE_ALREADY_EXISTS) { // Check if the directory already exists LIBSSH2_SFTP_ATTRIBUTES attr; do { rc = libssh2_sftp_stat_ex( this->sftpSession, strZ(path), (unsigned int)strSize(path), LIBSSH2_SFTP_STAT, &attr); } while (storageSftpWaitFd(this, rc)); if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(PathCreateError, "timeout stat'ing path '%s'", strZ(path)); // If rc = 0 then already exists if (rc == 0 && errorOnExists) { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &PathCreateError, strNewFmt("sftp error unable to create path '%s': path already exists", strZ(path)), NULL); } } // If the parent path does not exist then create it if allowed else if (sftpErrno == LIBSSH2_FX_NO_SUCH_FILE && !noParentCreate) { String *const pathParent = strPath(path); storageInterfacePathCreateP(this, pathParent, errorOnExists, noParentCreate, mode); storageInterfacePathCreateP(this, path, errorOnExists, noParentCreate, mode); strFree(pathParent); } else { storageSftpEvalLibSsh2Error( rc, sftpErrno, &PathCreateError, strNewFmt("sftp error unable to create path '%s'", strZ(path)), NULL); } } else { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &PathCreateError, strNewFmt("ssh2 error [%d] unable to create path '%s'", rc, strZ(path)), NULL); } } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static bool storageSftpPathRemove(THIS_VOID, const String *const path, const bool recurse, const StorageInterfacePathRemoveParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, recurse); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); bool result = true; MEM_CONTEXT_TEMP_BEGIN() { // Recurse if requested if (recurse) { const StorageList *const list = storageInterfaceListP(this, path, storageInfoLevelExists); if (list != NULL) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int listIdx = 0; listIdx < storageLstSize(list); listIdx++) { const String *const file = strNewFmt("%s/%s", strZ(path), strZ(storageLstGet(list, listIdx).name)); // Rather than stat the file to discover what type it is, just try to unlink it and see what happens int rc; do { rc = libssh2_sftp_unlink_ex(this->sftpSession, strZ(file), (unsigned int)strSize(file)); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(PathRemoveError, "timeout removing file '%s'", strZ(file)); // Attempting to unlink a directory appears to return LIBSSH2_FX_FAILURE or LIBSSH2_FX_PERMISSION_DENIED if (rc == LIBSSH2_ERROR_SFTP_PROTOCOL) { const uint64_t sftpErrno = libssh2_sftp_last_error(this->sftpSession); if (sftpErrno == LIBSSH2_FX_FAILURE || sftpErrno == LIBSSH2_FX_PERMISSION_DENIED) storageInterfacePathRemoveP(this, file, true); else { THROW_FMT( PathRemoveError, STORAGE_ERROR_FILE_REMOVE ": libssh sftp [%" PRIu64 "] %s", strZ(file), sftpErrno, libssh2SftpErrorMsg(sftpErrno)); } } else THROW_FMT(PathRemoveError, STORAGE_ERROR_FILE_REMOVE ": libssh ssh [%d]", strZ(file), rc); } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } } // Delete the path int rc; do { rc = libssh2_sftp_rmdir_ex(this->sftpSession, strZ(path), (unsigned int)strSize(path)); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(PathRemoveError, "timeout removing path '%s'", strZ(path)); if (rc == LIBSSH2_ERROR_SFTP_PROTOCOL) { const uint64_t sftpErrno = libssh2_sftp_last_error(this->sftpSession); if (sftpErrno != LIBSSH2_FX_NO_SUCH_FILE) { THROW_FMT( PathRemoveError, STORAGE_ERROR_PATH_REMOVE " sftp error [%" PRIu64 "] %s", strZ(path), sftpErrno, libssh2SftpErrorMsg(sftpErrno)); } // Path does not exist result = false; } else { // Path does not exist result = false; storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &PathRemoveError, strNewFmt(STORAGE_ERROR_PATH_REMOVE, strZ(path)), NULL); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN LIBSSH2_SESSION * storageSftpSession(StorageSftp *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN_TYPE_P(LIBSSH2_SESSION, this->session); } FN_EXTERN LIBSSH2_SFTP * storageSftpSessionSftp(StorageSftp *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN_TYPE_P(LIBSSH2_SFTP, this->sftpSession); } /*********************************************************************************************************************************** Is the libssh2 error a sign that the connection to the server has been lost? Socket-level errors always mean the connection is gone. LIBSSH2_ERROR_SOCKET_SEND is intentionally excluded since a failed send is ambiguous (and is used throughout the unit tests as a generic fatal error) -- the definitive "the peer is gone" signal is a failed receive. An sftp protocol error is ambiguous in general (e.g. mkdir returns LIBSSH2_FX_FAILURE when a directory already exists) but on a file or directory open a generic failure or explicit connection-lost status means the session is no longer usable, so treat those as a lost connection. This is only called from the open retry paths, so the ambiguous mkdir/rename cases do not reach here. ***********************************************************************************************************************************/ static bool storageSftpConnLost(StorageSftp *const this, const int rc) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_PARAM(INT, rc); FUNCTION_TEST_END(); ASSERT(this != NULL); switch (rc) { case LIBSSH2_ERROR_SOCKET_RECV: case LIBSSH2_ERROR_SOCKET_TIMEOUT: case LIBSSH2_ERROR_SOCKET_DISCONNECT: case LIBSSH2_ERROR_BAD_USE: FUNCTION_TEST_RETURN(BOOL, true); break; case LIBSSH2_ERROR_SFTP_PROTOCOL: { const uint64_t sftpErr = libssh2_sftp_last_error(this->sftpSession); FUNCTION_TEST_RETURN( BOOL, sftpErr == LIBSSH2_FX_FAILURE || sftpErr == LIBSSH2_FX_CONNECTION_LOST || sftpErr == LIBSSH2_FX_NO_CONNECTION); break; } default: break; } FUNCTION_TEST_RETURN(BOOL, false); } /*********************************************************************************************************************************** Free the current session without a graceful disconnect. Used before reopening since the connection is already known to be broken; errors are ignored because the only goal is to release resources before reconnecting. This is only ever called from storageSftpReopen() on a connection that has already been opened, so the session and socket are always set. ***********************************************************************************************************************************/ static void storageSftpConnFree(StorageSftp *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->session != NULL); ASSERT(this->ioSession != NULL); // Free the libssh2 session int rc; do { rc = libssh2_session_free(this->session); } while (storageSftpWaitFd(this, rc)); this->session = NULL; this->sftpSession = NULL; // Free the socket session, which closes the file descriptor ioSessionFree(this->ioSession); this->ioSession = NULL; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Open a socket session, perform the libssh2 handshake, host key verification, authentication, and init the sftp session. The connection parameters are read from the object so this can be called again by storageSftpReopen() after an idle connection is lost. ***********************************************************************************************************************************/ static void storageSftpOpen(StorageSftp *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // The socket/sftp session created here persists in the object context (not the caller's) so the single-return-object audit // does not apply. This also allows the session to be (re)opened from within another operation that is being audited. FUNCTION_AUDIT_HELPER(); // Open the socket session in the object context so it persists with the object MEM_CONTEXT_OBJ_BEGIN(this) { this->ioSession = ioClientOpen(this->ioClient); } MEM_CONTEXT_OBJ_END(); MEM_CONTEXT_TEMP_BEGIN() { this->session = libssh2_session_init(); if (this->session == NULL) THROW_FMT(ServiceError, "unable to init libssh2 session"); // Set session to non-blocking libssh2_session_set_blocking(this->session, 0); // Perform handshake int rc; do { rc = libssh2_session_handshake(this->session, ioSessionFd(this->ioSession)); } while (storageSftpWaitFd(this, rc)); if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(ServiceError, "timeout during libssh2 handshake [%d]", rc); if (rc != 0) { THROW_FMT( ServiceError, "libssh2 handshake failed [%d]: %s", rc, strZ(storageSftpLibSsh2SessionLastError(this->session))); } int hashType = LIBSSH2_HOSTKEY_HASH_SHA1; size_t hashSize = 0; // Verify that the fingerprint[N] buffer declared below is large enough when adding a new hashType switch (this->hostKeyHashType) { case hashTypeMd5: hashType = LIBSSH2_HOSTKEY_HASH_MD5; hashSize = HASH_TYPE_M5_SIZE; break; case hashTypeSha1: hashType = LIBSSH2_HOSTKEY_HASH_SHA1; hashSize = HASH_TYPE_SHA1_SIZE; break; #ifdef LIBSSH2_HOSTKEY_HASH_SHA256 case hashTypeSha256: hashType = LIBSSH2_HOSTKEY_HASH_SHA256; hashSize = HASH_TYPE_SHA256_SIZE; break; #endif // LIBSSH2_HOSTKEY_HASH_SHA256 default: THROW_FMT(ServiceError, "requested ssh2 hostkey hash type (%s) not available", zNewStrId(this->hostKeyHashType)); break; } // Compare fingerprint if provided else check known hosts files for a match if (this->hostKeyCheckType == SFTP_STRICT_HOSTKEY_CHECKING_FINGERPRINT) { const char *const binaryFingerprint = libssh2_hostkey_hash(this->session, hashType); if (binaryFingerprint == NULL) { THROW_FMT( ServiceError, "libssh2 hostkey hash failed: libssh2 errno [%d]", libssh2_session_last_errno(this->session)); } // 256 bytes is large enough to hold the hex representation of currently supported hash types. The hex encoded version // requires twice as much space (hashSize * 2) as the raw version. char fingerprint[256]; encodeToStr(encodingHex, (const uint8_t *)binaryFingerprint, hashSize, fingerprint); if (strcmp(fingerprint, strZ(this->hostFingerprint)) != 0) { THROW_FMT( ServiceError, "host [%s] and configured fingerprint (repo-sftp-host-fingerprint) [%s] do not match", fingerprint, strZ(this->hostFingerprint)); } } else if (this->hostKeyCheckType != SFTP_STRICT_HOSTKEY_CHECKING_NONE) { // Init the known host collection LIBSSH2_KNOWNHOSTS *const knownHostsList = libssh2_knownhost_init(this->session); if (knownHostsList == NULL) { const int rc = libssh2_session_last_errno(this->session); THROW_FMT( ServiceError, "failure during libssh2_knownhost_init: libssh2 errno [%d] %s", rc, strZ(storageSftpLibSsh2SessionLastError(this->session))); } // Get the list of known host files to search const StringList *const knownHostsPathList = storageSftpKnownHostsFilesList(this->knownHosts); // Loop through the list of known host files for (unsigned int listIdx = 0; listIdx < strLstSize(knownHostsPathList); listIdx++) { const char *const currentKnownHostFile = strZNull(strLstGet(knownHostsPathList, listIdx)); // Read the known hosts file entries into the collection, log message for readfile status. // libssh2_knownhost_readfile() returns the number of successfully loaded hosts or a negative value on error, an // empty known hosts file will return 0. if ((rc = libssh2_knownhost_readfile(knownHostsList, currentKnownHostFile, LIBSSH2_KNOWNHOST_FILE_OPENSSH)) <= 0) { if (rc == 0) LOG_DETAIL_FMT("libssh2 '%s' file is empty", currentKnownHostFile); else { LOG_DETAIL_FMT( "libssh2 read '%s' failed: libssh2 errno [%d] %s", currentKnownHostFile, rc, strZ(storageSftpLibSsh2SessionLastError(this->session))); } } else LOG_DETAIL_FMT("libssh2 read '%s' succeeded", currentKnownHostFile); } // Get the remote host key size_t hostKeyLen; int hostKeyType; const char *const hostKey = libssh2_session_hostkey(this->session, &hostKeyLen, &hostKeyType); // Check for a match in known hosts files else throw an error if no host key was retrieved if (hostKey != NULL) { rc = libssh2_knownhost_checkp( knownHostsList, strZ(this->host), (int)this->port, hostKey, hostKeyLen, LIBSSH2_KNOWNHOST_TYPE_PLAIN | LIBSSH2_KNOWNHOST_KEYENC_RAW, NULL); // Handle check success/failure if (rc == LIBSSH2_KNOWNHOST_CHECK_MATCH) LOG_DETAIL_FMT("known hosts match found for '%s'", strZ(this->host)); else { // Handle failure to match in a similar manner as ssh_config StrictHostKeyChecking. If this flag is set to // "strict", never automatically add host keys to the ~/.ssh/known_hosts file, and refuse to connect to hosts // whose host key has changed. This option forces the user to manually add all new hosts. If this flag is set to // "accept-new" then automatically add new host keys to the user known hosts files, but do not permit // connections to hosts with changed host keys. switch (this->hostKeyCheckType) { case SFTP_STRICT_HOSTKEY_CHECKING_STRICT: { // Throw an error when set to strict and we have any result other than match libssh2_knownhost_free(knownHostsList); THROW_FMT( ServiceError, "known hosts failure: '%s' %s [%d]: check type [%s]", strZ(this->host), storageSftpKnownHostCheckpFailureMsg(rc), rc, zNewStrId(this->hostKeyCheckType)); break; } default: { ASSERT(this->hostKeyCheckType == SFTP_STRICT_HOSTKEY_CHECKING_ACCEPT_NEW); // Throw an error when set to accept-new and match fails or mismatches else add the new host key to the // user's known_hosts file if (rc == LIBSSH2_KNOWNHOST_CHECK_MISMATCH || rc == LIBSSH2_KNOWNHOST_CHECK_FAILURE) { // Free the known hosts list libssh2_knownhost_free(knownHostsList); THROW_FMT( ServiceError, "known hosts failure: '%s': %s [%d]: check type [%s]", strZ(this->host), storageSftpKnownHostCheckpFailureMsg(rc), rc, zNewStrId(this->hostKeyCheckType)); } else storageSftpUpdateKnownHostsFile(this, hostKeyType, this->host, hostKey, hostKeyLen); break; } } } } else { THROW_FMT( ServiceError, "libssh2_session_hostkey failed to get hostkey: libssh2 error [%d]", libssh2_session_last_errno(this->session)); } // Free the known hosts list libssh2_knownhost_free(knownHostsList); } // Perform public key authorization, expand leading tilde key file paths if needed String *const privKeyPath = regExpMatchOne(STRDEF("^ *~"), this->keyPriv) ? storageSftpExpandTildePath(this->keyPriv) : strDup(this->keyPriv); String *const pubKeyPath = this->keyPub != NULL && regExpMatchOne(STRDEF("^ *~"), this->keyPub) ? storageSftpExpandTildePath(this->keyPub) : strDup(this->keyPub); do { rc = libssh2_userauth_publickey_fromfile( this->session, strZ(this->user), strZNull(pubKeyPath), strZ(privKeyPath), strZNull(this->keyPassphrase)); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(ServiceError, "timeout during public key authentication"); storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &ServiceError, STRDEF("public key authentication failed"), STRDEF( "HINT: libssh2 compiled against non-openssl libraries requires --repo-sftp-private-key-file and" " --repo-sftp-public-key-file to be provided\n" "HINT: libssh2 versions before 1.9.0 expect a PEM format keypair, try ssh-keygen -m PEM -t rsa -P \"\" to" " generate the keypair\n" "HINT: check authorization log on the SFTP server")); } // Init the sftp session do { this->sftpSession = libssh2_sftp_init(this->session); } while (this->sftpSession == NULL && storageSftpWaitFd(this, libssh2_session_last_errno(this->session))); if (this->sftpSession == NULL) { if (libssh2_session_last_errno(this->session) == LIBSSH2_ERROR_EAGAIN) THROW_FMT(ServiceError, "timeout during init of libssh2_sftp session"); else { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &ServiceError, strNewFmt("unable to init libssh2_sftp session"), NULL); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Reopen the session after the connection was lost ***********************************************************************************************************************************/ static void storageSftpReopen(StorageSftp *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); LOG_DEBUG_FMT("reconnect to lost sftp connection '%s:%u'", strZ(this->host), this->port); storageSftpConnFree(this); storageSftpOpen(this); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool storageSftpReconnect(StorageSftp *const this, const int rc) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_PARAM(INT, rc); FUNCTION_TEST_END(); ASSERT(this != NULL); bool result = storageSftpConnLost(this, rc); if (result) storageSftpReopen(this); FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ static const StorageInterface storageInterfaceSftp = { .feature = 1 << storageFeaturePath | 1 << storageFeatureInfoDetail, .info = storageSftpInfo, .list = storageSftpList, .newRead = storageSftpNewRead, .newWrite = storageSftpNewWrite, .pathCreate = storageSftpPathCreate, .pathRemove = storageSftpPathRemove, .remove = storageSftpRemove, }; FN_EXTERN Storage * storageSftpNew( const String *const path, const String *const host, const unsigned int port, const String *const user, const TimeMSec timeout, const String *const keyPriv, const StringId hostKeyHashType, const StorageSftpNewParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(STRING, user); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_PARAM(STRING, keyPriv); FUNCTION_LOG_PARAM(STRING_ID, hostKeyHashType); FUNCTION_LOG_PARAM(STRING, param.keyPub); FUNCTION_TEST_PARAM(STRING, param.keyPassphrase); FUNCTION_LOG_PARAM(STRING_ID, param.hostKeyCheckType); FUNCTION_LOG_PARAM(STRING, param.hostFingerprint); FUNCTION_LOG_PARAM(STRING_LIST, param.knownHosts); FUNCTION_LOG_PARAM(MODE, param.modeFile); FUNCTION_LOG_PARAM(MODE, param.modePath); FUNCTION_LOG_PARAM(BOOL, param.write); FUNCTION_LOG_PARAM(FUNCTIONP, param.pathExpressionFunction); FUNCTION_LOG_END(); ASSERT(path != NULL); ASSERT(host != NULL); ASSERT(port != 0); ASSERT(user != NULL); ASSERT(keyPriv != NULL); ASSERT(hostKeyHashType != 0); // Initialize user module userInit(); // Create the object OBJ_NEW_BEGIN(StorageSftp, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (StorageSftp) { .interface = storageInterfaceSftp, .timeout = timeout, // Retain connection parameters so the session can be reopened after the server drops an idle connection .host = strDup(host), .port = port, .user = strDup(user), .keyPriv = strDup(keyPriv), .keyPub = strDup(param.keyPub), .keyPassphrase = strDup(param.keyPassphrase), .hostKeyHashType = hostKeyHashType, .hostKeyCheckType = param.hostKeyCheckType, .hostFingerprint = strDup(param.hostFingerprint), .knownHosts = strLstDup(param.knownHosts), }; // Init SFTP session if (libssh2_init(0) != 0) THROW_FMT(ServiceError, "unable to init libssh2"); this->ioClient = sckClientNew(host, port, timeout, timeout); // Open the connection (socket, handshake, host key verification, authentication, sftp session) storageSftpOpen(this); // Ensure libssh2/libssh2_sftp resources freed memContextCallbackSet(objMemContext(this), storageSftpLibSsh2SessionFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( STORAGE, storageNew( STORAGE_SFTP_TYPE, path, param.modeFile == 0 ? STORAGE_MODE_FILE_DEFAULT : param.modeFile, param.modePath == 0 ? STORAGE_MODE_PATH_DEFAULT : param.modePath, param.write, 0, param.pathExpressionFunction, this, this->interface)); } #endif // HAVE_LIBSSH2 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/sftp/storage.h0000664000175100017510000000510015265101745020422 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage ***********************************************************************************************************************************/ #ifndef STORAGE_SFTP_STORAGE_H #define STORAGE_SFTP_STORAGE_H #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_SFTP_TYPE STRID5("sftp", 0x850d30) #ifdef HAVE_LIBSSH2 /*********************************************************************************************************************************** SFTP StrictHostKeyChecking ***********************************************************************************************************************************/ #define SFTP_STRICT_HOSTKEY_CHECKING_ACCEPT_NEW STRID5("accept-new", 0x2e576e9028c610) #define SFTP_STRICT_HOSTKEY_CHECKING_FINGERPRINT STRID5("fingerprint", 0x51c9942453b9260) #define SFTP_STRICT_HOSTKEY_CHECKING_NONE STRID5("none", 0x2b9ee0) #define SFTP_STRICT_HOSTKEY_CHECKING_STRICT STRID5("strict", 0x2834ca930) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct StorageSftpNewParam { VAR_PARAM_HEADER; bool write; mode_t modeFile; mode_t modePath; StoragePathExpressionCallback *pathExpressionFunction; const String *keyPub; const String *keyPassphrase; StringId hostKeyCheckType; const String *hostFingerprint; const StringList *knownHosts; } StorageSftpNewParam; #define storageSftpNewP(path, host, port, user, timeout, keyPriv, hostKeyHashType, ...) \ storageSftpNew(path, host, port, user, timeout, keyPriv, hostKeyHashType, (StorageSftpNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN Storage *storageSftpNew( const String *path, const String *host, unsigned int port, const String *user, TimeMSec timeout, const String *keyPriv, StringId hostKeyHashType, const StorageSftpNewParam param); #endif // HAVE_LIBSSH2 #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/sftp/storage.intern.h0000664000175100017510000000532515265101745021731 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage Internal ***********************************************************************************************************************************/ #ifndef STORAGE_SFTP_STORAGE_INTERN_H #define STORAGE_SFTP_STORAGE_INTERN_H #include #include #include "storage/sftp/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageSftp StorageSftp; /*********************************************************************************************************************************** Getters ***********************************************************************************************************************************/ // Current libssh2 session/sftp session, used by the read/write objects FN_EXTERN LIBSSH2_SESSION *storageSftpSession(StorageSftp *this); FN_EXTERN LIBSSH2_SFTP *storageSftpSessionSftp(StorageSftp *this); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN void storageSftpEvalLibSsh2Error( int ssh2Errno, uint64_t sftpErrno, const ErrorType *errorType, const String *msg, const String *hint); // Check which direction we are currently blocking on (reading, writing, or both) and wait for the fd to be ready accordingly FN_EXTERN bool storageSftpWaitFd(StorageSftp *this, int64_t rc); // If the error indicates the connection was lost then reopen the session so the operation can be retried. Returns true when the // session was reopened (the caller should retry the operation). FN_EXTERN bool storageSftpReconnect(StorageSftp *this, int rc); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_SFTP_TYPE \ StorageSftp * #define FUNCTION_LOG_STORAGE_SFTP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageSftp *", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/sftp/write.c0000664000175100017510000003411515265101745020113 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage File Write ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBSSH2 #include "common/debug.h" #include "common/log.h" #include "storage/sftp/write.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWriteSftp { const StorageWriteInterface *interface; // Interface const String *name; // File name mode_t modeFile; // File mode mode_t modePath; // Path mode bool createPath; // Create path if missing bool syncFile; // Sync file on close bool atomic; // Atomic write StorageSftp *storage; // Storage that created this object const String *nameTmp; // Temporary filename utilized for atomic ops const String *path; // Utilized for path operations LIBSSH2_SFTP_HANDLE *sftpHandle; // LibSsh2 session sftp handle }; /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static void storageWriteSftpOpen(THIS_VOID) { THIS(StorageWriteSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(storageSftpSessionSftp(this->storage) != NULL); const unsigned long int flags = LIBSSH2_FXF_CREAT | LIBSSH2_FXF_WRITE | LIBSSH2_FXF_TRUNC; // Retry the open once if the connection was lost (e.g. the server dropped an idle connection) and reopened successfully unsigned int retry = 0; int connErrno; do { // Open the file do { this->sftpHandle = libssh2_sftp_open_ex( storageSftpSessionSftp(this->storage), strZ(this->nameTmp), (unsigned int)strSize(this->nameTmp), flags, (int)this->modeFile, LIBSSH2_SFTP_OPENFILE); } while (this->sftpHandle == NULL && storageSftpWaitFd(this->storage, (connErrno = libssh2_session_last_errno(storageSftpSession(this->storage))))); // Attempt to create the path if it is missing if (this->sftpHandle == NULL && libssh2_session_last_errno(storageSftpSession(this->storage)) == LIBSSH2_ERROR_SFTP_PROTOCOL && libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)) == LIBSSH2_FX_NO_SUCH_FILE) { // Create the path storageInterfacePathCreateP(this->storage, this->path, false, false, this->modePath); // Open file again do { this->sftpHandle = libssh2_sftp_open_ex( storageSftpSessionSftp(this->storage), strZ(this->nameTmp), (unsigned int)strSize(this->nameTmp), flags, (int)this->modeFile, LIBSSH2_SFTP_OPENFILE); } while (this->sftpHandle == NULL && storageSftpWaitFd(this->storage, (connErrno = libssh2_session_last_errno(storageSftpSession(this->storage))))); } } while (this->sftpHandle == NULL && retry++ == 0 && storageSftpReconnect(this->storage, connErrno)); // Handle error if (this->sftpHandle == NULL) { const int rc = libssh2_session_last_errno(storageSftpSession(this->storage)); if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileOpenError, "timeout while opening file '%s'", strZ(this->name)); const uint64_t sftpErr = libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)); if (rc == LIBSSH2_ERROR_SFTP_PROTOCOL && sftpErr == LIBSSH2_FX_NO_SUCH_FILE) { THROW_FMT(FileMissingError, STORAGE_ERROR_WRITE_MISSING, strZ(this->name)); } else storageSftpEvalLibSsh2Error(rc, sftpErr, &FileOpenError, strNewFmt(STORAGE_ERROR_WRITE_OPEN, strZ(this->name)), NULL); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to the file ***********************************************************************************************************************************/ static void storageWriteSftp(THIS_VOID, const Buffer *const buffer) { THIS(StorageWriteSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_SFTP, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(this->sftpHandle != NULL); ssize_t rc; size_t remains = bufUsed(buffer); // Amount left to write size_t offset = 0; // Offset into the buffer // Loop until all the data is written do { do { rc = libssh2_sftp_write(this->sftpHandle, (const char *)bufPtrConst(buffer) + offset, remains); } while (storageSftpWaitFd(this->storage, rc)); // Break on error. Error will be thrown below the loop. if (rc < 0) break; // Offset for next write start point offset += (size_t)rc; // Update amount left to write remains -= (size_t)rc; } while (remains); if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileWriteError, "timeout writing '%s'", strZ(this->nameTmp)); if (rc < 0) { storageSftpEvalLibSsh2Error( (int)rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileWriteError, strNewFmt("unable to write '%s'", strZ(this->nameTmp)), NULL); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Unlink already existing file ***********************************************************************************************************************************/ static void storageWriteSftpUnlinkExisting(THIS_VOID) { THIS(StorageWriteSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); int rc; do { rc = libssh2_sftp_unlink_ex(storageSftpSessionSftp(this->storage), strZ(this->name), (unsigned int)strSize(this->name)); } while (storageSftpWaitFd(this->storage, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileWriteError, "timeout unlinking '%s'", strZ(this->name)); else storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileRemoveError, strNewFmt("unable to remove existing '%s'", strZ(this->name)), NULL); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Rename a file ***********************************************************************************************************************************/ static void storageWriteSftpRename(THIS_VOID) { THIS(StorageWriteSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); int rc; do { rc = libssh2_sftp_rename_ex( storageSftpSessionSftp(this->storage), strZ(this->nameTmp), (unsigned int)strSize(this->nameTmp), strZ(this->name), (unsigned int)strSize(this->name), LIBSSH2_SFTP_RENAME_OVERWRITE | LIBSSH2_SFTP_RENAME_ATOMIC | LIBSSH2_SFTP_RENAME_NATIVE); } while (storageSftpWaitFd(this->storage, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileWriteError, "timeout moving '%s' to '%s'", strZ(this->nameTmp), strZ(this->name)); else storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileRemoveError, strNewFmt("unable to move '%s' to '%s'", strZ(this->nameTmp), strZ(this->name)), NULL); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWriteSftpClose(THIS_VOID) { THIS(StorageWriteSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close if the file has not already been closed if (this->sftpHandle != NULL) { int rc; if (this->syncFile) { do { rc = libssh2_sftp_fsync(this->sftpHandle); } while (storageSftpWaitFd(this->storage, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileSyncError, "timeout syncing file '%s'", strZ(this->nameTmp)); else { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileSyncError, strNewFmt(STORAGE_ERROR_WRITE_SYNC, strZ(this->nameTmp)), NULL); } } } // Close the file do { rc = libssh2_sftp_close(this->sftpHandle); } while (storageSftpWaitFd(this->storage, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileCloseError, "timeout closing file '%s'", strZ(this->nameTmp)); storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileCloseError, strNewFmt(STORAGE_ERROR_WRITE_CLOSE, strZ(this->nameTmp)), NULL); } this->sftpHandle = NULL; // Rename from temp file if (this->atomic) { do { rc = libssh2_sftp_rename_ex( storageSftpSessionSftp(this->storage), strZ(this->nameTmp), (unsigned int)strSize(this->nameTmp), strZ(this->name), (unsigned int)strSize(this->name), LIBSSH2_SFTP_RENAME_OVERWRITE | LIBSSH2_SFTP_RENAME_ATOMIC | LIBSSH2_SFTP_RENAME_NATIVE); } while (storageSftpWaitFd(this->storage, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileCloseError, "timeout renaming file '%s'", strZ(this->nameTmp)); const uint64_t sftpErr = libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)); // Some/most sftp servers will not rename over an existing file, in testing this returned LIBSSH2_FX_FAILURE if (rc == LIBSSH2_ERROR_SFTP_PROTOCOL && (sftpErr == LIBSSH2_FX_FAILURE || sftpErr == LIBSSH2_FX_FILE_ALREADY_EXISTS)) { // Remove the existing file and retry the rename storageWriteSftpUnlinkExisting(this); storageWriteSftpRename(this); } else { storageSftpEvalLibSsh2Error( rc, sftpErr, &FileCloseError, strNewFmt("unable to move '%s' to '%s'", strZ(this->nameTmp), strZ(this->name)), NULL); } } } } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageWriteInterface storageWriteSftpInterface = { .close = storageWriteSftpClose, .open = storageWriteSftpOpen, .write = storageWriteSftp, }; FN_EXTERN StorageWriteSftp * storageWriteSftpNew( StorageSftp *const storage, const String *const name, const mode_t modeFile, const mode_t modePath, const bool createPath, const bool syncFile, const bool atomic) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(BOOL, createPath); FUNCTION_LOG_PARAM(BOOL, syncFile); FUNCTION_LOG_PARAM(BOOL, atomic); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); ASSERT(modeFile != 0); ASSERT(modePath != 0); OBJ_NEW_BEGIN(StorageWriteSftp, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageWriteSftp) { .interface = &storageWriteSftpInterface, .name = strDup(name), .modeFile = modeFile, .modePath = modePath, .createPath = createPath, .syncFile = syncFile, .atomic = atomic, .storage = storage, .nameTmp = atomic ? strNewFmt("%s." STORAGE_FILE_TEMP_EXT, strZ(name)) : strDup(name), .path = strPath(name), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE_SFTP, this); } #endif // HAVE_LIBSSH2 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/sftp/write.h0000664000175100017510000000354715265101745020125 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage File Write ***********************************************************************************************************************************/ #ifndef STORAGE_SFTP_WRITE_H #define STORAGE_SFTP_WRITE_H #include "common/io/session.h" #include "storage/sftp/storage.h" #include "storage/sftp/storage.intern.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWriteSftp StorageWriteSftp; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWriteSftp *storageWriteSftpNew( StorageSftp *storage, const String *name, mode_t modeFile, mode_t modePath, bool createPath, bool syncFile, bool atomic); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_WRITE_SFTP_TYPE \ StorageWriteSftp * #define FUNCTION_LOG_STORAGE_WRITE_SFTP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageWriteSftp", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/storage.c0000664000175100017510000007777015265101745017467 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Interface ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/type/list.h" #include "common/wait.h" #include "storage/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Storage { StoragePub pub; // Publicly accessible variables const String *path; mode_t modeFile; mode_t modePath; bool write; time_t targetTime; // Target repo reads by time List *cacheList; // Storage cache StoragePathExpressionCallback *pathExpressionFunction; }; // Storage list cache typedef struct StorageListCache { String *path; // Path StorageList *list; // List } StorageListCache; /**********************************************************************************************************************************/ FN_EXTERN Storage * storageNew( const StringId type, const String *const path, const mode_t modeFile, const mode_t modePath, const bool write, const time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, void *const driver, const StorageInterface interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(STORAGE_INTERFACE, interface); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(type != 0); ASSERT(strSize(path) >= 1 && strZ(path)[0] == '/'); ASSERT(driver != NULL); ASSERT(interface.info != NULL); ASSERT(interface.list != NULL); ASSERT(interface.newRead != NULL); ASSERT(interface.newWrite != NULL); ASSERT(interface.pathRemove != NULL); ASSERT(interface.remove != NULL); OBJ_NEW_BEGIN(Storage, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (Storage) { .pub = { .type = type, .driver = objMoveToInterface(driver, this, memContextPrior()), .interface = interface, }, .path = strDup(path), .modeFile = modeFile, .modePath = modePath, .write = write, .targetTime = targetTime, .pathExpressionFunction = pathExpressionFunction, }; // Create storage cache list if (this->targetTime != 0) this->cacheList = lstNewP(sizeof(StorageListCache), .comparator = lstComparatorStr); // If time limit is requested then versioning must be supported CHECK(AssertError, targetTime == 0 || storageFeature(this, storageFeatureVersioning), "versioning required for time limit"); // If time limit is requested then storage must be read-only CHECK(AssertError, targetTime == 0 || !write, "time limit requires read-only storage"); // If path sync feature is enabled then path feature must be enabled CHECK( AssertError, !storageFeature(this, storageFeaturePathSync) || storageFeature(this, storageFeaturePath), "path feature required"); // If hardlink feature is enabled then path feature must be enabled CHECK( AssertError, !storageFeature(this, storageFeatureHardLink) || storageFeature(this, storageFeaturePath), "path feature required"); // If symlink feature is enabled then path feature must be enabled CHECK( AssertError, !storageFeature(this, storageFeatureSymLink) || storageFeature(this, storageFeaturePath), "path feature required"); // If link features are enabled then linkCreate must be implemented CHECK( AssertError, (!storageFeature(this, storageFeatureSymLink) && !storageFeature(this, storageFeatureHardLink)) || interface.linkCreate != NULL, "linkCreate required"); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE, this); } /**********************************************************************************************************************************/ FN_EXTERN bool storageCopy(StorageRead *const source, StorageWrite *const destination) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_READ, source); FUNCTION_LOG_PARAM(STORAGE_WRITE, destination); FUNCTION_LOG_END(); ASSERT(source != NULL); ASSERT(destination != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { // Open source file if (ioReadOpen(storageReadIo(source))) { // Open the destination file now that we know the source file exists and is readable ioWriteOpen(storageWriteIo(destination)); // Copy data from source to destination ioCopyP(storageReadIo(source), storageWriteIo(destination)); // Close the source and destination files ioReadClose(storageReadIo(source)); ioWriteClose(storageWriteIo(destination)); // Set result to indicate that the file was copied result = true; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN bool storageExists(const Storage *const this, const String *const pathExp, const StorageExistsParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_PARAM(TIMEMSEC, param.timeout); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { Wait *const wait = waitNew(param.timeout); // Loop until file exists or timeout do { // storageInfoLevelBasic is required here because storageInfoLevelExists will not return the type and this function // specifically wants to test existence of a *file*, not just the existence of anything with the specified name. const StorageInfo info = storageInfoP( this, pathExp, .level = storageInfoLevelBasic, .ignoreMissing = true, .followLink = true); // Only exists if it is a file result = info.exists && info.type == storageTypeFile; } while (!result && waitMore(wait)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * storageGet(StorageRead *const file, const StorageGetParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_READ, file); FUNCTION_LOG_PARAM(SIZE, param.exactSize); FUNCTION_LOG_END(); ASSERT(file != NULL); Buffer *result = NULL; // If the file exists if (ioReadOpen(storageReadIo(file))) { MEM_CONTEXT_TEMP_BEGIN() { // If exact read if (param.exactSize > 0) { result = bufNew(param.exactSize); ioRead(storageReadIo(file), result); // If an exact read make sure the size is as expected if (bufUsed(result) != param.exactSize) THROW_FMT(FileReadError, "unable to read %zu byte(s) from '%s'", param.exactSize, strZ(storageReadName(file))); } // Else read entire file else { result = bufNew(0); Buffer *const read = bufNew(ioBufferSize()); do { // Read data ioRead(storageReadIo(file), read); // Add to result and free read buffer bufCat(result, read); bufUsedZero(read); } while (!ioReadEof(storageReadIo(file))); } // Move buffer to parent context on success bufMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); ioReadClose(storageReadIo(file)); } FUNCTION_LOG_RETURN(BUFFER, result); } /**********************************************************************************************************************************/ FN_EXTERN StorageInfo storageInfo(const Storage *const this, const String *const fileExp, StorageInfoParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, fileExp); FUNCTION_LOG_PARAM(ENUM, param.level); FUNCTION_LOG_PARAM(BOOL, param.ignoreMissing); FUNCTION_LOG_PARAM(BOOL, param.followLink); FUNCTION_LOG_PARAM(BOOL, param.noPathEnforce); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); ASSERT(this->pub.interface.info != NULL); StorageInfo result = {0}; MEM_CONTEXT_TEMP_BEGIN() { // Build the path const String *const file = storagePathP(this, fileExp, .noEnforce = param.noPathEnforce); // Set level when default if (param.level == storageInfoLevelDefault) param.level = storageFeature(this, storageFeatureInfoDetail) ? storageInfoLevelDetail : storageInfoLevelBasic; // If file is / then this is definitely a path so skip the call for drivers that do not support paths and do not provide // additional info to return. Also, some object stores (e.g. S3) behave strangely when getting info for /. if (strEq(file, FSLASH_STR) && !storageFeature(this, storageFeaturePath)) { result = (StorageInfo){.level = param.level}; } // Else call the driver else { // If targeting by time if (this->targetTime != 0) { // Find the version to read using the cache const String *const path = strPath(file); StorageListCache *cache = lstFind(this->cacheList, &path); if (cache == NULL) { MEM_CONTEXT_OBJ_BEGIN(this->cacheList) { StorageList *const list = storageInterfaceListP( storageDriver(this), path, storageInfoLevelBasic, .targetTime = this->targetTime); if (list != NULL) storageLstSort(list, sortOrderAsc); cache = memNew(sizeof(StorageListCache)); *cache = (StorageListCache){.path = strDup(path), .list = list}; lstAdd(this->cacheList, cache); } MEM_CONTEXT_OBJ_END(); lstSort(this->cacheList, sortOrderAsc); } if (cache->list != NULL) { result = storageLstFind(cache->list, strBase(file)); ASSERT(!result.exists || result.type != storageTypeFile || result.versionId != NULL); } } else result = storageInterfaceInfoP(storageDriver(this), file, param.level, .followLink = param.followLink); } // Error if the file missing and not ignoring if (!result.exists && !param.ignoreMissing) THROW_FMT(FileOpenError, STORAGE_ERROR_INFO_MISSING, strZ(file)); // Dup the strings into the prior context MEM_CONTEXT_PRIOR_BEGIN() { result.linkDestination = strDup(result.linkDestination); result.user = strDup(result.user); result.group = strDup(result.group); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ FN_EXTERN StorageIterator * storageNewItr(const Storage *const this, const String *const pathExp, StorageNewItrParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_PARAM(ENUM, param.level); FUNCTION_LOG_PARAM(BOOL, param.errorOnMissing); FUNCTION_LOG_PARAM(BOOL, param.recurse); FUNCTION_LOG_PARAM(BOOL, param.nullOnMissing); FUNCTION_LOG_PARAM(ENUM, param.sortOrder); FUNCTION_LOG_PARAM(STRING, param.expression); FUNCTION_LOG_PARAM(BOOL, param.recurse); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.interface.list != NULL); ASSERT(!param.errorOnMissing || storageFeature(this, storageFeaturePath)); StorageIterator *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Info level if (param.level == storageInfoLevelDefault) param.level = storageFeature(this, storageFeatureInfoDetail) ? storageInfoLevelDetail : storageInfoLevelBasic; result = storageItrMove( storageItrNew( storageDriver(this), storagePathP(this, pathExp), param.level, param.errorOnMissing, param.nullOnMissing, param.recurse, param.sortOrder, this->targetTime, param.expression), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_ITERATOR, result); } /**********************************************************************************************************************************/ FN_EXTERN void storageLinkCreate( const Storage *const this, const String *const target, const String *const linkPath, const StorageLinkCreateParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, target); FUNCTION_LOG_PARAM(STRING, linkPath); FUNCTION_LOG_PARAM(ENUM, param.linkType); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->write); ASSERT(target != NULL); ASSERT(linkPath != NULL); ASSERT(this->pub.interface.linkCreate != NULL); ASSERT( (param.linkType == storageLinkSym && storageFeature(this, storageFeatureSymLink)) || (param.linkType == storageLinkHard && storageFeature(this, storageFeatureHardLink))); MEM_CONTEXT_TEMP_BEGIN() { storageInterfaceLinkCreateP(storageDriver(this), target, linkPath, .linkType = param.linkType); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN StringList * storageList(const Storage *const this, const String *const pathExp, const StorageListParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_PARAM(BOOL, param.errorOnMissing); FUNCTION_LOG_PARAM(BOOL, param.nullOnMissing); FUNCTION_LOG_PARAM(STRING, param.expression); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!param.errorOnMissing || !param.nullOnMissing); StringList *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { StorageIterator *const storageItr = storageNewItrP( this, pathExp, .level = storageInfoLevelExists, .errorOnMissing = param.errorOnMissing, .nullOnMissing = param.nullOnMissing, .expression = param.expression); if (storageItr != NULL) { result = strLstNew(); while (storageItrMore(storageItr)) strLstAdd(result, storageItrNext(storageItr).name); strLstMove(result, memContextPrior()); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN void storageMove(const Storage *const this, StorageRead *const source, StorageWrite *const destination) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_READ, source); FUNCTION_LOG_PARAM(STORAGE_WRITE, destination); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.interface.move != NULL); ASSERT(source != NULL); ASSERT(destination != NULL); ASSERT(!storageReadIgnoreMissing(source)); ASSERT(storageType(this) == storageReadType(source)); ASSERT(storageReadType(source) == storageWriteType(destination)); MEM_CONTEXT_TEMP_BEGIN() { // If the file can't be moved it will need to be copied if (!storageInterfaceMoveP(storageDriver(this), source, destination)) // {vm_covered} { // Perform the copy storageCopyP(source, destination); // {vm_covered} // Remove the source file storageInterfaceRemoveP(storageDriver(this), storageReadName(source)); // {vm_covered} // Sync source path if the destination path was synced. We know the source and destination paths are different because // the move did not succeed. This will need updating when drivers other than Posix/CIFS are implemented because there's // no way to get coverage on it now. if (storageWriteSyncPath(destination)) // {vm_covered} storageInterfacePathSyncP(storageDriver(this), strPath(storageReadName(source))); // {vm_covered} } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN StorageRead * storageNewRead(const Storage *const this, const String *const fileExp, const StorageNewReadParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, fileExp); FUNCTION_LOG_PARAM(BOOL, param.ignoreMissing); FUNCTION_LOG_PARAM(BOOL, param.compressible); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(param.limit == NULL || varType(param.limit) == varTypeUInt64); StorageRead *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { const String *const path = storagePathP(this, fileExp); const String *versionId = NULL; // If targeting by time if (this->targetTime != 0) { versionId = storageInfoP(this, fileExp, .ignoreMissing = true).versionId; if (versionId == NULL && !param.ignoreMissing) THROW_FMT(FileMissingError, STORAGE_ERROR_READ_MISSING, strZ(path)); } result = storageReadMove( storageReadNew( this, path, param.ignoreMissing, false, param.compressible, param.offset, param.limit, this->targetTime != 0, versionId), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_READ, result); } /**********************************************************************************************************************************/ FN_EXTERN StorageReadMulti * storageNewReadMulti(const Storage *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN( STORAGE_READ_MULTI, storageReadMultiNew(this, this->pub.interface.prefetch, this->pub.interface.readOver)); } /**********************************************************************************************************************************/ FN_EXTERN StorageWrite * storageNewWrite(const Storage *const this, const String *const fileExp, const StorageNewWriteParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, fileExp); FUNCTION_LOG_PARAM(MODE, param.modeFile); FUNCTION_LOG_PARAM(MODE, param.modePath); FUNCTION_LOG_PARAM(STRING, param.user); FUNCTION_LOG_PARAM(STRING, param.group); FUNCTION_LOG_PARAM(INT64, param.timeModified); FUNCTION_LOG_PARAM(BOOL, param.noCreatePath); FUNCTION_LOG_PARAM(BOOL, param.noSyncFile); FUNCTION_LOG_PARAM(BOOL, param.noSyncPath); FUNCTION_LOG_PARAM(BOOL, param.noAtomic); FUNCTION_LOG_PARAM(BOOL, param.noTruncate); FUNCTION_LOG_PARAM(BOOL, param.compressible); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->write); // noTruncate does not work with atomic writes because a new file is always created for atomic writes ASSERT(!param.noTruncate || param.noAtomic); StorageWrite *result; MEM_CONTEXT_TEMP_BEGIN() { result = storageWriteMove( storageWriteNew( this, storagePathP(this, fileExp), param.modeFile != 0 ? param.modeFile : this->modeFile, param.modePath != 0 ? param.modePath : this->modePath, param.user, param.group, param.timeModified, !param.noCreatePath, !param.noSyncFile, !param.noSyncPath, !param.noAtomic, !param.noTruncate, param.compressible), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE, result); } /**********************************************************************************************************************************/ FN_EXTERN String * storagePath(const Storage *const this, const String *pathExp, const StoragePathParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE, this); FUNCTION_TEST_PARAM(STRING, pathExp); FUNCTION_TEST_PARAM(BOOL, param.noEnforce); FUNCTION_TEST_END(); ASSERT(this != NULL); String *result; // If there is no path expression then return the base storage path if (pathExp == NULL) { result = strDup(this->path); } else { // If the path expression is absolute then use it as is if ((strZ(pathExp))[0] == '/') { // Make sure the base storage path is contained within the path expression if (!strEqZ(this->path, "/")) { if (!param.noEnforce && (!strBeginsWith(pathExp, this->path) || !(strSize(pathExp) == strSize(this->path) || *(strZ(pathExp) + strSize(this->path)) == '/'))) { THROW_FMT(AssertError, "absolute path '%s' is not in base path '%s'", strZ(pathExp), strZ(this->path)); } } result = strDup(pathExp); } // Else path expression is relative so combine it with the base storage path else { // There may or may not be a path expression that needs to be evaluated String *pathEvaluated = NULL; // Check if there is a path expression that needs to be evaluated if ((strZ(pathExp))[0] == '<') { if (this->pathExpressionFunction == NULL) THROW_FMT(AssertError, "expression '%s' not valid without callback function", strZ(pathExp)); // Get position of the expression end const char *const end = strchr(strZ(pathExp), '>'); // Error if end is not found if (end == NULL) THROW_FMT(AssertError, "end > not found in path expression '%s'", strZ(pathExp)); // Create a string from the expression String *const expression = strNewZN(strZ(pathExp), (size_t)(end - strZ(pathExp) + 1)); // Create a string from the path if there is anything left after the expression String *path = NULL; if (strSize(expression) < strSize(pathExp)) { // Error if path separator is not found if (end[1] != '/') THROW_FMT(AssertError, "'/' should separate expression and path '%s'", strZ(pathExp)); // Only create path if there is something after the path separator if (end[2] == 0) THROW_FMT(AssertError, "path '%s' should not end in '/'", strZ(pathExp)); path = strNewZ(end + 2); } // Evaluate the path pathEvaluated = this->pathExpressionFunction(expression, path); // Evaluated path cannot be NULL if (pathEvaluated == NULL) THROW_FMT(AssertError, "evaluated path '%s' cannot be null", strZ(pathExp)); // Assign evaluated path to path pathExp = pathEvaluated; // Free temp vars strFree(expression); strFree(path); } if (strEqZ(this->path, "/")) result = strNewFmt("/%s", strZ(pathExp)); else result = strNewFmt("%s/%s", strZ(this->path), strZ(pathExp)); strFree(pathEvaluated); } } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void storagePathCreate(const Storage *const this, const String *const pathExp, const StoragePathCreateParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_PARAM(BOOL, param.errorOnExists); FUNCTION_LOG_PARAM(BOOL, param.noParentCreate); FUNCTION_LOG_PARAM(MODE, param.mode); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.interface.pathCreate != NULL && storageFeature(this, storageFeaturePath)); ASSERT(this->write); MEM_CONTEXT_TEMP_BEGIN() { // Call driver function storageInterfacePathCreateP( storageDriver(this), storagePathP(this, pathExp), param.errorOnExists, param.noParentCreate, param.mode != 0 ? param.mode : this->modePath); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool storagePathExists(const Storage *const this, const String *const pathExp) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(storageFeature(this, storageFeaturePath)); // storageInfoLevelBasic is required here because storageInfoLevelExists will not return the type and this function specifically // wants to test existence of a *path*, not just the existence of anything with the specified name. const StorageInfo info = storageInfoP(this, pathExp, .level = storageInfoLevelBasic, .ignoreMissing = true, .followLink = true); FUNCTION_LOG_RETURN(BOOL, info.exists && info.type == storageTypePath); } /**********************************************************************************************************************************/ FN_EXTERN void storagePathRemove(const Storage *const this, const String *const pathExp, const StoragePathRemoveParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_PARAM(BOOL, param.errorOnMissing); FUNCTION_LOG_PARAM(BOOL, param.recurse); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->write); ASSERT(!param.errorOnMissing || storageFeature(this, storageFeaturePath)); ASSERT(param.recurse || storageFeature(this, storageFeaturePath)); MEM_CONTEXT_TEMP_BEGIN() { // Build the path const String *const path = storagePathP(this, pathExp); // Call driver function if (!storageInterfacePathRemoveP(storageDriver(this), path, param.recurse) && param.errorOnMissing) THROW_FMT(PathRemoveError, STORAGE_ERROR_PATH_REMOVE_MISSING, strZ(path)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void storagePathSync(const Storage *const this, const String *const pathExp) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->write); // Not all storage requires path sync so just do nothing if the function is not implemented if (this->pub.interface.pathSync != NULL) { MEM_CONTEXT_TEMP_BEGIN() { storageInterfacePathSyncP(storageDriver(this), storagePathP(this, pathExp)); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void storagePut(StorageWrite *const file, const Buffer *const buffer) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_WRITE, file); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(file != NULL); ioWriteOpen(storageWriteIo(file)); ioWrite(storageWriteIo(file), buffer); ioWriteClose(storageWriteIo(file)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void storageRemove(const Storage *const this, const String *const fileExp, const StorageRemoveParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, fileExp); FUNCTION_LOG_PARAM(BOOL, param.errorOnMissing); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->write); ASSERT(!param.errorOnMissing || storageFeature(this, storageFeatureFileRemoveMissing)); MEM_CONTEXT_TEMP_BEGIN() { // Call driver function storageInterfaceRemoveP(storageDriver(this), storagePathP(this, fileExp), .errorOnMissing = param.errorOnMissing); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN time_t storageTargetTime(const Storage *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(TIME, this->targetTime); } /**********************************************************************************************************************************/ FN_EXTERN void storageToLog(const Storage *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "), strStcResultSizeInc(debugLog, strIdToLog(storageType(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt(debugLog, ", path: %s, write: %s}", strZ(this->path), cvtBoolToConstZ(this->write)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/storage.h0000664000175100017510000003150515265101745017456 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Interface ***********************************************************************************************************************************/ #ifndef STORAGE_STORAGE_H #define STORAGE_STORAGE_H #include /*********************************************************************************************************************************** Storage link type ***********************************************************************************************************************************/ typedef enum { // Symbolic (or soft) link storageLinkSym, // Hard link storageLinkHard, } StorageLinkType; /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Storage Storage; #include "common/io/filter/group.h" #include "common/time.h" #include "common/type/buffer.h" #include "common/type/param.h" #include "common/type/stringList.h" #include "storage/info.h" #include "storage/iterator.h" #include "storage/read.h" #include "storage/readMulti.h" #include "storage/storage.intern.h" #include "storage/write.h" /*********************************************************************************************************************************** Storage feature ***********************************************************************************************************************************/ typedef enum { // Does the storage support paths/directories as something that needs to be created and deleted? Object stores (e.g. S3) often // do not have paths/directories -- they are only inferred by the object name. Therefore it doesn't make sense to create or // remove directories since this implies something is happening on the storage and in the case of objects stores it would be a // noop. We'll error on any path operation (e.g. pathExists(), pathCreate(), non-recursive removes, error on missing paths, // etc.) for storage that does not support paths. storageFeaturePath, // Do paths need to be synced to ensure contents are durable? storageFeaturePath must also be enabled. storageFeaturePathSync, // Does the storage support hardlinks? Hardlinks allow the same file to be linked into multiple paths to save space. storageFeatureHardLink, // Does the storage support symlinks? Symlinks allow paths/files/links to be accessed from another path. storageFeatureSymLink, // Does the storage support detailed info, i.e. user, group, mode, link destination, etc. storageFeatureInfoDetail, // Does the storage support versioning? storageFeatureVersioning, // Can reads be retried? storageFeatureReadRetry, // Can files be detected as missing on remove? storageFeatureFileRemoveMissing, } StorageFeature; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Copy a file #define storageCopyP(source, destination) \ storageCopy(source, destination) FN_EXTERN bool storageCopy(StorageRead *source, StorageWrite *destination); // Does a file exist? This function is only for files, not paths. typedef struct StorageExistsParam { VAR_PARAM_HEADER; TimeMSec timeout; } StorageExistsParam; #define storageExistsP(this, pathExp, ...) \ storageExists(this, pathExp, (StorageExistsParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool storageExists(const Storage *this, const String *pathExp, StorageExistsParam param); // Read from storage into a buffer typedef struct StorageGetParam { VAR_PARAM_HEADER; size_t exactSize; } StorageGetParam; #define storageGetP(file, ...) \ storageGet(file, (StorageGetParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN Buffer *storageGet(StorageRead *file, StorageGetParam param); // File/path info typedef struct StorageInfoParam { VAR_PARAM_HEADER; StorageInfoLevel level; bool ignoreMissing; bool followLink; bool noPathEnforce; } StorageInfoParam; #define storageInfoP(this, fileExp, ...) \ storageInfo(this, fileExp, (StorageInfoParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN StorageInfo storageInfo(const Storage *this, const String *fileExp, StorageInfoParam param); // Iterator for all files/links/paths in a path which returns different info based on the value of the level parameter typedef struct StorageNewItrParam { VAR_PARAM_HEADER; StorageInfoLevel level; bool errorOnMissing; bool nullOnMissing; bool recurse; SortOrder sortOrder; const String *expression; } StorageNewItrParam; #define storageNewItrP(this, fileExp, ...) \ storageNewItr(this, fileExp, (StorageNewItrParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN StorageIterator *storageNewItr(const Storage *this, const String *pathExp, StorageNewItrParam param); // Get a list of files from a directory typedef struct StorageListParam { VAR_PARAM_HEADER; bool errorOnMissing; bool nullOnMissing; const String *expression; } StorageListParam; #define storageListP(this, pathExp, ...) \ storageList(this, pathExp, (StorageListParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN StringList *storageList(const Storage *this, const String *pathExp, StorageListParam param); // Move a file #define storageMoveP(this, source, destination) \ storageMove(this, source, destination) FN_EXTERN void storageMove(const Storage *this, StorageRead *source, StorageWrite *destination); // Open a file for reading typedef struct StorageNewReadParam { VAR_PARAM_HEADER; bool ignoreMissing; bool compressible; // Where to start reading in the file const uint64_t offset; // Limit bytes to read from the file (must be varTypeUInt64). NULL for no limit. const Variant *limit; } StorageNewReadParam; #define storageNewReadP(this, pathExp, ...) \ storageNewRead(this, pathExp, (StorageNewReadParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN StorageRead *storageNewRead(const Storage *this, const String *fileExp, StorageNewReadParam param); // Open multiple files for read #define storageNewReadMultiP(this) \ storageNewReadMulti(this) FN_EXTERN StorageReadMulti *storageNewReadMulti(const Storage *this); // Open a file for writing typedef struct StorageNewWriteParam { VAR_PARAM_HEADER; bool noCreatePath; bool noSyncFile; bool noSyncPath; bool noAtomic; // Do not truncate file if it exists. Use this only in cases where the file will be manipulated directly through the file // handle, which should always be the exception and indicates functionality that should be added to the storage interface. bool noTruncate; bool compressible; mode_t modeFile; mode_t modePath; time_t timeModified; const String *user; const String *group; } StorageNewWriteParam; #define storageNewWriteP(this, pathExp, ...) \ storageNewWrite(this, pathExp, (StorageNewWriteParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN StorageWrite *storageNewWrite(const Storage *this, const String *fileExp, StorageNewWriteParam param); // Get absolute path in the storage typedef struct StoragePathParam { VAR_PARAM_HEADER; bool noEnforce; } StoragePathParam; #define storagePathP(this, pathExp, ...) \ storagePath(this, pathExp, (StoragePathParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN String *storagePath(const Storage *this, const String *pathExp, StoragePathParam param); // Create a path typedef struct StoragePathCreateParam { VAR_PARAM_HEADER; bool errorOnExists; bool noParentCreate; mode_t mode; } StoragePathCreateParam; #define storagePathCreateP(this, pathExp, ...) \ storagePathCreate(this, pathExp, (StoragePathCreateParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void storagePathCreate(const Storage *this, const String *pathExp, StoragePathCreateParam param); // Does a path exist? #define storagePathExistsP(this, pathExp) \ storagePathExists(this, pathExp) FN_EXTERN bool storagePathExists(const Storage *this, const String *pathExp); // Remove a path typedef struct StoragePathRemoveParam { VAR_PARAM_HEADER; bool errorOnMissing; bool recurse; } StoragePathRemoveParam; #define storagePathRemoveP(this, pathExp, ...) \ storagePathRemove(this, pathExp, (StoragePathRemoveParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void storagePathRemove(const Storage *this, const String *pathExp, StoragePathRemoveParam param); // Sync a path #define storagePathSyncP(this, pathExp) \ storagePathSync(this, pathExp) FN_EXTERN void storagePathSync(const Storage *this, const String *pathExp); // Write a buffer to storage #define storagePutP(file, buffer) \ storagePut(file, buffer) FN_EXTERN void storagePut(StorageWrite *file, const Buffer *buffer); // Remove a file typedef struct StorageRemoveParam { VAR_PARAM_HEADER; bool errorOnMissing; } StorageRemoveParam; #define storageRemoveP(this, fileExp, ...) \ storageRemove(this, fileExp, (StorageRemoveParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void storageRemove(const Storage *this, const String *fileExp, StorageRemoveParam param); // Create a hard or symbolic link typedef struct StorageLinkCreateParam { VAR_PARAM_HEADER; // Flag to create hard or symbolic link StorageLinkType linkType; } StorageLinkCreateParam; #define storageLinkCreateP(this, target, linkPath, ...) \ storageLinkCreate(this, target, linkPath, (StorageLinkCreateParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void storageLinkCreate(const Storage *this, const String *target, const String *linkPath, StorageLinkCreateParam param); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Is the feature supported by this storage? FN_INLINE_ALWAYS bool storageFeature(const Storage *const this, const StorageFeature feature) { return THIS_PUB(Storage)->interface.feature >> feature & 1; } // Storage type (posix, cifs, etc.) FN_INLINE_ALWAYS StringId storageType(const Storage *const this) { return THIS_PUB(Storage)->type; } // Target reads by time (0 when not time-targeted) FN_EXTERN time_t storageTargetTime(const Storage *this); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void storageToLog(const Storage *this, StringStatic *debugLog); #define FUNCTION_LOG_STORAGE_TYPE \ Storage * #define FUNCTION_LOG_STORAGE_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, storageToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/storage.intern.h0000664000175100017510000005107715265101745020762 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Interface Internal Storage drivers are implemented using this interface. The interface has required and optional functions. Currently the optional functions are only implemented by the Posix driver which can store either a repository or a PostgreSQL cluster. Drivers that are intended to store repositories only need to implement the required functions. The behavior of required functions is further modified by storage features defined by the StorageFeature enum. Details are included in the description of each function. ***********************************************************************************************************************************/ #ifndef STORAGE_STORAGE_INTERN_H #define STORAGE_STORAGE_INTERN_H #include "common/type/param.h" #include "storage/info.h" #include "storage/list.h" #include "storage/read.h" #include "storage/write.h" /*********************************************************************************************************************************** Default file and path modes ***********************************************************************************************************************************/ #define STORAGE_MODE_FILE_DEFAULT 0640 #define STORAGE_MODE_PATH_DEFAULT 0750 /*********************************************************************************************************************************** Error messages ***********************************************************************************************************************************/ #define STORAGE_ERROR_READ_CLOSE "unable to close file '%s' after read" #define STORAGE_ERROR_READ_OPEN "unable to open file '%s' for read" #define STORAGE_ERROR_READ_MISSING "unable to open missing file '%s' for read" #define STORAGE_ERROR_READ_SEEK "unable to seek to %" PRIu64 " in file '%s'" #define STORAGE_ERROR_INFO "unable to get info for path/file '%s'" #define STORAGE_ERROR_INFO_MISSING "unable to get info for missing path/file '%s'" #define STORAGE_ERROR_LIST_INFO "unable to list file info for path '%s'" #define STORAGE_ERROR_LIST_INFO_MISSING "unable to list file info for missing path '%s'" #define STORAGE_ERROR_FILE_REMOVE "unable to remove file '%s'" #define STORAGE_ERROR_PATH_REMOVE "unable to remove path '%s'" #define STORAGE_ERROR_PATH_REMOVE_MISSING "unable to remove missing path '%s'" #define STORAGE_ERROR_PATH_SYNC "unable to sync path '%s'" #define STORAGE_ERROR_PATH_SYNC_CLOSE "unable to close path '%s' after sync" #define STORAGE_ERROR_PATH_SYNC_OPEN "unable to open path '%s' for sync" #define STORAGE_ERROR_PATH_SYNC_MISSING "unable to sync missing path '%s'" #define STORAGE_ERROR_WRITE_CLOSE "unable to close file '%s' after write" #define STORAGE_ERROR_WRITE_OPEN "unable to open file '%s' for write" #define STORAGE_ERROR_WRITE_MISSING "unable to open file '%s' for write in missing path" #define STORAGE_ERROR_WRITE_SEEK "unable to seek to %" PRIu64 " in file '%s'" #define STORAGE_ERROR_WRITE_SYNC "unable to sync file '%s' after write" /*********************************************************************************************************************************** Path expression callback function type - used to modify paths based on expressions enclosed in <> ***********************************************************************************************************************************/ typedef String *StoragePathExpressionCallback(const String *expression, const String *path); /*********************************************************************************************************************************** Storage info callback function type - used to return storage info ***********************************************************************************************************************************/ typedef void (*StorageListCallback)(void *callbackData, const StorageInfo *info); /*********************************************************************************************************************************** Required interface functions ***********************************************************************************************************************************/ // Get information about a file/link/path // // The level parameter controls the amount of information that will be returned. See the StorageInfo type and StorageInfoLevel enum // for details about information that must be provided at each level. The driver should only return the amount of information // requested even if more is available. All drivers must implement the storageInfoLevelExists and storageInfoLevelBasic levels. Only // drivers with the storageFeatureInfoDetail feature must implement the storageInfoLevelDetail level. typedef struct StorageInterfaceInfoParam { VAR_PARAM_HEADER; // Should symlinks be followed? Only required on storage that supports symlinks. bool followLink; } StorageInterfaceInfoParam; typedef StorageInfo StorageInterfaceInfo( void *thisVoid, const String *file, StorageInfoLevel level, StorageInterfaceInfoParam param); #define storageInterfaceInfoP(thisVoid, file, level, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).info(thisVoid, file, level, (StorageInterfaceInfoParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Create a hard or symbolic link typedef struct StorageInterfaceLinkCreateParam { VAR_PARAM_HEADER; // Flag to create hard or symbolic link StorageLinkType linkType; } StorageInterfaceLinkCreateParam; typedef void StorageInterfaceLinkCreate( void *thisVoid, const String *target, const String *linkPath, StorageInterfaceLinkCreateParam param); #define storageInterfaceLinkCreateP(thisVoid, target, linkPath, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).linkCreate(thisVoid, target, linkPath, \ (StorageInterfaceLinkCreateParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Create a file read object. The file should not be opened immediately -- open() will be called on the IoRead interface when the // file needs to be opened. typedef struct StorageInterfaceNewReadParam { VAR_PARAM_HEADER; // Is the file compressible? This is used when the file must be moved across a network and temporary compression is helpful. bool compressible; // Where to start reading in the file const uint64_t offset; // Limit bytes read from the file. NULL for no limit. const Variant *limit; // Target a specific file version (NULL for current version) const String *versionId; } StorageInterfaceNewReadParam; typedef void *StorageInterfaceNewRead(void *thisVoid, const String *file, StorageInterfaceNewReadParam param); #define storageInterfaceNewReadP(thisVoid, file, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).newRead(thisVoid, file, (StorageInterfaceNewReadParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Create a multi file read driver (optional). When implemented the driver handles all file reads, e.g. by forwarding them to a // remote where prefetch can be done more efficiently. When not implemented the default driver queues reads on the storage using the // prefetch and readOver values from the storage interface. typedef void *StorageInterfaceNewReadMulti(void *thisVoid); // --------------------------------------------------------------------------------------------------------------------------------- // Create a file write object. The file should not be opened immediately -- open() will be called on the IoWrite interface when the // file needs to be opened. typedef struct StorageInterfaceNewWriteParam { VAR_PARAM_HEADER; // File/path mode for storage that supports Posix-style permissions. modePath is only used in conjunction with createPath. mode_t modeFile; mode_t modePath; // User/group name const String *user; const String *group; // Modified time time_t timeModified; // Will paths be created as needed? bool createPath; // Sync file/path when required by the storage bool syncFile; bool syncPath; // Ensure the file is written atomically. If this is false it's OK to write atomically if that's all the storage supports // (e.g. S3). Non-atomic writes are used in some places where there is a performance advantage and atomicity is not needed. bool atomic; // Truncate file if it exists. Disable this only in cases where the file will be manipulated directly through the file handle, // which should always be the exception and shows functionality that should be added to the storage interface. bool truncate; // Is the file compressible? This is used when the file must be moved across a network and temporary compression is helpful. bool compressible; } StorageInterfaceNewWriteParam; typedef void *StorageInterfaceNewWrite(void *thisVoid, const String *file, StorageInterfaceNewWriteParam param); #define storageInterfaceNewWriteP(thisVoid, file, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).newWrite(thisVoid, file, (StorageInterfaceNewWriteParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Get info for all files/links/paths in a path (does not recurse or return info about the path passed to the function) // // See storageInterfaceInfoP() for usage of the level parameter. typedef struct StorageInterfaceListParam { VAR_PARAM_HEADER; // Regular expression used to filter the results. The expression is always checked in the callback passed to // storageInterfaceIterP() so checking the expression in the driver is entirely optional. The driver should only use the // expression if it can improve performance or limit network transfer. // // Partial matching of the expression is fine as long as nothing that should match is excluded, e.g. it is OK to prefix match // using the prefix returned from regExpPrefix(). This may cause extra results to be sent to the callback but won't exclude // anything that matches the expression exactly. const String *expression; // Target max version <= the specified time time_t targetTime; } StorageInterfaceListParam; typedef StorageList *StorageInterfaceList( void *thisVoid, const String *path, StorageInfoLevel level, StorageInterfaceListParam param); #define storageInterfaceListP(thisVoid, path, level, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).list( \ thisVoid, path, level, (StorageInterfaceListParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Remove a path (and optionally recurse) typedef struct StorageInterfacePathRemoveParam { VAR_PARAM_HEADER; } StorageInterfacePathRemoveParam; typedef bool StorageInterfacePathRemove(void *thisVoid, const String *path, bool recurse, StorageInterfacePathRemoveParam param); #define storageInterfacePathRemoveP(thisVoid, path, recurse, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).pathRemove( \ thisVoid, path, recurse, (StorageInterfacePathRemoveParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Remove a file typedef struct StorageInterfaceRemoveParam { VAR_PARAM_HEADER; // Error when the file to delete is missing bool errorOnMissing; } StorageInterfaceRemoveParam; typedef void StorageInterfaceRemove(void *thisVoid, const String *file, StorageInterfaceRemoveParam param); #define storageInterfaceRemoveP(thisVoid, file, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).remove(thisVoid, file, (StorageInterfaceRemoveParam){VAR_PARAM_INIT, __VA_ARGS__}) /*********************************************************************************************************************************** Optional interface functions ***********************************************************************************************************************************/ // Move a path/file atomically typedef struct StorageInterfaceMoveParam { VAR_PARAM_HEADER; } StorageInterfaceMoveParam; typedef bool StorageInterfaceMove(void *thisVoid, StorageRead *source, StorageWrite *destination, StorageInterfaceMoveParam param); #define storageInterfaceMoveP(thisVoid, source, destination, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).move( \ thisVoid, source, destination, (StorageInterfaceMoveParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Create a path typedef struct StorageInterfacePathCreateParam { VAR_PARAM_HEADER; } StorageInterfacePathCreateParam; typedef void StorageInterfacePathCreate( void *thisVoid, const String *path, bool errorOnExists, bool noParentCreate, mode_t mode, StorageInterfacePathCreateParam param); #define storageInterfacePathCreateP(thisVoid, path, errorOnExists, noParentCreate, mode, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).pathCreate( \ thisVoid, path, errorOnExists, noParentCreate, mode, (StorageInterfacePathCreateParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Sync a path typedef struct StorageInterfacePathSyncParam { VAR_PARAM_HEADER; } StorageInterfacePathSyncParam; typedef void StorageInterfacePathSync(void *thisVoid, const String *path, StorageInterfacePathSyncParam param); #define storageInterfacePathSyncP(thisVoid, path, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).pathSync(thisVoid, path, (StorageInterfacePathSyncParam){VAR_PARAM_INIT, __VA_ARGS__}) /*********************************************************************************************************************************** Storage type and helper function struct An array of this struct must be passed to storageHelperInit() to enable storage drivers other than built-in Posix. ***********************************************************************************************************************************/ typedef Storage *(*StorageHelperFunction)(unsigned int repoIdx, bool write, StoragePathExpressionCallback pathExpressionCallback); typedef struct StorageHelper { StringId type; // StringId that identifies the storage driver StorageHelperFunction helper; // Function that creates the storage } StorageHelper; #define STORAGE_END_HELPER {.type = 0} /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct StorageInterface { // Features implemented by the storage driver uint64_t feature; // Number of reads to prefetch unsigned int prefetch; // Bytes to read over rather than open file with new offset uint64_t readOver; // Required functions StorageInterfaceInfo *info; StorageInterfaceList *list; StorageInterfaceNewRead *newRead; StorageInterfaceNewWrite *newWrite; StorageInterfacePathRemove *pathRemove; StorageInterfaceRemove *remove; // Optional functions StorageInterfaceLinkCreate *linkCreate; StorageInterfaceMove *move; StorageInterfaceNewReadMulti *newReadMulti; StorageInterfacePathCreate *pathCreate; StorageInterfacePathSync *pathSync; } StorageInterface; #define storageNewP(type, path, modeFile, modePath, write, pathExpressionFunction, driver, ...) \ storageNew(type, path, modeFile, modePath, write, targetTime, pathExpressionFunction, driver, (StorageInterface){__VA_ARGS__}) FN_EXTERN Storage *storageNew( StringId type, const String *path, mode_t modeFile, mode_t modePath, bool write, time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, void *driver, StorageInterface interface); /*********************************************************************************************************************************** Common members to include in every storage driver and macros to extract the common elements ***********************************************************************************************************************************/ #define STORAGE_COMMON_MEMBER \ StorageInterface interface /* Storage interface */ typedef struct StorageCommon { STORAGE_COMMON_MEMBER; } StorageCommon; #define STORAGE_COMMON(thisVoid) \ ((const StorageCommon *)thisVoid) #define STORAGE_COMMON_INTERFACE(thisVoid) \ (STORAGE_COMMON(thisVoid)->interface) /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct StoragePub { StringId type; // Storage type void *driver; // Storage driver StorageInterface interface; // Storage interface } StoragePub; // Storage driver FN_INLINE_ALWAYS void * storageDriver(const Storage *const this) { return THIS_PUB(Storage)->driver; } // Storage interface FN_INLINE_ALWAYS StorageInterface storageInterface(const Storage *const this) { return THIS_PUB(Storage)->interface; } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_INTERFACE_TYPE \ StorageInterface #define FUNCTION_LOG_STORAGE_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "StorageInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/write.c0000664000175100017510000003074215265101745017141 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Write Interface ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWrite { StorageWritePub pub; // Publicly accessible variables void *driver; // Driver }; /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static void storageWriteOpen(THIS_VOID) { THIS(StorageWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); storageWriteDriverInterface(this->driver)->open(this->driver); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to a file ***********************************************************************************************************************************/ static void storageWrite(THIS_VOID, const Buffer *const buffer) { THIS(StorageWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); storageWriteDriverInterface(this->driver)->write(this->driver, buffer); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWriteClose(THIS_VOID) { THIS(StorageWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); storageWriteDriverInterface(this->driver)->close(this->driver); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get file descriptor ***********************************************************************************************************************************/ static int storageWriteFd(const THIS_VOID) { THIS(const StorageWrite); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, storageWriteDriverInterface(this->driver)->fd(this->driver)); } /*********************************************************************************************************************************** Seek in file ***********************************************************************************************************************************/ static void storageWriteSeek(THIS_VOID, const uint64_t position) { THIS(StorageWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE, this); FUNCTION_LOG_PARAM(UINT64, position); FUNCTION_LOG_END(); ASSERT(this != NULL); storageWriteDriverInterface(this->driver)->seek(this->driver, position); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const IoWriteInterface storageIoWriteInterface = { .close = storageWriteClose, .fd = storageWriteFd, .open = storageWriteOpen, .seek = storageWriteSeek, .write = storageWrite, }; FN_EXTERN StorageWrite * storageWriteNew( const Storage *const storage, const String *const name, const mode_t modeFile, const mode_t modePath, const String *const user, const String *const group, const time_t timeModified, const bool createPath, const bool syncFile, const bool syncPath, const bool atomic, const bool truncate, const bool compressible) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(STRING, user); FUNCTION_LOG_PARAM(STRING, group); FUNCTION_LOG_PARAM(TIME, timeModified); FUNCTION_LOG_PARAM(BOOL, createPath); FUNCTION_LOG_PARAM(BOOL, syncFile); FUNCTION_LOG_PARAM(BOOL, syncPath); FUNCTION_LOG_PARAM(BOOL, atomic); FUNCTION_LOG_PARAM(BOOL, truncate); FUNCTION_LOG_PARAM(BOOL, compressible); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(storage != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageWrite, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageWrite) { .driver = storageInterfaceNewWriteP( storageDriver(storage), name, .modeFile = modeFile, .modePath = modePath, .user = user, .group = group, .timeModified = timeModified, .createPath = createPath, .syncFile = syncFile, .syncPath = syncPath, .atomic = atomic, .truncate = truncate, .compressible = compressible), .pub = { .type = storageType(storage), .name = strDup(name), .createPath = createPath, .atomic = atomic, .truncate = truncate, .syncPath = syncPath, .syncFile = syncFile, .user = strDup(user), .group = strDup(group), .modePath = modePath, .modeFile = modeFile, .timeModified = timeModified, }, }; ASSERT(storageWriteDriverInterface(this->driver)->close != NULL); ASSERT(storageWriteDriverInterface(this->driver)->write != NULL); // Remove fd method if it does not exist in the driver IoWriteInterface storageIoWriteInterfaceCopy = storageIoWriteInterface; if (storageWriteDriverInterface(this->driver)->fd == NULL) storageIoWriteInterfaceCopy.fd = NULL; // Remove open method if it does not exist in the driver (cloud drivers don't have open) if (storageWriteDriverInterface(this->driver)->open == NULL) storageIoWriteInterfaceCopy.open = NULL; // Remove seek method if it does not exist in the driver if (storageWriteDriverInterface(this->driver)->seek == NULL) storageIoWriteInterfaceCopy.seek = NULL; this->pub.io = ioWriteNew(this, storageIoWriteInterfaceCopy); // Set filter group when interface function exists if (storageWriteDriverInterface(this->driver)->filterGroup != NULL) storageWriteDriverInterface(this->driver)->filterGroup(this->driver, ioWriteFilterGroup(storageWriteIo(this))); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE, this); } /**********************************************************************************************************************************/ #define STORAGE_CHUNK_SIZE_MAX (size_t)(1024 * 1024 * 1024) #define STORAGE_CHUNK_INCR (size_t)(8 * 1024 * 1024) FN_EXTERN size_t storageWriteChunkSize( const size_t chunkSizeDefault, const unsigned int splitDefault, const unsigned int splitMax, const unsigned int chunkIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, chunkSizeDefault); FUNCTION_TEST_PARAM(UINT, splitDefault); FUNCTION_TEST_PARAM(UINT, splitMax); FUNCTION_TEST_PARAM(UINT, chunkIdx); FUNCTION_TEST_END(); ASSERT(chunkSizeDefault > 0); ASSERT(STORAGE_CHUNK_SIZE_MAX >= chunkSizeDefault); ASSERT(STORAGE_CHUNK_INCR > 0); ASSERT(splitMax > splitDefault); // If below the default split then return default chunk size if (chunkIdx < splitDefault) { FUNCTION_TEST_RETURN(SIZE, chunkSizeDefault); } // Else if above max split then return max chunk size else if (chunkIdx > splitMax) FUNCTION_TEST_RETURN(SIZE, STORAGE_CHUNK_SIZE_MAX); // Calculate ascending chunk size uint64_t result = (uint64_t)(STORAGE_CHUNK_SIZE_MAX - STORAGE_CHUNK_INCR) * (chunkIdx - splitDefault + 1) / (splitMax - splitDefault); // If ascending chunk size is less than default then return default if (result <= chunkSizeDefault) { result = chunkSizeDefault; } // Else if ascending chunk size is not evenly divisible by chunk increment then round up else if (result % STORAGE_CHUNK_INCR != 0) result += STORAGE_CHUNK_INCR - (result % STORAGE_CHUNK_INCR); // Chunk size should never exceed SIZE_MAX (might happen on 32-bit platforms) CHECK(AssertError, result <= SIZE_MAX, "chunk size exceeds SIZE_MAX"); FUNCTION_TEST_RETURN(SIZE, (size_t)result); } /**********************************************************************************************************************************/ FN_EXTERN void storageWriteChunkBufferResize(const Buffer *const input, const size_t inputOffset, Buffer *const chunk, const size_t chunkSizeMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, input); FUNCTION_TEST_PARAM(SIZE, inputOffset); FUNCTION_TEST_PARAM(BUFFER, chunk); FUNCTION_TEST_PARAM(SIZE, chunkSizeMax); FUNCTION_TEST_END(); ASSERT(input != NULL); ASSERT(inputOffset <= bufUsed(input)); ASSERT(chunk != NULL); ASSERT(chunkSizeMax > 0); // Size required for the chunk buffer to hold the input that has not been copied yet const size_t chunkSizeRequired = bufUsed(chunk) + (bufUsed(input) - inputOffset); // Resize chunk buffer if it cannot hold the remaining input and is less than max chunk size if (chunkSizeRequired > bufSize(chunk) && bufSize(chunk) < chunkSizeMax) { size_t chunkSize; // If the input buffer is full or the chunk buffer is not empty there is very likely more data coming, so increase the chunk // buffer size aggressively rather than resizing repeatedly as small amounts of data arrive if (bufFull(input) || !bufEmpty(chunk)) { // If this is the first write set chunk size equal to double the input buffer size if (bufSize(chunk) == 0) { chunkSize = bufSize(input) * 2; } // Else double chunk size so as not to resize the chunk buffer too many times since prior data must be copied else chunkSize = bufSize(chunk) * 2; // Chunk size must be large enough to hold the remaining input if (chunkSize < chunkSizeRequired) chunkSize = chunkSizeRequired; } // Else no more writes are expected so allocate only what is needed else chunkSize = chunkSizeRequired; // Chunk size cannot be larger than max chunk size if (chunkSize > chunkSizeMax) chunkSize = chunkSizeMax; // Resize the chunk buffer bufResize(chunk, chunkSize); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void storageWriteToLog(const StorageWrite *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "); strStcResultSizeInc(debugLog, strIdToLog(storageWriteType(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt( debugLog, ", name: %s, modeFile: %04o, modePath: %04o, createPath: %s, syncFile: %s, syncPath: %s, atomic: %s}", strZ(storageWriteName(this)), storageWriteModeFile(this), storageWriteModePath(this), cvtBoolToConstZ(storageWriteCreatePath(this)), cvtBoolToConstZ(storageWriteSyncFile(this)), cvtBoolToConstZ(storageWriteSyncPath(this)), cvtBoolToConstZ(storageWriteAtomic(this))); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/write.h0000664000175100017510000001444715265101745017152 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Write Interface ***********************************************************************************************************************************/ #ifndef STORAGE_WRITE_H #define STORAGE_WRITE_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWrite StorageWrite; #include "common/io/write.h" #include "common/type/object.h" #include "common/type/string.h" #include "common/type/stringId.h" #include "storage/write.intern.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Move to a new parent mem context FN_INLINE_ALWAYS StorageWrite * storageWriteMove(StorageWrite *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct StorageWritePub { StringId type; // Storage type const String *name; // File name bool createPath; // Will the path be created if required? bool atomic; // Are writes atomic? bool truncate; // Truncate file if it exists bool syncPath; // Will path be synced? bool syncFile; // Will file be synced? const String *user; // Set user ownership const String *group; // Set group ownership mode_t modePath; // Path mode mode_t modeFile; // File mode time_t timeModified; // Set modification time IoWrite *io; // Write interface } StorageWritePub; // Will the file be written atomically? Atomic writes means the file will be complete or be missing. Filesystems have different ways // to accomplish this. FN_INLINE_ALWAYS bool storageWriteAtomic(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->atomic; } // Will the path be created if required? FN_INLINE_ALWAYS bool storageWriteCreatePath(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->createPath; } // Write interface FN_INLINE_ALWAYS IoWrite * storageWriteIo(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->io; } // File mode FN_INLINE_ALWAYS mode_t storageWriteModeFile(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->modeFile; } // Path mode (if the destination path needs to be create) FN_INLINE_ALWAYS mode_t storageWriteModePath(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->modePath; } // File name FN_INLINE_ALWAYS const String * storageWriteName(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->name; } // Will the file be synced before it is closed? FN_INLINE_ALWAYS bool storageWriteSyncFile(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->syncFile; } // Will the path be synced after the file is closed? FN_INLINE_ALWAYS bool storageWriteSyncPath(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->syncPath; } // Will the file be truncated if it exists? FN_INLINE_ALWAYS bool storageWriteTruncate(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->truncate; } // File type FN_INLINE_ALWAYS StringId storageWriteType(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->type; } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void storageWriteFree(StorageWrite *const this) { objFree(this); } /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Calculate chunk size for multipart upload to an object store FN_EXTERN size_t storageWriteChunkSize( size_t chunkSizeDefault, unsigned int defaultSplit, unsigned int maxSplit, unsigned int chunkIdx); // Resize chunk buffer for multipart upload to an object store. After the resize the chunk buffer is able to hold the input from // inputOffset to the end of the input buffer, unless max chunk size has been reached. FN_EXTERN void storageWriteChunkBufferResize(const Buffer *input, size_t inputOffset, Buffer *const chunk, size_t chunkSizeMax); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void storageWriteToLog(const StorageWrite *this, StringStatic *debugLog); #define FUNCTION_LOG_STORAGE_WRITE_TYPE \ StorageWrite * #define FUNCTION_LOG_STORAGE_WRITE_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, storageWriteToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/storage/write.intern.h0000664000175100017510000000522515265101745020442 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Write Interface Internal ***********************************************************************************************************************************/ #ifndef STORAGE_WRITE_INTERN_H #define STORAGE_WRITE_INTERN_H #include "common/io/write.h" #include "storage/storage.h" #include "version.h" /*********************************************************************************************************************************** Temporary file extension ***********************************************************************************************************************************/ #define STORAGE_FILE_TEMP_EXT PROJECT_BIN ".tmp" /*********************************************************************************************************************************** Interface ***********************************************************************************************************************************/ typedef struct StorageWriteInterface { void (*filterGroup)(void *driver, IoFilterGroup *filterGroup); // Set filter group (optional) void (*open)(void *driver); // Open write void (*write)(void *driver, const Buffer *buffer); // Write bytes void (*seek)(void *driver, uint64_t position); // Seek (optional) int (*fd)(const void *driver); // Write file descriptor (optional) void (*close)(void *driver); // Close write } StorageWriteInterface; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWrite *storageWriteNew( const Storage *storage, const String *name, mode_t modeFile, mode_t modePath, const String *user, const String *group, time_t timeModified, bool createPath, bool syncFile, bool syncPath, bool atomic, bool truncate, bool compressible); /*********************************************************************************************************************************** Getters ***********************************************************************************************************************************/ FN_INLINE_ALWAYS const StorageWriteInterface * storageWriteDriverInterface(const void *const driver) { return *(const StorageWriteInterface *const *)driver; } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/src/version.h0000664000175100017510000000526715265101745016041 0ustar00runnerrunner/*********************************************************************************************************************************** Version Numbers and Names ***********************************************************************************************************************************/ #ifndef VERSION_H #define VERSION_H /*********************************************************************************************************************************** Official name of the project ***********************************************************************************************************************************/ #define PROJECT_NAME "pgBackRest" /*********************************************************************************************************************************** Standard binary name ***********************************************************************************************************************************/ #define PROJECT_BIN "pgbackrest" /*********************************************************************************************************************************** Config file name. The path will vary based on configuration. ***********************************************************************************************************************************/ #define PROJECT_CONFIG_FILE PROJECT_BIN ".conf" /*********************************************************************************************************************************** Config include path name. The parent path will vary based on configuration. ***********************************************************************************************************************************/ #define PROJECT_CONFIG_INCLUDE_PATH "conf.d" /*********************************************************************************************************************************** Project version components. PROJECT_VERSION and PROJECT_VERSION_NUM are automatically generated from the component parts. ***********************************************************************************************************************************/ #define PROJECT_VERSION_MAJOR 2 #define PROJECT_VERSION_MINOR 60 #define PROJECT_VERSION_PATCH 0 #define PROJECT_VERSION_SUFFIX "" #define PROJECT_VERSION "2.60.0" #define PROJECT_VERSION_NUM 2060000 #endif ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1792312410.5400732 pgbackrest-2.60.0/test/0000775000175100017510000000000015265102133014351 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/test/meson.build0000664000175100017510000000376515265101745016536 0ustar00runnerrunner#################################################################################################################################### # Smoke test #################################################################################################################################### # Test a built pgbackrest through a backup/restore cycle against every PostgreSQL found on the system. Shipped in the distribution # so packagers can verify a build. Fails when there is no PostgreSQL to test against. test( 'smoke', python, args: [files('smoke.py'), '--pgbackrest', pgbackrest_exe], suite: 'smoke', timeout: 1800, ) #################################################################################################################################### # Fuzz targets #################################################################################################################################### # Link each target in fuzz with the objects of the pgbackrest binary, except main.c, and the fuzzing engine. The sources must be # compiled with the instrumentation flags of the engine, e.g. -fsanitize=fuzzer-no-link. if get_option('fuzz-engine') != '' fuzz_src = [] foreach src : src_pgbackrest if src != 'main.c' fuzz_src += src endif endforeach fuzz_object = pgbackrest_exe.extract_objects(fuzz_src) foreach fuzz : ['ini', 'json', 'pack'] executable( 'fuzz-' + fuzz, 'fuzz/fuzz.c', 'fuzz/' + fuzz + '.c', objects: fuzz_object, include_directories: include_directories('../src'), dependencies: [ lib_backtrace, lib_bz2, lib_openssl, lib_lz4, lib_systemd, lib_pq, lib_socket, lib_ssh2, lib_xml, lib_z, lib_zstd, ], link_args: get_option('fuzz-engine').split(), ) endforeach endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1792312293.0 pgbackrest-2.60.0/test/smoke.py0000775000175100017510000005743715265101745016074 0ustar00runnerrunner#!/usr/bin/env python3 """pgBackRest Smoke Test. A minimal, dependency-free (Python standard library only) sanity check that a built pgbackrest binary actually works. It finds every PostgreSQL installation on the system and drives pgbackrest through a full backup/restore cycle against each version found: initdb -> configure -> stanza-create -> check -> full backup -> load data -> incr backup -> wipe -> restore -> verify The test uses a local POSIX (filesystem) repository and a private unix socket so it never collides with a running PostgreSQL or an existing pgbackrest installation. It is a build-sanity check for packagers, not a functional test suite. Exit status: 0 = every supported version found was tested and passed, 1 = a version failed or nothing could be tested. Testing nothing is a failure rather than a skip, since a test that silently does nothing is not a successful test.""" #################################################################################################################################### import argparse import glob import os import pwd import re import shutil import subprocess import sys import tempfile import time # Stanza name used throughout the test STANZA = "smoke" # PostgreSQL port. There is no TCP listener (listen_addresses is empty) so this only names the socket file in the private socket # directory and never collides with a system PostgreSQL. PG_PORT = 5432 # Per-command timeout (seconds). The clusters are tiny so every step is fast; this only guards against a hang. COMMAND_TIMEOUT = 300 #################################################################################################################################### class SmokeError(Exception): """A smoke test step failed.""" #################################################################################################################################### class Runner: """Command runner that optionally drops privileges (when the script runs as root).""" def __init__(self, verbose, uid=None, gid=None, user=None): self.verbose = verbose self.uid = uid # target uid when dropping privileges, else None self.gid = gid self.user = user # OS user commands run as (also the PostgreSQL superuser) ################################################################################################################################ def _preexec(self): """Return a preexec function that drops to the target user, or None when running as the current user.""" if self.uid is None: return None uid, gid, user = self.uid, self.gid, self.user ############################################################################################################################ def demote(): os.setgid(gid) os.initgroups(user, gid) os.setuid(uid) return demote ################################################################################################################################ def _env(self, home, bin_path): """Build a clean environment for child processes.""" env = { "HOME": home, "PATH": bin_path + os.pathsep + os.environ.get("PATH", "/usr/bin:/bin"), "LC_ALL": "C", "LANG": "C", } return env ################################################################################################################################ def run(self, cmd, home, bin_path, input_text=None): """Run a command, capturing combined output. Raise SmokeError on failure or timeout. """ if self.verbose: sys.stderr.write("+ " + " ".join(cmd) + "\n") sys.stderr.flush() try: result = subprocess.run( cmd, cwd=home, env=self._env(home, bin_path), preexec_fn=self._preexec(), input=input_text, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True, timeout=COMMAND_TIMEOUT, ) except subprocess.TimeoutExpired: raise SmokeError("timed out after %ds: %s" % (COMMAND_TIMEOUT, " ".join(cmd))) if self.verbose and result.stdout: sys.stderr.write(result.stdout) sys.stderr.flush() if result.returncode != 0: raise SmokeError("command failed (exit %d): %s\n%s" % (result.returncode, " ".join(cmd), result.stdout)) return result.stdout ################################################################################################################################ def make_dir(self, path, mode=0o700): """Create a directory and hand ownership to the target user when dropping privileges.""" os.makedirs(path, mode=mode, exist_ok=True) if self.uid is not None: os.chown(path, self.uid, self.gid) ################################################################################################################################ def chown_tree(self, path): """Recursively chown a tree to the target user (used after creating the work dir skeleton as root).""" if self.uid is None: return os.chown(path, self.uid, self.gid) for root, dirs, files in os.walk(path): for name in dirs + files: os.chown(os.path.join(root, name), self.uid, self.gid) #################################################################################################################################### def pg_bin_candidates(extra_dirs): """Candidate bin directories to search for a PostgreSQL installation. This is a superset of the list in test/src/harness/host.c since the smoke test also runs where the integration tests cannot, i.e. MacOS, FreeBSD, and OmniOS.""" candidates = [] candidates += sorted(glob.glob("/usr/lib/postgresql/*/bin")) # Debian candidates += sorted(glob.glob("/usr/pgsql-*/bin")) # RHEL (PGDG) candidates += sorted(glob.glob("/usr/libexec/postgresql*")) # Alpine candidates += sorted(glob.glob("/opt/homebrew/opt/postgresql@*/bin")) # MacOS (Homebrew on ARM) candidates += sorted(glob.glob("/usr/local/opt/postgresql@*/bin")) # MacOS (Homebrew on Intel) candidates += ["/usr/local/bin"] # FreeBSD candidates += sorted(glob.glob("/opt/ooce/pgsql-*/bin")) # OmniOS candidates += ["/usr/bin"] # Native (e.g. RHEL application stream) candidates += list(extra_dirs) return candidates #################################################################################################################################### def pg_version(bin_dir): """Read the major version from a bin directory, or None if it is not a usable PostgreSQL install.""" for tool in ("initdb", "pg_ctl", "psql"): if not os.path.isfile(os.path.join(bin_dir, tool)) or not os.access(os.path.join(bin_dir, tool), os.X_OK): return None try: output = subprocess.check_output( [os.path.join(bin_dir, "pg_ctl"), "--version"], universal_newlines=True, stderr=subprocess.STDOUT ) except (subprocess.CalledProcessError, OSError): return None # e.g. "pg_ctl (PostgreSQL) 16.2" or "pg_ctl (PostgreSQL) 9.6.24". The version is encoded as major * 100 + minor so it sorts and # compares naturally: 9.6 -> 906, 10 -> 1000, 16 -> 1600. For version 10 and later the major is a single number and the trailing # component is only the patch release, so it is ignored. match = re.search(r"\)\s+(\d+)(?:\.(\d+))?", output) if match is None: return None major = int(match.group(1)) if major >= 10: return major * 100 # Versions before 10 have a two-part major version (e.g. 9.6) if match.group(2) is None: return None return major * 100 + int(match.group(2)) #################################################################################################################################### def pg_version_full(bin_dir): """Read the full version string from a bin directory, e.g. "pg_ctl (PostgreSQL) 19beta2". This is only called for an install returned by find_postgres(), where pg_version() has already run pg_ctl successfully. """ output = subprocess.check_output( [os.path.join(bin_dir, "pg_ctl"), "--version"], universal_newlines=True, stderr=subprocess.STDOUT ) return output.strip() #################################################################################################################################### def find_postgres(extra_dirs, only_versions, supported): """Return a sorted list of (version, bin_dir) and a sorted list of versions skipped because pgBackRest does not support them. The installs are deduplicated by the real path of initdb so a symlinked install is not run twice. """ found = {} skipped = set() for bin_dir in pg_bin_candidates(extra_dirs): if not os.path.isdir(bin_dir): continue version = pg_version(bin_dir) if version is None: continue # Skip versions that pgBackRest does not support, e.g. an old PostgreSQL that happens to be installed if supported is not None and version not in supported: skipped.add(version) continue if only_versions and version not in only_versions: continue key = os.path.realpath(os.path.join(bin_dir, "initdb")) if key not in found: found[key] = (version, bin_dir) return sorted(found.values()), sorted(skipped) #################################################################################################################################### def version_parse(version): """Parse a version string (e.g. "9.6" or "16") into the encoded form used internally.""" if "." in version: major, minor = version.split(".", 1) return int(major) * 100 + int(minor) return int(version) * 100 #################################################################################################################################### def supported_versions(): """Read the PostgreSQL versions supported by pgBackRest from the generated version header. The header is generated from build/postgres.yaml and, unlike that file, is shipped in the distribution, so this works from a source checkout or a distribution tarball. Return None when the header cannot be found so that every version found is tested. """ path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "src", "postgres", "version.auto.h") try: with open(path) as file: content = file.read() except OSError: return None result = {version_parse(match) for match in re.findall(r'^#define PG_VERSION_\w+_Z\s+"([0-9.]+)"', content, re.MULTILINE)} return result or None #################################################################################################################################### def version_str(version): """Render a version number for display (e.g. 906 -> "9.6", 1600 -> "16").""" major = version // 100 minor = version % 100 if major >= 10: return str(major) return "%d.%d" % (major, minor) #################################################################################################################################### def smoke_one(pgbackrest, version, bin_dir, runner, keep): """Run the smoke test for a single PostgreSQL version.""" work = tempfile.mkdtemp(prefix="pgbackrest-smoke-", dir=temp_base()) data = os.path.join(work, "data") repo = os.path.join(work, "repo") log = os.path.join(work, "log") socket = os.path.join(work, "socket") lock = os.path.join(work, "lock") config = os.path.join(work, "pgbackrest.conf") pg_log = os.path.join(log, "postgresql.log") # Helpers scoped to this run def pg(cmd): return runner.run([os.path.join(bin_dir, cmd[0])] + cmd[1:], work, bin_dir) def pgbr(args): return runner.run([pgbackrest, "--config=" + config, "--stanza=" + STANZA] + args, work, bin_dir) def psql(sql): return runner.run( [ os.path.join(bin_dir, "psql"), "-h", socket, "-p", str(PG_PORT), "-U", runner.user, "-d", "postgres", "-v", "ON_ERROR_STOP=1", "-tAqc", sql, ], work, bin_dir, ) pg_started = False success = False try: # Pre-create the directories the tools do not create themselves and hand the whole tree to the run user for path in (socket, log, lock): os.makedirs(path, mode=0o700, exist_ok=True) runner.chown_tree(work) # Initialize the cluster (checksums on, no fsync since it is throwaway, trust auth for local connections) pg(["initdb", "-k", "-N", "--pgdata=" + data, "--auth=trust"]) # Append test configuration to postgresql.conf conf = ( "\n" "unix_socket_directories = '%s'\n" "listen_addresses = ''\n" "port = %d\n" "wal_level = replica\n" "archive_mode = on\n" "archive_command = '%s --config=%s --stanza=%s archive-push %%p'\n" "fsync = off\n" % (socket, PG_PORT, pgbackrest, config, STANZA) ) append_file(os.path.join(data, "postgresql.conf"), conf) # Write the pgbackrest configuration write_file( config, "[global]\n" "repo1-path=%s\n" "log-path=%s\n" "lock-path=%s\n" "log-level-console=detail\n" "log-level-file=off\n" "start-fast=y\n" "compress-type=lz4\n" "\n" "[%s]\n" "pg1-path=%s\n" "pg1-port=%d\n" "pg1-socket-path=%s\n" "pg1-user=%s\n" % (repo, log, lock, STANZA, data, PG_PORT, socket, runner.user), runner, ) # Start PostgreSQL pg(["pg_ctl", "start", "-D", data, "-l", pg_log, "-w"]) pg_started = True # Load data psql("CREATE TABLE smoke AS SELECT generate_series(1, 1000) AS id") # Create the stanza and confirm archiving works end to end pgbr(["stanza-create"]) pgbr(["check"]) # Full backup pgbr(["--type=full", "backup"]) # Add more data and force the WAL segment to archive psql("INSERT INTO smoke SELECT generate_series(1001, 2000)") psql("SELECT %s()" % ("pg_switch_wal" if version >= 1000 else "pg_switch_xlog")) # Incremental backup. The backup command waits for the required WAL to archive, so the restore below has no race. pgbr(["--type=incr", "backup"]) expected = 2000 # Stop PostgreSQL and remove the data directory to prove the restore rebuilds it from the repository pg(["pg_ctl", "stop", "-D", data, "-w", "-m", "fast"]) pg_started = False shutil.rmtree(data) runner.make_dir(data, mode=0o700) # Restore the latest backup (recovery config is written automatically) and start PostgreSQL pgbr(["restore"]) pg(["pg_ctl", "start", "-D", data, "-l", pg_log, "-w"]) pg_started = True # Wait for recovery to replay all WAL and promote (there is no standby.signal so PostgreSQL promotes at end of WAL) wait_for_promote(psql) # Verify the restored data matches what was backed up actual = int(psql("SELECT count(*) FROM smoke").strip()) if actual != expected: raise SmokeError("restored row count %d does not match expected %d" % (actual, expected)) # Show backup info (printed by the runner when verbose) pgbr(["info"]) pg(["pg_ctl", "stop", "-D", data, "-w", "-m", "fast"]) pg_started = False success = True finally: # Always try to stop PostgreSQL so no stray processes are left behind if pg_started: try: runner.run([os.path.join(bin_dir, "pg_ctl"), "stop", "-D", data, "-w", "-m", "immediate"], work, bin_dir) except SmokeError: pass # Remove the work directory on success, else report where it was kept so the failure can be investigated if success and not keep: shutil.rmtree(work, ignore_errors=True) else: sys.stderr.write("work directory kept: %s\n" % work) #################################################################################################################################### def wait_for_promote(psql): """Poll until the cluster has finished recovery and accepts connections.""" deadline = time.time() + 60 while time.time() < deadline: try: if psql("SELECT pg_is_in_recovery()").strip() == "f": return except SmokeError: pass # server still starting or rejecting during recovery time.sleep(0.5) raise SmokeError("cluster did not finish recovery within 60s") #################################################################################################################################### def temp_base(): """Prefer /tmp for the work directory to keep unix socket paths short. A macOS TMPDIR can exceed the unix socket path limit.""" if os.path.isdir("/tmp") and os.access("/tmp", os.W_OK): return "/tmp" return tempfile.gettempdir() #################################################################################################################################### def write_file(path, contents, runner): """Write a file, giving it to the run user when the test dropped privileges.""" with open(path, "w") as file: file.write(contents) if runner.uid is not None: os.chown(path, runner.uid, runner.gid) #################################################################################################################################### def append_file(path, contents): """Append to a file. No ownership change is needed since the file being appended to was created by initdb as the run user.""" with open(path, "a") as file: file.write(contents) #################################################################################################################################### def resolve_runner(verbose, user_name): """Resolve the user to run as, dropping privileges when running as root.""" # Not root: run everything as the current user if os.geteuid() != 0: return Runner(verbose, user=pwd.getpwuid(os.geteuid()).pw_name) # Root: PostgreSQL refuses to run as root, so drop to an unprivileged user try: info = pwd.getpwnam(user_name) except KeyError: return None return Runner(verbose, uid=info.pw_uid, gid=info.pw_gid, user=info.pw_name) #################################################################################################################################### def binary_arch(path): """Read the machine architecture and endianness from an ELF binary. This confirms which architecture is actually being exercised, which matters when the binary is built for a foreign architecture and run under emulation.""" # ELF e_machine values for the architectures pgBackRest is built on machines = {0x03: "i386", 0x3E: "x86_64", 0x15: "ppc64", 0x16: "s390x", 0xB7: "aarch64", 0x28: "arm"} with open(path, "rb") as file: header = file.read(20) if len(header) < 20 or header[:4] != b"\x7fELF": return "unknown (not an ELF binary)" little = header[5] == 1 machine = int.from_bytes(header[18:20], "little" if little else "big") return "%s (%s-bit %s-endian)" % ( machines.get(machine, "machine 0x%02X" % machine), "64" if header[4] == 2 else "32", "little" if little else "big", ) #################################################################################################################################### def main(): """Main.""" parser = argparse.ArgumentParser(description="Run pgBackRest smoke tests against installed PostgreSQL versions.") parser.add_argument("--pgbackrest", required=True, help="path to the pgbackrest binary to test") parser.add_argument("--pg-bin-path", action="append", default=[], metavar="DIR", help="extra PostgreSQL bin directory") parser.add_argument("--version", action="append", default=[], metavar="MAJOR", help="only run this major version (repeatable)") parser.add_argument("--user", default="postgres", help="unprivileged user to drop to when run as root (default: postgres)") parser.add_argument("--keep", action="store_true", help="do not delete the work directory") parser.add_argument("-v", "--verbose", action="store_true", help="echo commands and their output") arg = parser.parse_args() pgbackrest = os.path.realpath(arg.pgbackrest) if not os.path.isfile(pgbackrest) or not os.access(pgbackrest, os.X_OK): sys.stderr.write("pgbackrest binary not found or not executable: %s\n" % pgbackrest) return 1 # Extra bin dirs from the command line and the environment (colon-separated) extra_dirs = list(arg.pg_bin_path) env_dirs = os.environ.get("PGBACKREST_SMOKE_PG_BIN_PATH", "") if env_dirs: extra_dirs += env_dirs.split(os.pathsep) # Parse the requested version filter (accepts "9.6" or "16") try: only_versions = {version_parse(item) for item in arg.version} except ValueError: sys.stderr.write("invalid --version: expected a major version such as 9.6 or 16\n") return 1 supported = supported_versions() installs, unsupported = find_postgres(extra_dirs, only_versions, supported) # Report any installs that were skipped because pgBackRest does not support them if unsupported: sys.stderr.write("skipped unsupported PostgreSQL: %s\n" % ", ".join(version_str(v) for v in unsupported)) # Testing nothing is a failure. If there is no PostgreSQL to test against then the build has not been verified. if not installs: sys.stderr.write("no supported PostgreSQL installation found to test against\n") return 1 runner = resolve_runner(arg.verbose, arg.user) if runner is None: sys.stderr.write("running as root but user '%s' does not exist so PostgreSQL cannot be run\n" % arg.user) return 1 sys.stdout.write( "pgbackrest smoke test: %s\nrunning as: %s\narchitecture: %s\nversions supported: %s\nversions found: %s\n\n" % ( pgbackrest, runner.user, binary_arch(pgbackrest), ", ".join(version_str(v) for v in sorted(supported)) if supported else "unknown", ", ".join(version_str(v) for v, _ in installs), ) ) passed = 0 failed = 0 for version, bin_dir in installs: label = "PostgreSQL %s (%s)" % (version_str(version), bin_dir) sys.stdout.write("=== %s ===\n" % label) # Log the version of PostgreSQL installed on the host. Only the major version is discovered above so this shows the minor # version, which is especially useful during the beta period. sys.stdout.write("%s\n" % pg_version_full(bin_dir)) sys.stdout.flush() start = time.time() try: smoke_one(pgbackrest, version, bin_dir, runner, arg.keep) passed += 1 sys.stdout.write("PASS %s (%.1fs)\n\n" % (label, time.time() - start)) except SmokeError as e: failed += 1 sys.stdout.write("FAIL %s (%.1fs)\n%s\n\n" % (label, time.time() - start, e)) sys.stdout.flush() sys.stdout.write("summary: %d passed, %d failed\n" % (passed, failed)) return 1 if failed else 0 if __name__ == "__main__": sys.exit(main())